Arto Heikkinen

Distributed multimedia content analysis with MapReduce

This paper introduces a scalable solution for distributing content-based video analysis tasks using the emerging MapReduce programming model. Scalable and efficient solutions are needed for this type of tasks, as the number of multimedia content is growing at an increasing rate. We present a novel implementation utilizing the popular Apache Hadoop MapReduce framework for both analysis job scheduling and video data distribution. We employ face detection as a case example because it represents a popular visual content analysis task. The main contribution of this paper is the performance evaluation of distribution models for video content processing in various configurations. In our experiments, we have compared the performance of our video data distribution method against two alternatives solutions on a seven node cluster. Hadoops performance overhead in video content analysis was also evaluated. We found Hadoop to be a data efficient solution with minimal computational overhead for the face detection task.

Augmented Reality Web Applications with Mobile Agents in the Internet of Things

Augmented reality (AR) is a promising technology for building applications in an Internet of Things (IoT) environment, utilized for visualizing information provided by IoT devices. In this paper, we enable Web-based mobile AR applications with mobile agents in a resource-oriented IoT system architecture. We present an adaptable mobile agent composition that contains the data representation logic and mappings between AR applications and system resources. Thus, mobile agents and AR application-specific data structures are exposed as global system resources. System resource linkages are considered between real-world objects and their virtual representations for mobile agent-based AR applications. The agent composition also complies with the REST principles for resource access and control system-wide. This allows dynamic runtime adaptation and addressing the device and resource heterogeneity, thus eliminating the need for application-specific communication protocols. Moreover, we utilize a Web-based mobile AR application framework, running completely in a Web browser, which facilitates straightforward AR application development. Lastly, a proof of concept mobile AR application is implemented, where a coffee maker with a visual tag is connected to a low-power resource-constrained wireless sensor network node as an IoT device. A mobile agent is injected into the IoT environment to expose the state changes of the coffee maker. Through the visual tag, AR applications are able to visualize the state changes of the coffee maker in their user interface.

RADE: Resource-aware distributed browser-to-browser 3D graphics delivery in the web

With the introduction of novel standardized web technologies such as WebGL, 3D virtual environments (VEs) are making their way into the web. Concurrently, resource rich mobile devices, such as smartphones and tablets, have become the primary access medium for the web. This paper introduces a method called RADE that enables resource-aware P2P-assisted 3D asset delivery in a web browser using WebRTC. Use of P2P technology in 3D asset delivery can (1) decrease the load on 3D asset servers; (2) decrease the application response times; and (3) reduce the operational costs of VE service providers. The performance and resource fairness of RADE was evaluated in real-life wireless networks using a prototype implementation. Based on the results, RADE can significantly reduce the server load and the 3D asset delivery times especially when 3D asset servers are under heavy load. For instance, with a scarce server bandwidth of 2Mbps, use of RADE resulted in 55% shorter 3D asset delivery times on average. Although RADE has been developed for 3D asset delivery, it is applicable for many types of web applications, including video delivery.

Performance evaluation of distributed data delivery on mobile devices using WebRTC

Direct peer-to-peer connectivity between web browsers is becoming reality with the emerging and constantly developing WebRTC technology stack. This opens possibilities for new kind of plugin-free web applications, such as browser-to-browser file transfers and multi-party conferencing. In this paper, the performance of WebRTC on mobile devices is evaluated with different mobile device, wireless network connectivity and web browser configurations. The evaluation was conducted with a WebRTC test environment that was implemented based on PeerJS JavaScript library and PeerServer signaling server. The measurements include session establishment delay and overhead, session maintenance overhead, resource consumption of multiple simultaneous file transfers and efficiency of different file transfer approaches. Based on the results, the delay for establishing a WebRTC connection may in the worst cases exceed even 10 seconds making it a serious bottleneck. However, from the standpoints of memory consumption and CPU load, high-end mobile devices are very capable of running multiple simultaneous WebRTC connections for data transfers. The results of this paper provide new insight to researchers, application and browser developers and WebRTC standardization bodies.

Kuukkeli-TV: Online content-based services and applications for broadcast TV with long-term user experiments

Online TV services have facilitated time-shifted TV viewing. Accordingly, new service concepts are needed to improve access to relevant information in the broadcast TV content. In this paper we introduce a system that indexes TV broadcast in near real-time from seven free-to-air television channels. Our system uses machine learning and data mining techniques to extract descriptive novelty word summaries and picture highlights automatically from TV program subtitles and uses them to provide non-linear content-based access to relevant TV content fragments in novel end-user services and applications. First end-user service allows browsing of recent time shifted TV content using time and program genre metaphors with extracted program summaries. Second end-user service provides free-text search of archival and time shifted TV content. Additionally, the system allows content-based recommendation of similar TV content from the database of 180 000 indexed programs. Recommendations are used in the enduser services and Mobile EPG Guide application. Over 5 000 user sessions have been collected to study how users adopt our content-based access metaphors to examine interesting TV content. User logs revealed that the proposed contentbased access techniques were more popular in TV program search and browsing activities than conventional techniques based on program title and description metadata.

Energy Efficient Event Driven Video Streaming Surveillance Using SleepyCAM

Wireless Multimedia Sensor Networks (WMSN) are one of the emerging paradigms of the Internet of Things (IoT) that are used to retrieve content including scalar data, video and audio streams and still images from the physical environment. In contrast to scalar sensor (such as temperature and humidity sensor) nodes, multimedia sensor nodes capture high volumes of data and perform far more complex tasks that can be highly power consuming. In this paper, we present the design of energy efficient high resolution camera sensor node, that is capable of capturing a full HD video at 30fps, using off-the-shelf hardware for an event driven video streaming surveillance application. In order to achieve long battery life, we use an energy efficient motion detection and power management mechanism, called sleepyCAM, which uses a lowpower scalar sensor node to detect motion and wake-up a high resolution camera node when needed. We used Libellium Waspmote platform and raspberry pi (RPi) to implement the functionality of the low-power sensor node and the HD camera node, respectively. We validate our work using a baseline setup on a standby RPi that uses scalar sensor for motion detection. The results demonstrate that with the used hardware platform, the power consumption can be reduced by more than 99%.

Time shifting patterns in browsing and search behavior for catch-up TV on the web

Catch-up TV services on the Web have facilitated time-shifted TV viewing. However, there is limited information about user search behavior with regard to recently time-shifted versus archival TV content. We deployed two distinct content-based web services to explore information retrieval of time-shifted TV content. The first web service is based on a browsing metaphor, while the second is based on free text content search metaphor. We analyzed more than 5000 user sessions from 12 months of logs and found that the programs accessed via browsing categorized program content summaries were typically less than one week old. In contrast, the programs accessed via free text search on subtitle content were typically more than a week old. Our findings provide a first assessment of user behavior in accessing time-shifted and archival TV content. Further research should develop the user experience for content-based TV access and explore the sharing patterns of archival TV content on social networks.

Distributed web service architecture for scalable content analysis: semi-automatic annotation of user generated content

This paper describes distributed web service architecture to allow web level scalability for multimedia content analysis. We introduce service component architecture that allows for orchestration and composition of services as content analysis units to enrich multimedia metadata using common data model built upon MPEG-7 content description standard. We report experiments with automatic load balancing and introduce an end user application that facilitates annotation of people in images and video.

DRUMM: Dynamic viewing of large-scale 3D city models on the web

3D city models have become an important user interface for various applications, ranging from entertainment to civil engineering. Today, 3D city models can also be accessed on the web without installing any additional software, which has significantly widened their potential audience. However, visually accurate 3D city models are typically large in terms of file size, and hence, require ample network bandwidth for minimizing download delays and providing a smooth user experience. In this paper, we introduce a method called DRUMM, which enables dynamic resource management for viewing 3D city models on the web. DRUMM supports both the use of varying criteria for prioritizing the download order of 3D graphics, and the division of 3D graphics into chunks facilitating parallel downloads that can be suspended and later continued. The performance of DRUMM was evaluated in terms of (1) starting delay; (2) used network bandwidth; and (3) the number of buildings with textures in the view using the developed prototype implementation. Based on the results, DRUMM improves the usability of 3D city applications, particularly when the network bandwidth is scarce.

Towards Augmented Reality Applications in a Mobile Web Context

Mobile augmented reality (MAR) applications are becoming reality due to the rapid evolution of mobile devices and their capacities. In this paper, we introduce MARW, a development framework for MAR applications on a Web browser. Use of the Web browser technology enables running the same MAR applications on all mobile platform and operating system configurations. The performance of the implemented MARW prototype was evaluated with several experiments using multiple mobile device and Web browser combinations. Based on the experimentation results, a maximum number of 12 frames per second were achieved with a high-end tablet device in vision-based tracking using markers. It was also discovered that the performance differences between Web browsers are significant in terms of MAR application performance.