Jari Hannuksela

Accelerating image recognition on mobile devices using GPGPU

The future multi-modal user interfaces of battery-powered mobile devices are expected to require computationally costly image analysis techniques. The use of Graphic Processing Units for computing is very well suited for parallel processing and the addition of programmable stages and high precision arithmetic provide for opportunities to implement energy-efficient complete algorithms. At the moment the first mobile graphics accelerators with programmable pipelines are available, enabling the GPGPU implementation of several image processing algorithms. In this context, we consider a face tracking approach that uses efficient gray-scale invariant texture features and boosting. The solution is based on the Local Binary Pattern (LBP) features and makes use of the GPU on the pre-processing and feature extraction phase. We have implemented a series of image processing techniques in the shader language of OpenGL ES 2.0, compiled them for a mobile graphics processing unit and performed tests on a mobile application processor platform (OMAP3530). In our contribution, we describe the challenges of designing on a mobile platform, present the performance achieved and provide measurement results for the actual power consumption in comparison to using the CPU (ARM) on the same platform.

Vision-based motion estimation for interaction with mobile devices

This paper introduces a novel interaction technique for handheld mobile devices which enables the user interface to be controlled by the motion of the users hand. A feature-based approach is proposed for global motion estimation that exploits gradient measures for both feature selection and feature motion uncertainty analysis. A voting-based scheme is presented for outlier removal. A Kalman filter is applied for smoothing motion trajectories. A fixed-point implementation of the method was developed due to the lack of floating-point hardware. Experiments testify the effectiveness of the approach on a camera-enabled mobile phone.

On-Chip Imaging of Schistosoma haematobium Eggs in Urine for Diagnosis by Computer Vision

Background Microscopy, being relatively easy to perform at low cost, is the universal diagnostic method for detection of most globally important parasitic infections. As quality control is hard to maintain, misdiagnosis is common, which affects both estimates of parasite burdens and patient care. Novel techniques for high-resolution imaging and image transfer over data networks may offer solutions to these problems through provision of education, quality assurance and diagnostics. Imaging can be done directly on image sensor chips, a technique possible to exploit commercially for the development of inexpensive “mini-microscopes”. Images can be transferred for analysis both visually and by computer vision both at point-of-care and at remote locations. Methods/Principal Findings Here we describe imaging of helminth eggs using mini-microscopes constructed from webcams and mobile phone cameras. The results show that an inexpensive webcam, stripped off its optics to allow direct application of the test sample on the exposed surface of the sensor, yields images of Schistosoma haematobium eggs, which can be identified visually. Using a highly specific image pattern recognition algorithm, 4 out of 5 eggs observed visually could be identified. Conclusions/Significance As proof of concept we show that an inexpensive imaging device, such as a webcam, may be easily modified into a microscope, for the detection of helminth eggs based on on-chip imaging. Furthermore, algorithms for helminth egg detection by machine vision can be generated for automated diagnostics. The results can be exploited for constructing simple imaging devices for low-cost diagnostics of urogenital schistosomiasis and other neglected tropical infectious diseases.

Document Image Mosaicing with Mobile Phones

This paper presents a novel user interaction concept for document image scanning with mobile phones. A high resolution mosaic image is constructed in two main stages. Firstly, online camera motion estimation is applied to the phone to assist the user to capture small image patches of the document page. Automatic image stitching process with the help of estimated device motion is carried out to reconstruct the full view of the document. Experiments on document images captured and processed with mosaicing software clearly show the feasibility of the approach.

A Vision-Based Approach for Controlling User Interfaces of Mobile Devices

We introduce a novel user interface solution for mobile devices which enables the display to be controlled by the motion of the user’s hand. A feature-based approach is proposed for dominant global motion estimation that exploits gradient measures for both feature selection and motion uncertainty analysis. We also present a voting-based scheme for outlier removal. A Kalman filter is utilized for smoothing motion trajectories. A fixed-point implementation of the method was made on a mobile device platform that sets computational restrictions for the algorithms used. Experiments with synthetic and real image sequences show the effectiveness of the method and demonstrate the practicality of the approach in a smartphone.

Graphics hardware accelerated panorama builder for mobile phones

Modern mobile communication devices frequently contain built-in cameras allowing users to capture highresolution still images, but at the same time the imaging applications are facing both usability and throughput bottlenecks. The difficulties in taking ad hoc pictures of printed paper documents with multi-megapixel cellular phone cameras on a common business use case, illustrate these problems for anyone. The result can be examined only after several seconds, and is often blurry, so a new picture is needed, although the view-finder image had looked good. The process can be a frustrating one with waits and the user not being able to predict the quality beforehand. The problems can be traced to the processor speed and camera resolution mismatch, and application interactivity demands. In this context we analyze building mosaic images of printed documents from frames selected from VGA resolution (640x480 pixel) video. High interactivity is achieved by providing real-time feedback on the quality, while simultaneously guiding the user actions. The graphics processing unit of the mobile device can be used to speed up the reconstruction computations. To demonstrate the viability of the concept, we present an interactive document scanning application implemented on a Nokia N95 mobile phone.

Face Tracking for Spatially Aware Mobile User Interfaces

This paper introduces a new face tracking approach for controlling user interfaces in hand-held mobile devices. The proposed method detects the face and the eyes of the user by employing a method based on local texture features and boosting. An extended Kalman filter combines local motion features extracted from the face region and the detected eye positions to estimate the 3-D position and orientation of the camera with respect to the face. The camera position is used as an input for the spatially aware user interface. Experimental results on real image sequences captured with a camera-equipped mobile phone validate the feasibility of the method.

Evaluation of real-time LBP computing in multiple architectures

Local binary pattern (LBP) is a texture operator that is used in several different computer vision applications requiring, in many cases, real-time operation in multiple computing platforms. The irruption of new video standards has increased the typical resolutions and frame rates, which need considerable computational performance. Since LBP is essentially a pixel operator that scales with image size, typical straightforward implementations are usually insufficient to meet these requirements. To identify the solutions that maximize the performance of the real-time LBP extraction, we compare a series of different implementations in terms of computational performance and energy efficiency, while analyzing the different optimizations that can be made to reach real-time performance on multiple platforms and their different available computing resources. Our contribution addresses the extensive survey of LBP implementations in different platforms that can be found in the literature. To provide for a more complete evaluation, we have implemented the LBP algorithms in several platforms, such as graphics processing units, mobile processors and a hybrid programming model image coprocessor. We have extended the evaluation of some of the solutions that can be found in previous work. In addition, we publish the source code of our implementations.

Adaptive motion-based gesture recognition interface for mobile phones

In this paper, we introduce a new vision based interaction technique for mobile phones. The user operates the interface by simply moving a finger in front of a camera. During these movements the finger is tracked using a method that embeds the Kalman filter and ExpectationMaximization (EM) algorithms. Finger movements are interpreted as gestures using Hidden Markov Models (HMMs). This involves first creating a generic model of the gesture and then utilizing unsupervised Maximum a Posteriori (MAP) adaptation to improve the recognition rate for a specific user. Experiments conducted on a recognition task involving simple control commands clearly demonstrate the performance of our approach.

Interactive multi-frame reconstruction for mobile devices

The small size of handheld devices, their video capabilities and multiple cameras are under-exploited assets. Properly combined, the features can be used for creating novel applications that are ideal for pocket-sized devices, but may not be useful in laptop computers, such as interactively capturing and analyzing images on the fly. In this paper we consider building mosaic images of printed documents and natural scenes from low resolution video frames. High interactivity is provided by giving a real-time feedback on the video quality, while simultaneously guiding the user’s actions. In our contribution, we analyze and compare means to reach interactivity and performance with sensor signal processing and GPU assistance. The viability of the concept is demonstrated on a mobile phone. The achieved usability benefits suggest that combining interactive imaging and energy efficient high performance computing could enable new mobile applications and user interactions.