Gamhewage C. de Silva

Evaluation of video summarization for a large number of cameras in ubiquitous home

A system for video summarization in a ubiquitous environment is presented. Data from pressure-based floor sensors are clustered to segment footsteps of different persons. Video handover has been implemented to retrieve a continuous video showing a person moving in the environment. Several methods for extracting key frames from the resulting video sequences have been implemented, and evaluated by experiments. It was found that most of the key frames the human subjects desire to see could be retrieved using an adaptive algorithm based on camera changes and the number of footsteps within the view of the same camera. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.

Experience retrieval in a ubiquitous home

We present a system for retrieval and summarization of continuously archived multimedia data from a home-like ubiquitous environment. Data from pressure-based floor sensors are analyzed to index video and audio from a large number of sources. Video and audio handover are implemented to retrieve continuous video streams with sound as a person is moving in the environment. Key frame extraction is proposed and several algorithms are implemented to obtain compact summaries corresponding to the activity of each person. Clustering algorithms and image analysis are used to identify actions and events. The system consists of a graphical user interface that can be used to retrieve video summaries interactively using simple queries.

Image-based Calorie Content Estimation for Dietary Assessment

In this paper, we present an image-analysis based approach to calorie content estimation for dietary assessment. We make use of daily food images captured and stored by multiple users in a public Web service called Food Log. The images are taken without any control or markers. We build a dictionary dataset of 6512 images contained in Food Log the calorie content of which have been estimated by experts in nutrition. An image is compared to the ground truth data from the point of views of multiple image features such as color histograms, color correlograms and SURF fetures, and the ground truth images are ranked by similarities. Finally, calorie content of the input food image is computed by linear estimation using the top n ranked calories in multiple features. The distribution of the estimation shows that 79% of the estimations are correct within ±40% error and 35% correct within ±20% error.

Retrieving multimedia travel stories using location data and spatial queries

We propose a system for retrieving multimedia related to a persons travel, using location data captured with a GPS receiver, mobile phone or a camera. The user makes simple sketches on a map displayed on a computer screen, to submit spatial, temporal or spatio-temporal queries regarding his travel. The system segments the location data and images, analyzes sketches made by a user, identifies the query, and retrieves relevant results. These results, combined with online maps and virtual tours rendered using street view panoramas, form multimedia travel stories. We present the systems current status and conclude with future directions.

Personalization of food image analysis

This paper presents a method to classify food images by updating the model of Bayesian network incrementally. We have been investigating a “food log” system which makes use of image analysis, and it can automatically detect food images and estimate the food balance (using a simple nutrition model). It also enables users to easily modify the results of the analysis when they contain errors. So far, the system does not make use of the corrections made by the users to improve the performance of classification. In this paper, we propose to incrementally update the classifier based on Baysian network so that the results of analysis will be improved by using the users corrections. With the incremental updating, the accuracy of food image detection is improved from 89% to 92%.

Ubiquitous Home: Retrieval of Experiences in a Home Environment

Automated capture and retrieval of experiences at home is interesting due to the wide variety and personal significance of such experiences. We present a system for retrieval and summarization of continuously captured multimedia data from Ubiquitous Home, a two-room house consisting of a large number of cameras and microphones. Data from pressure based sensors on the floor are analyzed to segment footsteps of different persons. Video and audio handover are implemented to retrieve continuous video streams corresponding to moving persons. An adaptive algorithm based on the rate of footsteps summarizes these video streams. A novel method for audio segmentation using multiple microphones is used for video retrieval based on sounds with high accuracy. An experiment, in which a family lived in this house for twelve days, was conducted. The system was evaluated by the residents who used the system for retrieving their own experiences; we report and discuss the results.

Multimedia and semantic technologies for future computing environments

Research progresses in multimedia computing and systems using semantic technologies have been recently and widely explored. This special issue on multimedia and semantic technologies for future computing environments provides high quality contributions addressing related theoretical and practical aspects of semantic multimedia. We have selected fourteen research papers whose topics are strongly related to the multimedia and semantic technology.

Food Log by snapping and processing images

We present the current status of FoodLog, a multimedia Internet application that enables easy capture and archival of information regarding our daily meals. The primary purpose of FoodLog is to facilitate dietary management support with minimum manual recording of information. It analyzes image archives that belong to a user to identify images of meals. Further image analysis determines the nutritional composition of these meals and stores the data to form a log. The user can view the data from this log in different formats, and also edit the data to correct any mistakes that occurred during image analysis. This application was recently opened to the public, and had accumulated approximately 25000 images during the first two months since its launch. We present the current status of this application, and discuss our future plans to extend it to allow interaction between users and more effective dietary management.

Sketch-based spatial queries for the retrieval of human locomotion patterns in smart environments

A system for retrieving video sequences created by tracking humans in a smart environment, by using spatial queries, is presented. Sketches made with a pointing device on the floor layout of the environment are used to formqueries corresponding to locomotion patterns. The sketches are analyzed to identify the type of the query. Directional search algorithms based on the minimum distance between points are applied for finding the best matches to the sketch. The results are ranked according to the similarity and presented to the user. The system was developed in two stages. An initial version of the system was implemented and evaluated by conducting a user study. Modifications were made where appropriate, according to the results and the feedback, to make the system more accurate and usable. We present the details of the initial system, the user study and the results, and the modifications thus made. The overall accuracy of retrieval for the initial system was approximately 93%, when tested on a collection of data from a real-life experiment. This is improved to approximately 97% after the modifications. The user interaction strategy and the search algorithms are usable in any environment for automated retrieval of locomotion patterns. The subjects who evaluated the system found it easy to learn and use. Their comments included several prospective applications for the user interaction strategy, providing valuable insight for future directions.

Person tracking and multicamera video retrieval using floor sensors in a ubiquitous environment

A system for video retrieval from a ubiquitous environment is presented. Data from pressure-based floor sensors are used as a supplementary input for retrieving video from a large number of cameras. An algorithm based on agglomerative hierarchical clustering is used to segment footpaths of individual persons. Video handover is proposed and two methods are implemented to retrieve video clips and key frame sequences showing a person moving inside the house. The video clips are further segmented according to the actions performed. We evaluate the performance of each stage of retrieval and present the results. The paper concludes with suggestions for improvements, and future directions.