Takuya Funatomi

Optimizing Mean Reciprocal Rank for person re-identification

Person re-identification is one of the most challenging issues in network-based surveillance. The difficulties mainly come from the great appearance variations induced by illumination, camera view and body pose changes. Maybe influenced by the research on face recognition and general object recognition, this problem is habitually treated as a verification or classification problem, and much effort has been put on optimizing standard recognition criteria. However, we found that in practical applications the users usually have different expectations. For example, in a real surveillance system, we may expect that a visual user interface can show us the relevant images in the first few (e.g. 20) candidates, but not necessarily before all the irrelevant ones. In other words, there is no problem to leave the final judgement to the users. Based on such an observation, this paper treats the re-identification problem as a ranking problem and directly optimizes a listwise ranking function named Mean Reciprocal Rank (MRR), which is considered by us to be able to generate results closest to human expectations. Using a maximum-margin based structured learning model, we are able to show improved re-identification results on widely-used benchmark datasets

Tracking Food Materials with Changing Their Appearance in Food Preparing

This paper describes our work in computer vision to track food materials in the food preparation process. Tracking such food materials is difficult, because they are often hidden when moved by hand. Furthermore, their appearance may change in hand when they are cut or peeled. For tracking these objects in such situations, we propose a novel method that matches an object on a cooking table to one grasped in the past. We use the following three criteria to match the objects even when they are cut or peeled: the similarity in their appearance, the validity of their change in appearance, and the grasped order. We experimentally evaluated our method by applying it to the scenes of cutting and peeling food materials. As a result, we achieved an accuracy of 83.6% in matching the objects.

Cooking Ingredient Recognition Based on the Load on a Chopping Board during Cutting

This paper presents a method for recognizing recipe ingredients based on the load on a chopping board when ingredients are cut. The load is measured by four sensors attached to the board. Each chop is detected by indentifying a sharp falling edge in the load data. The load features, including the maximum value, duration, impulse, peak position, and kurtosis, are extracted and used for ingredient recognition. Experimental results showed a precision of 98.1% in chop detection and 67.4% in ingredient recognition with a support vector machine (SVM) classifier for 16 common ingredients.

4D light field segmentation with spatial and angular consistencies

In this paper, we describe a supervised four-dimensional (4D) light field segmentation method that uses a graph-cut algorithm. Since 4D light field data has implicit depth information and contains redundancy, it differs from simple 4D hyper-volume. In order to preserve redundancy, we define two neighboring ray types (spatial and angular) in light field data. To obtain higher segmentation accuracy, we also design a learning-based likelihood, called objectness, which utilizes appearance and disparity cues. We show the effectiveness of our method via numerical evaluation and some light field editing applications using both synthetic and real-world light fields.

Recognizing ingredients at cutting process by integrating multimodal features

We propose a method for recognizing ingredients in food preparing activity. The research for object recognition mainly focuses on only visual information; however, ingredients are difficult to recognize only by visual information because of their limited color variations and larger within-class difference than inter-class difference in shapes. In this paper, we propose a method that involves some physical signals obtained in a cutting process by attaching load and sound sensors to the chopping board. The load may depend on an ingredients hardness. The sound produced when a knife passes through an ingredient reflects the structure of the ingredient. Hence, these signals are expected to facilitate more precise recognition. We confirmed the effectiveness of the integration of the three modalities (visual, auditory, and load) through experiments in which the developed method was applied to 23 classes of ingredients.

How Does User’s Access to Object Make HCI Smooth in Recipe Guidance?

This paper aims firstly to provide a flexible framework for developing recipe guiding system that displays information step-by-step along to events recognized in user’s activity, and secondly to introduce an example of our implementation on the proposed framework. Those who are working on a task requiring high concentration can be easily distracted by the interactive systems that require any kind of explicit manipulations. In such situation, recognizing events in the task is helpful as an alternative of the manipulations. The framework allows a system designer to incorporate his/her own recognizer to the guiding system. Based on this framework, we implemented a system working with user’s grabbing and releasing objects. A grabbed object tells the user’s intention of what is about to do next, and releasing the object indicates its completion. In the experiments using the WOZ method, we confirmed that these actions worked well as switches for the interface. We also summarize some of our efforts for automating the system.

Detecting start and end times of object-handlings on a table by fusion of camera and load sensors

We aim to extract object-handlings in real time for realizing a system supporting cooking activities. Traditional methods based solely on camera is principally poor at judging whether touched or just closed near objects, and those based on load sensing table cannot extract concurrent object-handlings separately. To extract object-handlings separately, we propose a sensor fusion framework of a camera and load sensor information. We applied our method to a simple cooking activity and confirmed that our method principally worked as expected.

Detection of social interaction from observation of daily living environments

In this article, we discuss how to detect occasional social interaction by a group of people in an open space such as a hall by observing the environment by cameras. Since it is known in the field of social psychology that some characteristic arrangement is maintained by each group of people during interaction, previous works have tried to detect social interaction based on the arrangement. However, these methods could confuse different groups especially when those groups are located close to each other, because the methods only consider direct relationship among the positions or orientations of the people for finding the characteristic arrangement. We propose a new region-based approach, which focuses on the spatial region to be occupied exclusively by each group of the people, introducing a technique for region extraction used in the field of image processing.

Developing a Real-Time System for Measuring the Consumption of Seasoning

In this paper, we propose a real-time system for measuring the consumption of various types of seasonings. In our system, all seasonings are placed on a scale, and we continuously take images of these items using a camera. Our system estimates the consumption of each condiment by calculating the difference between the weight when the seasoning was picked up and the weight when it was placed back on the scale. Our system identifies the type of seasoning that was used by determining whether or not the seasoning was present on the scale. By using our system, users can automatically log their usage of seasoning. Then, they can adjust the seasoning according to their desired taste.

3D shape reconstruction from incomplete silhouettes in multiple frames

3D shapes are reconstructed from silhouettes obtained by multiple cameras with the volume intersection method. In recent work, methods of integrating silhouettes in time sequences have been proposed. The number of silhouettes can be increased by integrating silhouettes in multiple frames. The silhouettes of a rigid object in multiple frames are integrated with its rigid motion. This motion is often estimated with 3D feature points extracted from silhouettes. When the estimated motion has large error, shapes are reconstructed with missing parts. This error is given by the incomplete extraction of 3D feature points, which is caused by additional and missing regions of extracted silhouettes. We cannot prevent silhouettes from being extracted with the additional and missing regions in real environments. Here, we propose an intelligent method of integrating incomplete silhouettes where outcrop points, which are 3D feature points for estimating motion, play an important role. The reconstructed shape can be evaluated referring to how many outcrop points have been included in the reconstructed shape of another frame. Although the evaluation does not represent the accuracy of estimated motion directly, it does guarantee that outstanding parts will be preserved in the reconstructed shape. Silhouettes in multiple frames can be integrated with fewer missing and additional parts based on this evaluation.