Rui M. Jesus

A gesture based game for image tagging

This paper proposes and discusses the ASAA (Application for Semi-Automatic Annotation) interface, a new computer game for image tagging. The application is composed by a 3D game interface, a game engine that uses a system for automatic image annotation and gestural input to play the game. The paper describes the rational and design principles for the application and presents ideas for future developments.

Methods for automatic and assisted image annotation

Personal memories composed of digital pictures are very popular at the moment. To retrieve these media items annotation is required. During the last years, several approaches have been proposed in order to overcome the image annotation problem. This paper presents our proposals to address this problem. Automatic and semi-automatic learning methods for semantic concepts are presented. The automatic method is based on semantic concepts estimated using visual content, context metadata and audio information. The semi-automatic method is based on results provided by a computer game. The paper describes our proposals and presents their evaluations.

Playing games as a way to improve automatic image annotation

Image annotation is hard to do in an automatic way. In this paper, we propose a framework for image annotation that combines the benefits of three paradigms: automatic annotation, human intervention and entertainment activities. We also describe our proposal inside this framework, the ASAA (application for semi-automatic annotation) interface, a new computer game for image tagging. The application has a 3D game interface, and is supported by a game engine that uses a system for automatic image classification and gestural input to play the game. We present results of the performance of semantic models obtained with a training set enlarged by images annotated during the game activity as well as usability tests of the application.

Tag around: a 3D gesture game for image annotation

Manual annotation is difficult to perform in large image databases and the automatic methods deliver results that are far from what is desired by the users. New and fun ways to provide image annotation are required. This paper describes Tag Around, a computer game for image tagging. The application is based on a 3D game interface, a game engine that uses a system for automatic image classification and gestural input to play the game. We present the game, how it is used to improve image annotation, the design principles of the interface and the usability tests that were carried out in order to validate the application.

Face recognition and gender classification in personal memories

Face annotation is an important concept for personal memories retrieval. Using automatic face recognition to annotate and find people in those memories provides an improvement of a personal memories management system. However, its results are limited by the uncontrolled conditions inherent to personal memories. In this paper, we propose a face recognition method to address these limitations which includes techniques of skin detection and pose estimation. It is also proposed a gender classification method to provide more information about the detected faces. Experimental results are presented that show that these methods improve the overall performance of face recognition and gender classification in personal memories.

Features combination for art authentication studies: brushstroke and materials analysis of Amadeo de Souza-Cardoso

This work presents a tool to support authentication studies of paintings attributed to the modernist Portuguese artist Amadeo de Souza-Cardoso (1887-1918). The strategy adopted was to quantify and combine the information extracted from the analysis of the brushstroke with information on the pigments present in the paintings. The brushstroke analysis was performed combining Gabor filter and Scale Invariant Feature Transform. Hyperspectral imaging and elemental analysis were used to compare the materials in the painting with those present in a database of oil paint tubes used by the artist. The outputs of the tool are a quantitative indicator for authenticity, and a mapping image that indicates the areas where materials not coherent with Amadeo’s palette were detected, if any. This output is a simple and effective way of assessing the results of the system. The method was tested in twelve paintings obtaining promising results.

ARZombie: A mobile augmented reality game with multimodal interaction

Augmented reality games have the power to extend virtual gaming into real world scenarios with real people, while enhancing the senses of the user. This paper describes the AR-Zombie game developed with the aim of studying and developing mobile augmented reality applications, specifically for tablets, using face recognition interaction techniques. The goal of the ARZombie player is to kill zombies that are detected through the display of the device. Instead of using markers as a mean of tracking the zombies, this game incorporates a facial recognition system, which will enhance the user experience by improving the interaction of players with the real world. As the player moves around the environment, the game will display virtual zombies on the screen if the detected faces are recognized as belonging to the class of the zombies. ARZombie was tested with users to evaluate the interaction proposals and its components were evaluated regarding the performance in order to ensure a better gaming experience.

Geographic image retrieval in mobile guides

This paper describes an application that is based on geographic queries to retrieve personal pictures when visiting points of interest using mobile devices such as PDAs. The retrieved information can be visualized in different ways, including the spatial visualization of the pictures of the places visited. The paper describes the user interface features and the experiments that were made to illustrate the proposed methods.

Tracking the Human Body Using Multiple Predictors

The objective of this work is to track the human body from a video sequence, assuming that the motion direction is parallel to the image plane. Tracking the human body is a difficult task because the human body may have unpredictable movements and it is difficult to accurately detect anatomic points in images without using artificial marks. Furthermore, self occlusions often prevent the observation of some body segments. This paper describes a tracking algorithm, which avoids the use of artificial marks. The proposed system is able to learn from previous experience, and therefore its performance improves during the tracking operation. The ability of the tracking system to gradually adapt to a particular type of human motion is obtained by using on-line learning methods based on multi-predictors. These predictors are updated in a supervised way using information provided by a human operator. Typically, the human operator corrects the model estimates several times during the first few seconds, but the corrections rate decreases as time goes by. Experimental results are presented in the paper to illustrate the performance of the proposed tracking system.

Clip art retrieval combining raster and vector methods

Clip art databases can be composed by raster images or by vector drawings. There are technologies for searching and retrieving clip arts for both image formats but research has been done separately, focusing on either format, without taking benefits of both research fields as a whole. This paper describes a study where the benefits of combining information extracted from vector and raster images to retrieve clip arts are evaluated and discussed. Color and texture features are extracted from raster images and geometry and topology features are extracted from vector images. The paper presents several comparisons between different combinations with several descriptors. The results of the study show the effectiveness of the solutions that combines both types of features.