Gerald Fritz

A Mobile Vision System for Urban Detection with Informative Local Descriptors

We present a computer vision system for the detection and identification of urban objects from mobile phone imagery, e.g., for the application of tourist information services. Recognition is based on MAP decision making over weak object hypotheses from local descriptor responses in the mobile imagery. We present an improvement over the standard SIFT key detector [7] by selecting only informative (i-SIFT) keys for descriptor matching. Selection is applied first to reduce the complexity of the object model and second to accelerate detection by selective filtering. We present results on the MPG-20 mobile phone imagery with severe illumination, scale and viewpoint changes in the images, performing with ≈ 98% accuracy in identification, efficient (100%) background rejection, efficient (0%) false alarm rate, and reliable quality of service under extreme illumination conditions, significantly improving standard SIFT based recognition in every sense, providing - important for mobile vision - runtimes which are ≈ 8 (≈24) times faster for the MPG-20 (ZuBuD) database.

Q-learning of sequential attention for visual object recognition from informative local descriptors

This work provides a framework for learning sequential attention in real-world visual object recognition, using an architecture of three processing stages. The first stage rejects irrelevant local descriptors based on an information theoretic saliency measure, providing candidates for foci of interest (FOI). The second stage investigates the information in the FOI using a codebook matcher and providing weak object hypotheses. The third stage integrates local information via shifts of attention, resulting in chains of descriptor-action pairs that characterize object discrimination. A Q-learner adapts then from explorative search and evaluative feedback from entropy decreases on the attention sequences, eventually prioritizing shifts that lead to a geometry of descriptor-action scanpaths that is highly discriminative with respect to object recognition. The methodology is successfully evaluated on indoors (COIL-20 database) and outdoors (TSG-20 database) imagery, demonstrating significant impact by learning, outperforming standard local descriptor based methods both in recognition accuracy and processing time.

3D attention: measurement of visual saliency using eye tracking glasses

Understanding and estimating human attention in different interactive scenarios is an important part of human computer interaction. With the advent of wearable eye-tracking glasses and Google glasses, monitoring of human visual attention will soon become ubiquitous. The presented work describes the precise estimation of human gaze fixations with respect to its environment, without the need of artificial landmarks in the field of view, and being capable of providing attention mapping onto 3D information. It enables full 3D recovery of the human view frustum and the gaze pointer in a previously acquired 3D model of the environment in real time. The key contribution is that our methodology enables mapping of fixations directly into an automatically computed 3d model. This innovative methodology will open new opportunities for human attention studies during interaction with its environment, bringing new potential into automated processing for human factors technologies.

Learning Predictive Features in Affordance based Robotic Perception Systems

This work is about the relevance of Gibsons concept of affordances for visual perception in interactive and autonomous robotic systems. In extension to existing functional views on visual feature representations, we identify the importance of learning in perceptual cueing for the anticipation of opportunities for interaction of robotic agents. We investigate how the originally defined representational concept for the perception of affordances - in terms of using either optical flow or heuristically determined 3D features of perceptual entities should be generalized to using arbitrary visual feature representations. In this context we demonstrate the learning of causal relationships between visual cues and predictable interactions, and emphasize on a novel framework for cueing and hypothesis verification of affordances that could play an important role in future robot control architectures. We argue that affordance based perception should enable systems to react to environment stimuli both more efficient and autonomous, and provide a potential to plan on the basis of responses to more complex perceptual configurations. We verify the concept with a concrete implementation applying state-of-the-art visual descriptors and regions of interest within a simulated robot scenario and prove that these features were successfully selected for predicting opportunities of robot interaction

Urban Object Recognition from Informative Local Features

Autonomous mobile agents require object recognition for high level interpretation and localization in complex scenes. In urban environments, recognition of buildings might play a dominant role in robotic systems that need object based navigation, that take advantage of visual feedback and multimodal information for self-localization, or that enable association to related information from the identified semantics. We present a new method – the informative local features approach – based on an information theoretic saliency measure that is rapidly derived from a local Parzen window density estimation in feature subspace. From the learning of a decision tree based mapping to informative features, it enables attentive access to discriminative information and thereby significantly speeds up the recognition process. This approach is highly robust with respect to severe degrees of partial occlusion, noise, and tolerant to some changes in scale and illumination. We present performance evaluation on our publicly available reference object database (TSG-20) that demonstrates the efficiency of this approach, case wise even outperforming the SIFT feature approach [1]. Building recognition will be advantageous in various application domains, such as, mobile mapping, unmanned vehicle navigation, and systems for car driver assistance.

Visual learning of affordance based cues

This work is about the relevance of Gibsons concept of affordances [1] for visual perception in interactive and autonomous robotic systems In extension to existing functional views on visual feature representations, we identify the importance of learning in perceptual cueing for the anticipation of opportunities for interaction of robotic agents We investigate how the originally defined representational concept for the perception of affordances – in terms of using either optical flow or heuristically determined 3D features of perceptual entities – should be generalized to using arbitrary visual feature representations In this context we demonstrate the learning of causal relationships between visual cues and predictable interactions, using both 3D and 2D information In addition, we emphasize a new framework for cueing and recognition of affordance-like visual entities that could play an important role in future robot control architectures We argue that affordance-like perception should enable systems to react to environment stimuli both more efficient and autonomous, and provide a potential to plan on the basis of responses to more complex perceptual configurations We verify the concept with a concrete implementation applying state-of-the-art visual descriptors and regions of interest that were extracted from a simulated robot scenario and prove that these features were successfully selected for their relevance in predicting opportunities of robot interaction.

Object recognition using local information content

Object identification from local information has recently been investigated with respect to its potential for robust recognition, e.g., in case of partial object occlusions, scale variation, noise, and background clutter in detection tasks. This work contributes to this research by a thorough analysis of the discriminative power of local appearance patterns and by proposing to exploit local information content for object representation and recognition. In a first processing stage, we localize discriminative regions in the object views from a posterior entropy measure, and then derive object models from selected discriminative local patterns. Object recognition is then applied to test patterns with associated low entropy using an efficient voting process. The method is evaluated by various degrees of partial occlusion and Gaussian image noise, resulting in highly robust recognition even in the presence of severe occlusion effects.

Learning to perceive affordances in a framework of developmental embodied cognition

Recently, the aspect of visual perception has been explored in the context of Gibsons concept of affordances in various ways. We focus in this work on the importance of developmental learning and the perceptual cueing for an agents anticipation of opportunities for interaction, in extension to functional views on visual feature representations. The concept for the incremental learning of complex from basic affordances is presented in relation to learning of specific affordance features. We demonstrate the learning of causal relations between visual cues and associated anticipated interactions by reinforcement learning of predictive perceptual states. The work pursues a recently presented framework for cueing and recognition of affordance-based visual entities that plays an important role in robot control architectures, in analogy to human perception. We experimentally verify the concept within a real world robot scenario by learning predictive features from delayed rewards, and prove that features were selected for their relevance in predicting opportunities for interaction.

Cascaded Sequential Attention for Object Recognition with Informative Local Descriptors and Q-learning of Grouping Strategies

The contribution of this work is to provide a three-stage architecture for sequential attention to provide a system being capable of sensorimotor object detection in real world environments. The first processing stage provides selected foci of interest in the image based on the extraction of information theoretic saliency of local image descriptors (i-SIFT). The second stage investigates the information in the local attention window using a codebook matcher, providing local weak hypotheses about the identity of the object under investigation. The third stage then proposes a shift of attention to a next attention window. The working hypothesis is to expect a better discrimination from the integration of both the individual local FOA patterns and the geometric relation between them, providing a model of more global information representation, and feeding into a recognition state in the Markov Decision Process (MDP). A reinforcement learner (Q-learner) performs then explorative search on useful actions, i.e., shifts of attention, towards locations of salient information, developing a strategy of useful action sequences being directed in state space towards the optimization of discrimination by information maximization. The method is evaluated in experiments using the COIL-20 database (indoor imagery) and the TSG-20 database (outdoor imagery) to demonstrate efficient performance in object detection tasks, proving the method being more accurate and computationally much less expensive than standard SIFT based recognition.

Attentive object detection using an information theoretic saliency measure

A major goal of selective attention is to focus processing on relevant information to enable rapid and robust task performance. For the example of attentive visual object recognition, we investigate here the impact of top-down information on multi-stage processing, instead of integrating generic visual feature extraction into object specific interpretation. We discriminate between generic and specific task based filters that select task relevant information of different scope and specificity within a processing chain. Attention is applied by tuned early features to selectively respond to generic task related visual features, i.e., to information that is in general locally relevant for any kind of object search. The mapping from appearances to discriminative regions is then modeled using decision trees to accelerate processing. The focus of attention on discriminative patterns enables efficient recognition of specific objects, by means of a sparse object representation that enables selective, task relevant, and rapid object specific responses. In the experiments the performance in object recognition from single appearance patterns dramatically increased considering only discriminative patterns, and evaluation of complete image analysis under various degrees of partial occlusion and image noise resulted in highly robust recognition, even in the presence of severe occlusion and noise effects. In addition, we present performance evaluation on our public available reference object database (TSG-20).