Maja Rudinac

Saliency Detection and Object Localization in Indoor Environments

In this paper we present a scene exploration method for the identification of interest regions in unknown indoor environments and the position estimation of the objects located in those regions. Our method consists of two stages: First, we generate a saliency map of the scene based on the spectral residual of three color channels and interest points are detected in this map. Second, we propose and evaluate a method for the clustering of neighboring interest regions, the rejection of outliers and the estimation of the positions of potential objects. Once the location of objects in the scene is known, recognition of objects/object classes can be performed or the locations can be used for grasping the object. The main contribution of this paper lies in a computationally inexpensive method for the localization of multiple salient objects in a scene. The performance obtained on a dataset of indoor scenes shows that our method performs good, is very fast and hence highly suitable for real-world applications, such as mobile robots and surveillance.

Learning and recognition of objects inspired by early cognition

In this paper, we present a unifying approach for learning and recognition of objects in unstructured environments through exploration. Taking inspiration from how young infants learn objects, we establish four principles for object learning. First, early object detection is based on an attention mechanism detecting salient parts in the scene. Second, motion of the object allows more accurate object localization. Next, acquiring multiple observations of the object through manipulation allows a more robust representation of the object. And last, object recognition benefits from a multi-modal representation. Using these principles, we developed a unifying method including visual attention, smooth pursuit of the object, and a multi-view and multi-modal object representation. Our results indicate the effectiveness of this approach and the improvement of the system when multiple observations are acquired from active object manipulation.

A fast and robust descriptor for multiple-view object recognition

In this paper we propose a fast and robust descriptor for multiple view object recognition using a small number of training examples. In order to design a descriptor to be discriminative between many different object appearances, we base it on a combination of invariant color, edge and texture descriptors. We use a color descriptor based on a HSV histogram — as it is robust to size and position of the object —, a gray level cooccurrence matrix as texture descriptor and an edge histogram as shape descriptor. After extraction of feature vectors, we perform normalization on all feature vectors from the training database in order to increase the importance of the most dominant feature components and reduce the less dominant ones. This normalization improves the recognition performance with almost 30% in case of a small number of training objects and in case of noise or occlusion. We tested our descriptor on the Columbia Object Image Library dataset (COIL 100) which presents objects in scaled, translated and rotated versions. Our recognition rate is extremely high: 99% in case of a large number of training objects and 93% for training with only 4 views of the object, or 5% of the database. The descriptor was also tested under various distortions: illumination change, noise corruption and occlusions. It proved to be very robust, with recognition rates decreasing only less then 5%. We compared our results with state of the art methods and we conclude that our descriptor achieves a better performance, both on the regular COIL database and on all distorted variants.

Empirical mode decomposition for saliency detection

We propose a novel method for saliency detection and attention selection inspired by processes in the human visual cortex. To mimic the varying spatial resolution of the human eye as well as the constant eye movements (saccades) and to model the effect of temporal adaptiveness, we use empirical mode decomposition and corresponding intrinsic mode functions (IMFs), instead of applying standard multi-scale framework as suggested in the state of the art. We derive IMFs between scales to calculate data driven center surround maps which locally reflect amount of information in the scene and we combine opposition color channels, luminosity information and orientation maps into a single saliency map calculated on IMFs. To equalize influence of different components contributing to the final saliency map, normalization steps are proposed. Finally, the MSER regions are calculated directly on the saliency map in order to obtain the most dominant points. We present results on both artificially generated images used in psychological experiments, natural images and application of our method for unknown object detection in robotics.

Portable, automatic water level estimation using mobile phone cameras

Frequent and more accurate water level measurement will allow for a more efficient distribution of water, resulting in less water loss. Therefore in this paper we propose a novel method for accurate water level detection and measurement applied on images of staff gauges, retrieved from mobile device camera. In the first step, we propose fast segmentation of the staff gauge using a 2-class random forest classifier based on a feature vector of textons. To obtain bars and numbers we apply Gaussian Mixture Model segmentation followed by optical character recognition based on random forest classifier and bar detection using shape moments. Based on the recognized lines and numbers a quadratic function for the water level measurement to obtain metric values is introduced. Finally, we propose a novel step for the water level line detection. The water level function and the detected water line provide the value of the water level based on the units on the staff-gauge. The water level can then be uploaded to a central server to determine if water flow needs to increase or decrease. Testing with a real world images from Dutch canals show very accurate detection with many different staff-gauge locations despite complex challenges of viewpoints variations, low quality images as well as changing illumination conditions.

Multimodal human centric object recognition framework for personal robots

In this paper we focus on a perception system for cognitive interaction between robots and humans especially for learning to recognize objects in household environments. Therefore we propose a novel three layered framework for object learning to bridge the gap between the robots recognition capabilities at lower neural level to the higher cognitive level of humans using the weighted fusion of multimodal sources like chromatic, structure and spatial information. In the first layer we propose the grounding of the raw sensory information into semantic concepts for each modality. We obtain a semantic color representation by using SLIC super-pixeling followed by a mapping learned from online images using a PLSA model. This results in a probability distribution over basic color names derived from cognitive linguistic studies. To represent structural information, we propose to cluster the ESF features obtained from Pointcloud data into primitive shape categories. This primitive shape knowledge is learned and expanded from the robots experience. For spatial information a metric map from the navigation system, demarcated into landmark locations is used. All these semantic representations are compliant with a humans description of his environment and further used in the second layer to generate probabilistic knowledge about the objects using random forest classifiers. In the third layer, we propose a novel weighted fusion of the obtained object probabilities, where the weights are derived from the prior experience of the robot. We evaluate our system in realistic domestic conditions provided at a Robocup@Home setting.

Online face recognition and learning for cognitive robots

For human-robot interaction users have to be robustly identified and their appearances learned online. Existing state of the art methods for face recognition do not support online learning of faces and lack the recognition performance required to be used in real-world situations. Hence a novel method is introduced in this paper as a descriptor, which provides the required performance by increasing the separability of the classes by maximizing the inter-class and minimizing intra-class variations. The robustness against variations in lighting and pose as well as the speed is increased by selecting only the most representative samples. Additionally to allow for classification of unknown faces, a novel method has been introduced. The main benefit over the state of the art methods is finding the relation between the distance of classification and the certainty of that classification. This relation is automatically calculated from the data belonging to each class. In that way novelty detection can be performed. To further improve recognition performance a method has been used that utilizes multiple frames in classification. To prove the benefits of the introduced methods extensive experiments have been performed on a state of the art face recognition database.

A robust real-time tracking system based on an adaptive selection mechanism for mobile robots

Extensive research has been conducted in the domain of object tracking. Among the existing tracking methods, most of them mainly focus on using various cues such as color, texture, contour, features, motion as well as depth information to achieve a robust tracking performance. The tracking methods themselves are highly emphasized while properties of the objects to be tracked are usually not exploited enough. In this paper, we first propose a novel adaptive tracking selection mechanism dependent on the properties of the objects. The system will automatically choose the optimal tracking algorithm after examining the textureness of the object. In addition, we propose a robust tracking algorithm for uniform objects based on color information which can cope with real world constraints. In the mean time, we deployed a textured object tracking algorithm which combines the Lucas-Kanade tracker and a model based tracker using the Random Forests classifier. The whole system was tested and the experimental results on a variety of objects show the effectiveness of the adaptive tracking selection mechanism. Moreover, the promising tracking performance shows the robustness of the proposed tracking algorithm. The computation cost of the algorithm is very low, which proves that it can be further used in various real-time robotics applications.

Item Recognition, Learning, and Manipulation in a Warehouse Input Station

One of the challenges of future retail warehouses is automating the order-picking process. To achieve this, items in an order tote must be automatically detected and grasped under various conditions. An inexpensive and flexible solution, presented in this chapter, is using vision systems to locate and identify items to be automatically grasped by a robot system in a bin-picking workstation. Such a vision system requires a single camera to be placed above an order tote, and software to perform the detection, recognition, and manipulation of products using robust image processing and pattern recognition techniques. In order to efficiently and robustly grasp a product by such a robot, both visual and grasping models of each item should be learnt off-line in a product input station. In current warehouse practice, all different types of products entering the warehouse are first measured manually in an input station and stored in the database of the warehouse management system. In this chapter, a method to automate this product input process is proposed: a system for automatic learning, measuring, and storing visual and grasping characteristics of the products is presented.

Knowing What You Don’t Know - Novelty Detection for Action Recognition in Personal Robots

Novelty detection is essential for personal robots to continuously learn and adapt in open environments. This paper specifically studies novelty detection in the context of action recognition. To detect unknown (novel) human action sequences we propose a new method called background models, which is applicable to any generative classifier. Our closed-set action recognition system consists of a new skeleton-based feature combined with a Hidden Markov Model (HMM)-based generative classifier, which has shown good earlier results in action recognition. Subsequently, novelty detection is approached from both a posterior likelihood and hypothesis testing view, which is unified as background models. We investigate a diverse set of background models: sum over competing models, filler models, flat models, anti-models, and some reweighted combinations. Our standard recognition system has an inter-subject recognition accuracy of 96% on the Microsoft Research Action 3D dataset. Moreover, the novelty detection module combining anti-models with flat models has 78% accuracy in novelty detection, while maintaining 78% standard recognition accuracy as well. Our methodology can increase robustness of any current HMM-based action recognition system against open environments, and is a first step towards an incrementally learning system.