Tristram Southey

Curious George: An attentive semantic robot

State-of-the-art methods have recently achieved impressive performance for recognising the objects present in large databases of pre-collected images. There has been much less focus on building embodied systems that recognise objects present in the real world. This paper describes an intelligent system that attempts to perform robust object recognition in a realistic scenario, where a mobile robot moving through an environment must use the images collected from its camera directly to recognise objects. To perform successful recognition in this scenario, we have chosen a combination of techniques including a peripheral-foveal vision system, an attention system combining bottom-up visual saliency with structure from stereo, and a localisation and mapping technique. The result is a highly capable object recognition system that can be easily trained to locate the objects of interest in an environment, and subsequently build a spatial-semantic map of the region. This capability has been demonstrated during the Semantic Robot Vision Challenge, and is further illustrated with a demonstration of semantic mapping. We also empirically verify that the attention system outperforms an undirected approach even with a significantly lower number of foveations.

Automated Spatial-Semantic Modeling with Applications to Place Labeling and Informed Search

This paper presents a spatial-semantic modeling system featuringautomated learning of object-place relations from an online annotateddatabase, and the application of these relations to a variety ofreal-world tasks. The system is able to label novel scenes with placeinformation, as we demonstrate on test scenes drawn from the same sourceas our training set. We have designed our system for future enhancementof a robot platform that performs state-of-the-art object recognitionand creates object maps of realistic environments. In this context, wedemonstrate the use of spatial-semantic information to performclustering and place labeling of object maps obtained from real homes.This place information is fed back into the robot system to inform anobject search planner about likely locations of a query object. As awhole, this system represents a new level in spatial reasoning andsemantic understanding for a physical platform.

Curious George: An Integrated Visual Search Platform

This paper describes an integrated robot system, known as Curious George, that has demonstrated state-of-the-art capabilities to recognize objects in the real world. We describe the capabilities of this system, including: the ability to access web-based training data automatically and in near real-time, the ability to model the visual appearance and 3D shape of a wide variety of object categories, navigation abilities such as exploration, mapping and path following, the ability to decompose the environment based on 3D structure, allowing for attention to be focused on regions of interest, the ability to capture high-quality images of objects in the environment, and finally, the ability to correctly label those objects with high accuracy. The competence of the combined system has been validated by entry into an international competition where Curious George has been among the top performing systems each year. We discuss the implications of such successful object recognition for society, and provide several avenues for potential improvement.

Place Classification Using Visual Object Categorization and Global Information

Places in an environment are locations where activities occur, and can be described by the objects they contain. This paper discusses the completely automated integration of object detection and global image properties for place classification. We first determine object counts in various place types based on Label Me images, which contain annotations of places and segmented objects. We then train object detectors on some of the most frequently occurring objects. Finally, we use object detection scores as well as global image properties to perform place classification of images. We show that our object-centric method is superior and more generalizable when compared to using global properties in indoor scenes. In addition, we show enhanced performance by combining both methods. We also discuss areas for improvement and the application of this work to informed visual search. Finally, through this work we display the performance of a state-of-the-art technique trained using automatically-acquired labeled object instances (i.e., bounding boxes) to perform place classification of realistic indoor scenes.

Automated Place Classification Using Object Detection

Places in an environment can be described by the objects they contain. This paper discusses the completely automated integration of object detection and place classification in a single system. We first perform automated learning of object-place relations from an online annotated database. We then train object detectors on some of the most frequently occurring objects. Finally, we use detection scores as well as learned object-place relations to perform place classification of images. We also discuss areas for improvement and the application of this work to informed visual search. As a whole, the system demonstrates the automated acquisition of training data containing labeled instances (i.e. bounding boxes) and the performance of a state-of-the-art object detection technique trained on this data to perform place classification of realistic indoor scenes.

3D spatial relationships for improving object detection

This work demonstrates how 3D qualitative spatial relationships can be used to improve object detection by differentiating between true and false positive detections. Our method identifies the most likely subset of 3D detections using seven types of 3D relationships and adjusts detection confidence scores to improve the average precision. A model is learned using a structured support vector machine [1] from examples of 3D layouts of objects in offices and kitchens. We test our method on synthetic detections to determine how factors such as localization accuracy, number of detections and detection scores change the effectiveness of 3D spatial relationships for improving object detection rates. Finally, we describe a technique for generating 3D detections from 2D image-based object detections and demonstrate how our method improves the average precision of these 3D detections.

Improving object detection using 3D spatial relationships

Reliable object detection is one of the most significant hurdles that must be overcome to develop useful household robots. Overall, the goal of this work is to demonstrate how effective 3D qualitative spatial relationships can be for improving object detection. We show that robots can utilize 3D qualitative spatial relationships to improve object detection by differentiating between true and false positive detections. The main body of the thesis focuses on an approach for improving object detection rates that identifies the most likely subset of 3D detections using seven types of 3D relationships and adjusts detection confidence scores to improve the average precision. These seven 3D qualitative spatial relationships are adapted from 2D qualitative spatial reasoning techniques. We learn a model for identifying the most likely subset using a structured support vector machine [Tsochantaridis et al., 2004] from examples of 3D layouts of objects in offices and kitchens. We produce 3D detections from 2D detections using a fiducial marker and images of a scene and show our model is successful at significantly improving overall detection rates on real world scenes of both offices and kitchens. After the real world results, we test our method on synthetic detections where the properties of the 3D detections are controlled. Our approach improves on the model it was based upon, that of [Desai et al., 2009], by utilizing a branch and bound tree search to improve both training and inference. Our model relies on sufficient true positive detections in the training data or good localization of the true positive detections. Finally, we analyze the cumulative benefits of the spatial relationships and determine that the most effective spatial relationships depend on both the scene type and localization accuracy. We demonstrate that there is no one relationship that is sufficient on its own or always outperforms others and that a mixture of relationships is always useful.

Maps, Objects and Contexts for Robots

When you have brought your new Apple iRobot1 home for the first time, you are faced with the challenging task of introducing the robot to its new home/workspace. Of course the robot knows about homes and typical tasks. That’s why you bought the sleek, stylish robot, in addition to the fact that it promised a simple interface. It prompts you to take it on a tour of the house, naming the rooms, pointing out the appliances, and identifying the occupants of the house. The promise of the now discontinued Aibo whose communication and basic behaviours show that even simple visual sensors using strong features (SIFT[17]) can enable visual tracking and recognition. Built in to the home robot will be the necessary concepts – tasks, objects, contexts, locations. Your home vacuum robot “knows” only about stairs, objects, infrared walls, and random search. Your iRobot knows about kitchens, doors, stairs, bedrooms, beer (you have the party version of the iRobot that can bring you beer in the entertainment room). How does the robot tie the sensory flow it receives to its plans, names, and goals in its repertoire? This fanciful thought experiment is not so far in the future. For some applications such as assistive technologies[21, 1] which operate in contexts where rich visual sensing is deployed, the range of objects may be limited to a care facility where patients’ rooms are typically constrained in their contents. Here it may be effective to learn the connection between features of the visual stream and the object in the scene, and how they influence the actions of the robot.