Hans-Joachim Boehme

TOOMAS: Interactive Shopping Guide robots in everyday use - final implementation and experiences from long-term field trials

The paper gives a comprehensive overview of our Shopping Guide project, which aims at the development of interactive mobile shopping companion robots for everyday use in challenging operating environments such as home improvement stores. It is spanning an arc from the expectations and requirements of store owners and customers, via the challenges of the shopping scenario and the operating environment, the implemented functionality of the shopping guide robots, up to the results of long-term field trials. The field trials started in April 2008 and still ongoing aim at studying whether and how a group of interactive mobile shopping guide robots can operate completely autonomously in such everyday environments and how they are accepted by uninstructed customers. In these field trials, where nine robotic shopping guides traveled together 2187 kilometers in three different home improvement stores in Germany, more than 8,600 customers were successfully guided to the locations of their products of choice. With the successful development of these shopping guide robots, a further important step towards assistive robotics for daily use has been done.

ShopBot: Progress in developing an interactive mobile shopping assistant for everyday use

The paper describes progress achieved in our long-term research project ShopBot, which aims at the development of an intelligent and interactive mobile shopping assistant for everyday use in shopping centers or home improvement stores. It is focusing on recent progress concerning two important methodological aspects: (i) the on-line building of maps of the operation area by means of advanced Rao-Blackwellized SLAM approaches using both sonar-based gridmaps as well as vision-based graph maps as representations, and (ii) a probabilistic approach to multi-modal user detection and tracking during the guidance tour. Experimental results of both the map building characteristics and the person tracking behavior achieved in an ordinary home improvement store demonstrate the reliability of both approaches. Moreover, we present first very encouraging results of long-term field trials which have been executed with three robotic shopping assistants in another home improvement store in Bavaria since March 2008. In this field test, the robots could demonstrate their suitability for this challenging real-world application, as well as the necessary user acceptance.

Vision-based Monte Carlo self-localization for a mobile service robot acting as shopping assistant in a home store

We present a novel omnivision-based robot localization approach which utilizes the Monte Carlo Localization (MCL), a Bayesian filtering technique based on a density representation by means of particles. The capability of this method to approximate arbitrary likelihood densities is a crucial property for dealing with highly ambiguous localization hypotheses as are typical for real-world environments. We show how omnidirectional imaging can be combined with the MCL-algorithm to globally localize and track a mobile robot given a taught graph-based representation of the operation area. In contrast to other approaches, the nodes of our graph are labeled with both visual feature vectors extracted from the omnidirectional image, and odometric data about the pose of the robot at the moment of the node insertion (position and heading direction). To demonstrate the reliability of our approach, we present first experimental results in the context of a challenging robotics application, the self-localization of a mobile service robot acting as shopping assistant in a very regularly structured, maze-like and crowded environment, a home store.

Omnivision-based probabilistic self-localization for a mobile shopping assistant continued

The basic idea of our omniview-based MCL approach and preliminary experimental results were presented in our previous paper [Proc. IROS 2002, pp. 256-262]. In continuing, this paper describes a number of methodical and technical improvements addressing challenges arising from the characteristics of our real-world application, the vision-based self-localization of a mobile robot that acts as a shopping assistant in the maze-like environment of a home store. To cope with highly variable illumination conditions, we present a reference-based correction approach that realizes a robust, automatic luminance stabilization and color adaptation already at the level of image formation. To deal with severe occlusions or disturbances of the omnidirectional image caused by, e.g. people standing near the robot or local illumination artifacts, we introduce a novel selective observation comparison method as prerequisite for a robust particle filter update. Further studies investigate the impact of the utilized observation model on the localization accuracy. The results of a series of localization experiments carried out in the home store confirm the robustness and superiority of our advanced, real-time approach.

PERSES-a vision-based interactive mobile shopping assistant

The paper describes the general idea, the application scenario, and selected methodological approaches of our long term research project PERSES (PERsonal SErvice System). The aim of the project consists of the development of an interactive mobile shopping assistant that allows a continuous and intuitively understandable interaction with a customer in a home improvement store. Typical tasks we have to tackle are to detect and contact potential users in the operation area, to guide them to desired areas or articles within the store or to follow them as a mobile information kiosk while continuously observing their behavior. Due to the specificity of the interaction-oriented scenario and the characteristics of the operation area, we have focused on vision based methods for both human-robot interaction and robot navigation. Besides some methodological approaches, we present preliminary results of experiments achieved with our mobile robot PERSES in the store with an emphasis on vision based methods for user localization, map building and self-localization.

User localisation for visually-based human-machine-interaction

Recently there is an increasing interest in video based interface techniques, allowing more natural interaction between users and systems than common interface devices do. We present a neural architecture for user localisation, embedded within a complex system for visually-based human machine interaction (HMI). User localisation is an absolute prerequisite to video based HMI. Due to the main objective, the greatest possible robustness of the localisation as well as the whole visual interface under highly varying environmental conditions, we propose a multiple cue approach. This approach combines the features of facial structure, head shoulder contour, skin color and motion, with a multiscale representation. The selection of the image region most likely containing a possible user is then realised via a WTA-process within the multiscale representation. Preliminary results show the reliability of the multiple cue approach.

Contribution to vision-based localization, tracking and navigation methods for an interactive mobile service-robot

Presents vision-based robot navigation and user localization techniques of our long-term research project PERSES (personal service system), which aims to develop an interactive mobile shopping assistant that allows a continuous and intuitively understandable interaction with customers in a home store. Against this background, the paper describes a number of new or improved approaches, addressing challenges arising from the characteristics of the operation area, and from the need to continuously interact with users in a complex environment. With our approaches to vision-based or visually-controlled map building, self-localization and navigation as well as user localization and tracking, we want to make a contribution to the real-world suitability of interactive mobile service-robots in non-trivial application areas and demanding human-robot interaction scenarios.

Extraction of Orientation from Floor Structure for Odometry Correction in Mobile Robotics

We are presenting a method for correcting odometry readings of a robot for increased accuracy of position estimation. Our method uses a simple pragmatic approach and exploits the distinct structure of the floor in our experimental area. By continually extracting orientation information from the floor view, we are able to correct the heading component of odometry, thereby eliminating the major source for position errors. Compared to other approaches the solution is computationally inexpensive. Our experiments show that by employing our correction method we are able to significantly increase position accuracy and consistently map paths up to several hundred meters.

A HYBRID STOCHASTIC-CONNECTIONIST APPROACH TO GESTURE RECOGNITION

In this paper a person-specific saliency system and subsequently two architectures for the recognition of dynamic gestures are described. The systems implemented are designed to take a sequence of images and to assign it to one of a number of discrete classes where each of them corresponds to a gesture from a predefined small vocabulary. Since we think that for a human-computer interaction the localization of the user is essential for any further step regarding the recognition and the interpretation of gestures, in the first part, we begin with describing our saliency system dedicated to the person localization task in cluttered environments. Successively, the intrinsic gesture recognition process is broken down into an initial preprocessing stage followed by a mapping from the preprocessed input variables to an output variable representing the class label. Subsequently, we utilize two different classifiers for mapping the ordered sequence of feature vectors to one gesture category. The first classifier utilizes a hybrid combination of Kohonen Self-Organizing Map (SOM) and Discrete Hidden Markov Models (DHMM). As second recognizer a system of Continuous Hidden Markov Models (CHMM) is used. Preliminary experiments with our baseline systems are demonstrated.

Awakening history: Preparing a museum tour guide robot for augmenting exhibits

While the idea of tour guide robots has been addressed several times, applying a video-projector based augmented reality component to this scenario is rather new. We show requirements of the localization system of the robot and how they can be fulfilled, as well as a basic system for projection correction and its suitability for this scenario.