Dean A. Pomerleau

Neural Network Perception for Mobile Robot Guidance

From the Publisher: Vision based mobile robot guidance has proven difficult for classical machine vision methods because of the diversity and real time constraints inherent in the task. This book describes a connectionist system called ALVINN (Autonomous Land Vehicle In a Neural Network) that overcomes these difficulties. ALVINN learns to guide mobile robots using the back-propagation training algorithm. Because of its ability to learn from example, ALVINN can adapt to new situations and therefore cope with the diversity of the autonomous navigation task. But real world problems like vision based mobile robot guidance present a different set of challenges for the connectionist paradigm. Among them are: how to develop a general representation from a limited amount of real training data, how to understand the internal representations developed by artificial neural networks, how to estimate the reliability of individual networks, how to combine multiple networks trained for different situations into a single system, and how to combine connectionist perception with symbolic reasoning. Neural Network Perception for Mobile Robot Guidance presents novel solutions to each of these problems. Using these techniques, the ALVINN system can learn to control an autonomous van in under 5 minutes by watching a person drive. Once trained, individual ALVINN networks can drive in a variety of circumstances, including single-lane paved and unpaved roads, and multi-lane lined and unlined roads, at speeds of up to 55 miles per hour. The techniques also are shown to generalize to the task of controlling the precise foot placement of a walking robot.

RALPH: rapidly adapting lateral position handler

Nearly 15,000 people die each year in the US in single vehicle roadway departure crashes. These accidents are often caused by driver inattention, or driver impairment (e.g. fatigued or intoxicated drivers). A system capable of warning the driver when the vehicle starts to depart the roadway, or controlling the lateral position of the vehicle to keep it in its lane, could potentially eliminate many of these crashes. This paper presents a system called RALPH (Rapidly Adapting Lateral Position Handler) which decomposes the problem of steering a vehicle into three steps, 1) sampling of the image, 2) determining the road curvature, and 3) determining the lateral offset of the vehicle relative to the lane center. The output of the later two steps are combined into a steering command, which can be compared with the human drivers current steering direction as part of a road departure warning system, or sent directly to the steering motor.

Rapidly adapting machine vision for automated vehicle steering

The Ralph vision system helps automobile drivers steer, by sampling an image, assessing the road curvature, and determining the lateral offset of the vehicle relative to the lane center. Ralph has performed well under extensive tests, including a coast-to-coast, 2,850-mile drive.

Efficient training of artificial neural networks for autonomous navigation

The ALVINN (Autonomous Land Vehicle In a Neural Network) project addresses the problem of training artificial neural networks in real time to perform difficult perception tasks. ALVINN is a backpropagation network designed to drive the CMU Navlab, a modified Chevy van. This paper describes the training techniques that allow ALVINN to learn in under 5 minutes to autonomously control the Navlab by watching the reactions of a human driver. Using these techniques, ALVINN has been trained to drive in a variety of circumstances including single-lane paved and unpaved roads, and multilane lined and unlined roads, at speeds of up to 20 miles per hour.

Food recognition using statistics of pairwise local features

Food recognition is difficult because food items are de-formable objects that exhibit significant variations in appearance. We believe the key to recognizing food is to exploit the spatial relationships between different ingredients (such as meat and bread in a sandwich). We propose a new representation for food items that calculates pairwise statistics between local features computed over a soft pixel-level segmentation of the image into eight ingredient types. We accumulate these statistics in a multi-dimensional histogram, which is then used as a feature vector for a discriminative classifier. Our experiments show that the proposed representation is significantly more accurate at identifying food than existing methods.

AURORA: a vision-based roadway departure warning system

AURORA is a vision-based system designed to warn a vehicle driver of possible impending roadway departure accidents. It employs a downward looking color video camera with a wide angle lens, a digitizer, and a portable Sun Sparc workstation. Using a binormalized adjustable template correlation algorithm, it reliably detects lane markers on structured roads at 60 Hz. A time-to-lane-crossing (TLC) measurement is calculated for each image based on the estimation of vehicles lateral position and velocity. This measurement is used to trigger an alarm when the TLC falls below a preset threshold. Promising results have been achieved under a variety of weather and lighting conditions, on many road types.

Neural Network Vision for Robot Driving

Autonomous navigation is a difficult problem for traditional vision and robotic techniques, primarily because of the noise and variability associated with real world scenes. Autonomous navigation systems based on traditional image processing and pattern recognition techniques often perform well under certain conditions, but have problems with others. Part of the difficulty stems from the fact that the processing performed by these systems remains fixed across various environments.

Expectation-based selective attention for visual monitoring and control of a robot vehicle

Reliable vision-based control of an autonomous vehicle requires the ability to focus attention on the important features in an input scene. Previous work with an autonomous lane following system, ALVINN (Pomerleau, 1993), has yielded good results in uncluttered conditions. This paper presents an artificial neural network based learning approach for handling difficult scenes which will confuse the ALVINN system. This work presents a mechanism for achieving task-specific focus of attention by exploiting temporal coherence. A saliency map, which is based upon a computed expectation of the contents of the inputs in the next time step, indicates which regions of the input retina are important for performing the task. The saliency map can be used to accentuate the features which are important for the task, and de-emphasize those which are not.

Visibility estimation from a moving vehicle using the RALPH vision system

Reduced visibility is a common casual factor in many traffic accidents. This paper describes a forward looking vision system which simultaneously tracks the lane and estimates visibility. The system estimates the visibility by measuring the attenuation of contrast between consistent road features at various distances ahead of the vehicle. Results of experiments on simulated images, as well as live vehicle tests are presented.

Knowledge-Based Training of Artificial Neural Networks for Autonomous Robot Driving

Many real world problems require a degree of flexibility that is difficult to achieve using hand programmed algorithms. One such domain is vision-based autonomous driving. In this task, the dual challenges of a constantly changing environment coupled with a real time processing constrain make the flexibility and efficiency of a machine learning system essential. This chapter describes just such a learning system, called ALVINN (Autonomous Land Vehicle In a Neural Network). It presents the neural network architecture and training techniques that allow ALVINN to drive in a variety of circumstances including single-lane paved and unpaved roads, multilane lined and unlined roads, and obstacle-ridden on-and off-road environments, at speeds of up to 55 miles per hour.