Christoph Gustav Keller

Making Bertha Drive?An Autonomous Journey on a Historic Route

125 years after Bertha Benz completed the first overland journey in automotive history, the Mercedes Benz S-Class S 500 INTELLIGENT DRIVE followed the same route from Mannheim to Pforzheim, Germany, in fully autonomous manner. The autonomous vehicle was equipped with close-to-production sensor hardware and relied solely on vision and radar sensors in combination with accurate digital maps to obtain a comprehensive understanding of complex traffic situations. The historic Bertha Benz Memorial Route is particularly challenging for autonomous driving. The course taken by the autonomous vehicle had a length of 103 km and covered rural roads, 23 small villages and major cities (e.g. downtown Mannheim and Heidelberg). The route posed a large variety of difficult traffic scenarios including intersections with and without traffic lights, roundabouts, and narrow passages with oncoming traffic. This paper gives an overview of the autonomous vehicle and presents details on vision and radar-based perception, digital road maps and video-based self-localization, as well as motion planning in complex urban scenarios.

Active Pedestrian Safety by Automatic Braking and Evasive Steering

Active safety systems hold great potential for reducing accident frequency and severity by warning the driver and/or exerting automatic vehicle control ahead of crashes. This paper presents a novel active pedestrian safety system that combines sensing, situation analysis, decision making, and vehicle control. The sensing component is based on stereo vision, and it fuses the following two complementary approaches for added robustness: 1) motion-based object detection and 2) pedestrian recognition. The highlight of the system is its ability to decide, within a split second, whether it will perform automatic braking or evasive steering and reliably execute this maneuver at relatively high vehicle speed (up to 50 km/h). We performed extensive precrash experiments with the system on the test track (22 scenarios with real pedestrians and a dummy). We obtained a significant benefit in detection performance and improved lateral velocity estimation by the fusion of motion-based object detection and pedestrian recognition. On a fully reproducible scenario subset, involving the dummy that laterally enters into the vehicle path from behind an occlusion, the system executed, in more than 40 trials, the intended vehicle action, i.e., automatic braking (if a full stop is still possible) or automatic evasive steering.

The Benefits of Dense Stereo for Pedestrian Detection

This paper presents a novel pedestrian detection system for intelligent vehicles. We propose the use of dense stereo for both the generation of regions of interest and pedestrian classification. Dense stereo allows the dynamic estimation of camera parameters and the road profile, which, in turn, provides strong scene constraints on possible pedestrian locations. For classification, we extract spatial features (gradient orientation histograms) directly from dense depth and intensity images. Both modalities are represented in terms of individual feature spaces, in which discriminative classifiers (linear support vector machines) are learned. We refrain from the construction of a joint feature space but instead employ a fusion of depth and intensity on the classifier level. Our experiments involve challenging image data captured in complex urban environments (i.e., undulating roads and speed bumps). Our results show a performance improvement by up to a factor of 7.5 at the classification level and up to a factor of 5 at the tracking level (reduction in false alarms at constant detection rates) over a system with static scene constraints and intensity-only classification.

Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.

A new benchmark for stereo-based pedestrian detection

Pedestrian detection is a rapidly evolving area in the intelligent vehicles domain. Stereo vision is an attractive sensor for this purpose. But unlike for monocular vision, there are no realistic, large scale benchmarks available for stereo-based pedestrian detection, to provide a common point of reference for evaluation. This paper introduces the Daimler Stereo-Vision Pedestrian Detection benchmark, which consists of several thousands of pedestrians in the training set, and a 27-min test drive through urban environment and associated vehicle data. The data, including ground truth, is made publicly available for non-commercial purposes. The paper furthermore quantifies the benefit of stereo vision for ROI generation and localization; at equal detection rates, false positives are reduced by a factor of 4–5 with stereo over mono, using the same HOG/linSVM classification component.

Video based localization for Bertha

In August 2013, the modified Mercedes-Benz SClass S500 Intelligent Drive (“Bertha”) completed the historic Bertha-Benz-Memorial-Route fully autonomously. The self-driving 103 km journey passed through urban and rural areas. The system used detailed geometric maps to supplement its online perception systems. A map based approach is only feasible if a precise, map relative localization is provided. The purpose of this paper is to give a survey on this corner stone of the system architecture. Two supplementary vision based localization methods have been developed. One of them is based on the detection of lane markings and similar road elements, the other exploits descriptors for point shaped features. A final filter step combines both estimates while handling out-of-sequence measurements correctly.

Will the pedestrian cross?: probabilistic path prediction based on learned motion features

Future vehicle systems for active pedestrian safety will not only require a high recognition performance, but also an accurate analysis of the developing traffic situation. In this paper, we present a system for pedestrian action classification (walking vs. stopping) and path prediction at short, sub-second time intervals. Apart from the use of positional cues, obtained by a pedestrian detector, we extract motion features from dense optical flow. These augmented features are used in a probabilistic trajectory matching and filtering framework. The vehicle-based system was tested in various traffic scenes. We compare its performance to that of a state-of-the-art IMM Kalman filter (IMM-KF), and for the action classification task, to that of human observers, as well. Results show that human performance is best, followed by that of the proposed system, which outperforms the IMM-KF and the simpler system variants.

Dense Stereo-Based ROI Generation for Pedestrian Detection

This paper investigates the benefit of dense stereo for the ROI generation stage of a pedestrian detection system. Dense disparity maps allow an accurate estimation of the camera height, pitch angle and vertical road profile, which in turn enables a more precise specification of the areas on the ground where pedestrians are to be expected. An experimental comparison between sparse and dense stereo approaches is carried out on image data captured in complex urban environments (i.e. undulating roads, speed bumps). The ROI generation stage, based on dense stereo and specific camera and road parameter estimation, results in a detection performance improvement of factor five over the state-of-the-art based on ROI generation by sparse stereo. Interestingly, the added processing cost of computing dense disparity maps is at least partially amortized by the fewer ROIs that need to be processed at the system level.

Multi trajectory pose adjustment for life-long mapping

State of the art highly automated and self-driving vehicles heavily depend on detailed maps since they free the system from many otherwise complex onboard processing tasks. However, depending on the environment and the fineness of the map, the validity span of maps is often short and a periodic remapping of large areas with sensor-packed mapping vehicles is beyond any feasibility. Crowd based mapping approaches using low cost sensors appear more practicable. Herein we propose a general method to align several trajectories of the same area which is fundamental for any life-long mapping. Our algorithm requires previously acquired pose differences as input. These differences induce a pose graph which is aligned yielding a minimum least-squares residual. Therefore, our method is independent from the underlying sensor technology. For evaluation purposes, we align pose graphs from simulated pose differences and compare it against the ground truth. Furthermore, stereo cameras are used to obtain pose difference estimates by common visual odometry methods. We present quantitative results of the robustness and accuracy of our method based on these pose differences. The results are compared against a high precision GPS receiver. Our approach clearly outperforms this costly reference sensor.

Robust localization based on radar signal clustering

Significant advances have been achieved in mobile robot localization and mapping in dynamic environments, however these are mostly incapable of dealing with the physical properties of automotive radar sensors. In this paper we present an accurate and robust solution to this problem, by introducing a memory efficient cluster map representation. Our approach is validated by experiments that took place on a public parking space with pedestrians, moving cars, as well as different parking configurations to provide a challenging dynamic environment. The results prove its ability to reproducibly localize our vehicle within an error margin of below 1% with respect to ground truth using only point based radar targets. A decay process enables our map representation to support local updates.