Experimental Comparison of Global Motion Planning Algorithms for Wheeled Mobile Robots
Eric Heiden, Luigi Palmieri, Kai O. Arras, Gaurav S. Sukhatme, Sven Koenig
EExperimental Comparison of Global Motion Planning Algorithmsfor Wheeled Mobile Robots
Eric Heiden ∗ , Luigi Palmieri ∗ , Kai O. Arras , Gaurav S. Sukhatme and Sven Koenig Abstract — Planning smooth and energy-efficient motions forwheeled mobile robots is a central task for applicationsranging from autonomous driving to service and intralogisticrobotics. Over the past decades, a wide variety of motionplanners, steer functions and path-improvement techniqueshave been proposed for such non-holonomic systems. Withthe objective of comparing this large assortment of state-of-the-art motion-planning techniques, we introduce a novelopen-source motion-planning benchmark for wheeled mobilerobots, whose scenarios resemble real-world applications (suchas navigating warehouses, moving in cluttered cities or parking),and propose metrics for planning efficiency and path quality.Our benchmark is easy to use and extend, and thus allowspractitioners and researchers to evaluate new motion-planningalgorithms, scenarios and metrics easily. We use our benchmarkto highlight the strengths and weaknesses of several commonstate-of-the-art motion planners and provide recommendationson when they should be used.
I. I
NTRODUCTION
Motion planning is a central component in the applicationof mobile robots to various important real-world domains,such as autonomous driving, warehouse logistics, and servicerobotics [1]. Besides finding complex paths in obstacle-richenvironments, they need to account for the kinodynamicconstraints that a wheeled system enforces on the state space.In particular, in this work, we focus on global motionplanning algorithms that find paths in large, cluttered andcomplex environments, often by considering only static orsemi-static information of the environment and an approxi-mate robot dynamics model.Over the years, various motion planning algorithms, steerfunctions, and path improvement (so-called post-smoothing)methods have been introduced, while the interest in au-tonomous robots navigating complex spaces is ever increas-ing. To investigate the current state of the art in motion plan-ning for wheeled mobile robots, in this work, we establish abenchmarking framework that is tailored toward these kindsof kinodynamic systems and their application in real-worldscenarios.As shown in Figure 1, our benchmarking is based onthe following ingredients: motion planners, post-smoothingmethods, steer functions and collision checkers. The combi-nation of these building blocks is then evaluated in a variety ∗ Equal contribution E. Heiden, G. S. Sukhatme, S. Koenig are with the Department ofComputer Science, University of Southern California, Los Angeles, USA { heiden, gaurav, skoenig } @usc.edu. L. Palmieri and K. O. Arras are with Robert Bosch GmbH,Corporate Research, Stuttgart, Germany { Luigi.Palmieri,KaiOliver.Arras } @de.bosch.com Often in literature denoted also as steering or extend function.
Planners Sampling-based planners Anytime planners State lattice-based planners
Metrics Exact / collision-free solutions Path length Maximum curvature Computation time Mean clearing distance Number of cusps
Post-smoothing algorithms Simplify Max B-Spline Shortcut GRIPS
Steer functions Dubins Reeds-Shepp CC Reeds-Shepp POSQ
Scenarios Parking (parallel, forward, reverse) Warehouse navigation Procedural grids of varying obstacle densities, corridor sizes Grid environments from Moving AI Cities benchmark: Berlin (256x256) NewYork (512x512) Boston (1024x1024)
Python front-end
C++ back-end
Collision checking Polygon-based
Fig. 1. Architecture of the proposed motion-planning benchmarkingframework. The components necessary for motion planning are shown inthe box on the left (green), and the ingredients used in the evaluation areshown in the box on the right (red). The implementation is split into a C++back-end for running the resource-intensive motion-planning components,and a Python front-end for providing a flexible interface to the design andevaluation of the benchmarking scenarios through Jupyter notebooks. of scenarios (environments with start and goal configura-tions) along various metrics.In a typical experiment, the scenario determines theenvironment and the start and goal configuration. A mo-tion planner is instantiated with a defined steer functionto connect two vertices during the search while ensuringkinodynamic feasibility. Throughout the planning phase, acollision checker validates the currently considered solutionwith regards to the shape of the robot and the obstacles in theenvironment. After the planner has found a feasible solution,it can optionally be improved through post-smoothing algo-rithms that modify the path in order to reduce path lengthand curvature.Each of these ingredients are chosen carefully to ensurethey match our application constraints. For example, wefocus solely on polygon-based collision checking that placesan additional burden on planners that make heavy use ofthe state validation test. In other experiments, we investigatehow post-smoothing methods can benefit the efficiency ofthe planning framework and show that in some cases the fastsolutions found by feasible sampling-based motion planners[2], [3] can be smoothed in a way that they outperform theirslower, albeit asymptotically optimal [4], anytime motionplanning counterparts. Drawing from conclusions of thesefindings, we give recommendations on the combination of a r X i v : . [ c s . R O ] M a r he considered motion-planning components for particularapplication areas.While much of our benchmarking software largely buildson the Open Motion Planning Library (OMPL) [5], weprovide interfaces to implementations of planners (such asSBPL planners and Theta ∗ ) and steer functions (POSQand continuous-curvature steering) outside of OMPL. Thisenables us to get a more comprehensive picture of the currentprogress in motion planning for wheeled mobile robots,while being more agnostic to particular implementations ofthe building blocks.II. R ELATED W ORK
Several benchmarks have been proposed recently for ana-lyzing the performance of different planning algorithms fora large variety of robotic systems [6], [7], [8], [9], [10], [11],[12]. All of them are generic and do not deeply analyse, as inour case, algorithms’ planning performance for the specificcase of wheeled mobile robots. Following we detail the mostprominent ones.The Pathfinding Benchmarks [10] from the Moving AI labare designed for 2D path finders which consider no kinematicconstraints of the robotics system. The benchmark offers alarge set of scenarios (start, goal positions with length ofoptimal 2D path) for each grid map (of different nature i.e.mazes, cities). Contrarily to the latter, our benchmark, whichuses some of its maps, considers different nonholonomicsystems and different evaluation metrics rather than only pathlength.Althoff et al. [6] propose a composable benchmark formotion planning and control on roads, specific for carsautonomously driving on road networks’ lanes. Differentlyour benchmark focuses its attention on static open spaces(indoor and outdoor) where a planner finds a global path formaneuvering an autonomous system (i.e. differential driverobot or car).Moll et al. [7] introduce a generic benchmarking tool formotion planning algorithms highly coupled with OMPL [5].This benchmark suite is highly customizable (it is straight-forward to integrate novel collision checkers or sampling-based planners) but lacks of specific benchmark scenarios formobile robotics applications. On the contrary, we proposea benchmark that contains a set of scenarios, problems tosolve and metrics specific for mobile robotics settings. Also[11] collects a set of classical benchmarks (e.g. alpha puzzle,bug-trap) for different systems, but the benchmark offersa relatively small amount of examples for wheeled mobilerobots.Similarly to the benchmark presented by Cohen etal. [8] and differently from [7], our approach enablesresearchers and practitioners to test different classes ofmotion planners, i.e., sampling-based planners (e.g. RRT ∗ ,PRM ∗ [4], RRT [13]), discrete-search approaches (e.g.A ∗ [14], Theta ∗ [15]), state-lattice based planners (e.g.ARA ∗ [16], ANA ∗ [17]). Differently from both of them,we provide several definitions of publicly-available steering functions for wheeled mobile robots (e.g. POSQ [18], Con-tinuous Curvature [19], Reeds-Shepp [20], and Dubins [21]).Luo et al. [9] introduce a benchmark on asymptoticallyoptimal planners. These are compared on four differentenvironments, with a single pair of predefined start and goalposes. The study considers only straight line connections(no particular kinematics or nonholonomic constraints). Inthis work, we propose a benchmark with a much largerselection of diverse environments, and consider differentnonholonomic constraints.Several other works [22], [23], [24], [25] have presentedapproaches to benchmark motion planning algorithms ofrobots moving in dynamic environments. Our work focusesits attention to planning considering a current static descrip-tion of the environment: a fundamental single planning stepperformed during robot navigation in dynamic environments.III. A PPROACH
In this paper, we benchmark global motion planningalgorithms commonly used for wheeled mobile robots, andprovide general recommendations on the usage of thesemethods, considering their combination with post-smoothingmethods and various steer functions. Our benchmark is basedon two fundamental pillars: the components involved inmotion planning and the evaluation procedures (shown inthe box on the left and right, respectively, in Figure 1). Inparticular, evaluating the performance of a motion planningalgorithm requires selecting the appropriate testing environ-ments (e.g., considering different types of map representa-tions) and metrics (related to planning efficiency and qualityof the results). We carefully selected these components byconsidering their scientific impact, and their recognition andpopularity in the open source community [5], [26], [19]. Ourchoices are thoroughly presented in Sections IV-V and willbe used to solve the following motion planning problems.
A. Motion Planning Problem
Let X ⊂ R D be a manifold defining a configuration space, U ⊂ R M the symmetric control space, X obs ⊂ X the obstaclespace and X free = X \ X obs the free space. A wheeled mobilerobot can be described by an ordinary differential equationdenoting a driftless control-affine system [27]:˙ x ( t ) = M ∑ j = g j ( x ( t )) u ( t ) (1)where x ( t ) ∈ X is the state of the system, u ( t ) ∈ U the controlapplied to it, for all t , and g , . . . , g M are the system vectorfields on X .Let γ denote a planning query, defined by its initial state x start ∈ X and goal state x goal ∈ X . We define the set of allpossible solution paths for a given query γ as Σ γ , with σ ∈ Σ γ : [ , ] → X free being one of the possible solutions suchthat σ ( ) = x start and σ ( ) = x goal . The arc-length of a path σ is defined by l ( σ ) = (cid:82) || ˙ σ ( t ) || dt . The arc-length inducesa sub-Riemannian distance dist on X : dist ( x , z ) = inf σ l ( σ ) ,i.e., the length of the optimal path connecting x to z , whichdue to our assumptions is also symmetric. Let σ ∗ denote theet of all points along a path σ . The dist-clearance of a path σ is defined as δ dist ( σ ) = sup (cid:8) r ∈ R | R dist ( x , r ) ⊆ X free ∀ x ∈ σ ∗ (cid:9) (2)where R dist ( x , r ) is the cost-limited reachable set for thesystem in Eq. 1 centered at x within a path length of r (e.g.,a sphere for Euclidean systems): R dist ( x , r ) = (cid:8) z ∈ X | dist ( x , z ) ≤ r (cid:9) . (3)The dist-clearance of a query γ is defined as δ dist ( γ ) = sup (cid:8) δ dist ( σ ) | σ ∈ Σ γ (cid:9) (4)and denotes the maximum clearance that a solution path toa query can have. A planning algorithm solves the followingˆ δ dist -robustly feasible motion planning problem P : given aquery ˆ γ with a dist-clearance of δ dist ( ˆ γ ) > ˆ δ dist , find a control u ( t ) ∈ U with domain [ , ] such that the unique trajectory σ satisfying Equation 1 is fully contained in the free space X free ⊆ X and connects x start to x goal . Moreover, in caseof (asymptotically) optimal planning, the planner minimizes(as the number of samples goes to infinity) a defined costfunction c : Σ γ → R ≥ . Hereinafter, we will use the term steer function to indicate a function that generates a path in X connecting two specified states.IV. P LANNING C OMPONENTS
In this section we detail the ingredients used in our bench-marking framework, i.e., motion planners, post-smoothingmethods, collision checkers, and steer functions (see Fig-ure 1).
A. Motion Planners
We compare a variety of planners belonging to fourdifferent families, namely feasible sampling-based motionplanners , any-angle path planners , anytime or asymptoticallyoptimal motion planners and state-lattice-based planners .We choose the most prominent open-source available plan-ners for each class.
1) Feasible Sampling-based Motion Planners:
To thisclass of planners belong all the planners that are onlyprobabilistically complete, i.e., the planner will find a pathwith a probability of one if the number of samples goesto infinity. We adopt the following ones from the OMPLlibrary: RRT [13], Stable Sparse RRT (SST) [28], EST [29],SBL [30], PDST [31], PRM [3], SPARS [32], SPARS2 [33].For all of them we use a uniform distribution with goalbiasing, we plan in future to extend it also to deterministicsampling approaches [34], [35], [36]. For the sake of brevity we leave out detailed explanations of the planningalgorithms and direct the reader to the corresponding references.
2) Anytime or Asymptotically Optimal Sampling-basedMotion Planners:
In contrast to the planners from Sec. IV-A.1, anytime, or optimal, sampling-based motion plannersare asymptotically optimal planners (i.e. the probability offinding an optimal solution approaches one as the num-ber of samples increases to infinity). This category in-cludes the following planners from OMPL: RRT ∗ [4], In-formed RRT ∗ [37], SORRT ∗ [38], BIT ∗ [39], RRT ∗ [41], PRM ∗ [4], and CForest [42]. In this class, weadditionally include planners that perform informed search(Informed RRT ∗ , SORRT ∗ , BIT ∗ ) or use multiple trees inparallel (CForest). We configure these algorithms to samplefrom a uniform distribution with goal biasing.
3) Any-Angle Path Planners:
In contrast to classical grid-based path finding approaches, such as A ∗ , any-angle plan-ners do not constrain their solutions to grid edges. Dueto their advantageous smoothness and planning efficiency,we choose this class of planners, instead of classical pathplanners on the grid. In particular, in our benchmarking,we adopt the algorithm Theta ∗ [15] by also consideringconnections between grid points (thus samples from theconfiguration space) generated by a steer function. To enableTheta ∗ to use steer functions, we leverage the approachpresented in [43].
4) State-Lattice-based Planners:
In this benchmark, weinclude state-lattice-based planners that use deterministicsampling. In particular, they approximate the configurationby using a state lattice generated through a forward ap-proach [44]. We make use of the SBPL library [16] withthe following planners: ARA ∗ [16], AD ∗ [26], MHA ∗ [45]. B. Steer Functions
Throughout this benchmark we consider wheeled mo-bile robots with nonholonomic constraints. Connecting twostates for this class of systems is known as solving a two-point boundary value problem (2P-BVP) which is typicallyaccomplished by a steer function [27]. In the following,we introduce the steer functions used in our benchmark,namely: Dubins, Reeds-Sheep, Continuous Curvature, POSQand motion primitives. For all of them we use the followingkinematic model: ˙ x = v cos θ , ˙ y = v sin θ , ˙ θ = ω , with x , y being the robot Euclidean coordinates measured against afixed world frame, θ the robot heading, v its tangentialvelocity, and ω its angular velocity.
1) Dubins Curves:
Dubins et al. [21] (DS) assume a cardriving with constant speed v = ω = ω = ω = −
2) Reeds-Shepp:
Reeds-Shepp curves [20] (RS) are anextension of Dubins steering. Besides the Dubins primitives,Reeds-Shepp curves consider a car that can also movebackwards with constant speed (thus v can be − +
3) Continuous Curvature Steer Functions:
Due to theirsystem definition, Reeds-Shepp and Dubins steering requirethe system to be stopped each time a new turn is requested.To counteract this issue, Fraichard et al. [19] propose anew class of steer functions for car-like kinematics called continuous curvature (CC) steering functions. Differentlyfrom Reeds-Shepp and Dubins, continuous-curvature steerfunctions enforce continuity on the curvature κ of the paths(extending the state to also considering the curvature). Byconsidering the curvature, the complexity of finding an opti-mal path slightly increases when compared to Reeds-Sheppand Dubins. Banzhaf et al. [46] further extend this class byallowing the continuous-curvature functions to fall back tothe Reeds-Shepp family, thus having curvature discontinu-ities at switches in the driving direction, a useful propertywhen operating in very cluttered environments (i.e., the caris allowed to turn the steering wheel while not moving). Asrepresentative of the continuous-curvature steer functions, weinclude continuous-curvature Reeds-Shepp steering (referredto as CC Reeds-Shepp) in this benchmark.
4) POSQ:
In [18], the authors exponentially solve the2P-BVP for the kinematic car-like system by extending thediscontinuous control approach developed by Astolfi et al.[47]. The approach, unlike Dubins or Reeds-Shepp, doesnot produce optimal paths, but it was nonetheless shownto produce smooth paths. Moreover it does not considerconstant velocities, thus allowing the robot to move morefreely in cluttered environments.
5) Motion Primitives:
State lattice planning uses a setof precomputed motion primitives (pairs of v , ω ) instead ofsteer functions. Hence, the approach does not fully solve the2P-BVP. In this benchmark, we use a forward-propagationapproach and use unicycle motion primitives for all the SBPLplanners, which are available from the SBPL repository. C. Post-smoothing Methods
Besides the planners, in this benchmark, we take algo-rithms for path improvement into consideration. We adoptexisting post-smoothing methods from the OMPL library,namely the B-Spline, Shortcut and Simplify Max algo-rithms [5]. In addition, we compare them against the recentlyintroduced gradient-informed post smoothing (GRIPS) algo-rithm [48], a hybrid approach that uses short-cutting andlocally optimizes vertexes placement.
D. Collision Checking
Throughout all our experiments, we use two-dimensionalpolygon-based collision models (see Figure 2) where therobot is represented by a convex shape. Based on the
Separating Axis Theorem [49], we check for intersectionsbetween the robot and the obstacle polygons. Compared totesting for point collision, this model is significantly moredemanding to evaluate, resulting in larger computation timesfor state validation checks. However, since we are interestedin motion planners that are relevant to mobile robots, suchcollision models need to be taken into account.
Experiment / Sec-tion Environment Collisionmodel Description cross corridor subsubsection VII-C.1 100 ×
100 grid(procedural) car Evaluation on procedurally gen-erated corridor environments withvarying corridor diameters (Fig-ure 3 bottom) cross turning subsubsection VII-C.2 100 ×
100 grid(procedural) car Evaluation on procedurally gener-ated grid environments with vary-ing turning radii in Reeds Sheppsteering cross density subsubsection VII-C.3 100 ×
100 grid(procedural) car Evaluation on procedurally gener-ated grid environments with vary-ing obstacle densities (Figure 3top) sam vs any subsection VII-D 150 ×
150 grid(procedural) car Comparison of anytime plannersvs. a combination of sampling-based planners and post-smoothingmethods
Berlin 0 256 subsection VII-A 256 ×
256 grid(MovingAI) car Evaluation of the 50 hardestscenarios from the Berlin 0 256MovingAI benchmark
NewYork 1 512 subsection VII-A 512 ×
512 grid(MovingAI) car Evaluation of the 50 hardest sce-narios from the NewYork 1 512MovingAI benchmark
Boston 1 1024 subsection VII-A 1024 × parking 1 subsubsection VII-B.1 polygon car Evaluation on the polygon-basedenvironment parking 1 (Figure 4) parking 2 subsubsection VII-B.1 polygon car Evaluation on the polygon-basedenvironment parking 2 (Figure 4) parking 3 subsubsection VII-B.1 polygon car Evaluation on the polygon-basedenvironment parking 3 (Figure 4) warehouse subsubsection VII-B.2 polygon warehousebot Evaluation on the polygon-basedenvironment warehouse (Figure 4) TABLE IO
VERVIEW OF EXPERIMENTS CONDUCTED IN THIS BENCHMARK . V. E
VALUATION
In this section, we describe the set of experiments, en-vironments and the metrics used to evaluate the plannersand post-smoothing methods in terms of planning efficiencyand in returned path quality. The list of all experiments,pointing to the related results’ sections, is reported in Table I.The table collects eleven different types of experiment werun: three of them use the Moving-AI grid environmentsdescribed in Section V-A.1.a, four the procedurally generatedgrids detailed in Section V-A.1.b, and four use a polygonalrepresentation of the environment and robot (see Section V-A.2). Of the latter, three study the behavior of the plannerswhen the environment complexity change and one propertiesof post-smoothers and planners’ combinations.
A. Environments
In the following, we describe the two types of envi-ronments we consider throughout our benchmarking, aswell as the how the scenarios are defined, i.e. the startand goal configurations for each environment. We considerthe two main classes of environmental representation usednowadays for motion planning: grids and the polygon-basedones. Section V-A.1 details a set of experiments based ongrid representations of the obstacles, a typical approachused in robotics navigation, in particular when planningin large environments (i.e. cities, airports, train stations orlarge office-like environments). Polygon-based environments,described in Section V-A.2, are often adopted when planningin tight and small environments (i.e. parking and warehouse ig. 2. The two different polygon-based collision models used throughoutthis benchmark. like environments), where the planning system should morecarefully and precisely consider obstacles’ geometry.
1) Grid-based Environments:
We design two sets of en-vironments, a sub-selection of the grids form the Moving AIbenchmark [10] and a set of grids procedurally generated byvarying corridors’ size or obstacles density. a) Moving AI environments:
The Cities Dataset fromMoving AI Lab’s Pathfinding Benchmarks [10] containsoccupancy grid maps of various cities at varying resolutions,ranging between 256 ×
256 and 1024 × b) Procedurally-generated environments: Besides theMoving AI environments, we generate grid mazes procedu-rally to investigate how the planners behave under specificconditions that influence the available free space. As shownin the bottom row of Figure 3, the grid worlds we generateresemble typical indoor scenes with complex networks ofrectangular spaces, such as rooms and corridors. We generatethese environments by starting with a completely occupiedgrid and apply a few iterations from a modified RRT ex-ploration that only connects the nearest tree node to therandomly sampled point via either horizontal or vertical linesof a certain width. This allows us to generate environments,with different corridor sizes (see bottom row in Figure 3).Following from the RRT tree that generates the free spacein our procedural grid environments, we select the furthesttwo points in the tree as the start and goal positions foreach scenario. Additionally, to compare the planners on moregeneric environments, we implemented environments wherecells are randomly sampled from a uniform distribution andset to being occupied. This random process is repeated untila desired ratio of occupied versus free cells has been reached,as shown in the top row of Figure 3.
2) Polygon-based Environments:
Furthermore our bench-mark includes environments where the obstacles are repre-sented by convex shapes and the robot itself is analogouslyrepresented by a polygon. Scenarios of this kind come closeto real-world, two-dimensional navigation scenarios wherethe collision checker has to take into account the geometryof the robot and its environment to evaluate the validityof state. For example, in the case of a robot representedby an elongated rectangle, the orientation angle can greatlyinfluence whether a narrow pass in the environment can be
Fig. 3. Examples of the procedurally generated grid environments. Top:varying obstacle ratios. Bottom: varying corridor sizes.Fig. 4. The four polygon-based environments where obstacles are repre-sented by convex shapes. traversed, whereas in the point-based collision model suchconsiderations need not be made.We show the four types of polygon-based environments wedesigned in Figure 4 and with example paths in Figure 5. Wechoose five start-and-goal configurations and validate themby ensuring that the planner BFMT finds exact solutionsusing the Reeds Shepp steer function (Figure 4). In thefirst three cases, we consider the scenario of an autonomouscar-like vehicle that needs to park itself among a set ofsurrounding parked cars and other obstacles. We considerthe three common cases of parking: (1) parking forward, (2)parking backward into a parking lot, and (3) parallel-parkin a street of parked cars. In the last type of polygon-basedenvironments (4), a complex warehouse-like environment issimulated where the robot has to navigate between shelvesof various sizes and irregular orientations. B. Metrics
We compare the planners based on a selection of metricsrelevant to wheeled mobile robotics applications, such asautonomous driving, service and intralogistic robotics. Inparticular, we evaluate the planners in terms of quality of thereturned solutions and in planning efficiency by consideringthe following metrics: • Success statistics that measure the ratio of found,collision-free, and exact solutions. • Path length of the obtained solution in the workspace W . All the asymptotically optimal planners are config-ured to minimize path length, thus we measure how wellthe planners performs based on their main objective. Through preliminary experiments, we found BFMT to be among themost reliable planning algorithms that gave high-quality solutions in shorttime on the polygon-based environments. A trajectory is exact if it connects the start and goal nodes. .0 2.5 5.0 7.5 10.0 12.5 15.0−8−6−4−2024 0.0 2.5 5.0 7.5 10.0 12.5 15.0 0.0 2.5 5.0 7.5 10.0 12.5 15.0 0.0 2.5 5.0 7.5 10.0 12.5 15.0 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Informed RRT*StartGoal0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5−8−6−4−2024 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Informed RRT*StartGoal−5 0 5 10 15 20 25−15−10−505 −5 0 5 10 15 20 25 −5 0 5 10 15 20 25 −5 0 5 10 15 20 25 −5 0 5 10 15 20 25 Informed RRT*StartGoal−10 0 10 20 30 40 50 60 70−60−50−40−30−20−10010 −10 0 10 20 30 40 50 60 70 −10 0 10 20 30 40 50 60 70 −10 0 10 20 30 40 50 60 70 −10 0 10 20 30 40 50 60 70 Informed RRT*StartGoal
Fig. 5. Exemplary results for the polygon-based environments parking1 , parking2 , parking3 , and warehouse (from top to bottom) with all five differentstart/goal configurations. Each subplot shows the computed trajectories from the Informed RRT ∗ planner using the CC Reeds-Shepp steer function. • Curvature ( κ ) and Maximum curvature ( κ max ): as a wayto measure the induced comfort and smoothness of theobtained paths. Keeping the maximum curvature at alow level corresponds to smoother maneuvers, thereforeless control effort and energy to steer the robot. • Computation time to find the first solution. • Mean clearing distance ( δ dist ( γ ) ): with lower valuesindicating that the solutions are closer to the obstacles. • Number of cusps following [46]: maneuvering in diffi-cult environments may require the robots to stop andturn the wheels in the opposite direction, thus yieldinga cusp in the trajectory. Having more cusps correspondto less smooth and more difficult to drive paths.VI. B
ENCHMARK I MPLEMENTATION
We develop our benchmarking system in C++ and providea high-level front-end in Python . The experiments are imple-mented in Jupyter notebooks that leverage our Python front-end and enable the user to monitor through rich progressreports and plotting capabilities the status of the execution.We are collecting the experimental results and derived plotson our website at https://robot-motion.github.io/mpb/ where the complete data can be analyzed.We run the benchmark on a server featuring 256 GB RAM,two Intel Xeon Gold 6154 @ 3.00GHz CPUs offering 72threads in total, running on Ubuntu 18.04 (kernel version4.15.0). Each experiment is run using 20 parallel processesthat correspond to different environment seeds, in the caseof the procedurally generated environments. Each process Our code will be made open-source at https://github.com/robot-motion/mpb . runs a sequence of planners and post-smoothing methods onits predefined environment. We limit the parallelism to 20out of 72 available CPU cores due to the fact that plannerssuch as CForest spawn multiple threads on its own to find asolution. By further randomizing the order in which each ofour benchmark processes executes the planners, we can keepthe number of parallel threads in check (e.g. avoid running20 parallel CForest instances). Each process is automaticallycancelled if twice of its time limit has been exceeded (timeout), or if its memory consumption has exceeded 18 GB.VII. R ESULTS
This section summarizes the results obtained in our exper-iments, while focusing on the main findings. The completestatistical analysis will be published on our website.
A. Moving AI Scenarios
This section reports the results obtained of the gridsselected from the Moving AI benchmark, see Tables II,III and IV. The solution column contains two numbersseparated by a ‘/’: the second number indicates the numberof solutions found (highlighted by the orange bar in thebackground), the first number indices how many of thesesolutions are collision-free. Each planner is run on a total of51 scenarios. The following columns indicate the planningstatistics in the format mean ± standard deviation across themetrics (planning time, path length, maximum curvature andaverage curvature along the paths). The last column showsthe total number of cusps in all solutions combined. Wegroup these statistics by the steer functions, for which weselected different time limits, as shown in the tables next F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution
Fig. 6. Statistics for the parking1 scenarios. First row: Reeds Shepp steering, second row: CC Reeds Shepp steering, third row: POSQ steering, fourthrow: Dubins steering. to the group labels. These time limits have been determinedempirically to ensure that many solutions could be found.The SBPL planners are treated separately since they did notuse any of the provided steer functions but their particularunicycle motion primitive. a) Path Length and Smoothness:
Results are detailedin Table II. In terms of path length and smoothness, any-time path planners achieve better performance within thegiven maximum planning time for all the steer functions.Feasible planners generate often longer and less smoothpaths (higher curvature and number of cusps). Specificallyfor the scenario
Berlin 0 256 , BFMT achieves the shortestpath lengths within the fastest time with average curvature,except with POSQ steering where it has poor runtime andpath length. KPIECE throughout all experiments finishesamong the fastest but consistently has the longest pathsand among the worst maximum curvature. For the Dubinscurves, it was considerably more difficult for the planners to find feasible solutions – sampling-based planners, suchas EST, SST, PDST and KPIECE were the most successfulin finding exact, collision-free paths. In the
NewYork 1 512 scenarios, RRT ∗ , RRT , SORRT ∗ , Informed RRT ∗ and CFor-est achieve the shortest solutions with the lowest curvature.While CForest generally finds short solutions with lowcomputation times, few of them are collision-free. EST findsthe most and shortest solutions with POSQ, although witha significant number of cusps resulting in relatively highcurvature. Boston 1 1024 is the largest of the grid-basedenvironments and requires significantly longer computationtimes for the majority of planners to return an exact andcollision-free path. In most cases, only the feasible sampling-based motion planners, such as EST, SST and RRT manageto find valid solutions, whereas CForest, Informed RRT ∗ andSORRT ∗ do not find any collision-free paths with Reeds-Shepp steering within a time limit of 7 .
25 50 75 100 125 150 175 200 225 2500255075100125150175200225250 0 25 50 75 100 125 150 175 200 225 250StartGoal
Fig. 7. Many of the challenging Moving AI Cities scenarios define start andgoal locations that are too close to obstacles to be solvable by a polygon-based collision model of the robot. Shown here are two scenarios from the
Berlin 0 256 map with highlighted start and goal positions.
Reeds-Shepp with among the shortest path length and lowestcurvature (among planners which also find valid paths). WithPOSQ and Dubins as steer functions, however, it fails to findany (POSQ) or more than one (Dubins) solutions. b) Post-Smoothing Results:
In Figure 15 we sum-marize the post-smoothing results across all planners inthe
Berlin 0 256 scenarios, which is representative for theother Moving AI benchmark environments. GRIPS oftenoutperforms the other methods in maximum curvature whileachieving similar path length as SimplifyMax. In compu-tation times, B-Spline, Shortcut and SimplifyMax performsimilarly, except with POSQ steering where the latter issignificantly slower with a median computation time almosttwice as high as the other methods. SimplifyMax yieldssolutions which often have very small clearing distance.B-Spline solutions have considerably more cusps than theresults obtained with the other methods. c) Theta ∗ and SBPL Issues: On the larger-scale envi-ronments considered throughout this benchmark (particularlythe Moving AI scenarios), we noticed that our currentimplementation of Theta ∗ makes heavy use of the collisionchecker that significantly deteriorates its computation time.As can be seen in Table II, only in the case of a 6 min timelimit for a fast-to-evaluate steer function, such as Dubins,does this algorithm find a competitive number of collision-free, exact solutions. In other cases, our implementation doesnot yield a solution before the time limit is up. Similarly,the planners AD ∗ , ARA ∗ and MHA ∗ from SBPL were oftenunable to find feasible solutions within the time limit. On the Boston 1 1024 scenario, MHA ∗ found only a single solutionwithin 60 min, while non of the other SBPL planners returnedany feasible path. We therefore excluded these results fromTable IV. B. Polygon-based Environments
The following scenarios are particularly tailored towardautonomous driving. Instead of navigating grid world, theenvironments use arbitrary convex shapes to represent ob-stacles.
1) Parking scenarios: a) Path Length and Smoothness:
Similarly to the grid-based environments, in these scenarios, anytime plannersachieve better performance in terms of path length andsmoothness than feasible planners, although at the priceof being slower. In the scenarios for the first parking en-vironments, we notice that RRT, Informed RRT ∗ , RRT ∗ and SORRT ∗ always find solutions, across all tested steerfunctions, as shown in Figure 6. SST, Theta ∗ , SPARSand SPARS2, however, do often not find any solutions.Particularly SPARS2 is the only planner that cannot findany solutions for CC Reeds Shepp steering, SST is theonly algorithm that is unable to solve any scenarios withPOSQ steering. The Dubins steer function appears to beparticularly challenging, as SPARS, SPARS2 and Theta ∗ cannot find any paths, while various other planners, such asPRM, PRM ∗ , BFMT and BIT ∗ only solve a small fractionof the scenarios exactly. We observe similar behavior onthe second parking environment (Figure 16). The parallelparking environment ( parking3 ) proves more challenging(Figure 17) for most planners which leads to considerableless collision-free and exact solutions, particularly under thekinodynamic constraints of Dubins steering.
2) Warehouse scenarios:
We visualize example solutionsobtained from all planners on the fourth scenario fromthe warehouse environment with Reeds Shepp steering inFigure 8. BFMT, CForest, Informed RRT ∗ and SORRT ∗ findthe shortest solutions which all lie in the same homotopyclass.Compared to most parking scenarios, the warehouse en-vironment typically requires longer computation times forthe planners (especially anytime planners) to find solutions.It offers considerably more opportunity for the planners tofind solutions of varying homotopy classes (cf. Figure 8),resulting in a larger variance of path length. CForest, In-formed RRT ∗ , and SORRT ∗ consistently find among theshortest paths, although Informed RRT ∗ has among thelongest computation times (cf. Figure 9). C. Procedurally-generated grid environments
As described in subsection V-A, we procedurally generateenvironments to have full control over the shape of thefree space within the planners need to find solutions. Thisallows us to precisely analyze how varying features of theenvironments influence the planning results.
1) Varying corridor sizes:
As shown on the abscissa inFigure 10, the corridor sizes are expressed in the number ofgrid cells. We sample five 100 ×
100 grid environments foreach corridor radius (Figure 3 bottom row), sampled fromthe same starting seed over radii between three and eightgrid cells. As we increase the corridor size, the path lengthsof all planners decrease, as well as the number of cusps.The curvature metric remains mostly unaffected, except forPDST, PRM and SPARS2 where it considerable decreases.Theta ∗ , the SBPL planners, Informed RRT ∗ and RRT ig. 8. Example trajectories for the different planners in one of the five warehouse scenarios with Reeds-Shepp steering. B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution
Fig. 9. Statistics for the warehouse scenarios. First row: Reeds Shepp steering, second row: CC Reeds Shepp steering, third row: POSQ steering, fourthrow: Dubins steering.
PARS, SPARS2 and PRM have poor curvature, but PDSTimproves by a factor of two toward the maximum corridorsize. While PDST and RRT initially find four and nine outof ten possible solutions, only at a corridor radius of fourcells do all planners find exact solutions in every case.
2) Varying turning radii:
We vary the turning radius usedby the Reeds Shepp steer function and evaluate the plannerson a 100 ×
100 indoor-like grid environment with a corridorradius of five grid cells (cf. Figure 3 bottom row). The changein turning radius has a surprisingly little effect on the pathquality, see Figure 11. Slight developments can be observedwhere the path lengths tend to increase as the turningradius becomes larger. Especially PRM has a pronouncedinclination in the number of cusps. The curvature is generallynot tending in any direction significantly. The number ofexact solutions is at zero for Theta ∗ , PRM constantly findstwo out of ten solutions, while the other planners find all ofthe solutions exactly.
3) Varying obstacle densities:
As described in subsec-tion V-A, in this experiment, we randomly set cells of a100 ×
100 grid environment to be occupied until a selecteddensity, i.e. ratio between occupied and free cells, hasbeen achieved (see Figure 3 top row). Through variousexperiments, we determined the ranges between 1% and 3%to yield meaningful results. We successively increase theobstacle density in steps of 0 . ∗ have a relatively strong increase. Thepath lengths are not as much affected, although increasing inmany cases, such as EST, SPARS2, SPARS, KPIECE. Thecurvature is increasing for many planners, such as PDST,PRM, PRM ∗ . D. Planning and Post-Smoothing
Based on our experiments with varying time limits overa range of time limits between zero and 30 seconds, we areinvestigating how post-smoothing methods can benefit themotion planning pipeline. In combination with sampling-based planners, which quickly find feasible solutions, canthese improvement techniques yield results that are quali-tatively competitive with the solutions obtained by anytimeplanners within shorter computation times?To answer this question, we run a set of sampling-basedplanners (EST, RRT, SBL, STRIDE) with all post-smoothingmethods considered in this benchmark, and compare itagainst anytime planners run at time limits ranging betweenfive and 60 seconds.As shown in Figure 13, we observe that the algorithmsGRIPS and Simplify Max yield significant improvements inpath length and maximum curvature. They both reduce thepath length typically by a factor of two and similarly smooththe path in a way that the maximum curvature drops by close to a factor of two. In most cases, Simplify Max isconsiderably faster than GRIPS to obtain these results. TheB-spline algorithm does not always improve the path quality,which may be explained by the problem that B-splines do nottranslate well to curves that can be followed by Reeds Sheppsteering, leading to slight turns that increase the curvature.Overall, there exist several couplings between sampling-based planners and post-smoothers that outperform anytimeplanners in speed and solution quality. For example, withinthree seconds RRT combined with Simplify Max smoothingachieves a maximum curvature at the same level as ananytime planner such as Informed RRT ∗ after 60 seconds,while yielding a shorter path length (Figure 14).VIII. G ENERAL O BSERVATIONS
Based on the results detailed in section VII, in this sectionwe provide a general analysis across the experiments andgive specific recommendations.
A. Planning Time
Feasible planners are much faster and reliable in find-ing a single solution. RRT consistently ranked among thefastest of the planners we evaluated. While anytime, i.e.asymptotically optimal, planners require more time to findsolutions, these are of higher quality than the paths foundby feasible planners. The complexity of the steer functionalso severely impacts the performance of the planners. Du-bins curves, for example, are computationally challengingsystems for planning in very cluttered environments. Anadded burden on the runtime complexity stems from thepolygon-based collision model, that, in contrast to typicalpoint-based collision checkers, further penalizes algorithmsthat are not implemented in a way to make as few statevalidity checks as possible, such as our non-optimized Theta ∗ implementation. Collision checking often consumed most ofthe allotted planning time such that this planner, in manycases, did not find any solution. B. Quality of Anytime Solutions
Overall the results confirm what we know from the theory:on average, anytime planners obtain better solutions in termsof path length, number of cusps and maximum curvature. In-formed anytime approaches (e.g., Informed-RRT*, SORRT*,BIT*) can achieve sometimes shorter paths throughout alltested steer functions. However, this is not always the case.These approaches are still impacted by larger complexityin the environment, and do not perform faster in highlyconstrained environments.
C. Variability of the Results
The main concern regarding sampling-based planners (fea-sible and anytime ones) is the high variance of the obtainedresults, which may lead also occasionally to low perfor-mance. In particular, we believe that the stochasticity of thesampling phase is a major drawback that should be addressedfrom the community. Deterministic sampling [34], [35], [50]is an approach that mitigates this issue. State-lattice planners
Exact Solutions
Cusps
Path Length
Maximum Curvature
BFMTBIT*CForestESTInformed RRT*KPIECEPDSTPRMPRM*RRTRRT
Fig. 10. Various planning statistics for the Reeds Shepp steer function in the procedurally grid environments with varying corridor sizes.
Exact Solutions
Cusps
Path Length
Maximum Curvature
BFMTBIT*CForestESTFMTInformed RRT*KPIECEPDSTPRMPRM*RRTRRT
Fig. 11. Various planning statistics for different turning radii (in meters) of the Reeds Shepp steer function in the procedurally grid environments (size:100 × Exact Solutions
Cusps
Path Length
Maximum Curvature
BFMTBIT*CForestESTInformed RRT*KPIECEPDSTPRMPRM*RRTRRT
Fig. 12. Planning statistics of the Reeds Shepp steer function in the procedurally generated grid environments (size: 100 × Computation Time [s]
Path Length
Computation Time [s]
Maximum Curvature
RRT (GRIPS)RRT (B-Spline)RRT (Shortcut)RRT (SimplifyMax)EST (GRIPS)EST (B-Spline)EST (Shortcut)EST (SimplifyMax)SBL (GRIPS)SBL (B-Spline)SBL (Shortcut)SBL (SimplifyMax)STRIDE (GRIPS)STRIDE (B-Spline)STRIDE (Shortcut)STRIDE (SimplifyMax)RRT*Informed RRT*SORRT*PRM*CForestBIT*SPARSSPARS2
Fig. 13. Comparison of sampling-based planners in combination with post-smoothing methods and anytime planners evaluated over maximum time limits5, 10, 15, 30, 45, 60 seconds on a 150 ×
150 grid environment. The initial solution found by the sampling-based planners is indicated by a (cid:72) symbol,the post-smoothers GRIPS ( (cid:108) ), B-Spline ( (cid:54) ), Shortcut ( (cid:58) ) and SimplifyMax ( (cid:116) ) are marked according to the legend. The anytime planners are shown assolid lines with · markers. Left: path length of the respective solutions.
Right: maximum curvature.
25 50 75 100 125 1500255075100125150
RRT
Reeds-Shepp
Path Length: 256.817Computation Time: 3.232Maximum Curvature: 3.932
Informed RRT*
Reeds-Shepp
Path Length: 223.335Computation Time: 60.001Maximum Curvature: 0.250
Fig. 14. Trajectories resulting from the comparison of sampling-based planners in combination with post-smoothing methods against anytime planners. Thesolution on the left is obtained from the sampling-based planner RRT after 3 .
232 s. Using the SimplifyMax algorithm, this solution is smoothed (center),within a total time (including RRT planning) of 3 .
232 s. On the right, the solution from Informed RRT ∗ is shown, which is computed after 60 .
001 s. are an example of deterministic techniques, which, at theprice of the solution quality, offer deterministic performance.
D. Post-smoothing Synergies
Post-smoothing combined with feasible planners is a goodstrategy in terms of planning efficiency and final path quality(sub-optimal and may not completely fulfill kinodynamicrequirements). The results show that there exist severalcouplings of feasible sampling-based planners and post-smoothers that outperform anytime planners both in com-putation time and solution quality.
E. Environment Complexity
Our benchmarking confirms that the environments sig-nificantly influence the performance of the planners. Envi-ronments, such as the polygon-based warehouse scenarios,revealed vastly different solutions between the planners (seeFigure 8).As pointed out in Section VII-C, the planning performanceis further impacted by different environment characteristics,such as narrow corridors and spaces cluttered with smallobstacles. Plain state-of-the-art approaches that do not im-plement additional sampling heuristics, such as goal biasing,often fail to return solutions in very difficult environmentswhere the corridors are small or the obstacle density is high.
F. Influence of the Steer Function
Regarding the steer functions, we have observed two mainphenomena which confirm previous theoretical claims [27].Computationally complex steer functions, such as CC Reeds-Shepp, severly impact the planning efficiency of all the algo-rithms. Solving planning queries for systems with complexnonholonomic constraints in very cluttered environments alsorequires more planning time, i.e. particularly for systemswhich are not small-time locally controllable, such as Dubinscurves. On the larger-scale experiments (e.g. subsection VII-A) we observed a significant variance in the planningtime allotment necessary for the planners to find solutionswith different steer functions, ranging from 1 . ONCLUSION
Following the need for more reproducible evaluations ofcommonly used AI algorithms, and with the goal of compar-ing a large set of state-of-the-art motion planning techniques,the presented paper establishes a benchmark for motion plan-ners that focuses on problems with nonholonomic systems, inparticular wheeled mobile robots. From our experiments, wedraw guidelines and highlight use-cases that are close to real-world scenarios of autonomous navigation systems. We areplanning to open-source the data and implementation of thebenchmarking framework, including the tooling to reproduceall presented results.ACKNOWLEDGMENTSThe authors thank Ziang Liu for his contributions to thesoftware repository and testing of various algorithms.R
EFERENCES[1] B. Paden, M. ˇC´ap, S. Z. Yong, D. Yershov, and E. Frazzoli, “Asurvey of motion planning and control techniques for self-drivingurban vehicles,”
IEEE Transactions on intelligent vehicles , vol. 1,no. 1, pp. 33–55, 2016.[2] S. M. LaValle, “Rapidly-exploring random trees: A new tool for pathplanning,” Computer Science Department, Iowa State University, Tech.Rep., 1998.[3] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars, “Prob-abilistic roadmaps for path planning in high-dimensional configurationspaces,”
IEEE transactions on Robotics and Automation , vol. 12, no. 4,pp. 566–580, 1996.[4] S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimalmotion planning,”
The international journal of robotics research ,vol. 30, no. 7, pp. 846–894, 2011.[5] I. A. S¸ucan, M. Moll, and L. E. Kavraki, “The Open Motion PlanningLibrary,”
IEEE Robotics & Automation Magazine , vol. 19, no. 4, pp.72–82, December 2012, http://ompl.kavrakilab.org.[6] M. Althoff, M. Koschi, and S. Manzinger, “CommonRoad: Com-posable benchmarks for motion planning on roads,” in . IEEE, 2017, pp. 719–726.[7] M. Moll, I. A. Sucan, and L. E. Kavraki, “Benchmarking motionplanning algorithms: An extensible infrastructure for analysis andvisualization,”
IEEE Robotics & Automation Magazine , vol. 22, no. 3,pp. 96–102, 2015.[8] B. Cohen, I. A. S¸ucan, and S. Chitta, “A generic infrastructurefor benchmarking motion planners,” in . IEEE, 2012, pp. 589–595.[9] J. Luo and K. Hauser, “An empirical study of optimal motionplanning,” in . IEEE, 2014, pp. 1761–1768. lanner Solutions Time [s] Path Length Curvature Clearance CuspsScenario:
Berlin 0 256 (SBPL, 12 min time limit)SBPL AD ∗ . ± .
00 361 . ± .
23 1 . ± .
74 10 . ± .
88 34SBPL ARA ∗ . ± .
01 360 . ± .
94 0 . ± .
07 10 . ± .
35 11SBPL MHA ∗
10 / 10 4 . ± .
53 397 . ± .
58 2 . ± .
62 12 . ± .
90 37
Scenario:
Berlin 0 256 (Reeds-Shepp steering, 1 . . ± .
89 369 . ± .
72 1 . ± .
96 8 . ± .
87 204BIT ∗
13 / 50 90 . ± .
09 362 . ± .
34 1 . ± .
01 7 . ± .
11 159CForest 5 / 51 90 . ± .
07 347 . ± .
54 0 . ± .
73 7 . ± .
83 70EST 46 / 51 2 . ± .
47 566 . ± .
92 1 . ± .
43 11 . ± .
57 769Informed RRT ∗
13 / 51 90 . ± .
01 350 . ± .
89 0 . ± .
33 7 . ± .
02 67KPIECE 45 / 51 1 . ± .
95 1077 . ± .
55 1 . ± .
51 11 . ± .
38 1791PDST 43 / 51 3 . ± .
55 580 . ± .
46 1 . ± .
61 11 . ± .
74 555PRM 24 / 51 90 . ± .
07 364 . ± .
69 1 . ± .
12 8 . ± .
40 329PRM ∗
33 / 51 90 . ± .
06 357 . ± .
46 0 . ± .
83 8 . ± .
51 156RRT 48 / 51 4 . ± .
43 475 . ± .
71 1 . ± .
53 11 . ± .
44 636RRT . ± .
47 346 . ± .
78 0 . ± .
75 7 . ± .
97 90RRT ∗
18 / 51 90 . ± .
01 347 . ± .
50 0 . ± .
55 7 . ± .
94 80SORRT ∗
21 / 51 90 . ± .
01 350 . ± .
64 0 . ± .
59 7 . ± .
89 75SPARS 46 / 51 90 . ± .
43 471 . ± .
80 1 . ± .
71 11 . ± .
95 586SPARS2 44 / 50 90 . ± .
01 410 . ± .
50 2 . ± .
56 10 . ± .
46 490SST 48 / 51 90 . ± .
01 506 . ± .
41 2 . ± .
52 9 . ± .
61 1348Theta ∗ Scenario:
Berlin 0 256 (CC Reeds-Shepp steering, 18 min time limit)BFMT 49 / 50 35 . ± .
26 366 . ± .
15 0 . ± .
77 8 . ± .
14 126BIT ∗
26 / 28 1080 . ± .
85 372 . ± .
39 0 . ± .
70 7 . ± .
83 101CForest 0 / 51 5 . ± .
36 334 . ± .
35 0 . ± .
86 7 . ± .
60 122EST 49 / 51 48 . ± .
31 670 . ± .
72 1 . ± .
32 11 . ± .
89 1793Informed RRT ∗
39 / 51 1080 . ± .
18 353 . ± .
32 0 . ± .
56 7 . ± .
00 73KPIECE 26 / 51 77 . ± .
60 1022 . ± .
38 1 . ± .
35 11 . ± .
25 3221PDST 40 / 51 132 . ± .
65 523 . ± .
90 1 . ± .
61 11 . ± .
69 540PRM 38 / 51 1062 . ± .
51 389 . ± .
72 0 . ± .
80 9 . ± .
89 411PRM ∗
38 / 51 1080 . ± .
19 379 . ± .
99 0 . ± .
89 9 . ± .
94 260RRT 49 / 51 35 . ± .
67 500 . ± .
97 1 . ± .
66 11 . ± .
59 527RRT . ± .
62 351 . ± .
94 0 . ± .
39 7 . ± .
96 69RRT ∗
47 / 51 1080 . ± .
08 348 . ± .
16 0 . ± .
48 7 . ± .
95 55SORRT ∗ . ± .
28 352 . ± .
92 0 . ± .
00 6 . ± .
62 48SPARS 48 / 49 1083 . ± .
92 468 . ± .
90 1 . ± .
73 11 . ± .
30 429SPARS2 0 N/A N/A N/A N/A N/ASST 49 / 51 1080 . ± .
02 605 . ± .
93 1 . ± .
10 9 . ± .
57 6950Theta ∗ Scenario:
Berlin 0 256 (POSQ steering, 12 min time limit)BFMT 4 / 10 582 . ± .
16 1010 . ± .
89 0 . ± .
33 13 . ± .
39 137BIT ∗
35 / 41 141 . ± .
90 790 . ± .
17 0 . ± .
36 12 . ± .
23 396CForest 28 / 51 149 . ± .
51 341 . ± .
55 0 . ± .
41 13 . ± .
26 171EST 50 / 51 720 . ± .
03 138 . ± .
90 1 . ± .
37 15 . ± .
65 1563Informed RRT ∗
49 / 51 663 . ± .
44 177 . ± .
16 0 . ± .
46 14 . ± .
63 636KPIECE 21 / 51 24 . ± .
24 1236 . ± .
43 1 . ± .
32 10 . ± .
57 1662PDST 7 / 51 65 . ± .
95 579 . ± .
92 1 . ± .
48 10 . ± .
25 1487PRM 4 / 51 88 . ± .
91 643 . ± .
00 1 . ± .
67 7 . ± .
84 1150PRM ∗ . ± .
56 607 . ± .
27 1 . ± .
61 8 . ± .
87 817RRT 49 / 50 688 . ± .
08 496 . ± .
33 1 . ± .
76 13 . ± .
12 378RRT . ± .
87 145 . ± .
17 1 . ± .
47 16 . ± .
63 1087RRT ∗
46 / 51 692 . ± .
98 152 . ± .
35 1 . ± .
43 15 . ± .
15 1240SORRT ∗
45 / 50 669 . ± .
53 162 . ± .
61 1 . ± .
48 16 . ± .
13 563SPARS 0 / 47 165 . ± .
92 662 . ± .
32 0 . ± .
32 10 . ± .
44 317SPARS2 7 / 51 28 . ± .
18 549 . ± .
71 1 . ± .
43 10 . ± .
82 517SST 0 N/A N/A N/A N/A N/ATheta ∗ . ± .
32 364 . ± .
22 0 . ± .
19 3 . ± .
53 36
Scenario:
Berlin 0 256 (Dubins steering, 6 min time limit)BFMT 3 / 39 39 . ± .
14 517 . ± .
21 0 . ± .
00 9 . ± .
73 343BIT ∗
21 / 36 360 . ± .
08 363 . ± .
81 0 . ± .
00 7 . ± .
15 18CForest 6 / 51 360 . ± .
03 352 . ± .
64 0 . ± .
02 7 . ± .
18 43EST 39 / 51 59 . ± .
88 675 . ± .
61 0 . ± .
00 12 . ± .
54 372Informed RRT ∗
19 / 50 360 . ± .
03 346 . ± .
74 0 . ± .
01 7 . ± .
19 52KPIECE 37 / 51 44 . ± .
76 1210 . ± .
20 0 . ± .
02 12 . ± .
41 885PDST 31 / 49 48 . ± .
29 524 . ± .
19 0 . ± .
00 11 . ± .
12 124PRM 0 / 51 360 . ± .
03 580 . ± .
38 0 . ± .
00 8 . ± .
58 554PRM ∗ . ± .
05 545 . ± .
86 0 . ± .
00 7 . ± .
97 532RRT 37 / 51 59 . ± .
73 518 . ± .
43 0 . ± .
01 12 . ± .
31 126RRT . ± .
02 338 . ± .
86 0 . ± .
01 7 . ± .
09 50RRT ∗
23 / 51 360 . ± .
03 337 . ± .
68 0 . ± .
00 7 . ± .
15 43SORRT ∗
18 / 51 360 . ± .
04 347 . ± .
29 0 . ± .
01 7 . ± .
24 59SPARS 0 / 29 360 . ± .
77 569 . ± .
17 0 . ± .
00 9 . ± .
60 104SPARS2 0 / 30 360 . ± .
01 504 . ± .
48 0 . ± .
00 9 . ± .
51 155SST 39 / 49 360 . ± .
02 593 . ± .
50 0 . ± .
11 10 . ± .
38 1375Theta ∗
14 / 14 177 . ± .
84 454 . ± .
19 0 . ± .
00 7 . ± .
55 70
TABLE IIP
LANNING STATISTICS USING DIFFERENT STEER FUNCTIONS FROM THE B ERLIN
SCENARIO FROM THE M OVING AI BENCHMARK . R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Path Length G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Maximum Curvature G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Computation Time G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Mean Clearing Distance G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Cusps
MeanMedian G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Path Length G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Maximum Curvature G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Computation Time G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Mean Clearing Distance G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Cusps
MeanMedian G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Path Length G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Maximum Curvature G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Computation Time G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Mean Clearing Distance G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Cusps
MeanMedian G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Path Length G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Maximum Curvature G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Computation Time G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Mean Clearing Distance G R I P S B - Sp li n e S h o r t c u t S i m p li f y M a x Cusps
MeanMedian
Fig. 15. Planning statistics for the post-smoothing algorithms GRIPS, B-Spline, Shortcut, SimplifyMax (left to right per subplot) using different steerfunctions from the
Berlin 0 256 scenario from the Moving AI benchmark. These are the 50 most difficult start-goal configurations from the benchmark.First row: Reeds Shepp steering, second row: Dubins steering, third row: CC Reeds Shepp steering, fourth row: POSQ steering.[10] N. Sturtevant, “Benchmarks for grid-based pathfinding,”
Transactionson Computational Intelligence and AI in Games , vol. 4, no. 2, pp. 144– 148, 2012. [Online]. Available: http://web.cs.du.edu/ ∼ sturtevant/papers/benchmarks.pdf[11] “Algorithms and Applications Group motion planning benchmark,”hhttps://parasol.tamu.edu/groups/amatogroup/benchmarks/index.php.[12] “PathBench: a benchmarking platform for classic and learned pathplanning algorithms,” https://github.com/djl11/PathBench.[13] S. M. LaValle and J. J. Kuffner Jr, “Randomized kinodynamic plan-ning,” The international journal of robotics research , vol. 20, no. 5,pp. 378–400, 2001.[14] P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for theheuristic determination of minimum cost paths,”
IEEE transactions onSystems Science and Cybernetics , vol. 4, no. 2, pp. 100–107, 1968.[15] A. Nash, K. Daniel, S. Koenig, and A. Felner, “Thetaˆ*: Any-anglepath planning on grids,” in
AAAI , vol. 7, 2007, pp. 1177–1183.[16] M. Likhachev, G. J. Gordon, and S. Thrun, “ARA*: Anytime A* withprovable bounds on sub-optimality,” in
Advances in neural information processing systems , 2004, pp. 767–774.[17] J. Van Den Berg, R. Shah, A. Huang, and K. Goldberg, “Anytimenonparametric A*,” in
Twenty-Fifth AAAI Conference on ArtificialIntelligence , 2011.[18] L. Palmieri and K. O. Arras, “A novel RRT extend function forefficient and smooth mobile robot motion planning,” in . IEEE,2014, pp. 205–211.[19] T. Fraichard and A. Scheuer, “From Reeds and Shepp’s to continuous-curvature paths,”
IEEE Transactions on Robotics , vol. 20, no. 6, pp.1025–1035, 2004.[20] J. Reeds and L. Shepp, “Optimal paths for a car that goes bothforwards and backwards,”
Pacific journal of mathematics , vol. 145,no. 2, pp. 367–393, 1990.[21] L. E. Dubins, “On curves of minimal length with a constraint onaverage curvature, and with prescribed initial and terminal positionsand tangents,”
American Journal of mathematics , vol. 79, no. 3, pp.497–516, 1957.22] D. Calisi and D. Nardi, “Performance evaluation of pure-motiontasks for mobile robots with respect to world models,”
AutonomousRobots , vol. 27, no. 4, p. 465, Sep 2009. [Online]. Available:https://doi.org/10.1007/s10514-009-9150-y[23] J. Weisz, Y. Huang, F. Lier, S. Sethumadhavan, and P. Allen,“RoboBench: Towards sustainable robotics system benchmarking,” in . IEEE, 2016, pp. 3383–3389.[24] I. Ra˜n´o and J. Minguez, “Steps toward the automatic evaluation ofrobot obstacle avoidance algorithms,” in
In Workshop of Benchmarkingin Robotics, in the IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS . Citeseer, 2006.[25] C. Sprunk, J. R¨owek¨amper, G. Parent, L. Spinello, G. D. Tipaldi,W. Burgard, and M. Jalobeanu, “An experimental protocol forbenchmarking robotic indoor navigation,” in
Experimental Robotics .Springer, 2016, pp. 487–504.[26] M. Likhachev, D. I. Ferguson, G. J. Gordon, A. Stentz, and S. Thrun,“Anytime Dynamic A*: An anytime, replanning algorithm.” in
ICAPS ,vol. 5, 2005, pp. 262–271.[27] J.-P. Laumond, S. Sekhavat, and F. Lamiraux, “Guidelines in nonholo-nomic motion planning for mobile robots,” in
Robot motion planningand control . Springer, 1998, pp. 1–53.[28] Y. Li, Z. Littlefield, and K. E. Bekris, “Asymptotically optimalsampling-based kinodynamic planning,”
The International Journal ofRobotics Research , vol. 35, no. 5, pp. 528–564, 2016.[29] D. Hsu, J.-C. Latombe, and R. Motwani, “Path planning in expansiveconfiguration spaces,” in
Proceedings of International Conference onRobotics and Automation , vol. 3. IEEE, 1997, pp. 2719–2726.[30] S. Balakirsky and D. Dimitrov, “Single-query, bi-directional, lazyroadmap planner applied to car-like robots,” in . IEEE, 2010, pp.5015–5020.[31] A. M. Ladd and L. E. Kavraki, “Fast tree-based exploration ofstate space for robots with dynamics,” in
Algorithmic Foundationsof Robotics VI . Springer, 2004, pp. 297–312.[32] A. Dobson, A. Krontiris, and K. E. Bekris, “Sparse roadmap spanners,”in
Algorithmic Foundations of Robotics X . Springer, 2013, pp. 279–296.[33] A. Dobson and K. E. Bekris, “Improving sparse roadmap spanners,”in .IEEE, 2013, pp. 4106–4111.[34] L. Janson, B. Ichter, and M. Pavone, “Deterministic sampling-basedmotion planning: Optimality, complexity, and performance,”
The In-ternational Journal of Robotics Research , vol. 37, no. 1, pp. 46–61,2018.[35] A. Yershova and S. M. LaValle, “Deterministic sampling methods forspheres and SO(3),” in
IEEE International Conference on Roboticsand Automation, 2004. Proceedings. ICRA’04. 2004 , vol. 4. IEEE,2004, pp. 3974–3980.[36] L. Palmieri, L. Bruns, M. Meurer, and K. O. Arras, “Dispertio:Optimal sampling for safe deterministic motion planning,”
IEEERobotics and Automation Letters , pp. 1–1, 2019.[37] J. D. Gammell, S. S. Srinivasa, and T. D. Barfoot, “Informed RRT*:Optimal sampling-based path planning focused via direct sampling ofan admissible ellipsoidal heuristic,” in . IEEE, 2014, pp. 2997–3004.[38] J. D. Gammell, T. D. Barfoot, and S. S. Srinivasa, “Informed samplingfor asymptotically optimal path planning,”
IEEE Transactions onRobotics , vol. 34, no. 4, pp. 966–984, 2018.[39] J. D. Gammell, S. S. Srinivasa, and T. D. Barfoot, “Batch informedtrees (BIT*): Sampling-based optimal planning via the heuristicallyguided search of implicit random geometric graphs,” in . IEEE,2015, pp. 3067–3074.[40] O. Arslan and P. Tsiotras, “Use of relaxation methods in sampling-based algorithms for optimal motion planning,” in . IEEE, 2013, pp.2421–2428.[41] J. A. Starek, J. V. Gomez, E. Schmerling, L. Janson, L. Moreno,and M. Pavone, “An asymptotically-optimal sampling-based algorithmfor bi-directional motion planning,” in . IEEE, 2015,pp. 2072–2078. [42] M. Otte and N. Correll, “C-forest: Parallel shortest path planning withsuperlinear speedup,”
IEEE Transactions on Robotics , vol. 29, no. 3,pp. 798–806, 2013.[43] L. Palmieri, S. Koenig, and K. O. Arras, “RRT-based nonholonomicmotion planning using any-angle path biasing,” in . IEEE, 2016,pp. 2775–2781.[44] M. Pivtoraiko, R. A. Knepper, and A. Kelly, “Differentially constrainedmobile robot motion planning in state lattices,”
Journal of FieldRobotics , vol. 26, no. 3, pp. 308–333, 2009.[45] F. Islam, V. Narayanan, and M. Likhachev, “Dynamic multi-heuristicA*,” in . IEEE, 2015, pp. 2376–2382.[46] H. Banzhaf, L. Palmieri, D. Nienh¨user, T. Schamm, S. Knoop, andJ. M. Z¨ollner, “Hybrid curvature steer: A novel extend function forsampling-based nonholonomic motion planning in tight environments,”in . IEEE, 2017, pp. 1–8.[47] A. Astolfi, “Exponential stabilization of a wheeled mobile robot viadiscontinuous control,”
Journal of dynamic systems, measurement, andcontrol , vol. 121, no. 1, pp. 121–126, 1999.[48] E. Heiden, L. Palmieri, S. Koenig, K. O. Arras, and G. S. Sukhatme,“Gradient-informed path smoothing for wheeled mobile robots,” in . IEEE, 2018, pp. 1710–1717.[49] S. Gottschalk, “Separating axis theorem,” Department of ComputerScience, UNC Chapel Hill, Tech. Rep., 1996.[50] L. Palmieri, L. Bruns, M. Meurer, and K. O. Arras, “Dispertio:Optimal sampling for safe deterministic motion planning,”
IEEERobotics and Automation Letters , vol. 5, no. 2, pp. 362–368, April2020.
PPENDIX B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * −1.00−0.75−0.50−0.250.000.250.50 1e−11+2.5000000001e−1 Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution
Fig. 16. Statistics for the parking2 scenarios. First row: Reeds Shepp steering, second row: CC Reeds Shepp steering, third row: POSQ steering, fourthrow: Dubins steering. lanner Solutions Time [s] Path Length Curvature Clearance CuspsScenario:
NewYork 1 512 (SBPL, 20min time limit)SBPL AD ∗ ∗ ∗
12 / 12 18 . ± .
29 780 . ± .
66 0 . ± .
05 8 . ± .
74 59
Scenario:
NewYork 1 512 (Reeds-Shepp steering, 2 . . ± .
48 717 . ± .
39 0 . ± .
45 11 . ± .
22 275BIT ∗
11 / 51 148 . ± .
48 711 . ± .
96 0 . ± .
52 10 . ± .
72 100CForest 1 / 51 148 . ± .
48 685 . ± .
93 0 . ± .
26 9 . ± .
56 40EST 45 / 51 0 . ± .
15 1125 . ± .
17 0 . ± .
24 13 . ± .
10 839Informed RRT ∗ . ± .
48 691 . ± .
08 0 . ± .
44 9 . ± .
54 72KPIECE 43 / 51 0 . ± .
60 2215 . ± .
02 0 . ± .
21 13 . ± .
17 2016PDST 36 / 51 14 . ± .
00 982 . ± .
41 0 . ± .
35 12 . ± .
71 560PRM 33 / 51 148 . ± .
49 722 . ± .
27 0 . ± .
47 11 . ± .
73 393PRM ∗
35 / 51 148 . ± .
49 706 . ± .
38 0 . ± .
47 11 . ± .
65 154RRT 47 / 51 0 . ± .
20 890 . ± .
49 0 . ± .
34 12 . ± .
87 621RRT . ± .
49 690 . ± .
77 0 . ± .
39 10 . ± .
44 67RRT ∗
17 / 51 148 . ± .
48 689 . ± .
10 0 . ± .
41 10 . ± .
46 73SORRT ∗ . ± .
48 690 . ± .
34 0 . ± .
35 9 . ± .
76 46SPARS 47 / 51 148 . ± .
60 989 . ± .
95 0 . ± .
31 14 . ± .
49 573SPARS2 33 / 51 148 . ± .
48 756 . ± .
56 1 . ± .
48 11 . ± .
28 511SST 42 / 51 148 . ± .
47 966 . ± .
01 1 . ± .
19 9 . ± .
26 2424
Scenario:
NewYork 1 512 (CC Reeds-Shepp steering, 15min time limit)BFMT 50 / 51 32 . ± .
97 717 . ± .
37 0 . ± .
41 12 . ± .
82 135BIT ∗
10 / 26 817 . ± .
97 744 . ± .
77 0 . ± .
39 9 . ± .
66 98CForest 0 / 51 7 . ± .
65 710 . ± .
08 0 . ± .
46 9 . ± .
57 127EST 47 / 48 65 . ± .
91 1143 . ± .
58 0 . ± .
29 11 . ± .
08 1360Informed RRT ∗
19 / 46 818 . ± .
96 698 . ± .
82 0 . ± .
40 10 . ± .
66 93KPIECE 41 / 48 102 . ± .
41 2044 . ± .
96 0 . ± .
38 12 . ± .
70 3204PDST 37 / 48 213 . ± .
13 944 . ± .
86 0 . ± .
50 12 . ± .
57 578PRM 39 / 42 789 . ± .
76 761 . ± .
03 0 . ± .
51 11 . ± .
00 307PRM ∗
35 / 43 804 . ± .
57 742 . ± .
96 0 . ± .
51 11 . ± .
96 239RRT 41 / 48 19 . ± .
98 914 . ± .
77 0 . ± .
38 11 . ± .
50 491RRT . ± .
42 703 . ± .
21 0 . ± .
39 11 . ± .
41 56RRT ∗
43 / 44 814 . ± .
41 701 . ± .
19 0 . ± .
23 11 . ± .
31 50SORRT ∗ . ± .
54 653 . ± .
20 0 . ± .
34 10 . ± .
67 12SPARS 36 / 36 872 . ± .
32 972 . ± .
22 0 . ± .
35 13 . ± .
87 335SPARS2 0 N/A N/A N/A N/A N/ASST 39 / 40 805 . ± .
18 1046 . ± .
20 0 . ± .
52 9 . ± .
14 2323
Scenario:
NewYork 1 512 (POSQ steering, 20min time limit)BFMT 0 N/A N/A N/A N/A N/ABIT ∗ . ± .
70 823 . ± .
96 0 . ± .
15 16 . ± .
78 10CForest 29 / 43 272 . ± .
81 472 . ± .
60 0 . ± .
54 17 . ± .
51 143EST 47 / 47 1138 . ± .
91 204 . ± .
09 1 . ± .
48 17 . ± .
57 908Informed RRT ∗
44 / 46 1088 . ± .
43 243 . ± .
34 0 . ± .
52 15 . ± .
24 937KPIECE 9 / 51 29 . ± .
21 2282 . ± .
03 0 . ± .
30 12 . ± .
25 2033PDST 6 / 49 337 . ± .
74 1055 . ± .
35 1 . ± .
76 11 . ± .
66 1488PRM 10 / 38 911 . ± .
37 1137 . ± .
91 1 . ± .
68 8 . ± .
56 1454PRM ∗ . ± .
15 1012 . ± .
99 1 . ± .
55 8 . ± .
45 1259RRT 43 / 45 1032 . ± .
26 617 . ± .
35 0 . ± .
40 18 . ± .
17 392RRT . ± .
51 320 . ± .
92 0 . ± .
50 16 . ± .
05 831RRT ∗
42 / 45 1035 . ± .
34 317 . ± .
95 0 . ± .
50 16 . ± .
93 1046SORRT ∗
46 / 47 1090 . ± .
73 228 . ± .
97 0 . ± .
51 15 . ± .
61 1116SPARS 0 / 24 228 . ± .
73 924 . ± .
62 0 . ± .
18 12 . ± .
92 224SPARS2 3 / 32 286 . ± .
74 1177 . ± .
27 0 . ± .
64 11 . ± .
07 353SST 0 N/A N/A N/A N/A N/A
Scenario:
NewYork 1 512 (Dubins steering, 10min time limit)BFMT 2 / 45 38 . ± .
58 831 . ± .
59 0 . ± .
01 11 . ± .
74 343BIT ∗
11 / 31 600 . ± .
32 722 . ± .
88 0 . ± .
00 10 . ± .
92 39CForest 3 / 50 583 . ± .
31 696 . ± .
79 0 . ± .
01 9 . ± .
53 33EST 29 / 51 77 . ± .
12 1228 . ± .
31 0 . ± .
01 12 . ± .
20 383Informed RRT ∗
25 / 51 583 . ± .
52 703 . ± .
47 0 . ± .
03 11 . ± .
91 64KPIECE 28 / 51 39 . ± .
51 2114 . ± .
26 0 . ± .
01 15 . ± .
54 1837PDST 33 / 49 38 . ± .
62 901 . ± .
32 0 . ± .
02 13 . ± .
72 148PRM 0 / 51 583 . ± .
53 864 . ± .
27 0 . ± .
01 10 . ± .
76 408PRM ∗ . ± .
31 830 . ± .
61 0 . ± .
00 10 . ± .
47 342RRT 41 / 51 36 . ± .
97 968 . ± .
20 0 . ± .
01 13 . ± .
16 190RRT . ± .
53 694 . ± .
03 0 . ± .
03 10 . ± .
67 27RRT ∗
12 / 50 583 . ± .
30 697 . ± .
37 0 . ± .
02 10 . ± .
56 24SORRT ∗
19 / 51 583 . ± .
53 703 . ± .
36 0 . ± .
02 11 . ± .
90 44SPARS 0 / 39 591 . ± .
57 1115 . ± .
55 0 . ± .
00 11 . ± .
43 292SPARS2 0 / 35 588 . ± .
97 913 . ± .
60 0 . ± .
02 10 . ± .
70 247SST 41 / 50 583 . ± .
31 1028 . ± .
21 0 . ± .
00 11 . ± .
09 1438
TABLE IIIP
LANNING STATISTICS USING DIFFERENT STEER FUNCTIONS FROM THE N EW Y ORK
SCENARIO FROM THE M OVING AI BENCHMARK . F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Path Length B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Maximum Curvature B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Computation Time B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Cusps B F M T B I T * C F o r e s t E S T I n f o r m e d RR T * K P I E C E P D S T P R M P R M * RR T RR T RR T * S O RR T * S P A R S S P A R S SS T T h e t a * Aggregate
Total runsFound solutionsCollision-freeExact solution
Fig. 17. Statistics for the parking3 scenarios. First row: Reeds Shepp steering, second row: CC Reeds Shepp steering, third row: POSQ steering, fourthrow: Dubins steering. lanner Solutions Time [s] Path Length Curvature Clearance CuspsScenario:
Boston 1 1024 (Reeds-Shepp steering, 7 . . ± .
52 1521 . ± .
06 0 . ± .
16 24 . ± .
04 430BIT ∗
12 / 50 450 . ± .
17 1470 . ± .
15 0 . ± .
18 27 . ± .
71 102CForest 0 / 51 450 . ± .
02 1430 . ± .
05 0 . ± .
12 21 . ± .
74 63EST 46 / 51 12 . ± .
55 2361 . ± .
57 0 . ± .
11 32 . ± .
87 689Informed RRT ∗ . ± .
01 1436 . ± .
42 0 . ± .
17 21 . ± .
79 69KPIECE 44 / 51 0 . ± .
13 4054 . ± .
07 0 . ± .
10 32 . ± .
93 1432PDST 32 / 51 64 . ± .
23 2363 . ± .
57 0 . ± .
27 33 . ± .
32 626PRM 28 / 50 450 . ± .
19 1492 . ± .
06 0 . ± .
49 26 . ± .
13 541PRM ∗
30 / 50 450 . ± .
13 1460 . ± .
00 0 . ± .
20 22 . ± .
60 258RRT 46 / 51 0 . ± .
91 1859 . ± .
52 0 . ± .
24 30 . ± .
49 446RRT . ± .
03 1438 . ± .
92 0 . ± .
11 21 . ± .
77 58RRT ∗ . ± .
01 1436 . ± .
65 0 . ± .
15 21 . ± .
94 62SORRT ∗ . ± .
01 1435 . ± .
22 0 . ± .
15 21 . ± .
80 53SPARS 35 / 50 451 . ± .
06 1933 . ± .
78 0 . ± .
10 32 . ± .
13 460SPARS2 34 / 51 450 . ± .
01 1559 . ± .
65 0 . ± .
29 33 . ± .
50 525SST 46 / 51 450 . ± .
01 1855 . ± .
14 0 . ± .
19 25 . ± .
97 3031
Scenario:
Boston 1 1024 (CC Reeds Shepp steering, 45min time limit)BFMT 47 / 47 66 . ± .
26 1526 . ± .
58 0 . ± .
21 28 . ± .
28 264BIT ∗
14 / 26 2604 . ± .
81 1497 . ± .
97 0 . ± .
23 25 . ± .
81 110CForest 0 / 50 1 . ± .
73 1406 . ± .
73 0 . ± .
33 22 . ± .
50 86EST 46 / 50 367 . ± .
86 2344 . ± .
80 0 . ± .
10 32 . ± .
32 1454Informed RRT ∗
12 / 43 2583 . ± .
75 1385 . ± .
89 0 . ± .
14 23 . ± .
22 56KPIECE 43 / 48 252 . ± .
32 3533 . ± .
62 0 . ± .
17 30 . ± .
94 2476PDST 36 / 45 1034 . ± .
62 2326 . ± .
37 0 . ± .
24 33 . ± .
43 569PRM 30 / 39 2571 . ± .
87 1552 . ± .
06 0 . ± .
27 31 . ± .
68 478PRM ∗
30 / 40 2574 . ± .
23 1520 . ± .
76 0 . ± .
19 28 . ± .
37 378RRT 26 / 50 37 . ± .
79 1857 . ± .
10 0 . ± .
21 31 . ± .
42 508RRT . ± .
55 1460 . ± .
50 0 . ± .
15 23 . ± .
38 73RRT ∗
14 / 42 2580 . ± .
45 1451 . ± .
30 0 . ± .
15 22 . ± .
21 61SORRT ∗ . ± .
13 1162 . ± .
04 0 . ± .
05 26 . ± .
22 13SPARS 28 / 29 2702 . ± .
87 1798 . ± .
15 0 . ± .
11 34 . ± .
37 292SPARS2 0 N/A N/A N/A N/A N/ASST 36 / 36 2700 . ± .
02 1898 . ± .
62 0 . ± .
37 25 . ± .
79 1937
Scenario:
Boston 1 1024 (POSQ steering, 60min time limit)BFMT 0 N/A N/A N/A N/A N/ABIT ∗ . ± .
06 3177 . ± .
80 0 . ± .
21 33 . ± .
87 118CForest 29 / 43 2198 . ± .
31 1166 . ± .
03 0 . ± .
43 30 . ± .
67 231EST 41 / 42 3526 . ± .
09 593 . ± .
04 0 . ± .
37 25 . ± .
77 210Informed RRT ∗
41 / 45 3387 . ± .
45 901 . ± .
98 0 . ± .
35 30 . ± .
17 119KPIECE 11 / 51 133 . ± .
82 4480 . ± .
00 0 . ± .
26 27 . ± .
77 1698PDST 3 / 49 1284 . ± .
17 2201 . ± .
55 1 . ± .
50 26 . ± .
34 978PRM 17 / 40 2603 . ± .
00 1895 . ± .
32 0 . ± .
49 21 . ± .
13 1377PRM ∗ . ± .
08 2046 . ± .
60 0 . ± .
46 20 . ± .
51 1172RRT 42 / 46 3286 . ± .
39 2305 . ± .
04 0 . ± .
37 35 . ± .
92 429RRT . ± .
07 988 . ± .
13 0 . ± .
42 29 . ± .
86 107RRT ∗
43 / 48 3323 . ± .
09 967 . ± .
57 0 . ± .
34 31 . ± .
74 117SORRT ∗
36 / 40 3600 . ± .
69 925 . ± .
30 0 . ± .
41 32 . ± .
48 104SPARS 0 / 26 588 . ± .
84 2298 . ± .
28 0 . ± .
19 30 . ± .
87 210SPARS2 0 / 46 660 . ± .
18 2528 . ± .
22 0 . ± .
43 27 . ± .
33 527SST 0 N/A N/A N/A N/A N/A
Scenario:
Boston 1 1024 (Dubins steering, 30min time limit)BFMT 1 / 50 75 . ± .
19 1650 . ± .
43 0 . ± .
00 31 . ± .
01 303BIT ∗
12 / 40 1800 . ± .
11 1502 . ± .
51 0 . ± .
01 28 . ± .
06 90CForest 1 / 48 1800 . ± .
07 1446 . ± .
22 0 . ± .
05 21 . ± .
83 24EST 38 / 51 249 . ± .
65 2356 . ± .
25 0 . ± .
00 32 . ± .
88 1066Informed RRT ∗ . ± .
07 1453 . ± .
24 0 . ± .
03 24 . ± .
40 39KPIECE 37 / 51 18 . ± .
68 4117 . ± .
46 0 . ± .
01 35 . ± .
56 2052PDST 25 / 42 50 . ± .
76 2412 . ± .
49 0 . ± .
01 32 . ± .
03 327PRM 0 / 47 1800 . ± .
11 1620 . ± .
43 0 . ± .
00 31 . ± .
27 262PRM ∗ . ± .
27 1600 . ± .
10 0 . ± .
00 30 . ± .
99 216RRT 36 / 51 16 . ± .
95 2175 . ± .
94 0 . ± .
01 33 . ± .
17 511RRT . ± .
53 1433 . ± .
70 0 . ± .
02 21 . ± .
21 22RRT ∗ . ± .
05 1417 . ± .
88 0 . ± .
03 22 . ± .
71 31SORRT ∗
10 / 46 1800 . ± .
06 1452 . ± .
02 0 . ± .
02 24 . ± .
52 37SPARS 2 / 39 1805 . ± .
54 1842 . ± .
67 0 . ± .
00 30 . ± .
37 198SPARS2 2 / 42 1800 . ± .
01 1721 . ± .
07 0 . ± .
00 32 . ± .
90 371SST 42 / 45 1800 . ± .
04 1838 . ± .
55 0 . ± .
25 28 . ± .
24 1263
TABLE IVP
LANNING STATISTICS USING DIFFERENT STEER FUNCTIONS FROM THE B OSTON