SceML - A Graphical Modeling Framework for Scenario-based Testing of Autonomous Vehicles
SS CE ML - A G
RAPHICAL M ODELING F RAMEWORK FOR S CENARIO - BASED T ESTING OF A UTONOMOUS V EHICLES
A P
REPRINT
Barbara Schütt
FZI Research Center for Information Technology [email protected]
Thilo Braun
FZI Research Center for Information Technology [email protected]
Stefan Otten
FZI Research Center for Information Technology [email protected]
Eric Sax
FZI Research Center for Information Technology [email protected]
February 9, 2021 A BSTRACT
Ensuring the functional correctness and safety of autonomous vehicles is a major challenge for theautomotive industry. However, exhaustive physical test drives are not feasible, as billions of drivenkilometers would be required to obtain reliable results. Scenario-based testing is an approach totackle this problem and reduce necessary test drives by replacing driven kilometers with simulationsof relevant or interesting scenarios. These scenarios can be generated or extracted from recorded datawith machine learning algorithms or created by experts. In this paper, we propose a novel graphicalscenario modeling language. The graphical framework allows experts to create new scenarios orreview ones designed by other experts or generated by machine learning algorithms. The scenariodescription is modeled as a graph and based on behavior trees. It supports different abstraction levelsof scenario description during software and test development. Additionally, the graph-based structureprovides modularity and reusable sub-scenarios, an important use case in scenario modeling. Agraphical visualization of the scenario enhances comprehensibility for different users. The presentedapproach eases the scenario creation process and increases the usage of scenarios within developmentand testing processes.
Keywords
Scenario-based · Graphical Modeling Language · Automotive Software Engineering · Testing · Graph · Modularit
In software development, testing is an important step to realize develop systems. In particular, in the automotive industrysoftware validation and verification cannot be neglected and constitutes an integral part of the development process tomake sure requirements are met and the system fulfills the intended use cases [Wood et al., 2019, p. 76]. According toWachenfeld and Winner [2016], real world test drives, known as distance-based testing, are a valid testing method andgive the most reliable results. However, distance-based testing is not feasible: Almost 8 billion kilometers are necessaryto make a statistically significant comparison at a confidence of 95% to show the failure rate of an autonomous vehicleis lower than the human driver failure rate. This results in 225 years of driving for a fleet of 100 vehicles and if there arechanges or variations in the software, all the driving has to be repeated Kalra and Paddock [2016], Wachenfeld andWinner [2016].An alternative is offered by scenario-based testing. Compared to the random test cases emerging during distance-basedtesting, scenario-based testing is a systematic approach to find new test cases Pegasus Project [2019]. The goal is to setup a collection of validated and relevant scenarios, depending on the test subject. This approach can be used at differentstages during the development process Menzel et al. [2018]. Scenarios are developed at an abstract level during the a r X i v : . [ c s . S E ] F e b ceML Modeling Framework A P
REPRINT d) e) a) ScenarioPassInit <=d2<30 km/h<=d1 <12 km/h c) b) ScenarioCollisonInit <30 km/h<=d1 <12 km/h
Figure 1: Different presentations of the urban intersection scenario: a) top view of scenario start situation; b) TrafficSequence Charts (TSL) of the scenario resulting in a collision; c) TSL of the scenario without collision; d) SceMLrepresentation; e) simulation results in the CARLA simulator.concept phase, e.g. functional scenarios as shown in Fig. 2, and get more and more refined throughout the softwaredevelopment and testing process. Distance-based testing is a statistical verification approach, whereas scenario-basedtesting offers a more direct way to find and examine relevant or critical traffic situations. But unlike formal verificationit cannot claim completeness of the input space over all scenarios.Wiecher et al. [2019] proposed an example for the usage of scenarios within the development process, called Test-DrivenScenario Specification, where new requirements specification and analysis techniques are based on executable scenariosand automated testing. But in general, scenarios for testing have to be painstakingly created or collected. There arevarious methods for scenario generation or extraction, e.g. generating test scenarios with machine learning Abdessalemet al., combinatorial test generation with constraints Rocklage et al., extracting scenarios from test drives [Pfeffer et al.,2019, Hartjen et al., 2019] or expert-based scenario creation via one of several simulation tools that provide a scenarioeditor Automotive [2020], dSpace [2020], VTD [2020]. An appropriate scenario description and modeling language isneeded during the scenario acquisition and software development and test process, but widely supported standards arejust about to emerge.Here, we propose a scenario modeling language with the following features:• It has a graphical scenario modeling concept suitable for human experts.• A scenario is modeled with a graph structure, that can be easily converted in a machine readable data structure.• It allows different depth of information modeling to support different abstraction levels during the developmentprocess Menzel et al. [2018].• It supports modularity in form of sub-modules that can be reused in other scenarios or collected in moduledata bases or libraries.• The scenario outcome is not defined in the scenario description, since it might not be known before executionand can also be defined in pass and fail criteria or evaluation metrics of a test case where the scenario is used.Throughout this work, the following common urban scenario is used to explain state-of-the-art methods and our ownapproach: The ego vehicle, the autonomous vehicle to be tested, and a bike start at an intersection. The bike plans tocross the road, whereas the ego vehicle wants to take a right turn.In Section 2, a scenario terminology and design for scenario-based testing of automated and autonomous vehiclesis described. Section 3 introduces our graph-based modeling language notation, our metamodel and an exemplaryrealization of running our modeled example scenario before Section 4 gives a short conclusion.2ceML Modeling Framework
A P
REPRINT
Functional scenarios Logical scenarios Concrete scenariosLevel of abstraction Number of scenarios
Setting: intersectionEgo vehicle takes rightturn.Bike crosses street. Lane width: [2.5m-3.5m]Ego speed: [1.0m/s-2.0m/s]Bike speed [0.8m/s-1.2m/s] Lane width: 3.0mEgo speed: 1.3m/sBike speed 1.0m/s
Figure 2: All three abstraction levels as proposed by Menzel et al. [2018].
Menzel et al. [2018] suggest three abstraction levels for scenarios that can accompany the development process and beconverted into each other. Fig. 2 illustrates these three levels with the help of our intersection scenario:The most abstract level of scenario representation are called functional scenarios and described via linguistic scenarionotation using natural, non-structured language terminology. The main goal for this level is to create scenarios easilyunderstandable for experts. It describes the base road network, in our case the urban intersection map, and all actorswith their maneuvers, right turning ego vehicle and road crossing cyclist.The next abstraction level are logical scenarios and refine the representation of functional scenarios with the help ofparameters. These parameters can, for instance, be ranges for road width, vehicle positions and velocity or time andweather variables. The parameters can either be described as parameter ranges or with a probability distribution. Thelast level are concrete scenarios and depict operating scenarios with concrete values for each parameter in the parameterspace. This means, that one logical scenario can yield a large number of concrete scenarios, depending on the numberof variables, size of range and step size for these ranges.
Motivated from the many different use cases of scenarios, there exist several methods and languages to describescenarios. One important trade-off is between intuitive usage of a language that is human readable and a well-definedmachine-readable executable language that can be used for simulation tools. Graphical description and formalizedscenario description languages are the two main approaches in describing scenarios. A common approach to describe atraffic scenario between experts is a 2-dimensional view as shown in Fig. 1a) Rosenberger et al. [2019], Wachenfeld et al.[2016], Yasar et al. [2008]. This picture conveys an idea of what is supposed to happen, but lacks crucial information,e.g., which actor is going first, synchronization points or conditional behavior. Open Autonomous Safety furtherdeveloped this concept to a combination of this 2-dimensional view and a natural language description of the scenarioand its result. Damm et al. [2017] proposed Traffic Sequence Charts (TSL), which are depicted in Fig. 1b) and c). Thismethod solves the missing information problem and also defines a powerful graphical notation in describing a scenarioand an underlying first-order logic. A scenario consists of several snapshots with information about each importantevent ordered as a sequence. However, it defines the scenario outcome which might not be known before a simulationwas executed. Therefore, both outcomes have to be modeled as scenarios in TSL, shown in Fig. 1b) and c). Also, thenumber of possible outcomes grows in case a scenario contains more than ego vehicle and one actor, since the egovehicle has more collision opportunities during execution.The outcome is influenced by scenario variables, e.g. actors’ position, velocity and acceleration, but also by weatherand lighting conditions and used agent implementations or driving functions. Additionally, two different agent3ceML Modeling Framework
A P
REPRINT implementation can lead to different scenario outcomes even if the same variables are used. Finding the outcomeof a scenario can be viewed as a multidimensional optimization problem, since all this has to be considered beforethe scenario modelling process and can be used to create new concrete scenarios or to train driving functions viareinforcement learning Abeysirigoonawardena et al. [2019]. Two possible and most obvious outcomes of the urbanintersection scenario could be collision or no collision . However, the outcome is usually evaluated by pass and failcriteria that are define by the System-under-Test (SuT) and the reason a scenario is used, e.g. validation or verification,reinforcement learning, etc., and can be of a more complex nature than a collision between two actors, e.g. timedependent conflict measures as mentioned by Astarita and Giofré [2019]. Another approach to model scenarios wasproposed by Majzik et al. [2019], where a context model and a situation model define a scenario. The context modeldescribes the scene and the environment, and the situation model holds all object information. During the execution ofa scenario, the situation model is constantly updated with the help of the context model, but a clear sequence of what ishappening in a scenario is not supported.A tool-independent scenario description standard currently under development is ASAM OpenSCENARIO. In March2020 version 1.0 of this standard was released. It works together with ASAM OpenDRIVE and OpenCGR whichcomplement each other to describe all aspects of a traffic scenario: from the road topology to all participating vehiclesand their maneuvers ASAM OpenSCENARIO [2020]. OpenSCENARIO only supports concrete scenarios, but allowsparameterization in order to load different parameter sets to realize logical scenarios. Additionally, it supports a catalogconcept for maneuvers, actions, trajectories among others ASAM OpenSCENARIO [2020] for a better reusability.OpenSCENARIO uses a complex XML syntax, which makes it difficult for humans to understand or even design a newscenario by hand.The CARLA simulator, a simulator based on the Unreal gaming engine, and its scenario runner already support a textualbehavior tree structure based on a python behavior tree language Dosovitskiy et al. [2017], CARLA Team [2020].A scenario is modeled without position data or actor type. This information is given in an extra configuration file,which contains initial values for several different parameter sets. Since not all parameters can be varied in this file, e.g.vehicle velocities, CARLA offers concrete and partial logical scenarios. However, as of now CARLA does not supportmodularity or reusable code, since a new scenario has to be completely designed and written from scratch by an expert.All of the mentioned scenario description and modeling languages fulfill one ore more of our proposed features, butnone fulfills all of them. Figure 3: Second urban intersection scenario with ego vehicle, car and pedestrian.4ceML Modeling Framework
A P
REPRINT
Our scenario modeling language (SceML) is explained by using two examples. The first example scenario is the alreadyused urban intersection scenario (UIS1) depicted in Fig. 1d), which shows this scenario in our proposed graph notation.The start situation and the actors’ routes are depicted in Fig. 1a). The ego vehicle (blue car) wants to make a right turn.The bike, driving on a parallel lane to the ego vehicle, crosses the street right after the intersection. Additionally, thescenario contains three synchronization conditions to schedule the actors’ actions within an scenario execution:•
Sync 1:
Ego vehicle reaches destination around the corner, depicted as the top blue
InLocationRadius node inFig. 1d) .•
Sync 2:
Ego vehicle approaches the bike within a certain radius to signal the bike to start, depicted in the leftbottom blue
InVehicleRadius node.•
Sync 3:
Bike reaches destination across the road, depicted as the right blue
InLocationRadius node in Fig. 1d).Figure 4: Expanded CrossingManeuver which is shown collapsed in Fig. 3.The second example scenario takes place at the same urban intersection and is shown in SceML graph notation in Fig. 3(UIS2). The ego vehicle starts in the same position as in UIS1, however, instead of a right turn it wants to turn left. Asecond actor is an oncoming car, which simply wants to cross the intersection without turning left or right. The last actorin this scenario is a pedestrian crossing the intersection in the same manner as the bike from UIS1 with the differencethat it is moving on the left side walk instead of the right. UIS2 also has a number of synchronization conditions:•
Sync 1:
Ego vehicle reaches destination around the corner, depicted as the top blue
InLocationRadius node inFig. 3.•
Sync 2:
Ego vehicle approaches location within the intersection to signal the second car to start, depicted bythe blue
InLocationRadius node set before the DriveDistance nod.•
Sync 3:
Second car is close to pedestrian to signal the pedestrian to start crossing the street, depicted by first
InVehicleRadius node in CrossingManeuver shown in Fig. 4.•
Sync 4:
Pedestrian reaches destination across the road, depicted as the right blue
InLocationRadius node inFig. 4.
The notation of SceML is graph-based and has similarities to reversed behavior trees as they were proposed byColledanchise and Ögren [2016] and Marcotte and Hamilton [2017], an established way of expressing and structuringartificial intelligence in games. Usually, a behavior tree starts at exactly one root node and has several children, thatcan either be behavior nodes or sub-trees with different execution policies: sequential execution, displayed by a singlearrow node, parallel execution with demanded success on one child (question mark node) or parallel execution withdemanded success on all children (double arrow node). Fig. 5 shows UIS1 with the aforementioned synchronizationconditions as behavior tree.SceML utilizes the same three execution policies, however, due to human readability depicts them in a different way.Sequential execution is shown by a sequential chain of maneuvers and conditions and, thus, does not need an extra5ceML Modeling Framework
A P
REPRINT
Root
InVehicleLocation ! InLocationRadius InLocationRadius ! TurnRightEgoVehicleAccelerateBike KeepVelocityBike
Figure 5: Urban intersection scenario from Fig. 1d) represented in behavior tree notation as it was proposed byColledanchise and Ögren [2016]. Table 1: Node Notation LegendNode Description Node DescriptionLongitudinal maneuver Join: one finishedLateral maneuver Join: all finishedCondition End nodesymbol. Parallel executed maneuvers and conditions join in dedicated join nodes: to proceed further at least one pathhas to be finished or all paths have to be finished.
Each scenario has one root node, that holds important scenario information, e.g. name, actor list, etc. In addition, eachscenario has an end node (see Tab. 1), where all maneuver paths have to end.
The left hand side of Tab. 1 shows a list of possible maneuver nodes: longitudinal - movements, parallel to the actor’smovement direction (yellow), lateral - movements perpendicular to the actor’s movement direction (red). More complexmaneuvers, which consist of a combination of longitudinal and lateral movements are depicted in green. However, theyare not shown here, since these composite maneuvers are not used in UIS1 and UIS2.Condition nodes help to synchronize maneuvers or to make sure certain parts of a graph are only executed when a set ofrequirements is fulfilled and are colored blue.
Join nodes symbolize the parallel execution of nodes or sequences. A
Join-all-finished -Node denotes a successfulexecution of all incoming nodes or sequences is required. A
Join-one-finished -Node is viewed as successfully executedwhen one possible path to this node is executed. Both join node types are shown in Tab. 1.6ceML Modeling Framework
A P
REPRINT
Modules allow to summarize maneuvers and conditions for various reasons:1. A combination of basic maneuvers results in a more complex maneuver, e.g., lane change to the left, passvehicle, lane change to the right. This maneuver sequence can be put in a module frame, which is called
Overtaking Maneuver . Modules allow collapsed versions of a tree and make it easier to understand a scenario.2. Modules can be saved in a library and reused in further scenarios.Additionally, modules are allowed to contain sub-modules. Fig. 3 shows a scenario using the
CrossingManeuver module, which is the same sequence of actions and conditions the bike in UIS1 is executing. A module has in- andoutput ports in order to manage incoming and outgoing connections.
ModuleGraphElement ModuleElement ActionNode ConditionManeuverJoiningPolicyJoinAllFinished JoinOneFinished RootNode0..*0..1 Has0..* 0..1HasSubmodules HasElements GraphElement0..* 0..*ConnectedTo EndNode
Figure 6: Partial metamodel of SceML’s graph structure.Fig. 6 shows a partial metamodel of SceML’s graph structure. It mostly follows UML class diagram conventions andcontains all important classes. Classes colored in grey are abstract classes, whereas orange classes are not. Blue boxesrepresent relations between classes.
The whole scenario description is based on a hierarchical graph structure. Each graph node inherits from
GraphElement ,which can be further categorized into
Module , ModuleElement , RootNode and
EndNode . The graph’s root, called
RootNode , holds information about actors, their starting positions, environment, scenario name, map name etc. Allpossible paths have to end in one
EndNode . If no further information about joining is given, the end node works as ajoin-all-finished node. Every scenario can only contain one root and one end.A
Module is a combination of different
ModuleElement s. Only root and end nodes cannot be part of a module, since amodule is defined as a complex maneuver.
ModuleElement s are all nodes that can be part of a scenario graph and areeither
ActionNode s or
JoiningPolicy nodes. Action nodes again are divided into
Maneuver nodes, defining differentmaneuvers or
Condition nodes, which impose different runtime conditions on an executed scenario.
JoiningPolicy define the type of a parallel execution: success on all or on one. The
ModuleElement can be viewed as an artificial layerin order to make all tree node types available as parts of a module.7ceML Modeling Framework
A P
REPRINT
The main feature of SceML is a modular scenario concept and scenario description, i.e., to build and design scenariosand reusable sub-scenarios. A
Module can contain none to several sub-modules and an instance of a module can onlybe part of one parent module. However, different instances of a module can be parts of different parent modules.Additionally, a module can hold none to several module elements, but again the instance of a module element canonly be part of one module. For example, a module m can contain several instances of TurnRight , but each instance of
TurnRight can only be part of module m . The relation HasSubmodules in Fig. 6 describes the ability for a module tocontain other modules as sub-modules. The possible depth for sub-modules within sub-modules is in theory infinite andonly limited by computational resources and clarity of a scenario.
ActionNode ActorHas1..1 0..*Actor PedestrianTwoWheelerFourWheeler MotorbikeBikeEgoVehicle CarTruck
Figure 7: Metamodel describing actors and their relationship to action nodes.Actors are divided into three categories: pedestrians, two wheelers and four wheelers. The proposed class-baseddefinitions do not claim completeness, but show an intuitive way to structure participant types in an expandable way.Each maneuver or condition has a relation to exactly one actor, as shown in Fig. 7. This actor is called reference actorand describes the actor executing a maneuver or in need to fulfill a condition. Each actor can be reference actor of noneup to several action nodes. An action node can also have a relation to a second actor, called target actor. Examples forsuch maneuvers are
Follow or Approach . The abstraction level is set as a parameter in the root node and valid for all elements of a scenario. A concrete scenariois modeled by a graph with defined parameters for each variable and if no value is given uses default values definedby the language. The goal of a concrete scenario is to have an executable scenario and, therefore, is parsed intoOpenSCENARIO file format as described in Sec. 3.5. The modeling of a logical scenario is similar to the concretescenario, but instead of certain values a range can be given. Nonetheless, is also possible to use concrete values if novalue range is needed. Additionally, a functional scenario is the graph model of a scenario without the necessity ofgiven variable ranges or values, but both can be given if wanted. Functional and logical scenarios cannot be used togenerate proper OpenSCENARIO files, since the description language demands it to be a concrete scenario.
The graph structure and the scenario content need to be validated in order to get an executable scenario. Somemetamodel decisions are made due to model or graph validity. A scenario has to have a one root node and a root nodebelongs to one model. The same is true for end nodes. All maneuver and condition nodes have to be guaranteed to bepart of a path through the directed graph, that is connected to the end node. However, further requirements have to bemet to get a valid scenario:1. The root node cannot have an incoming connection and the end node cannot have an outgoing connection.8ceML Modeling Framework
A P
REPRINT
2. Parameters must be given and fit the scenario abstraction level.3. Join nodes need at least two incoming sequences.4. (optional) Certain maneuvers are not executed simultaneously, e.g. an actor cannot accelerate and decelerate atthe same time.5. (optional) Parameters fit to the actor and maneuver, e.g. a pedestrian cannot move with a velocity of 50 kmh .SceML has yet to be extended with custom validation rules to implement these requirements.
To develop an experimental scenario modeling framework, which allows to model a graphical graph-based scenariorepresentation, we used the Microsoft Visual Studio Modeling SDK Docs [2016]. Fig. 1 d) shows a scenario graphmodeled with our framework. It is possible to define further information for each node used in a scenario than what isFigure 8: Exemplary property definition for
Accelerate maneuver executed by the bike.shown in UIS1 and UIS2 graph notation of Fig. 1d) and Fig. 3. Fig. 8 presents a properties menu for the
Accelerate maneuver assigned to the bike or pedestrian in the CrossingManeuver. It is possible to customize all given parametersunder
Parameters , e.g. target velocity and throttle for acceleration.
Misc describes general maneuver settings, such asname or maneuver type, but also maneuver dependent information that cannot be changed, e.g. maneuver category andcomplexity level. In order to have a tool compliant scenario file, we build a custom OpenSCENARIO 1.0 parser toparse the graph structure into a valid OpenSCENARIO file.We used the CARLA simulator version 0.9.9 Dosovitskiy et al. [2017] and its scenario runner expansion CARLA Team[2020] to execute UIS1 and UIS2. CARLA’s scenario runner partially supports OpenSCENARIO 1.0 since version0.9.9 as possible scenario input. Fig. 1e) shows two screenshots from the scenario execution. The upper image wastaken directly after scenario initiation and the lower image during execution time, while both actors drive into theintersection but do not collide. For both scenarios the option exists to change variables in order to create collisionsor otherwise critical situations. UIS1 could end in a collision between ego vehicle and bike, whereas UIS2 can endwithout a collision, a collision between ego vehicle and car, or a collision between ego vehicle and pedestrian.In theory it is possible to support different data types, e.g. world or road coordinates, for actor position and enabledifferent presentation modi in maneuver and condition nodes. However, we did not implement it at this stage, sinceCARLA in contrast to OpenSCENARIO is not supporting it at this time.9ceML Modeling Framework
A P
REPRINT
This work extends previous scenario modeling approaches by combining a graphical approach with the possibility tomodel different abstraction levels. In detail, a concept for a graphical scenario modeling framework was presented. Ituses a graph structure to represent scenarios for the development and testing process of autonomous vehicles. In thepresent state, it allows to model scenarios in different abstraction levels. A functional scenario can be modeled by usingnode representations without further information about given maneuvers and variables. The properties of all nodes canbe filled with certain values, in case an executable concrete scenario is needed or with value ranges if a logical scenariois desired. Our approach supports modularity and reusable sub-scenarios, that can be stored in a library. A customOpenSCENARIO parser was used to store concrete scenarios and enable the CARLA simulator to load a scenario.Our modeling approach can be used to create concrete scenarios for test cases used in the validation process, but furtherholds the possibility to only create logical scenarios for a later exploration to find critical and interesting concretescenarios, since the outcome is not defined before execution. Hence, this approach can be used to find new scenarios tobuild and complete data bases in particular by providing the possibility to combine human and computational efforts.Finally, SceML could be used to inspect and modify automatically generated and extracted scenarios from recordeddata sets.As a next step we plan to implement missing validation requirements and to combine our modeling framework witha scenario evaluation framework, in order to only model a logical scenario which is then used to find all interestingconcrete scenarios, i.e. collisions, near collisions and other critical conflict situations.
References
Mathew Wood, David Wittmann, Siyuan Liu, Christian Knobel, Sandro Syguda, Thomas Wiltschko,Neil Garbacik, Michael O’Brien, Udo Dannebaum, Jack Weast, and Bernd Dorniede. Safetyfirst for automated driving, 2019. URL .Walther Wachenfeld and Hermann Winner. The release of autonomous vehicles. In
Autonomous driving , pages 425–449.Springer, 2016.Nidhi Kalra and Susan M Paddock. Driving to safety: How many miles of driving would it take to demonstrateautonomous vehicle reliability?
Transportation Research Part A: Policy and Practice , 94:182–193, 2016.Pegasus Project. Pegasus Method - An Overview. , 2019. [Online; accessed 16-April-2020].Till Menzel, Gerrit Bagschik, and Markus Maurer. Scenarios for development, test and validation of automated vehicles.In , pages 1821–1827. IEEE, 2018.Carsten Wiecher, Joel Greenyer, and Jan Korte. Test-driven scenario specification of automotive software components.In , pages 12–17. IEEE, 2019.Raja Ben Abdessalem, Shiva Nejati, Lionel C Briand, and Thomas Stifter. Testing vision-based control systems usinglearnable evolutionary algorithms. In , pages 1016–1026. IEEE.Elias Rocklage, Heiko Kraft, Abdullah Karatas, and Jörg Seewig. Automated scenario generation for regression testingof autonomous vehicles. In ,pages 476–483. IEEE.Raphael Pfeffer, Eric Sax, and Steffen Schmidt. Realdatenbasierte simulationsgestützte Absicherung hochautomatisierterFahrfunktionen.
ATZelektronik , 14(11):24–29, nov 2019. ISSN 1862-1791. doi:10.1007/s35658-019-0116-3.Lukas Hartjen, Robin Philipp, Fabian Schuldt, Bernhard Friedrich, and Falk Howar. Classification of driving ma-neuvers in urban traffic for parametrization of test scenarios. In
9. Tagung Automatisiertes Fahren . Lehrstuhl fürFahrzeugtechnik mit TÜV SÜD Akademie, 2019.IPG Automotive. CarMaker: Virtual testing of automobiles and light-duty vehicles. https://ipg-automotive.com/products-services/simulation-software/carmaker/ , 2020. [Online; accessed 16-April-2020].dSpace. dSpace - ASM Traffic. , 2020. [Online; accessed 16-April-2020].VTD. VTD - Virtual Test Drive. https://vires.com/vtd-vires-virtual-test-drive/ , 2020. [Online;accessed 16-April-2020]. 10ceML Modeling Framework
A P
REPRINT
Philipp Rosenberger, Jan Timo Wendler, Martin Friedrich Holder, Clemens Linnhoff, Moritz Berghöfer, HermannWinner, and Markus Maurer. Towards a generally accepted validation methodology for sensor models - challenges,metrics, and first results. In
Grazer Symposium Virtuelles Fahrzeug , May 2019. URL http://tuprints.ulb.tu-darmstadt.de/8653/ .Walther Wachenfeld, Philipp Junietz, Raphael Wenzel, and Hermann Winner. The worst-time-to-collision metric forsituation identification. In , pages 729–734. IEEE, 2016.Ansar-Ul-Haque Yasar, Yolande Berbers, and Davy Preuveneers. A computational analysis of driving variationson distributed multiuser driving simulators. In
IASTED International Conference on Modelling and Simulation ,number 19, pages 178–186. ACTA Press; Canada, 2008.Open Autonomous Safety. OAS Scenario Testing. https://oas.voyage.auto/scenarios . Accessed: 2010-09-30.W Damm, S Kemper, E Möhlmann, T Peikenkamp, and A Rakow. Traffic sequence charts-from visualization tosemantics. Technical report, AVACS Technical Report 117 (Oct 2017), 2017.Yasasa Abeysirigoonawardena, Florian Shkurti, and Gregory Dudek. Generating adversarial driving scenarios inhigh-fidelity simulators. In , pages 8271–8277.IEEE, 2019.Vittorio Astarita and Vincenzo Pasquale Giofré. From traffic conflict simulation to traffic crash simulation: Introducingtraffic safety indicators based on the explicit simulation of potential driver errors.
Simulation Modelling Practice andTheory , 94:215–236, 2019.István Majzik, Oszkár Semeráth, Csaba Hajdu, Kristóf Marussy, Zoltán Szatmári, Zoltán Micskei, András Vörös, Aren ABabikian, and Dániel Varró. Towards system-level testing with coverage guarantees for autonomous vehicles. In ,pages 89–94. IEEE, 2019.ASAM OpenSCENARIO. ASAM OpenSCENARIO. , 2020. [Online; accessed 17-April-2020].Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban drivingsimulator. arXiv preprint arXiv:1711.03938 , 2017.CARLA Team. ScenarioRunner for CARLA. https://github.com/carla-simulator/scenario_runner , 2020.[Online; accessed 19-April-2020].Michele Colledanchise and Petter Ögren. How behavior trees modularize hybrid control systems and generalizesequential behavior compositions, the subsumption architecture, and decision trees.
IEEE Transactions on robotics ,33(2):372–389, 2016.Ryan Marcotte and Howard J Hamilton. Behavior trees for modelling artificial intelligence in games: A tutorial.
TheComputer Games Journal , 6(3):171–184, 2017.Microsoft Docs. Microsoft Visual Studio Visualization and Modeling SDK. https://docs.microsoft.com/en-us/visualstudio/modeling/getting-started-with-domain-specific-languages?view=vs-2019https://docs.microsoft.com/en-us/visualstudio/modeling/getting-started-with-domain-specific-languages?view=vs-2019