Sebastian Siegl

Formal specification and systematic model-driven testing of embedded automotive systems

Increasingly intelligent energy-management and safety systems are developed to realize safe and economic automobiles. The realization of these systems is only possible with complex and distributed software. This development poses a challenge for verification and validation. Upcoming standards like ISO 26262 provide requirements for verification and validation during development phases. Advanced test methods are requested for safety critical functions. Formal specification of requirements and appropriate testing strategies in different stages of the development cycle are part of it. In this paper we present our approach to formalize the requirements specification by test models. These models serve as basis for the following testing activities, including the automated derivation of executable test cases from it. Test cases can be derived statistically, randomly on the basis of operational profiles, and deterministically in order to perform different testing strategies. We have applied our approach with a large German OEM in different development stages of active safety and energy management functionalities. The test cases were executed in model-in-the-loop and in hardware-in-the-loop simulation. Errors were identified with our approach both in the requirement specification and in the implementation that were not discovered before.

Model Based Requirements Analysis and Testing of Automotive Systems with Timed Usage Models

In the automotive industry requirements are often still composed of natural language text, spreadsheets, drawings, and formal models. Models are often used to describe partial aspects from the whole set of requirements. Hence, ???aws and vagueness in requirements are common and hard to discover. Upcoming standards like ISO 26262 request the automotive industry to be more strict and formal on the requirements. Formal notation and unambiguitiy is explicitely stated. In the field of system and acceptance testing requirements are the basis for all activities. Although, requirements are often not close to testing. To overcome this we introduced the Timed Usage Model (TUM) as a formal representation of requirements specification. During the creation of the model the requirements are analyzed and brought into an unambiguous and formal representation. Traceability is achieved, as each path in the model must be based upon a requirement. The formulation of the requirements in form of an unambiguous model clarifies the requirements and helps to detect design errors. During the creation of the model omissions and ???aws in the requirements are discovered. The model serves as a communication medium when functionality responsibles are involved to clarify these aspects. Timed Usage Models were created for power train functionality and the energy management. Moreover, the model as a formalized representation of the requirements served as the basis for the whole testing process, including test planning, test case generation, and test campaign analysis.

Introduction of time dependencies in usage model based testing of complex systems

Model-driven testing based on Markov chain usage models (MCUM) is an established method to address testing issues. It is not possible, however, to describe by means of MCUMs the timing of stimuli and time dependencies between inputs and outputs of a system. Additional concepts and information is needed. In this paper it is presented how Timed Usage Models (TUM) can be used to solve this issue. Concepts for stimuli and responses are introduced, that comprise that something must happen before a time interval, after a time interval or within a time interval. It is presented how this can be described by TUMs. TUMs are enhanced MCUMs that allow the usage of distributions of time. The computations for TUMs are based on semi-Markov processes and are therefore not restricted to discrete steps in time. Therefore, indicators and metrics for the test planning and management can be derived that take into account time. Test cases can be derived that reflect variability in inputs and, additionally, variability in timing of inputs. Complex real time systems require a test method that provides the possibility to handle the effect of timing and variability in timing of inputs to the system.

A novel industry grade dataset for fault prediction based on model-driven developed automotive embedded software

In this paper, we present a novel industry dataset on static software and change metrics for Matlab/Simulink models and their corresponding auto-generated C source code. The data set comprises data of three automotive projects developed and tested accordingly to industry standards and restrictive software development guidelines. We present some background information of the projects, the development process and the issue tracking as well as the creation steps of the dataset and the used tools during development. A specific highlight of the dataset is a low measurement error on change metrics because of the used issue tracking and commit policies.

Automated testing of embedded automotive systems from requirement specification models

Embedded software for modern automotive and avionic systems is increasingly complex. In early design phases, even when there is still uncertainty about the feasibility of the requirements, valuable information can be gained from models that describe the expected usage and the desired system reaction. The generation of test cases from these models indicates the feasibility of the intended solution and helps to identify scenarios for which the realization is hardly feasible or the intended system behavior is not properly defined. In this paper we present the formalization of requirements by models to simulate the expected field usage of a system. These so called usage models can be enriched by information about the desired system reaction. Thus, they are the basis for all subsequent testing activities: First, they can be used to verify the first implementation models and design decisions w.r.t. the fulfillment of requirements and second, test cases can be derived in a random or statistic manner. The generation can be controlled with operational profiles that describe different classes of field usage. We have applied our approach at a large German car manufacturer in the early development phase of active safety functionalities. Test cases were generated from the usage models to assess the implementation models in MATLAB/Simulink. The parametrization of the systems could be optimized and a faulty transition in the implementation models was revealed. These design and implementation faults had not been discovered with the established test method.

Improving model-based verification of embedded systems by analyzing component dependences

Embedded systems in automobiles become increasingly complex as they are intended to make vehicles even more safe, comfortable, and efficient. International norms like ISO 26262 and IEC 61165 postulate methods for the development and verification of safety critical systems. These standards should ensure that the dependability and quality of the embedded systems is maintained while their complexity and interdependence increases. Yet, the standards do not contain concrete methods or tools for their fulfillment. As concerns classic techniques for dependability analysis they either base on system analysis by means of Markov analysis or on reliability estimation from a usage perspective. Treating the system only from one perspective, however, is a drawback as the system analysis neglects functional or non-functional dependences of the system. These dependences can directly influence the reliability in the field usage. In this paper we present our approach to combine component dependency models with usage models to overcome these deficiencies. It is possible to identify usage scenarios which aim for critical dependences and to analyze the interaction of components inside the system. On the other hand usage scenarios can be assessed whether they meet the desired verification purpose. The component dependency models reveal dependences that were not identified before, because it allows the extraction of implications across functional and non functional dependences like memory, timing and processor utilization.

Mathematical Test Effort Estimation for Dependability Assessment of Sensor-Based Driver Assistance Systems

The development of modern driver assistance systems in the automotive domain requires extensive testing, for safety as well as for legal reasons. This is especially the case for sensor-based systems that provide support for autonomous driving: if they fail, an accident may occur with fatal consequences. In the majority of cases, the required test effort is roughly estimated by means of previous project data, expert knowledge or based on economical factors. As an alternative, a general mathematical approach is presented in this paper, which is focused on black box systems with sensor data fusion. It enables new projects related with sensor data fusion to estimate the required test effort for a given scenario.

Modeling and Statistical Testing of Real Time Embedded Automotive Systems by Combination of Test Models and Reference Models in MATLAB/Simulink

Embedded systems become increasingly complex and distributed. Although there is necessity for thourough testing, exhaustive validiation and verification is hardly possible in industry due to time and resource restrictions. In the past the reason for this has often been that it was to time-consuming to specify, to execute, and to evaluate test cases for the first design models and the integrated embedded system. In the meantime methods have become popular in industry that allow the automated generation, execution, and evaluation of test cases. In order to be able to automate these steps all necessary information must be integrated into the models that are the basis for the following steps. The complexity of the system, however, makes the evaluation and assessment of the behavior of the system even more complex. The growth of information needed for this comes along with it. The integration of this information into the model which is used for the generation of test cases is hardly feasible. In this paper we describe how this issue can be addressed by the combination of reference models in MATLAB/Simulink with test models. Time Usage Models (TUM) are employed as test models and provide the basis to generate all possible test scenarios. Model based statistical testing with consideration of time and durations is supported by TUMs. The reference models are used like an executable specification, providing information for the evaluation of the system to be tested. The test model can therefore be kept generic in order to be able to derive virtually any test case from the model, taking account of the potentially infinite sequence of inputs reactive systems might process. We applied the presented approach with a german automotive OEM for the validation and verification of the energy management system.

On Error-Class Distribution in Automotive Model-Based Software

Software fault prediction promises to be a powerful tool in supporting test engineers upon their decision where to define testing hotspots. However, there are limitations on a cross project prediction and a lack of reports upon application to industrial software, as well as the power of metrics to represent bugs. In this paper, we present a novel analysis based upon faults discovered in model-based automotive software projects and their relationship to metrics used to perform fault prediction. Using our previously released dataset on software metrics, we report bug classes discovered during heavy testing of those automotive software. As the software has been developed following strict coding and development guidelines, we present the results based on a comparison between the discovered error classes and those which might derive a reduced potential error set. Using the three projects from our dataset we determine if any of these bug classes are project specific.

Partitioning the requirements of embedded systems by input/output dependency analysis for compositional creation of parallel test models

In this paper we present a novel approach to reduce the effort for creating the model and facilitate its appliance in industry. We stick to the foundations of constructive enumeration to create a complete, traceably correct, and consistent model, but we do first decompose the task into manageable units by input/output dependency analysis. The expected behavior is formalized in temporal logic. The resulting model is a composition of all models, that run in parallel. As time is explicitly considered during the creation of the model, timing information is available for structured testing of non functional, e.g. real time requirements, as well as for the determination of measures and dependability estimators. By this approach, the subsequent activities for quality assurance, such as validation and verification, measurement of coverage criteria, and dependability estimators, e.g. of reliability, safety, and risk, profit from this approach, as they rely on a provably correct basis. We applied the method to an embedded system of a German automotive OEM, that was designed in Matlab Simulink and architectured with AUTOSAR 3.2 methodology. An existing test suite was at hand, that was created with the established method. This existing test suite served as benchmark to assess the quality of the new test suite, derived from the model. We compared the reachability of the test cases inside the implementation with code coverage measures and examined the variance of use imposed by the test suites. We present the promising results in this paper.