Mehrdad Sabetzadeh

A SysML-based approach to traceability management and design slicing in support of safety certification: Framework, tool support, and case studies

Context: Traceability is one of the basic tenets of all safety standards and a key prerequisite for software safety certification. In the current state of practice, there is often a significant traceability gap between safety requirements and software design. Poor traceability, in addition to being a non-compliance issue on its own, makes it difficult to determine whether the design fulfills the safety requirements, mainly because the design aspects related to safety cannot be clearly identified. Objective: The goal of this article is to develop a framework for specifying and automatically extracting design aspects relevant to safety requirements. This goal is realized through the combination of two components: (1) A methodology for establishing traceability between safety requirements and design, and (2) an algorithm that can extract for any given safety requirement a minimized fragment (slice) of the design that is sound, and yet easy to understand and inspect. Method: We ground our framework on System Modeling Language (SysML). The framework includes a traceability information model, a methodology to establish traceability, and mechanisms for model slicing based on the recorded traceability information. The framework is implemented in a tool, named SafeSlice. Results: We prove that our slicing algorithm is sound for temporal safety properties, and argue about the completeness of slices based on our practical experience. We report on the lessons learned from applying our approach to two case studies, one benchmark and one industrial case. Both studies indicate that our approach substantially reduces the amount of information that needs to be inspected for ensuring that a given (behavioral) safety requirement is met by the design.

An extended systematic literature review on provision of evidence for safety certification

Abstract Context Critical systems in domains such as aviation, railway, and automotive are often subject to a formal process of safety certification. The goal of this process is to ensure that these systems will operate safely without posing undue risks to the user, the public, or the environment. Safety is typically ensured via complying with safety standards. Demonstrating compliance to these standards involves providing evidence to show that the safety criteria of the standards are met. Objective In order to cope with the complexity of large critical systems and subsequently the plethora of evidence information required for achieving compliance, safety professionals need in-depth knowledge to assist them in classifying different types of evidence, and in structuring and assessing the evidence. This paper is a step towards developing such a body of knowledge that is derived from a large-scale empirically rigorous literature review. Method We use a Systematic Literature Review (SLR) as the basis for our work. The SLR builds on 218 peer-reviewed studies, selected through a multi-stage process, from 4963 studies published between 1990 and 2012. Results We develop a taxonomy that classifies the information and artefacts considered as evidence for safety. We review the existing techniques for safety evidence structuring and assessment, and further study the relevant challenges that have been the target of investigation in the academic literature. We analyse commonalities in the results among different application domains and discuss implications of the results for both research and practice. Conclusion The paper is, to our knowledge, the largest existing study on the topic of safety evidence. The results are particularly relevant to practitioners seeking a better grasp on evidence requirements as well as to researchers in the area of system safety. As a major finding of the review, the results strongly suggest the need for more practitioner-oriented and industry-driven empirical studies in the area of safety certification.

Characterizing the Chain of Evidence for Software Safety Cases: A Conceptual Model Based on the IEC 61508 Standard

Increasingly, licensing and safety regulatory bodies require the suppliers of software-intensive, safety-critical systems to provide an explicit software safety case – a structured set of arguments based on objective evidence to demonstrate that the software elements of a system are acceptably safe. Existing research on safety cases has mainly focused on how to build the arguments in a safety case based on available evidence; but little has been done to precisely characterize what this evidence should be. As a result, system suppliers are left with practically no guidance on what evidence to collect during software development. This has led to the suppliers having to recover the relevant evidence after the fact – an extremely costly and sometimes impractical task. Although standards such as the IEC 61508 – which is widely viewed as the best available generic standard for managing functional safety in software – provide some guidance for the collection of relevant safety and certification information, this guidance is mostly textual, not expressed in a precise and structured form, and is not easy to specialize to context-specific needs. To address these issues, we present a conceptual model to characterize the evidence for arguing about software safety. Our model captures both the information requirements for demonstrating compliance with IEC 61508 and the traceability links necessary to create a seamless chain of evidence. We further describe how our generic model can be specialized according to the needs of a particular context, and discuss some important ways in which our model can facilitate software certification.

Automated Checking of Conformance to Requirements Templates Using Natural Language Processing

Templates are effective tools for increasing the precision of natural language requirements and for avoiding ambiguities that may arise from the use of unrestricted natural language. When templates are applied, it is important to verify that the requirements are indeed written according to the templates. If done manually, checking conformance to templates is laborious, presenting a particular challenge when the task has to be repeated multiple times in response to changes in the requirements. In this article, using techniques from natural language processing (NLP), we develop an automated approach for checking conformance to templates. Specifically, we present a generalizable method for casting templates into NLP pattern matchers and reflect on our practical experience implementing automated checkers for two well-known templates in the requirements engineering community. We report on the application of our approach to four case studies. Our results indicate that: (1) our approach provides a robust and accurate basis for checking conformance to templates; and (2) the effectiveness of our approach is not compromised even when the requirements glossary terms are unknown. This makes our work particularly relevant to practice, as many industrial requirements documents have incomplete glossaries.

Supporting the verification of compliance to safety standards via model-driven engineering: Approach, tool-support and empirical validation

Context: Many safety-critical systems are subject to safety certification as a way to provide assurance that these systems cannot unduly harm people, property or the environment. Creating the requisite evidence for certification can be a challenging task due to the sheer size of the textual standards based on which certification is performed and the amenability of these standards to subjective interpretation. Objective: This paper proposes a novel approach to aid suppliers in creating the evidence necessary for certification according to standards. The approach is based on Model-Driven Engineering (MDE) and addresses the challenges of using certification standards while providing assistance with compliance. Method: Given a safety standard, a conceptual model is built that provides a succinct and explicit interpretation of the standard. This model is then used to create a UML profile that helps system suppliers in relating the concepts of the safety standard to those of the application domain, in turn enabling the suppliers to demonstrate how their system development artifacts comply with the standard. Results: We provide a generalizable and tool-supported solution to support the verification of compliance to safety standards. Empirical validation of the work is presented via an industrial case study that shows how the concepts of a sub-sea production control system can be aligned with the evidence requirements of the IEC61508 standard. A subsequent survey examines the perceptions of practitioners about the solution. Conclusion: The case study indicates that the supplier company where the study was performed found the approach useful in helping them prepare for certification of their software. The survey indicates that practitioners found our approach easy to understand and that they would be willing to adopt it in practice. Since the IEC61508 standard applies to multiple domains, these results suggest wider applicability and usefulness of our work.

Combining Goal Models, Expert Elicitation, and Probabilistic Simulation for Qualification of New Technology

New technologies typically involve innovative aspects that are not addressed by the existing normative standards and hence are not assessable through common certification procedures. To ensure that new technologies can be implemented in a safe and reliable manner, a specific kind of assessment is performed, which in many industries, e.g., the energy sector, is known as Technology Qualification (TQ). TQ aims at demonstrating with an acceptable level of confidence that a new technology will function within specified limits. Expert opinion plays an important role in TQ, both to identify the safety and reliability evidence that needs to be developed, and to interpret the evidence provided. Hence, it is crucial to apply a systematic process for eliciting expert opinions, and to use the opinions for measuring the satisfaction of a technologys safety and reliability objectives. In this paper, drawing on the concept of assurance cases, we propose a goal-based approach for TQ. The approach, which is supported by a software tool, enables analysts to quantitatively reason about the satisfaction of a technologys overall goals and further to identify the aspects that must be improved to increase goal satisfaction. The three main components enabling quantitative assessment are goal models, expert elicitation, and probabilistic simulation. We report on an industrial pilot study where we apply our approach for assessing a new offshore technology.

Evidence management for compliance of critical systems with safety standards: A survey on the state of practice

Abstract Context Demonstrating compliance of critical systems with safety standards involves providing convincing evidence that the requirements of a standard are adequately met. For large systems, practitioners need to be able to effectively collect, structure, and assess substantial quantities of evidence. Objective This paper aims to provide insights into how practitioners deal with safety evidence management for critical computer-based systems. The information currently available about how this activity is performed in the industry is very limited. Method We conducted a survey to determine practitioners’ perspectives and practices on safety evidence management. A total of 52 practitioners from 15 countries and 11 application domains responded to the survey. The respondents indicated the types of information used as safety evidence, how evidence is structured and assessed, how evidence evolution is addressed, and what challenges are faced in relation to provision of safety evidence. Results Our results indicate that (1) V&V artefacts, requirements specifications, and design specifications are the most frequently used safety evidence types, (2) evidence completeness checking and impact analysis are mostly performed manually at the moment, (3) text-based techniques are used more frequently than graphical notations for evidence structuring, (4) checklists and expert judgement are frequently used for evidence assessment, and (5) significant research effort has been spent on techniques that have seen little adoption in the industry. The main contributions of the survey are to provide an overall and up-to-date understanding of how the industry addresses safety evidence management, and to identify gaps in the state of the art. Conclusion We conclude that (1) V&V plays a major role in safety assurance, (2) the industry will clearly benefit from more tool support for collecting and manipulating safety evidence, and (3) future research on safety evidence management needs to place more emphasis on industrial applications.

A relationship-based approach to model integration

A key problem in model-based development is integrating a collection of models into a single, larger, specification as a way to construct a functional system, to develop a unified understanding, or to enable automated reasoning about properties of the resulting system. In this article, we suggest that the choice of a particular model integration operator depends on the inter-model relationships that hold between individual models. Based on this observation, we distinguish three key integration operators studied in the literature—merge, composition and weaving—and describe each operator along with the notion of relationship that underlies it. We then focus on the merge activity and provide a detailed look at the factors that one must consider in defining a merge operator, particularly the way in which the relationships should be captured during merge. We illustrate these factors using two merge operators that we have developed in our earlier work for combining models that originate from distributed teams.

Matching and Merging of Variant Feature Specifications

Model Management addresses the problem of managing an evolving collection of models by capturing the relationships between models and providing well-defined operators to manipulate them. In this paper, we describe two such operators for manipulating feature specifications described using hierarchical state machine models: Match, for finding correspondences between models, and Merge, for combining models with respect to known or hypothesized correspondences between them. Our Match operator is heuristic, making use of both static and behavioral properties of the models to improve the accuracy of matching. Our Merge operator preserves the hierarchical structure of the input models, and handles differences in behavior through parameterization. This enables us to automatically construct merges that preserve the semantics of hierarchical state machines. We report on tool support for our Match and Merge operators, and illustrate and evaluate our work by applying these operators to a set of telecommunication features built by AT&T.

Change impact analysis for Natural Language requirements: An NLP approach

Requirements are subject to frequent changes as a way to ensure that they reflect the current best understanding of a system, and to respond to factors such as new and evolving needs. Changing one requirement in a requirements specification may warrant further changes to the specification, so that the overall correctness and consistency of the specification can be maintained. A manual analysis of how a change to one requirement impacts other requirements is time-consuming and presents a challenge for large requirements specifications. We propose an approach based on Natural Language Processing (NLP) for analyzing the impact of change in Natural Language (NL) requirements. Our focus on NL requirements is motivated by the prevalent use of these requirements, particularly in industry. Our approach automatically detects and takes into account the phrasal structure of requirements statements. We argue about the importance of capturing the conditions under which change should propagate to enable more accurate change impact analysis. We propose a quantitative measure for calculating how likely a requirements statement is to be impacted by a change under given conditions. We conduct an evaluation of our approach by applying it to 14 change scenarios from two industrial case studies.