Alberto Pasquini

A method for software quality planning, control, and evaluation

Squid is a method and a tool for quality assurance and a control that allows a software development organization to plan and control product quality during development. The Telescience software development project used it to build a remote monitoring and control system based in Antarctica.

Sensitivity of reliability-growth models to operational profile errors vs. testing accuracy [software testing]

This paper investigates: 1) the sensitivity of reliability-growth models to errors in the estimate of the operational profile (OP); and 2) the relation between this sensitivity and the testing accuracy for computer software. The investigation is based on the results of a case study in which several reliability-growth models are applied during the testing phase of a software system. The faults contained in the system are known in advance; this allows measurement of the software reliability-growth and comparison with the estimates provided by the models. Measurement and comparison are repeated for various OPs, thus giving information about the effect of a possible error in the estimate of the OP. The results show that: 1) the predictive accuracy of the models is not heavily affected by errors in the estimate of the OP; and 2) this relation depends on the accuracy with which the software system has been tested.

Test set size minimization and fault detection effectiveness: a case study in a space application

Abstract An important question in software testing is whether it is reasonable to apply coverage-based criteria as a filter to reduce the size of a test set. An empirical study was conducted using a test set minimization technique to explore the effect of reducing the size of a test set, while keeping block coverage constant, on the fault detection strength of the resulting minimized test set. Two types of test sets were examined. For those with respect to a fixed size, no test case screening was conducted during the generation, whereas for those with respect to a fixed coverage, each subsequent test case had to improve the overall coverage in order to be included. The study reveals that regardless of how a test set is generated, with or without any test case screening, block minimized test sets have a size/effectiveness advantage, in terms of a significant reduction in test set size and with almost the same fault detection effectiveness, over the original non-minimized test sets.

The SQUID approach to defining a quality model

This paper describes an attempt to use the approach developed by the SQUIDproject, which was part of the ESPRIT 3 programme, to define the software quality requirements of the Telescience project. The SQUID project developed its approach to quality modelling in parallel with ongoing feedback from testing that approach on the Telescience project, which was both large and software intensive. As part of this exercise we used the ISO software quality standard ISO 9126. It was an assessment of this and other existing quality models that caused us to re-assess what was meant by a quality model and led to a decomposition of existing ‘quality models’ into a composite model reflecting the different aspects of the model and its mapping onto a specific project or product. We break existing quality models into components which reflect the structure and content of the model. This composite model must then be customized for an individual product/project, we call this customized model a ‘Product Quality Model’. Application of this approach to the Telescience project identified a number of practical problems that the SQUID project needed to address. It also indicated a number of problems inherent in the current version of ISO 9126.

Sensitivity of reliability growth models to operational profile errors

The estimation of the operational profile is one of the key factors during the use of software reliability growth models. However, the operational profile can be very difficult to estimate in particular applications such as software used for process control. In other cases, a single operational profile may not be sufficient to describe the use of the product by a number of different customers. An operational profile may also change during the development of software or during its operational usage. All these cases may lead to errors in the estimation of the operational profile. The paper describes an empirical evaluation of the sensitivity of reliability growth models to errors in the estimation of the operational profiles. Some reliability growth models are applied during the testing phase of a software system. The particular characteristics of the case study allow the measurement of the actual reliability growth of the software and its comparison with the estimations provided by the models. Measurement and comparison are repeated for different operational profiles giving information about the effect of a possible error in the estimation of the operational profile. Results show that errors in the operational profile estimation do not heavily affect reliability estimates and that their influence is strongly dependent on the accuracy with which the software system has been tested.

Evaluation of air traffic management procedures: safety assessment in an experimental environment

This paper presents and discusses the application of safety assessment methodologies to a pre-operational project in the Air Traffic Control field. In the case analysed in the present paper a peculiar aspect was the necessity to effectively assess new operational procedures and tools. In particular we exploited an integrated methodology to evaluate computer-based applications and their interactions with the operational environment. Current ATC safety practices, methodologies, guidelines and standards were critically revised, in order to identify how they could be applied to the project under consideration. Thus specific problematic areas for the safety assessment in a pre-operational experimental project are highlighted and, on the basis of theoretical principles, some possible solutions taken into consideration. The latter are described highlighting the rationale of most relevant decisions, in order to provide guidance for generalisation or re-use.

Interface Mutation Test Adequacy Criterion: An Empirical Evaluation

An experiment was conducted to evaluate an inter-procedural test adequacy criterion named Interface Mutation. Program SPACE, developed for the European Space Agency (ESA), was used in this experiment. The development record available for this program was used to find the faults uncovered during its development. Using this information the test process was reproduced starting with a version of SPACE containing several faults and then applying Interface Mutation. Thus we could evaluate the fault revealing effectiveness of Interface Mutation. Results from the experiment suggest that (a) the application of Interface Mutation favors the selection of fault revealing test cases when they exist and (b) Interface Mutation tends to select fault revealing test cases more efficiently than in the case where random selection is used.

Experimental evaluation of a fuzzy-set based measure of software correctness using program mutation

Experimental evaluation of software reliability models that depend on the source code of the target program is expensive due to the need for a large sample of programs. The authors have used program mutation to generate many versions of one of the more complex components comprising a hypothetical but realistic nuclear reactor safety control program. Trivial mutants were filtered by using branch and path testing. These programs were used to assess a fuzzy set based measure of program correctness. The results confirmed that the model is conservative. In addition, the experiments provided new insights into the model, including reassessment of its assumptions and directions for refining it.<<ETX>>

A critical view of severity classification in risk assessment methods

Abstract The knowledge of operational experts plays a fundamental role in performing safety assessments in safety critical organizations. The complexity and socio-technical nature of such systems produce hazardous situations which require a thorough understanding of concrete operational scenarios and cannot be anticipated by simply analysing single failures of specific functions. This paper addresses some limitations regarding state-of-the-art safety assessment techniques, with special reference to the adoption of “chain of event” models in accident causation (widely criticised by many authors), to the use of severity classes and to the adoption of the worst credible effect criterion. Such methods tend to assume a linear link between single hazards considered in isolation and corresponding consequences for safety, thus neglecting the intrinsic complexity of the systems under analysis and reducing the opportunities for an effective involvement of operational experts. An alternative approach is proposed to overcome these limitations, by distinguishing different typologies of hazards and integrating the analysis of single functions with the study of concrete operational scenarios.

SHELFS: Managing critical issues through experience feedback

Knowledge is considered the most relevant asset of modern organizations. Most of this knowledge is embodied in the human practices and interactions among people and artifacts, and can become organizational knowledge only if properly captured, managed, and reused. Modern organizations strive to capture this knowledge because they consider it an important factor for improving the safety of their processes. Yet many organizations privilege a reactive approach to learn from experience; the one based on the analysis of reports from accidents, incidents, and near misses. This approach is considered to be too limited, too late, and too slow for supporting an efficient experience feedback. We present a proactive method tailored for introducing human factors in a safety critical company, which is based on a distributed knowledge view of the working processes. This method stresses the positive face of safety: It should allow a positive return of experience from the human practices.