Robin L. Dillon

How Near-Misses Influence Decision Making Under Risk: A Missed Opportunity for Learning

Although organizations appear to learn from obvious failures, we argue that it is harder for them to learn from “near-misses”---events in which chance played a role in averting failure. In this paper, we formalize the concept of near-misses and hypothesize that organizations and managers fail to learn from near-misses because they evaluate such events as successes and thus feel safer about the situation. We distinguish perceived (“felt”) risk from calculated statistical risk and propose that lower levels of perceived risk encourage people with near-miss information to make riskier subsequent decisions compared to people without near-miss information. In our first study, we confirm the tendency to evaluate near-misses as successes by having participants rate a project manager whose decisions result in either (a) mission success, (b) near-miss, or (c) failure. Participants (both students and NASA employees and contractors) give similar ratings to managers whose decisions produced near-misses and to managers whose decisions resulted in successes, and both ratings are significantly different from ratings of managers who experienced failures. We suggest that the failure to hold managers accountable for near-misses is a foregone learning opportunity for both the manager and the organization. In our second set of studies, we confirm that near-miss information leads people to choose a riskier alternative because of a lower perceived risk following near-miss events. We explore several alternative explanations for these findings, including the role of Bayesian updating in processing near-miss data. Ultimately, the analysis suggests that managers and organizations are reducing their perception of the risk, although not necessarily updating (lowering) the statistical probability of the failure event. We speculate that this divergence arises because perceived risk is the product of associative processing, whereas statistical risk arises from rule-based processing.

Probabilistic risk analysis for the NASA space shuttle: a brief history and current work

Abstract While NASA managers have always relied on risk analysis tools for the development and maintenance of space projects, quantitative and especially probabilistic techniques have been gaining acceptance in recent years. In some cases, the studies have been required, for example, to launch the Galileo spacecraft with plutonium fuel, but these successful applications have helped to demonstrate the benefits of these tools. This paper reviews the history of probabilistic risk analysis (PRA) by NASA for the space shuttle program and discusses the status of the on-going development of the Quantitative Risk Assessment System (QRAS) software that performs PRA. The goal is to have within NASA a tool that can be used when needed to update previous risk estimates and to assess the benefits of possible upgrades to the system.

How Near-Miss Events Amplify or Attenuate Risky Decision Making

In the aftermath of many natural and man-made disasters, people often wonder why those affected were underprepared, especially when the disaster was the result of known or regularly occurring hazards (e.g., hurricanes). We study one contributing factor: prior near-miss experiences. Near misses are events that have some nontrivial expectation of ending in disaster but, by chance, do not. We demonstrate that when near misses are interpreted as disasters that did not occur, people illegitimately underestimate the danger of subsequent hazardous situations and make riskier decisions (e.g., choosing not to engage in mitigation activities for the potential hazard). On the other hand, if near misses can be recognized and interpreted as disasters that almost happened, this will counter the basic “near-miss” effect and encourage more mitigation. We illustrate the robustness of this pattern across populations with varying levels of real expertise with hazards and different hazard contexts (household evacuation for a hurricane, Caribbean cruises during hurricane season, and deep-water oil drilling). We conclude with ideas to help people manage and communicate about risk. This paper was accepted by Teck Ho, decision analysis.

Risk-based decision making for terrorism applications.

This article describes the anti-terrorism risk-based decision aid (ARDA), a risk-based decision-making approach for prioritizing anti-terrorism measures. The ARDA model was developed as part of a larger effort to assess investments for protecting U.S. Navy assets at risk and determine whether the most effective anti-terrorism alternatives are being used to reduce the risk to the facilities and war-fighting assets. With ARDA and some support from subject matter experts, we examine thousands of scenarios composed of 15 attack modes against 160 facility types on two installations and hundreds of portfolios of 22 mitigation alternatives. ARDA uses multiattribute utility theory to solve some of the commonly identified challenges in security risk analysis. This article describes the process and documents lessons learned from applying the ARDA model for this application.

The Respective Roles of Risk and Decision Analyses in Decision Support

Decision support models help structure and inform complex choices under uncertainty. Two classic models are risk analysis and decision analysis. Risk analysis is understood here as risk characterization, and in some cases, the identification and benefit assessment of some risk management options. It is based on systems analysis and probability, and it excludes the actual decision phase, which requires the preferences, e.g., the utility function, of the decision maker(s). Risk analysis and decision analysis have some similarities and are often complementary. To model uncertainties, both rely on probability, generally a subjective Bayesian degree of belief. A decision analysis can include a risk analysis component, and the design of a risk management plan may require decision analysis support. The challenge for risk analysts is to characterize potential failure problems before decision options have been identified, and when there is no single decision maker, or group of decision makers, who can provide preference functions and degrees of belief. Yet, a correct and complete model of uncertainties in the probabilistic risk analysis phase is important if the results are to be used later for decision support, especially when the number of systems involved and the duration of their operations is unknown. In this paper, we explore some of the challenges inherent to probabilistic risk analysis that should be of interest to the decision analysts who intend to use risk analysis results.

Optimal use of budget reserves to minimize technical and management failure risks during complex project development

Project managers are recognizing that adequate resource reserves are a critical success factor in a project development environment that is complex and uncertain. Yet, justifying the need for project reserves is still a challenge, as is the optimal allocation of any available resources to minimize development uncertainties. This paper presents a multiperiod decision model designed to support the management of reserves considering the risks of failures including technical, managerial, i.e., exceeding budget and schedule, or strategic, i.e., meeting budget, schedule, and technical specifications but not achieving the full strategic value of the project. In this paper, we examine the tradeoffs among these risks and their implications for resource allocation during a projects development phase. This decision support model is referred to as Dynamic Advanced Probabilistic Risk Analysis Model. It provides decision makers with a quantitative tool to allocate reserves (beyond the bare-bone minimum project costs) among project reserves, technical reinforcements of the engineered system, and product enhancements, with the advantage of flexibility over time. The model yields first, coarse estimates of the value of deferring some commitments about the products design until critical uncertainties are resolved and second, an estimate of the optimal amount to be invested in testing and reviews. We show that the greater the uncertainties at the onset of the development phase, the greater the value of this information.

Assessment of Cost Uncertainties for Large Technology Projects: A Methodology and an Application

Large projects, especially those planned and managed by government agencies, often incur substantial cost overruns. The tolerance, particularly on the part of members of Congress, for these cost overruns has decreased, thus increasing the need for accurate, defensible cost estimates. Important aspects of creating responsible cost estimates are accounting for the uncertainties in these estimates, expressing the estimates clearly, and communicating them to decision makers. Our method for estimating cost uncertainties can be used at all stages of a project. It combines the principles of probabilistic risk analysis with procedures for expert elicitation to incorporate uncertainties and extraordinary events in cost estimates. The Department of Energy implemented this process to select a new tritium supply source. During this implementation, we identified four key issues in modeling cost risks: how to consider correlations among cost components, how to aggregate assessments of multiple experts, how to manage communication and information sharing among experts, and what is an appropriate discount rate for cost estimates.

APRAM: an advanced programmatic risk analysis method

While engineering projects are expected to accomplish more with fewer resources, the dependencies among these projects have continually increased. At NASA, for example, faster-better-cheaper (FBC) projects are being grouped into programmes to enhance the benefits of synergies among smaller projects, and to attenuate, through diversification, the risks of single large missions. Under strict resource constraints, FBC project managers must balance several types of failure risks including the effect of their projects performance on future missions. Existing risk analysis models generally focus on the quantification of either technical or management risks. These tools, while beneficial, are generally used in isolation, and tightening the constraints generally encourages decisions that can increase the risks of technical failure. Since it is difficult to balance simultaneously cost, schedule and performance of a given project as well as their effects on other projects, managers who face these problems can benefit from an integrated programmatic risk analysis. In this paper, we present an advanced programmatic risk analysis model (APRAM) that is based on probability and systems analysis. It includes both managerial and technical risks, and can support decisions such as the choice of a design and of a budget reserve at the onset of a project. The model starts with the analysis of the trade-off between reserves and system development, then incorporates the effects of testing and reviews, partial mission failures, and project dependencies within programmes. The method is illustrated by the schematic example of two projects in an unmanned space programme.

The Impact of Technology on Relationships within Organizations

While much research has been undertaken to examine how technology impacts individuals and groups in organizations, as well as how it impacts organizations in their entirety, attention has not been directed at how technology impacts the relationships between these different levels in an organization—individual, group, and organization-wide. This paper seeks to address this void. Specifically, a grounded theory approach is used to begin development of a theory for how technology impacts these relationships. Based on a semi-structured survey, the resultant model is presented. Eight technology variables are identified in the model as responsible for impacting the relationships between the different levels of an organization. As a first foray into this area, the findings in this paper can serve as the foundation for further theory development and empirical testing.

Including technical and security risks in the management of information systems: A programmatic risk management model: A PROGRAMMATIC RISK MANAGEMENT MODEL

Developing and managing an information systems project has always been challenging, but with increased security concerns and tight budget resources, the risks are even greater. With more networks, mobility, and telecommuting, there is an increased need for an assessment of the technical and security risks. These risks if realized can have devastating impacts: interruptions of service, data theft or corruption, embezzlement and fraud, and compromised customer privacy. The software risk assessment literature (for example, Barki et al. 2001; Lyytinen et al. 1998; Schmidt et al. 2001) has focused primarily on managerial (i.e., development) risks, while the security risk models (for example, Cohen et al. 1998; Straub and Welke 1998) do not include the development risks and implementation costs. Theoretical risk models need to be developed that can provide a framework for assessing and managing the critical technical failure and security risk factors in conjunction with the managerial and development risks. This research seeks to model this problem by extending risk models originally developed for large-scale engineering systems.