William G. Stillwell

A comparison of weight approximation techniques in multiattribute utility decision making

Abstract The notion of “dominance” in multiattribute utility decision contexts leads to a change in the considered alternative set. The implications of this set change are discussed in relation to the conditions of Wainers (1976) “equal-weights theorem,” and the possibility of sensitivity to weighting of importance dimensions is demonstrated. Data from three multiattribute decision-making studies are examined using four rank weighting techniques as well as equal weights in order to examine the practical significance of this sensitivity. Correlations between measures of overall utility produced by each weighting technique suggest that rank weighting of dimensions results in some improvement over equal weighting. This improvement was not always, however, reflected by significant changes in decisions.

Evaluating credit applications: A validation of multiattribute utility weight elicitation techniques

Abstract Multiattribute Utility Measurement (MAUM) provides a set of tools and procedures that are designed to aid the decision maker who is faced with decision problems of such complexity and ambiguity that unaided, intuitive judgment is likely to lead to the selection of suboptimal alternatives. Attempts to validate MAUM procedures have been primarily of three types: (1) behavioral tests of axiom systems derived from assumptions about what constitutes reasonable behavior; (2) convergent validation, in which the results of different procedures or even different subjects are compared; and (3) criterion validation, in which judgments and their resultant decisions are compared with some external criterion. From a behavioral point of view, the last of these, criterion validity, is by far the strongest. Past efforts at criterion validation of MAUM have suffered from three limitations: the subjects were not experts, alternative weight elicitation procedures were not compared, and the strength of the criterion used in each case is open to question. The purpose of this experiment was to provide an empirical comparison of a number of alternative MAUM weight elicitation procedures in a situation that offered a meaningful external crtierion along with subjects expert in its use. High quality decisions resulted from weight judgments provided in response to all weight elicitation procedures as long as single dimensions were first individually scaled and then weighted for aggregation. A procedure in which alternatives were rated holistically and weights and single dimension utility functions derived statistically showed poorer quality decisions. Thus, the “divide and conquer” theme of MAUM was upheld.

Psychological scaling of expert estimates of human error probabilities: application to nuclear power plant operation

The U.S. Nuclear Regulatory Commission and Sandia National Laboratories sponsored a project to evaluate psychological scaling techniques for use in qenerating estimates of human error probabilities. The project evaluated two techniques: direct numerical estimation and paired comparisons. Expert estimates were found to be consistent across and within judaes. Converaent validity was good, in comparison to estimates in a handbook of human reliability. Predictive validity could not be established because of the lack of actual relative frequencies of error (which will be a difficulty inherent in validation of any procedure used to estimate HEPs). Application of expert estimates in probabilistic risk assessment and in human factors is discussed.

Expert Judgment for Human Reliability in Risk Analysis

As more and more attention has been focused on quantitative risk analysis, the role of human reliability in the operation of complex technological systems has become increasingly apparent. The potential for human errors is now widely acknowledged as a significant, if not the major, contributor to risk. Human errors in the accident at Three Mile Island (Kemeny. 1979; Rogovin and Frampton. 1980) have led to considerable attention to human reliability in one of the most visible and controversial technological systems—nuclear power plants.