Leonard Adelman

Experiments, quasi-experiments, and case studies: A review of empirical methods for evaluating decision support systems

Developers of decision support systems (DSS) often fail to present empirical data supporting the claimed merits of their systems. Discussions with developers indicate that they often do not consider or know how to perform the required empirical evaluations. That problem is addressed by reviewing the issues inherent in using experiments, quasi-experiments, and case studies to evaluate DSSs. The discussion revolves around the issues of reliability and four types of validity: internal, construct, statistical conclusion, and external. The discussion is focused upon but not restricted to expert systems. >

Research with Patriot Air Defense Officers: Examining Information Order Effects:

In an experiment using the Patriot air defense training simulator, we found that the order in which the same information was presented to experienced Patriot officers significantly affected their probability estimates, identification judgments, and engagement decisions. In all cases the results represented a primacy effect, contradicting the recency effect predictions of the Hogarth and Einhorn (1992) model. In the present study we found that early in an aircrafts track history, the order effect is caused by the overweighting of prior information; later in the history, the late order effect is caused by participants reinterpreting the meaning of information based on what preceded it. These findings help to explain the differential effectiveness of a proposed display and the importance of naturalistic research for testing the generality of models of decision making.

How Web site decision technology affects consumers

Using decision technology on a Web site to support alternative comparison increases satisfaction and the use of compensatory strategies among users but, surprisingly, does not affect their decision confidence or time-performing tasks.

Measurement issues in knowledge engineering

There are five sources (or determinants) of knowledge base quality: domain experts, knowledge engineers, knowledge representation schemes, knowledge elicitation methods, and problem domains. The knowledge base for many expert systems is developed for a problem domain using one domain expert, one knowledge engineer, one knowledge representation scheme, and one elicitation method. Since there is minimal research demonstrating that the possible variation in each of these sources does not significantly affect the quality of the knowledge base, the generalizability (or validity) of such systems in real-world settings is questionable. Consequently, research is needed to assess the extent to which system validity is affected by these sources of variability. Toward this end, the results of reanalyzing the data from an experiment varying domain experts, knowledge engineers, and elicitation methods when developing a multiattributed representation scheme for combat readiness are presented. No significant effects were obtained for elicitation method or knowledge engineer. >

Confirmation Bias in Complex Analyses

Most research works investigating the confirmation bias has used abstract experimental tasks where participants drew inferences from just a few items of evidence. The experiment reported in this paper investigated the confirmation bias in a complex analysis task that is more characteristic of law enforcement investigations, financial analysis, and intelligence analysis. Participants were professionals, half of whom had intelligence analysis experience. The effectiveness of a procedure designed to mitigate the confirmation bias, called analysis of competing hypotheses (ACH), was tested. Results showed a confirmation bias for both experience groups, but ACH significantly reduced bias only for participants without intelligence analysis experience. Confirmation bias manifested as a weighting bias, not as an interpretation bias. Participants tended to agree on the interpretation of evidence (i.e., whose hypothesis was supported by the evidence) but tended to disagree on the importance of the evidence-giving more weight to the evidence that supported their preferred hypothesis and less weight to evidence that disconfirmed it.

Distributed Tactical Decisionmaking: Conceptual Framework and Empirical Results

A psychologically oriented conceptual framework of distributed tactical decisionmaking (DTDM) that is applicable to decision environments such as a future Naval Battle Group /Force operating with a flexible and reconfigurable command and control system in a multithreat environment is proposed. The conceptual framework distinguishes between individual, group, process-control, and task-level input factors; characteristics of the groups interaction process; and various outcome variables. The critical assumption underlying the framework is that the interaction process mediates the influence of input factors on performance outcomes. Four intervening variables are proposed as important mediators of group process and performance in distributed environments: cognitive model similarity and consistency, incentives for cooperation, control over division of labor and resources, and task complexity. The adequacy of the first three intervening variables and the larger conceptual framework were tested in an initial experiment. The experiment provided support for the underlying assumption in general, and the incentives for cooperation mediating variable in particular. Moreover, the experiment demonstrated the importance of a multitrait-multimethod measurement approach, for a focus on information flow efficiency alone would not have uncovered the importance of voluntary cooperation, a behavioral process variable, to group performance. Implications for future research are discussed.

Demonstrating the effect of context on order effects for an army air defense task using the patriot simulator

The results reported herein support the hypotheses that (1) situation-specific, contextual features of a task can cause people to use explanation-based reasoning (Pennington and Hastie, 1993); (2) such reasoning can cause experienced personnel, both individually and in two-person teams, to reinterpret the meaning of the same information when it is presented in two different ordered sequences; and (3) the result will be primacy or recency (or no) effects depending on whether the most recent conflicting information can be explained away or not, respectively. These results extend the belief-adjustment model proposed by Hogarth and Einhorn (1992), which does not address information reinterpretations, and always predicts recency effects for an evaluation task with a short series of conflicting information. More generally, the results demonstrate the importance of situation-specific, contextual features in understanding judgment processes.

Using Brunswikian theory and a longitudinal design to study how hierarchical teams adapt to increasing levels of time pressure

Brunswikian theory and a longitudinal design were used to study how three-person, hierarchical teams adapted to increasing levels of time pressure and, thereby, try to understand why previous team research has not necessarily found a direct relationship between team processes and performance with increasing time pressure. We obtained four principal findings. First, team members initially adapted to increasing time pressure without showing any performance decrements by accelerating their cognitive processing, increasing the amount of their implicit coordination by sending more information without being asked and, to a lesser extent, filtering (omitting) certain activities. Second, teams began and continued to perform the task differently with increasing time pressure, yet often achieved comparable levels of performance. Third, time pressure did affect performance because there was a level of time pressure beyond which performance could not be maintained, although that level differed for different teams. And, fourth, some adaptation strategies were more effective than others at the highest time pressure level. Taken together, these findings support the Brunswikian perspective that one should not necessarily expect a direct relationship between team processes and performance with increasing time pressure because teams adapt their processes in different, yet often equally effective ways, in an effort to maintain high and stable performance.

Examining the effect of information sequence on Patriot air defense officers' judgments

Abstract This paper describes a recent experiment conducted with Patriot air defense officers and employing the Patriot air defense simulators at Ft. Bliss, Texas. The experiment found that, under certain conditions, the participants made different identification judgments and took different engagement actions depending on the sequence in which the same information was presented to them. This finding was consistent with the theoretical predictions of the Hogarth-Einhorn belief updating model. However, there were large individual differences, and experience mitigated the effect. These findings replicate previous research by the authors using a paper-and-pencil instrument. Limitations in the current experiment are discussed, as are directions for future research.

The role of task properties in determining the relative effectiveness of multiattribute weighting techniques

Abstract Most studies comparing the relative effectiveness of multiattribute weighting techniques have found few differences. These studies have, however, failed to systematically vary attribute properties such as number of attributes and distribution of correct attribute weights. It was hypothesized that any weighting technique would be more effective at arriving at the correct weights the smaller the number of attributes because of lessened information processing requirements. In addition, it was hypothesized that the relative effectiveness of different weighting techniques would depend on the peakedness of the distribution of correct attribute weights because different weighting techniques should generate more peaked distributions than others. Two experiments were conducted to test these hypotheses. The first experiment focused on the accuracy of the weights assigned to attributes by individuals; the second on the accuracy of groups. Both experiments confirmed the first hypothesis regarding the number of attributes, but only the first experiment confirmed the second hypothesis regarding the peakedness of the distribution of attribute weights.