Alex Kirlik

Supervisory control in a dynamic and uncertain environment: a process model of skilled human-environment interaction

An understanding of how both psychological and environmental factors mutually constrain skilled behavior is required to effectively support human activity. As a step toward meeting this need, a process model of skilled human interaction with a dynamic and uncertain environment is presented. The model was able to mimic human behavior in a laboratory task requiring one- and two-person crews to direct the activities of a fleet of agents to locate and process valued objects in a simulated world. The process model is a pair of highly interactive components that together mimic the behavior of the human-environment system. One component is a representation of the external environment as a dynamically changing set of opportunities for action. The second component is a dynamic representation of skilled human decision making and planning behavior within the environment so described. The process model is an expression of a general theory of skilled interaction assuming that perception and action mechanisms sensitive to environmental constraints are responsible for generating much of behavior, and where the need for additional cognitive processing of internal representations may result from environmental designs that do not adequately support the perceptual guidance of activity. >

SUPPORTING PERCEPTION IN THE SERVICE OF DYNAMIC DECISION MAKING

Skilled performers in complex environments rely heavily on heuristic strategies to cope with the time pressure and complexity of dynamic tasks. We suggest that the use of task simplification strategies based largely on perception and pattern recognition is fundamental to the novice-expert shift in dynamic decision making. We therefore suggest that interface training interventions should support the development of highly effective and robust heuristic strategies, rather than the development of more abstract, cognitively intensive strategies. A pair of empirical studies are presented that investigated the benefits of training interventions aimed at supporting perceptual and pattern-recognitional activities in dynamic environments. Results suggest that the acquisition of skilled performance in dynamic environments can be accelerated by supporting perceptual activities in the service of dynamic decision making. Implications of these results for training, aiding, and interface design are discussed.

The Ecological Expert: acting to create information to guide action

Technological interfaces often provide restricted access to controlled systems, requiring human operators to compensate by developing and reasoning with internal models. One solution to this problem is to enhance interface displays to provide the operator with an external system model or models. This design approach presumes complete and accurate display models can always be created and also ignores how inadequate interface resources for action, as opposed to perception, may be contributing to interaction difficulties. The article considers an alternative approach focussing on enhancing interface resources for action, motivated by the observation that people can sometimes compensate for inadequate perceptual conditions by acting to generate novel sources of perceptual information. The majority of the paper consists of a description and mathematical analysis of how performers at various skill levels use action to create perceptual information during system control, in the context of an everyday control task: short order cooking. The analysis also demonstrates how the most expert performers used action to create an external model of task dynamics that could be used in lieu of an internal model. The scope and limits of this action based design approach are considered.

Inferring rule-based strategies in dynamic judgment tasks: toward a noncompensatory formulation of the lens model

Performers in time-stressed, information-rich tasks develop rule-based, simplification strategies to cope with the severe cognitive demands imposed by judgment and decision making. Linear regression modeling, proven useful for describing judgment in a wide range of static tasks, may provide misleading accounts of these heuristics. That approach assumes cue-weighting and cue-integration are well described by compensatory strategies. In contrast, evidence suggests that heuristic strategies in dynamic tasks may instead reflect rule-based, noncompensatory cue usage. We therefore, present a technique called genetics-based policy capturing (GBPC) for inferring noncompensatory rule-based heuristics from judgment data as an alternative to regression. In GBPC, rule-base representation and search uses a genetic algorithm, and fitting the model to data using multiobjective optimization to maximize fit on three dimensions: completeness (all human judgments are represented); specificity (maximal concreteness); and parsimony (no unnecessary rules are used). GBPC is illustrated using data from the highest and lowest scoring participants in a simulated dynamic, combat information center (CIC) task. GBPC inferred rule-bases for these two performers that shed light on both skill and error. We compare the GBPC results with regression-based lens modeling of the same data set, and discuss how the GBPC results allowed us to interpret the high scoring performers highly significant use of unmodeled knowledge (C=1) revealed by lens model analysis. The GBPC findings also allow us to now interpret a similarly high use of unmodeled knowledge (C=1)in a previously published lens model analysis of a different data set collected in the same experimental task. We conclude by discussing training implications, and also prospects for the development of integrated GBPC models of both human judgment and the task environment, thus providing a noncompensatory formulation of the lens model (a genetics-based lens model, or GBLM) of the integrated human-environment system.

Modelling and simulation of complex sociotechnical systems: envisioning and analysing work environments

Accurate comprehension and analysis of complex sociotechnical systems is a daunting task. Empirically examining, or simply envisioning the structure and behaviour of such systems challenges traditional analytic and experimental approaches as well as our everyday cognitive capabilities. Computer-based models and simulations afford potentially useful means of accomplishing sociotechnical system design and analysis objectives. From a design perspective, they can provide a basis for a common mental model among stakeholders, thereby facilitating accurate comprehension of factors impacting system performance and potential effects of system modifications. From a research perspective, models and simulations afford the means to study aspects of sociotechnical system design and operation, including the potential impact of modifications to structural and dynamic system properties, in ways not feasible with traditional experimental approaches. This paper describes issues involved in the design and use of such models and simulations and describes a proposed path forward to their development and implementation. Practitioner Summary: The size and complexity of real-world sociotechnical systems can present significant barriers to their design, comprehension and empirical analysis. This article describes the potential advantages of computer-based models and simulations for understanding factors that impact sociotechnical system design and operation, particularly with respect to process and occupational safety.

Loss-of-Control prediction and prevention for NASA's transport class model

In an effort to improve the overall safety of the current and future air transportation system, researchers have been investigating the causes and series of events that can potentially lead to aircraft loss of control (LoC) as well as methods to prevent the occurrence of such events. Motivated by this challenging problem, this paper presents preliminary work on the development of technologies for in-flight LoC prediction and prevention. The developed scheme is based on a quantitative approach for LoC characterization introduced by the Boeing Company and NASA Langley Research Center, and is designed to arrest the development of an LoC sequence through an integrated LoC prediction and prevention control system. The paper also presents initial piloted simulations that analyze the ability of the developed system to assist the pilot in flying the aircraft through adverse environmental conditions, and to preserve safe aircraft operation by preventing erroneous crew input that could potentially lead to an LoC situation.

Brunswikian theory and method as a foundation for simulation-based research on clinical judgment.

Alex Kirlik, PhD An article on clinicians’ abilities to accurately and reliably make judgments of Apgar scores by Nadler et al1 (hereafter, “the authors”) published in a recent issue of this journal presented the first use of Egon Brunswik’s2– 4 (also see Refs. 5 and 6) theory of probabilistic functioning and methodology of representative design in Simulation in Healthcare. The authors noted that while the Apgar score has become an internationally recognized, standard method for rating judgments of the physiological status of newborns, little research had been done to date to determine how accurately these clinical judgments are made. Perhaps more importantly, the authors noted that although video recordings of neonatal resuscitations in actual clinical settings have been used for auditing and training, a recent study7 found that “clinicians cannot assign Apgar scores to video recordings of actual neonatal resuscitations with acceptable levels of interobserver reliability, and that observers’ scores had little agreement with the original scores assigned by the clinicians who performed the resuscitation.”1 In an attempt to clarify this state of affairs, the authors posed the question of whether the poor performance of the clinicians in the study just mentioned7 might, in large part, be an artifact of their restricted perceptual access to, or information about, the physiological states of the newborns whose resuscitations were video recorded. As such, the authors generated their own resuscitation videos that they believed to make more perceptually available the actual clinical signs (judgment cues) needed to make accurate Apgar judgments in a clinical context. To do so, they used a representatively designed sample of 51 resuscitation scenarios and a newborn mannequin-simulator to make their own video recordings. In describing the motivation for their research method, the authors wrote “According to Brunswik, experiments—including experiments in laboratories—should have a representative design where participants are exposed to situations that represent the range and distribution of situations and cues (clinical signs) in their natural environment.”1 Using this methodology, the authors found that the clinicians participating in their study made Apgar rating judgments that were highly (0.78 – 0.91) correlated with true Apgar scores programmed into their patient simulator (ie, ground truth). The authors concluded that the relatively high level of clinical judgment performance observed in their study, as compared with the prior study,7 owed largely to a relatively higher degree of representativeness2– 4 in their experimental design. Finally, the authors also concluded that the high level of correlation (0.79 – 0.97) found between clinician’s ratings and individualized linear models created by regressing these ratings against the perceptual cues or clinical signs available from the videotapes indicated that clinicians demonstrated “systematic” judgment. That is, the authors found a high level of consistency in how clinicians executed their cognitive strategies for making these judgments.8 This finding was taken to lend credence to the use of Brunswik’s theory of probabilistic functionalism to provide a useful way to analyze and model clinical judgment in representatively designed experimental conditions.

Cognition in Human—Machine Systems: Experiential and Environmental Aspects of Adaptation

Publisher Summary This chapter discusses cognition in human―machine system (HMS) that explains the experimental and environmental aspects of adaption. Advances in information technology have altered the human ecology in fundamental ways, with many of these changes presenting demands upon cognition in learning, problem solving, decision making, and skilled performance. Cognitive research in human–machine systems differs from other research on cognition as its goal of practical relevance forces investigators to come to grips with two methodological issues: relative inability to select and control the task-relevant knowledge and the experience of its research participants; and a relative inability to select and control the task environment in which research is conducted. These constraints place limits on the possibility for systematic experimental design and hypothesis testing. On the positive side, however, empirical studies in HMS necessarily sample the regions of the human cognitive ecology that can go unexplored by investigations conducted with a more cautious attitude toward participant selection and environmental control. And finally, the unique and perhaps most important contributions HMS research has made to the study of cognition deals with the experiential and environmental aspects of cognition in adaptive behavior.Publisher Summary This chapter discusses cognition in human―machine system (HMS) that explains the experimental and environmental aspects of adaption. Advances in information technology have altered the human ecology in fundamental ways, with many of these changes presenting demands upon cognition in learning, problem solving, decision making, and skilled performance. Cognitive research in human–machine systems differs from other research on cognition as its goal of practical relevance forces investigators to come to grips with two methodological issues: relative inability to select and control the task-relevant knowledge and the experience of its research participants; and a relative inability to select and control the task environment in which research is conducted. These constraints place limits on the possibility for systematic experimental design and hypothesis testing. On the positive side, however, empirical studies in HMS necessarily sample the regions of the human cognitive ecology that can go unexplored by investigations conducted with a more cautious attitude toward participant selection and environmental control. And finally, the unique and perhaps most important contributions HMS research has made to the study of cognition deals with the experiential and environmental aspects of cognition in adaptive behavior.

MODE USAGE IN AUTOMATED COCKPITS: SOME INITIAL OBSERVATIONS

Abstract Mode confusion is increasingly becoming a significant contributor to accidents and incidents involving highly automated airliners; in the last seven years there have been four airline accidents in which mode problems were present. This paper attempts to provide some initial observations about modes and how pilots use them. The authors define the terms “mode“,“mode transitions“,“mode configurations“, and propose a framework for describing and classifying modes. Preliminary results from a field study that documented mode usage in “Glass Cockpit” aircraft are presented. The data were collected during 30 flights onboard Boeing 757/767 type aircraft. Summary of the data depicts the paths that pilots use in transitioning from one mode to another. Analysis of the data suggest that these mode transitions are influenced by changes in aircraft altitude as well as by two factors in the operational environment: the type of air traffic control facility supervising the flight, and the type of instruction (clearance) issued.

Interactive visualizations to improve Bayesian reasoning

Proper Bayesian reasoning is critical in a variety of domains that require practitioners to make predictions about the probability of events contingent upon earlier actions or events. However, much research on judgment has shown that people who are unfamiliar with Bayes’ Theorem often reason quite poorly with conditional probabilities due to various cognitive biases. Owing to previous successes of visualization techniques for debiasing judges and improving judgment performance, we created an interactive computer visualization designed to aid Bayes-naïve people in solving conditional probability problems that would not require a training period to use, and would be flexible enough to accommodate many problem types. Results are suggestive that participants using our interactive visualization were able to substantially improve their Bayesian reasoning performance above that of previous debiasing methods. This finding has significant implications for expanding the toolbox of techniques that can be used to more accurately elicit predictions and forecasts from judges whose expertise lies beyond the realm of statistics.