Stacey Langfitt Hendrickson

Issues in Benchmarking Human Reliability Analysis Methods: A Literature Review

There is a diversity of human reliability analysis (HRA) methods available for use in assessing human performance within probabilistic risk assessments (PRA). Due to the significant differences in the methods, including the scope, approach, and underlying models, there is a need for an empirical comparison investigating the validity and reliability of the methods. To accomplish this empirical comparison, a benchmarking study comparing and evaluating HRA methods in assessing operator performance in simulator experiments is currently underway. In order to account for as many effects as possible in the construction of this benchmarking study, a literature review was conducted, reviewing past benchmarking studies in the areas of psychology and risk assessment. A number of lessons learned through these studies is presented in order to aid in the design of future HRA benchmarking endeavors.

Improving Extreme-Scale Problem Solving: Assessing Electronic Brainstorming Effectiveness in an Industrial Setting

Objective: An experiment was conducted to compare the effectiveness of individual versus group electronic brainstorming to address difficult, real-world challenges. Background: Although industrial reliance on electronic communications has become ubiquitous, empirical and theoretical understanding of the bounds of its effectiveness have been limited. Previous research using short-term laboratory experiments have engaged small groups of students in answering questions irrelevant to an industrial setting. The present experiment extends current findings beyond the laboratory to larger groups of real-world employees addressing organization-relevant challenges during the course of 4 days. Methods: Employees and contractors at a national laboratory participated, either in a group setting or individually, in an electronic brainstorm to pose solutions to a real-world problem. Results: The data demonstrate that (for this design) individuals perform at least as well as groups in producing quantity of electronic ideas, regardless of brainstorming duration. However, when judged with respect to quality along three dimensions (originality, feasibility, and effectiveness), the individuals significantly (p < .05) outperformed the group. Conclusion: When quality is used to benchmark success, these data indicate that work-relevant challenges are better solved by aggregating electronic individual responses rather than by electronically convening a group. Application: This research suggests that industrial reliance on electronic problem-solving groups should be tempered, and large nominal groups may be more appropriate corporate problem-solving vehicles.

Individual differences and the science of human performance

This study comprises the third year of the Robust Automated Knowledge Capture (RAKC) project. In the previous two years, preliminary research was conducted by collaborators at the University of Notre Dame and the University of Memphis. The focus of this preliminary research was to identify relationships between cognitive performance aptitudes (e.g., short-term memory capacity, mental rotation) and strategy selection for laboratory tasks, as well as tendencies to maintain or abandon these strategies. The current study extends initial research by assessing electrophysiological correlates with individual tendencies in strategy selection. This study identifies regularities within individual differences and uses this information to develop a model to predict and understand the relationship between these regularities and cognitive performance.

Robust Automated Knowledge Capture

This report summarizes research conducted through the Sandia National Laboratories Robust Automated Knowledge Capture Laboratory Directed Research and Development project. The objective of this project was to advance scientific understanding of the influence of individual cognitive attributes on decision making. The project has developed a quantitative model known as RumRunner that has proven effective in predicting the propensity of an individual to shift strategies on the basis of task and experience related parameters. Three separate studies are described which have validated the basic RumRunner model. This work provides a basis for better understanding human decision making in high consequent national security applications, and in particular, the individual characteristics that underlie adaptive thinking.

Individual and Group Electronic Brainstorming in an Industrial Setting

An experiment was conducted comparing the effectiveness of individual versus group electronic brainstorming in addressing real-world “wickedly difficult” challenges. Previous laboratory research has engaged small groups of students in answering questions irrelevant to an industrial setting. The current experiment extended this research to larger, real-world employee groups engaged in addressing organization-relevant challenges. Within the present experiment, the data demonstrated that individuals performed at least as well as groups in terms of number of ideas produced and significantly (p<.02) outperformed groups in terms of the quality of those ideas (as measured along the dimensions of originality, feasibility, and effectiveness).

A Framework for Understanding Operator Decision Making in Simulated Nuclear Power Plant Cyber Attacks

Malicious cyber-attacks are becoming increasingly prominent due to the advance of technology and methods over the last decade. These attacks have the potential to bring down critical infrastructures, such as nuclear power plants (NPP’s), which are so vital to the country that their incapacitation would have debilitating effects on national security, public health, or safety. Despite the devastating effects a cyber-attack could have on NPP’s, there is a lack of understanding as to the effects on the plant from a discreet failure or surreptitious sabotage of components and a lack of knowledge in how the control room operators would react to such a situation. In this project, the authors are collaborating with NPP operators to discern the impact of cyber-attacks on control room operations and lay out a framework to better understand the control room operators’ tasks and decision points.

Improving human effectiveness for extreme-scale problem solving : final report (assessing the effectiveness of electronic brainstorming in an industrial setting).

An experiment was conducted comparing the effectiveness of individual versus group electronic brainstorming in order to address difficult, real world challenges. While industrial reliance on electronic communications has become ubiquitous, empirical and theoretical understanding of the bounds of its effectiveness have been limited. Previous research using short-term, laboratory experiments have engaged small groups of students in answering questions irrelevant to an industrial setting. The current experiment extends current findings beyond the laboratory to larger groups of real-world employees addressing organization-relevant challenges over the course of four days. Findings are twofold. First, the data demonstrate that (for this design) individuals perform at least as well as groups in producing quantity of electronic ideas, regardless of brainstorming duration. However, when judged with respect to quality along three dimensions (originality, feasibility, and effectiveness), the individuals significantly (p<0.05) out performed the group working together. The theoretical and applied (e.g., cost effectiveness) implications of this finding are discussed. Second, the current experiment yielded several viable solutions to the wickedly difficult problem that was posed.