Christina Rusnock

Objective-Analytical Measures of Workload – the Third Pillar of Workload Triangulation?

The ability to assess operator workload is important for dynamically allocating tasks in a way that allows efficient and effective goal completion. For over fifty years, human factors professionals have relied upon self-reported measures of workload. However, these subjective-empirical measures have limited use for real-time applications because they are often collected only at the completion of the activity. In contrast, objective-empirical measurements of workload, such as physiological data, can be recorded continuously, and provide frequently-updated information over the course of a trial. Linking the low-sample-rate subjective-empirical measurement to the high-sample-rate objective-empirical measurements poses a significant challenge. While the series of objective-empirical measurements could be down–sampled or averaged over a longer time period to match the subjective-empirical sample rate, this process discards potentially relevant information, and may produce meaningless values for certain types of physiological data. This paper demonstrates the technique of using an objective-analytical measurement produced by mathematical models of workload to bridge the gap between subjective-empirical and objective-empirical measures. As a proof of concept, we predicted operator workload from physiological data using VACP, an objective-analytical measure, which was validated against NASA-TLX scores. Strong predictive results pave the way to use the objective-empirical measures in real-time augmentation (such as dynamic task allocation) to improve operator performance.

Incorporating automation: using modeling and simulation to enable task re-allocation

Models for evaluating changes in human workload as a function of task allocation between humans and automation are investigated. Specifically, SysML activity diagrams and IMPRINT workload models are developed for a tablet-based game with the ability to incorporate automation. Although a first order model could be created by removing workload associated with tasks that are allocated away from the human and to the computer, we discuss the need to improve the activity diagrams and models by capturing workload associated with communicating state information between the human and the automation. Further, these models are extended to capture additional human tasks, which permit the user to maintain situation awareness, enabling the human to monitor the robustness of the automation. Through these model extensions, it is concluded that human workload will be affected by the degree the human relies upon the automation to accurately perform its allocated tasks.

Improving Model Cross-Applicability for Operator Workload Estimation

When operators are overwhelmed, judicious employment of automation can help. Ideally, an adaptive system which can accurately estimate current operator workload can more effectively employ automation. Supervised machine learning models can be trained to estimate workload from operator-state parameters sensed by on-body sensors which, for example, collect heart rate or brain activity information. Unfortunately, estimating operator workload using trained models is limited: using a model trained in one context can yield poor estimation of workload in another. This research examines the efficacy of using two regression-tree alternatives (random forests and pruned regression trees) to decrease workload estimation cross-application error. The study is conducted for a remotely piloted aircraft simulation under two context-switch scenarios 1) across human operators and 2) across task conditions. While cross-task results were inconclusive, both algorithms significantly reduced cross-application error in estimating workload across operators, and random forests performed best in cross-operator applicability.

Timing within human-agent interaction and its effects on team performance and human behavior

Current systems incorporating human-agent interaction typically place the human in a supervisory role and the agent as a subordinate. However, a key aspect of teaming is the dynamic shift in roles. Depending on the situation at hand, teaming could lead to a peer relationship where the human and agent are working together on the same task. This research investigates how the timing of agent actions impacts team performance, as well as human workload and behavior. A human-in-the-loop experiment demonstrated that when the agent performs tasks faster than the human, the human tends to become reliant upon the automation and assumes a supervisory role. A human performance model predicts that extending agent execution time will decrease human reliance on the automation. However, in the environment under investigation, a tradeoff exists between team performance and human involvement.

Extending System Readiness Levels to Assess and Communicate Human Readiness

Program managers need to understand the maturity of all aspects of their system to make intelligent investment decisions during product development. Technology Readiness Levels TRL are regularly applied to classify and communicate the state of technology development and to guide investment during technology and system development. This concept has recently been extended to include integration readiness to understand the maturity of the interfaces between technology components in a system. However, these methods do not consider whether human interfaces to the system have been fully matured. The current research extends the TRL and Integration Readiness Level IRL concepts to include the human elements of the system. The method proposes two concepts: the Human Capability Level and the Human IRL. The Human Capability Level refers to the capability of a system to reliably provide well equipped personnel with sufficient skills and the ability to perform successfully with the system. The Human Integration Readiness Level refers to the demonstration of effective human integration with system technologies. Each of these concepts are then combined with previous work to demonstrate the development of a System Readiness Level, which includes technology and human elements. We show that these Human Capability and Integration Level concepts can be used in conjunction with the existing TRL and IRL concepts to compare the relative maturity of components of the system for supporting each class of human users as well as performing its technical functions.

Simulation-Based Evaluation of the Effects of Patient Load on Mental Workload of Healthcare Staff

Introduction In parts of Ohio, Veterans Affairs Medical Centers are working to handle patient load issues by sending patient overflows to the Wright-Patterson Medical Center. The Wright-Patterson Medical Center will benefit from the increase in patients; however, there are concerns that the patient quality of care may suffer. If the increase in patients results in the healthcare staff experiencing high mental workload levels, staff performance could be reduced. The objective of this research is to evaluate the influence of patient load on the mental workload of staff in an inpatient unit at the Wright-Patterson Medical Center. Methods This research uses discrete-event simulation to quantitatively model the mental workload of healthcare staff in an inpatient unit of the Wright-Patterson Medical Center. The model was used to find the idle time, average workload, and overload time of healthcare staff under current and future patient loads. In addition, the performance of individual tasks was evaluated. Results The results of this research find a linear relationship between patient load and three workload metrics (idle time, average workload, and overload time) with each worsening as patient load increases. Nurses and technicians experience the greatest negative impacts to mental workload as patient load increases with those staff members who have the most workload at the baseline condition experiencing greater increase in workload as patient load increases. In addition, the time spent in an overload state increases disproportionately with patient load increases, with overload time increases being worse for urgent tasks than for nonurgent tasks. Conclusions Based on this study, the researchers found that the modeled inpatient unit can safely handle the expected patient load increases. The study provides the unit with information to proactively prepare and reduce healthcare staff overloading.

Human-Centered Design Using System Modeling Language

The human user is important to consider during system design. However, common system design models, such as the system modeling language, typically represent human users and operators as external actors, rather than as internal to the system. This research presents a method for integrating human considerations into system models through human-centered design. A specific system is selected to serve as the case study for demonstrating the methodology. The sample system is analyzed to identify the task and information flow. Then, both system- and human-centered diagrams are separately created to represent different viewpoints of the system. These diagrams are compared and analyzed, and new diagrams are created that incorporate both system and human considerations into one concordant representation of the system model. These new views allow systems engineers and human factors engineers to effectively communicate the role of the user during early system design trades.

Measuring Human Trust Behavior in Human-Machine Teams

This paper presents a paradigm for distilling trust behaviors in human-machine teams. The paradigm moves beyond diagnostic alarms-based automation definitions of compliance and reliance toward a view of trust behavior that includes automations where the machine has authority to act on behalf of the human-machine team in the environment. The paradigm first determines the purpose of the automation and then relies on three types of authority within the human-machine team to identify what trust behaviors will look like in specific instances. An example using the Space Navigator environment demonstrates how trust behaviors can be measured.

Assessing Continuous Operator Workload With a Hybrid Scaffolded Neuroergonomic Modeling Approach

Objective: We aimed to predict operator workload from neurological data using statistical learning methods to fit neurological-to-state-assessment models. Background: Adaptive systems require real-time mental workload assessment to perform dynamic task allocations or operator augmentation as workload issues arise. Neuroergonomic measures have great potential for informing adaptive systems, and we combine these measures with models of task demand as well as information about critical events and performance to clarify the inherent ambiguity of interpretation. Method: We use machine learning algorithms on electroencephalogram (EEG) input to infer operator workload based upon Improved Performance Research Integration Tool workload model estimates. Results: Cross-participant models predict workload of other participants, statistically distinguishing between 62% of the workload changes. Machine learning models trained from Monte Carlo resampled workload profiles can be used in place of deterministic workload profiles for cross-participant modeling without incurring a significant decrease in machine learning model performance, suggesting that stochastic models can be used when limited training data are available. Conclusion: We employed a novel temporary scaffold of simulation-generated workload profile truth data during the model-fitting process. A continuous workload profile serves as the target to train our statistical machine learning models. Once trained, the workload profile scaffolding is removed and the trained model is used directly on neurophysiological data in future operator state assessments. Application: These modeling techniques demonstrate how to use neuroergonomic methods to develop operator state assessments, which can be employed in adaptive systems.

Effects of agent timing on the human-agent team

Abstract As technology becomes more sophisticated, autonomous agents are applied more frequently to improve system performance. The current research employed a five step method, including modeling, simulation, and human experimentation to explore the effect of an artificial agent’s timing on the performance of a human-agent team within a highly dynamic task environment. Agent timing significantly influenced the role assumed by the human within the team. Further, agent timing changed system performance by approximately 40% within the experimental conditions. Results indicate that an artificial agent’s timing can be varied as a function of the task demands placed upon the human-agent team to maintain an appropriate level of human activity and engagement. Therefore, agent timing may be controlled to adapt autonomy to provide an apparent continuum along which to control human engagement in systems employing human-agent teaming within dynamic environments.