Eric T. Greenlee

Which Eye Tracker Is Right for Your Research? Performance Evaluation of Several Cost Variant Eye Trackers

Though not often mentioned, the price point of many eye tracking systems may be a factor limiting their adoption in research. Recently, several inexpensive eye trackers have appeared on the market, but to date little systematic research has been conducted to validate these systems. The present experiment attempted to address this gap by evaluating and comparing five different eye trackers, the Eye Tribe Tracker, Tobii EyeX, Seeing Machines faceLAB, Smart Eye Pro, and Smart Eye Aurora for their gaze tracking accuracy and precision. Results suggest that all evaluated trackers maintained acceptable accuracy and precision, but lower cost systems frequently also experienced high rates of data loss, suggesting that researchers adopting low cost systems such as those evaluated here should be judicious in their research usage.

Coordinated Displays to Assist Cyber Defenders

Cyber network analysts must gather evidence from multiple sources and ultimately decide whether or not suspicious activity represents a threat to network security. Information relevant to this task is usually presented in an uncoordinated fashion, meaning analysts must manually correlate data across multiple databases. The current experiment examined whether analyst performance efficiency would be improved by coordinated displays, i.e., displays that automatically link relevant information across databases. We found that coordinated displays nearly doubled performance efficiency, in contrast to the standard uncoordinated displays, and coordinated displays resulted in a modest increase in threat detections. These results demonstrate that the benefits of coordinated displays are significant enough to recommend their inclusion in future cyber defense software.

Driver Vigilance in Automated Vehicles: Hazard Detection Failures Are a Matter of Time:

Objective: The primary aim of the current study was to determine whether monitoring the roadway for hazards during automated driving results in a vigilance decrement. Background: Although automated vehicles are relatively novel, the nature of human-automation interaction within them has the classic hallmarks of a vigilance task. Drivers must maintain attention for prolonged periods of time to detect and respond to rare and unpredictable events, for example, roadway hazards that automation may be ill equipped to detect. Given the similarity with traditional vigilance tasks, we predicted that drivers of a simulated automated vehicle would demonstrate a vigilance decrement in hazard detection performance. Method: Participants “drove” a simulated automated vehicle for 40 minutes. During that time, their task was to monitor the roadway for roadway hazards. Results: As predicted, hazard detection rate declined precipitously, and reaction times slowed as the drive progressed. Further, subjective ratings of workload and task-related stress indicated that sustained monitoring is demanding and distressing and it is a challenge to maintain task engagement. Conclusion: Monitoring the roadway for potential hazards during automated driving results in workload, stress, and performance decrements similar to those observed in traditional vigilance tasks. Application: To the degree that vigilance is required of automated vehicle drivers, performance errors and associated safety risks are likely to occur as a function of time on task. Vigilance should be a focal safety concern in the development of vehicle automation.

Does the Vigilance Decrement Occur for Elementary Features

The aim of the current study was to determine whether the vigilance decrement occurs when observers search for a critical signal that consists of an elementary feature (line orientation). Elementary features are processed quickly and presumably with minimal attentional resources (Treisman & Gormican, 1988). Such features should be resistant to the vigilance decrement, according to theories of the decrement that posit depletion of attentional resources as the underlying mechanism. Observers completed a vigilance task in which they reported the presence of a critical signal, which consisted of a slanted line presented amidst vertical lines. A vigilance decrement was evident in correct detections and reaction time. In a follow-up study, a pop-out effect for the slanted line was replicated in a traditional search task. To the extent that an elementary feature is processed preattenatively, the occurrence of a vigilance decrement challenges theories of the decrement based on depleted attentional resources. However, whether elementary features are processed without attention has been debated and further studies are needed. Identifying display characteristics that can be processed automatically without depleting attention will enhance monitoring performance by eliminating the decrement in safety critical tasks such as aviation and baggage screening.

Is Neuroergonomic Monitoring Non-Intrusive? An Examination of Transcranial Doppler Sonography

One of the primary aims of neuroergonomic research is to identify neural and psychophysiological measures that can be used to index operator state in real time (e.g., Stephens, Scerbo, & Pope, 2012; Tripp & Warm, 2007). Such measures are desired because they may have capabilities that exceed traditional subjective and performance methods of estimating operator state. Physiological monitoring has been proffered as a remedy for the shortcomings of subjective and performance measures of operator state, because it is arguably less intrusive than subjective measures, allows for continuous measurement, and provides estimates of operator state even when performance efficiency is unobservable (Stephens et al., 2012). One area where physiological monitoring may be particularly beneficial is within tasks that require vigilance. Vigilance tasks are common within many military, industrial, and medical settings and generally require an operator to maintain attentional focus for the detection of critical events that tend to occur rarely and unpredictably. However, the ability to maintain vigilance typically declines over time, which is problematic when detection is critical to system safety. Additionally, the demands of vigilance tasks are associated with high workload and task-induced stress (Warm, Parasuraman, & Matthews, 2008). Research suggests that physiological measures may allow for continuous, real-time tracking of an operator’s level of vigilance, and facilitate adaptive interventions (e.g., rest breaks, adaptive automation) to ensure that a loss of vigilance does not lead to errors or accidents (Warm & Parasuraman, 2007). Transcranial Doppler Sonography (TCD) is a physiological measure that is especially well-suited to real-time assessment of operator vigilance (Warm & Parasuraman, 2007). TCD can be used to monitor hemodynamic activity within the brain, which has been closely linked to level of vigilance performance. In short, TCD may serve to predict and prevent loss of vigilance, non-intrusively in operational settings (Warm & Parasuraman, 2007). Before deploying TCD for vigilance monitoring in operational settings users should consider whether it is truly non-intrusive. If TCD monitoring negatively impacts operator state or performance, practical applications may be limited. The current study was designed to determine whether TCD use has intrusive effects on operator state and performance. To assess operator state, we utilized three measures that have been used previously in vigilance tasks: The Short Stress State Questionnaire (SSSQ; Helton, 2004), the Simulator Sickness Questionnaire (Kennedy, Lane, & Berbaum, 1993), and the NASA-TLX. These questionnaires were used to assess task-induced stress, symptoms of simulator sickness, and workload, respectively. Performance was assessed in terms of percentage correct detections. Forty-two students (34 Women, 8 Men) were assigned at random to either a Doppler Monitoring condition or to a Control (No Physiological Monitoring) condition with the restriction that the group size (n = 21) and the ratio of participant sex be identical in both conditions. All participants completed a 40-minute vigilance task based on the procedure used by Greenlee and colleagues (2015), wherein participants monitored a computerized gauge for cases in which the gauge needle deviated from a vertical position (20% signal probability, 50 events per minute). Participants in the Doppler Monitoring condition wore the TCD during the vigil, those in the Control condition did not. Results from both conditions mirror typical vigilance findings. Percentage correct detections declined over time, distress and symptoms of simulator sickness increased, engagement decreased, and workload was rated as high. However, the performance, task-induced stress, reported levels of simulator sickness, and workload of participants equipped with TCD were indistinguishable from values taken from participants in the control condition. With the caveat that these are statistically null results, this supports the possibility that TCD monitoring may be non-intrusive. This finding, combined with research demonstrating the link between TCD measurements and vigilance (e.g., Warm & Parasuraman, 2007), supports the recommendation that TCD should be used in operational settings where operator vigilance is of paramount importance.

When actions speak louder than words: Using changes in operator behavior and system efficiency measures to detect the presence of a cyber attack

Increases in cyber incidents have required substantial investments in cyber defense for national security. However, adversaries have begun moving away from traditional cyber tactics in order to escape detection by network defenders. The aim of some of these new types of attacks is not to steal information, but rather to create subtle inefficiencies that, when aggregated across a whole system, result in decreased system effectiveness. The aim of such attacks is to evade detection for long durations, allowing them to cause as much harm as possible. As a result, such attacks are sometimes referred to as “low and slow” (e.g., Mancuso et al., 2013). It is unknown how effective operators are likely to be at detecting and correctly diagnosing the symptoms of low and slow cyber attacks. Recent research by Hirshfield and colleagues (2015) suggests that the symptoms of the attack may need to be extreme in order to gain operator recognition. This calls into question the utility of relying on operators for detection altogether. Therefore, one goal for this research was to provide an initial exploration of attack deception and magnitude on operator behavior, performance, and potential detection of the attack. Operators in these systems are not passive observers, however, but active agents attempting to further their task goals. As a result, operators may alter their behavior in response to degraded system capabilities. This suggests that changes in the pattern and frequency of operator behavior following the inception of a cyber attack could potentially be used to detect its onset, even without the operator being fully aware of those changes (Mancuso et al., 2014). Similarly, since low and slow attacks are designed to degrade overall system effectiveness, performance measures of system efficiency, such as frequency and duration of tasks completed, may provide additional means to detect an ongoing cyber attack. As such, a second goal for the present research was to determine whether changes in operator behavior and system efficiency metrics could act as indicators of an active low and slow cyber attack. Participants in this experiment performed a multiunmanned aerial vehicle (UAV) supervisory control task. During the task, participant control over their UAVs was disrupted by a simulated cyber attack that caused affected UAVs to stop flying toward participant- selected destinations and enter an idle state. Aside from halting along their designated flight path, idled UAVs displayed no other indication of the cyber attack. The frequency of cyber attacks increased with time-on-task, such that attacks were relatively infrequent at the beginning of the task, occurring once in every five destination assignments made, and were ubiquitous by the end of the task, occurring after each destination assignment. Attack deception was manipulated with regard to participants’ approximate screen gaze location at the time of a cyber attack. In the overt condition, UAVs entered the idle state near the participant’s current focal area (indexed by the location of operator mouse interactions with the simulation), thereby providing some opportunity for operators to directly observe the effects of the cyber attack. In the covert condition, the attack occurred outside the operator’s current focal area, forcing them to rely on memory to detect the cyber attack. In the control condition, no cyber attacks occurred during the experiment. Following the UAV supervisory control task, participants were asked a series of debriefing questions to determine if they had noticed the UAV manipulation during the task. Most participants (approximately 64%) reported noticing the manipulation, but only after a series of questions prompting them to think of any problems they encountered during the task. The remaining participants reported noticing no errors during the task. Results regarding measures of performance and system efficiency indicated that performance decreased as the magnitude of the cyber attack increased. Measures of efficiency were calculated using fan-out (Olsen & Goodrich, 2003) which provided information regarding how many UAVs operators were able to control and how long UAVs were in an idle state during the trial. Operators controlled fewer vehicles, and vehicles sat idle for longer durations, as the magnitude of the cyber attack increased. However, these differences in efficiency were not statistically significantly different until relatively late in the trial. Overall, operators seemed insensitive to the presence of the cyber attack, only disclosing the problem after being prompted several times through guided questions by the experimenter. However, significant changes in operator behavior and system efficiency were observed as the magnitude of the cyber attack increased. These results demonstrate that subtle cyber attacks designed to slowly degrade human performance were measurable, but these changes were not apparent until late in the experiment when the attack was at its midpoint in magnitude. This experiment suggests that even though measurable changes in operator behavior may not occur until late in an attack, these metrics are more effective than reliance on operator detection.

Evaluation of the Team Workload Questionnaire (TWLQ) in a Team-Choice Task

Objective: The present study was designed to evaluate the team workload questionnaire (TWLQ) in a task that was distinct from the task used to create it. Background: The TWLQ was created from workload ratings generated by members of athletic sports teams. Given that such teams represent only a portion of the diversity of operational teams, we aimed to assess the generalizability of the TWLQ. Method: The present study applied the TWLQ in a collaborative choice task (hiring decision) to determine whether the factor structure reported in the initial publication of the scale would generalize from the execution tasks it was developed from to a disparate team task focused on consensus building. Results: Confirmatory factor analysis indicated that the present data (N = 144) were a poor fit for the three-factor structure of the TWLQ. Subsequent exploratory factor analysis revealed a much more interrelated model of team workload with no clear division between the three conceptual factors described in the original validation of the TWLQ. Conclusion: The factor structure of the TWLQ did not generalize to the present team-choice task. Application: Given that the duties of operational teams vary, it is critical that future research examine how the conceptual structure of team workload may be altered by task type.

Driver Vigilance in Automated Vehicles: Effects of Demands on Hazard Detection Performance

Objective: The current study investigated driver vigilance in partially automated vehicles to determine whether increased task demands reduce a driver’s ability to monitor for automation failures and whether the vigilance decrement associated with hazard detections is due to driver overload. Background: Drivers of partially automated vehicles are expected to monitor for signs of automation failure. Previous research has shown that a driver’s ability to perform this duty declines over time. One possible explanation for this vigilance decrement is that the extreme demands of vigilance causes overload and leads to depletion of limited attentional resources required for vigilance. Method: Participants completed a 40-min drive in a simulated partially automated vehicle and were tasked with monitoring for hazards that represented potential automation failures. Two factors were manipulated to test the impact of monitoring demands on performance: Spatial uncertainty and event rate. Results: As predicted, hazard detection performance was poorer when monitoring demands were increased, and performance declined as a function of time on task. Subjective reports also indicated high workload and task-induced stress. Conclusion: Drivers of partially automated vehicles are impaired by the vigilance decrement and elevated task demands, meaning that safe operation becomes less likely when the demands associated with monitoring automation increase and as a drive extends in duration. This study also supports the notion that vigilance performance in partially automated vehicles is likely due to driver overload. Application: Developers of automation technologies should consider countermeasures that attenuate a driver’s cognitive load when tasked with monitoring automation.

Evaluation of the Team Workload Questionnaire (TWLQ) in a Team Choice Task

To date, conceptual explanations of workload and development of workload measures have been focused primarily on individual workload, the workload of a single operator as they perform a task. Yet, this focus on individual workload does not consider the many situations in which operators are required to collaborate, communicate, and operate as a team to achieve successful performance outcomes. In short, conceptualization and development of team workload measures have lagged behind those of individual workload. In an effort to meet the need for a conceptually-driven team workload measure, Sellers, Helton, Näswall, Funke, and Knott (2014) recently developed the team workload questionnaire (TWLQ). In developing the measure, Sellers and colleagues asked rugby players to rate their workload on TWLQ items. Subsequent exploratory factor analysis suggested that team workload was best described by three latent factors: Taskwork, the demands for task execution on the individual; Teamwork, the demands for team members to cooperate and coordinate with other teammates; and Team-Task Balancing, the demands associated with the need to manage both taskwork and teamwork – reflective of the dual task nature of working within a team. As with any novel measure of workload, it is important to continue evaluation of the measure’s sensitivity to task demands, diagnosticity regarding sources of task demands, and correlation with performance outcomes. Early research with the TWLQ has demonstrated that the measure is sensitive to changes in team task demands and the effects of training in a team UAV control task (Helton, Epling, de Joux, Funke, & Knott, 2015; Sellers, Helton, Näswall, Funke, & Knott, 2015). An additional, critical component of continued validation of the TWLQ is confirmation of the factor structure initially observed by Sellers and colleagues (2014) with data generated from a novel task. Concerns regarding generalizability are particularly germane because of variability in the nature of tasks that teams engage. Whereas some teams are tasked with executing coordinated physical activities, such as is the case in athletic contests (e.g., rugby), the task of other teams is to talk, plan, and decide (e.g., committees; McGrath, 1984). In the current study, we applied the TWLQ in a collaborative choice task (a personnel hiring decision). This team choice task required a high degree of communication, discussion, and joint decision making – team dynamics that contrast sharply with those required during an execution task. In short, the nature of the teamwork in the current study was significantly different from the teamwork evaluated by Sellers and colleagues (2014) when generating the TWLQ. Our goal in this study was to continue validation of the TWLQ by examining its factor structure with a novel dataset derived from a task requiring qualitatively different team dynamics. Confirmatory factor analysis indicated that the present data (N = 144) were a poor fit for the three-factor structure of the TWLQ. Subsequent exploratory factor analysis revealed a much more interrelated model of team workload with no clear division between the three conceptual factors described in the original validation of the TWLQ. This finding indicates that the factor structure of the TWLQ did not generalize to the present team choice task. Given that the duties of operational teams vary, it is critical that future research examine how the conceptual structure of team workload may be altered by task type.

Driver Vigilance in Automated Vehicles: Investigating Hazard Detection Performance

Current automated driving technologies are limited to semi-automated function, meaning that the human drivers are still responsible for aspects of vehicle operation and must supervise the functions of the automated vehicle. When automation is engaged, drivers transition from manually controlling the vehicle to passively monitoring the vehicular environment for any signs of impending automation failure. Thus, operation of semi-autonomous vehicles requires the drivers to sustain attention, to remain vigilant, so that they can intervene and quickly resume manual control of the vehicle if the need arises. This human-automation interaction has the classic hallmarks of a vigilance task. Because automated vehicle drivers appear to be engaged in a traditional vigilance task, we predicted that they would exhibit the vigilance decrement as a time-related decline in hazard detection performance (percent correct detections). Further, we predicted that automated vehicle drivers would experience high levels of task-induced workload and stress, two additional consequences of vigilance tasks. To test these predictions, eleven licensed drivers (5 men, 6 women) completed a 40-minute simulated driving task in which an automated driving system performed all vehicle control functions, while the drivers monitored the roadway to detect hazards which represented the possibility of potential automation failure and collision. Workload was assessed using the NASA-TLX (Hart & Staveland, 1988) and taskinduced stress was measured using the Short Stress State Questionnaire (SSSQ; Helton, 2004). As predicted, hazard detection performance declined over time. As the drive progressed, the average rate of correct detections declined by more than 20%. Regarding workload, the automated driving task elicited high ratings of Mental Demand, Temporal Demand, Effort, and Frustration. These findings align with previous vigilance research, wherein workload is generally high and the typical contributors to workload are mental demand and effort, and sometimes, temporal demand and frustration (Finomore et al., 2013; Greenlee et al., 2016; Warm et al., 2008). In addition to being demanding, it also appears that the automated driving task may have been stressful. Results from the SSSQ indicated that engagement decreased during the task. A decline in Engagement is one component of the stress response that characteristically occurs during a vigilance task (Warm et al., 2008). In sum, the current results reveal that sustained monitoring of the roadway during automated driving leads to deleterious effects on driver performance and subjective state. Given that, in current automated vehicles, human drivers are expected to engage in supervisory tasks the observed decrement in drivers’ hazard detection performance has dire implications for the safety of automated vehicle technology. To ensure safe semi-automated vehicle technologies, researchers and developers must consider the limitations of human vigilance. Otherwise, performance errors and consequent safety risks are likely to occur as a function of time on task, when human drivers passively monitor the activities of semi-automated vehicles. Countermeasures to such risks should be developed based on extant research within the vigilance domain. Toward that end, future research is needed to better understand the factors that influence the vigilance decrement during automated driving so that risks can be minimized and safety can be maximized.