Brian J. Melloy

Using virtual reality technology for aircraft visual inspection training: presence and comparison studies

The aircraft maintenance industry is a complex system consisting of several interrelated human and machine components. Recognizing this, the Federal Aviation Administration (FAA) has pursued human factors related research. In the maintenance arena the research has focused on the aircraft inspection process and the aircraft inspector. Training has been identified as the primary intervention strategy to improve the quality and reliability of aircraft inspection. If training is to be successful, it is critical that we provide aircraft inspectors with appropriate training tools and environment. In response to this need, the paper outlines the development of a virtual reality (VR) system for aircraft inspection training. VR has generated much excitement but little formal proof that it is useful. However, since VR interfaces are difficult and expensive to build, the computer graphics community needs to be able to predict which applications will benefit from VR. To address this important issue, this research measured the degree of immersion and presence felt by subjects in a virtual environment simulator. Specifically, it conducted two controlled studies using the VR system developed for visual inspection task of an aft-cargo bay at the VR Lab of Clemson University. Beyond assembling the visual inspection virtual environment, a significant goal of this project was to explore subjective presence as it affects task performance. The results of this study indicated that the system scored high on the issues related to the degree of presence felt by the subjects. As a next logical step, this study, then, compared VR to an existing PC-based aircraft inspection simulator. The results showed that the VR system was better and preferred over the PC-based training tool.

3-D eye movement analysis.

This paper presents a novel three-dimensional (3-D) eye movement analysis algorithm for binocular eye tracking within virtualreality (VR). The user’s gaze direction, head position, and orientation are tracked in order to allow recording of the user’s fixations within the environment. Although the linear signal analysis approach is itself not new, its application to eye movement analysis in three dimensions advances traditional two-dimensional approaches, since it takes into account the six degrees of freedom of head movements and is resolution independent. Results indicate that the 3-D eye movement analysis algorithm can successfully be used for analysis of visual process measures in VR. Process measures not only can corroborate performance measures, but also can lead to discoveries of the reasons for performance improvements. In particular, analysis of users’ eye movements in VR can potentially lead to further insights into the underlying cognitive processes of VR subjects.

Binocular eye tracking in virtual reality for inspection training

This paper describes the development of a binocular eye tracking Virtual Reality system for aircraft inspection training. The aesthetic appearance of the environment is driven by standard graphical techniques augmented by realistic texture maps of the physical environment. A “virtual flashlight” is provided to simulate a tool used by inspectors. The users gaze direction, as well as head position and orientation, are tracked to allow recording of the users gaze locations within the environment. These gaze locations, or scanpaths, are calculated as gaze/polygon intersections, enabling comparison of fixated points with stored locations of artificially generated defects located in the environment interior. Recorded scanpaths provide a means of comparison of the performance of experts to novices, thereby gauging the effects of training.

Binocular eye tracking in VR for visual inspection training

This paper presents novel software techniques for binocular eye tracking within Virtual Reality and discusses their application to aircraft inspection training. The aesthetic appearance of the environment is driven by standard graphical techniques augmented by realistic texture maps of the physical environment. The users gaze direction, as well as head position and orientation, are tracked to allow recording of the users fixations within the environment. Methods are given for (1) integration of the eye tracker into a Virtual Reality framework, (2) stereo calculation of the users 3D gaze vector, (3) a new 3D calibration technique developed to estimate the users inter-pupillary distance post-facto, and (4) a new technique for eye movement analysis in 3-space. The 3D eye movement analysis technique is an improvement over traditional 2D approaches since it takes into account the 6 degrees of freedom of head movements and is resolution independent. Results indicate that although the current signal analysis approach is somewhat noisy and tends to underestimate the identified number of fixations, recorded eye movements provide valuable human factors process measures complementing performance statistics used to gauge training effectiveness.

3D eye movement analysis for VR visual inspection training

This paper presents an improved 3D eye movement analysis algorithm for binocular eye tracking within Virtual Reality for visual inspection training. The users gaze direction, head position and orientation are tracked to allow recording of the users fixations within the environment. The paper summarizes methods for (1) integrating the eye tracker into a Virtual Reality framework, (2) calculating the users 3D gaze vector, and (3) calibrating the software to estimate the users inter-pupillary distance post-facto. New techniques are presented for eye movement analysis in 3D for improved signal noise suppression. The paper describes (1) the use of Finite Impulse Response (FIR) filters for eye movement analysis, (2) the utility of adaptive thresholding and fixation grouping, and (3) a heuristic method to recover lost eye movement data due to miscalibration. While the linear signal analysis approach is itself not new, its application to eye movement analysis in three dimensions advances traditional 2D approaches since it takes into account the 6 degrees of freedom of head movements and is resolution independent. Results indicate improved noise suppression over our previous signal analysis approach.

Evaluation of different training strategies to improve decision-making performance in inspection

Various studies in inspection have demonstrated the usefulness of feedforward (FF) and feedback (FB) in improving the performance of inspectors. However, the results of these studies are not conclusive as to their effect on the sensitivity and response strategy of inspectors. Hence, this study evaluates the individual and collective effect of FF and FB on the sensitivity and response strategy of the inspectors using a computer simulation of a contact lens inspection task. Twenty-four subjects, randomly assigned to various conditions, performed an inspection task wherein the FF and FB conditions were manipulated between the subjects. In addition, the defect probability and the discriminability were manipulated within the subjects. Subsequently, the results were interpreted using signal detection theory. Although feedback improved the sensitivity, a finding consistent with earlier studies, feedforward was more effective in assisting the inspectors in setting their response strategy closer to the optimum. Moreover, providing both FF and FB resulted in improved sensitivity as well as a more optimal response strategy.

A comparison of three levels of training designed to promote systematic search behavior in visual inspection

Abstract Three levels of training, intended to improve visual search proficiency during inspection by promoting systematic search, are investigated. The three levels of training are verbal instruction on systematic search, verbal instruction with a static diagram of a systematic search, and verbal instruction and a diagram with practice using a dynamic visual stimulus tracing a systematic search. The levels are compared using both performance and process measures in order to identify the least complex means of improving search behavior. While all the training levels improved visual search proficiency, none did so more than any other. Evidence suggests that the levels were not differentiated due to the search task employed—a task that inherently lent itself to systematic search. Thus the least complex level is preferred for this task and those with similar attributes. Relevance to industry Human visual inspection is necessary to many industrial processes. Verbal instructions on how to perform a systematic search can be an effective method of training to lower inspection time under the conditions used in this study.

The effects of per-lot and per-item pacing on inspection performance

Abstract Inspection is a necessary activity in manufacturing to ensure process and quality control. Since some type of pacing is generally incorporated into the manufacturing setting, discovering how the speed and rigidity of pacing affect inspection accuracy is essential. In the context of this study, the level of rigidity is defined by either a maximum inspection time per item or a maximum time per lot of items (equivalent to the product of the number of items and the maximum per-item time). This study was conducted using 10 subjects on a simulated visual inspection task under varying degrees of pacing speed and rigidity. Although pacing speed was found to affect accuracy, rigidity did not. Overall, per-item inspection was found to be the most applicable to the majority of industrial inspection tasks. Relevance to industry The research conducted here is applicable to inspection tasks in manufacturing for both paced and unpaced inspection. The results obtained reveal the relative effectiveness of per-item and per-lot pacing on inspection quality. These results can be used by practitioners to enhance inspection performance and ultimately inspection quality.

Using Virtual Reality Technology to Improve Aircraft Inspection Performance: Presence and Performance Measurement Studies

Research in aircraft inspection and maintenance has revealed the criticality of human inspection performance in improving aviation safety. If we are to provide the general public with a safe and reliable air transportation system, inspection must be performed effectively, efficiently and consistently. Even though it is difficult to eliminate errors completely, continuing emphasis must be placed on identifying interventions to reduce errors and improve consistency in performance. Training has been identified as the primary intervention strategy in improving the quality and reliability of aircraft inspection performance. If training is to be successful, it is clear that we need to provide aircraft inspectors with tools to help enhance their inspection skills and improve performance. In response to this need a Virtual Reality (VR) based simulator was developed for visual inspection task of an aft cargo bay. Presence and performance validation studies were conducted to evaluate the simulator and are described as part of this paper.

On spatiochromatic visual sensitivity and peripheral color LOD management

Empirical findings from a gaze-contingent color degradation study report the effects of artificial reduction of the human visual systems sensitivity to peripheral chromaticity on visual search performance. To our knowledge, this is the first such investigation of peripheral color reduction. For unimpeded performance, results suggest that, unlike spatiotemporal content, peripheral chromaticity cannot be reduced within the central 20° visual angle. Somewhat analogous to dark adaptation, reduction of peripheral color tends to simulate scotopic viewing conditions. This holds significant implications for chromatic Level Of Detail management. Specifically, while peripheral spatiotemporal detail can be attenuated without affecting visual search, often dramatically (e.g., spatial detail can be so reduced up to 50% at about 5°), peripheral chromatic reduction is likely to be noticed much sooner. Therefore, color LOD reduction (e.g., via compression), should be maintained isotropically across the central 20° visual field.