Johannes M. Ernst

A concept for a virtual flight deck shown on an HMD

A combination of see-through head-worn or helmet-mounted displays (HMDs) and imaging sensors is frequently used to overcome the limitations of the human visual system in degraded visual environments (DVE). A visual-conformal symbology displayed on the HMD allows the pilots to see objects such as the landing site or obstacles being invisible otherwise. These HMDs are worn by pilots sitting in a conventional cockpit, which provides a direct view of the external scene through the cockpit windows and a user interface with head-down displays and buttons. In a previous publication, we presented the advantages of replacing the conventional head-down display hardware by virtual instruments. These virtual aircraft-fixed cockpit instruments were displayed on the Elbit JEDEYE system, a binocular, see-through HMD. The idea of our current work is to not only virtualize the display hardware of the flight deck, but also to replace the direct view of the out-the-window scene by a virtual view of the surroundings. This imagery is derived from various sensors and rendered on an HMD, however without see-through capability. This approach promises many advantages over conventional cockpit designs. Besides potential weight savings, this future flight deck can provide a less restricted outside view as the pilots are able to virtually see through the airframe. The paper presents a concept for the realization of such a virtual flight deck and states the expected benefits as well as the challenges to be met.

Conformal Displays: Human Factors Analysis of Innovative Landing Aids

Abstract. In the past couple of years, research on display content for helicopter operations headed in a new direction. The already reached goals could evolve into a paradigm change for information visualization. Technology advancements allow implementing three-dimensional and conformal content on a helmet-mounted see-through device. This superimposed imagery inherits the same optical flow as the environment. It is supposed to ease switching between display information and environmental cues. The concept is neither pathbreaking nor new, but it has not been successfully established in aviation yet. Nevertheless, there are certainly some advantages to expect—at least from perspective of a human-centered system design. Within the following pages, the next generation displays will be presented and discussed with a focus on human factors. Beginning with recalling some human factor related research facts, an experiment comparing the former two-dimensional research displays will be presented. Before introducing the DLR conformal symbol set and the three experiments about an innovative drift, indication related research activities toward conformal symbol sets will be addressed.

Integration of an Exocentric Orthogonal Coplanar 360 Degree Top View in a Head Worn See-Through Display Supporting Obstacle Awareness for Helicopter Operations

The objective was the development of an HMI for helicopter obstacle awareness and warning systems in order to improve the situational and spatial awareness as well as the workload of helicopter pilots. The related work concerning obstacle awareness and warning systems, situational awareness, orthogonal coplanar and perspective representations plus previous work done by DLR was depicted and discussed. The two main aspects of the developed HMI concept were explained, i.e., the combination of the exocentric orthogonal coplanar top view with the egocentric perspective view, and secondly three ways for the integration of the obstacle awareness display inside a head-worn see-through display. The developed HMI concept was applied to two helicopter offshore operations and its specific obstacle situation. The first operation is a hoist operation at the lower access point of an offshore wind turbine. The second regards the landing operation on an offshore platform. From a technical point of view, especially concerning available sensor technologies, helicopter might be fitted with obstacle awareness systems in future. The HMI design is still under investigation in order to support the pilot in a holistic and balanced way.

360 Degree Top View inside a Helmet Mounted Display providing Obstacle Awareness for Helicopter Operations

This paper introduces a display concept for helicopter obstacle awareness and warning systems. The key feature of the concept is the integration of a 360-degree coplanar orthogonal top view in the egocentric perspective of a helmet mounted see-through display. The concept intends to provide obstacle awareness while pilots are looking outside. The concept should further improve the situational and spatial awareness as well as the workload of helicopter pilots when operating in challenging surroundings. The display concept is applied to two helicopter off-shore operations and its specific obstacle situation. The first operation represents a hoist operation at the lower access point of an off-shore wind turbine. The second regards an off-shore platform landing operation. The paper depicts the two use cases, related work concerning obstacle awareness and warning systems, and recapitulates situational awareness plus the properties of orthogonal coplanar in comparison to the properties of perspective representations. Thereafter the two main aspects of the developed HMI concept were presented, i.e., the combination of the exocentric orthogonal coplanar top view with the egocentric perspective view, and secondly three ways for the integration of the top view inside the helmet mounted display. The implemented HMI design represents work in progress, i.e., looking forward to develop an optimal holistic and balanced display concept featuring helicopter obstacle awareness and warning systems.

Synthetic vision on a head-worn display supporting helicopter offshore operations

Helicopters play an important role during construction and operation of o shore wind farms. Most of the time helicopter offshore operations are conducted over open water and often in degraded visual environment. Such scenarios provide very few usable visual cues for the crew to safely pilot the aircraft. For instance, no landmarks exist for navigation and orientation is hindered by weather phenomena that reduce visibility and obscure the horizon. To overcome this problem, we are developing an external vision system which uses a non-see-through, head-worn display (HWD) to show fused sensor and database information about the surroundings. This paper focuses on one aspect of our system: the computer-generated representation of relevant visual cues of the water surface. Our motivation is to develop a synthetic view of the surroundings that is superior to the real out-the-window view. The moving water surface does not provide fixed references for orientation and sometimes even produces wrong motion cues. Thus, we replace it by a more valuable, computer-generated clear view. Since pilots estimate wind direction and speed by checking the movement characteristics of the water surface, our synthetic display also integrates this information. This paper presents several options for a synthetic vision display supporting offshore operations. Further, it comprises results from simulator trials, where helicopter pilots performed final approaches and landings on an offshore platform supported by our display. The results will contribute to the advancement of our HWD-based virtual cockpit concept. Additionally, our findings may be relevant to conventional, head-down synthetic vision displays visualizing offshore environments.

Integrating Legacy ESVS Displays in the Unity Game Engine

Current augmented reality (AR) and virtual reality (VR) displays are targeted at entertainment and home education purposes. However, these headsets can be used for prototyping and developing display concepts to be used in aviation. In previous papers we have demonstrated the use of helmet mounted enhanced and synthetic vision systems (ESVS) displays that have been implemented on commercially available VR displays. One of the most widely used engines for developing VR and AR applications is the Unity game engine. While it supports a broad range of display hardware it can be challenging to integrate legacy ESVS software, since its main purpose is the fast development of virtual worlds. To avoid a complete re-write of such displays we demonstrate techniques to integrate legacy software in Unity. In detail, we show how render plugins or texture buffers can be used to display existing ESVS output in a Unity project. We show advantages and drawbacks of these different approaches. Further, we detail problems in case the source software is written for a different platform, for example, when integrating OpenGL displays in a DirectX environment. While the demonstrated techniques are implemented and tested with the Unity game engine, they can be used for other target game and render engines, too.

Review of colored conformal symbology in head-worn displays

The usage of conformal symbology in color head-worn displays (HWDs) opens up a range of new possibilities on modern flight decks. The capability of color augmentation seems especially useful for low flights in degraded visual environments. Helicopter flights in these conditions, including brownout by swirling dust or sand particles, can often lead to spatial disorientation (SD) and result in a significant amount of controlled flight into terrain (CFIT). While first generation color-capable conformal displays are deployed, practical guidelines for the use of color in these see-through interfaces are yet to be established. A literature survey is carried out to analyze available knowledge of color use in conformal displays and to identify established methodologies for human-factors experimentation in this domain. Firstly the key human factors involved in color HWDs are outlined, including hardware design aspects as well as perceptual and attentional aspects. Secondly research on color perception is mapped out, focusing on investigations of luminance contrast requirements, modeling of color space blending and development of color correction solutions. Thirdly application-based research of colored conformal symbology is reviewed, including several simulations and flight experiments. Analysis shows that established luminance contrast requirements need to be validated and that performance effects of colored HWD symbology need more objective measurements. Finally practical recommendations are made for further research. This literature study has thus established a theoretical framework for future experimental efforts in colored conformal symbology. The Institute of Flight Guidance of the German Aerospace Center (DLR) anticipates conducting experiments within this framework.

VR and AR Environments for Virtual Cockpit Enhancements

The recent evolution of cockpit design has moved from the established glass cockpits into new directions. Among them is the virtual enhancement of cockpits by augmented reality (AR) and virtual reality (VR) displays. Well known in aviation are helmet mounted see-through displays, but opaque VR displays are of increasing interest also. This technology enables the pilot to use virtual instrumentation as an add-on to the real cockpit. Even a totally virtualized instrumentation is possible. Furthermore, VR technology allows the fast prototyping and pilot training in cockpit environments that are still in development before even a single real instrument is built. We show how commercial off-the-shelf VR hardware can be used to build a prototyping environment. We demonstrate advantages and challenges when using software engines usually built for the games industry. We describe our own integration concept, which re-uses as much of our own software as possible and allows integration with minimal parallel development.

Designing a virtual cockpit for helicopter offshore operations

In recent years the number of offshore wind farms is rapidly increasing. Especially coastal European countries are building numerous offshore wind turbines in the Baltic, the North, and the Irish Sea. During both construction and operation of these wind farms, many specially-equipped helicopters are on duty. Due to their flexibility, their hover capability, and their higher speed compared to ships, these aircraft perform important tasks like helicopter emergency medical services (HEMS) as well as passenger and freight transfer flights. The missions often include specific challenges like platform landings or hoist operations to drop off workers onto wind turbines. However, adverse weather conditions frequently limit helicopter offshore operations. In such scenarios, the application of aircraft-mounted sensors and obstacle databases together with helmet-mounted displays (HMD) seems to offer great potential to improve the operational capabilities of the helicopters used. By displaying environmental information in a visual conformal manner, these systems mitigate the loss of visual reference to the surroundings. This helps the pilots to maintain proper situational awareness. This paper analyzes the specific challenges of helicopter offshore operations in wind parks by means of an online survey and a structured interview with pilots and operators. Further, the work presents how our previously introduced concept of an HMD-based virtual flight deck could enhance helicopter offshore missions. The advantages of this system – for instance its “see-through the airframe”-capability and its highly-flexible cockpit setup – enable us to design entirely novel pilot assistance systems. The gained knowledge will be used to develop a virtual cockpit that is tailor-made for helicopter offshore maneuvers