Sven Schmerwitz

Visual-conformal display format for helicopter guidance

Helicopter guidance in situations where natural vision is reduced is still a challenging task. Beside new available sensors, which are able to “see” through darkness, fog and dust, display technology remains one of the key issues of pilot assistance systems. As long as we have pilots within aircraft cockpits, we have to keep them informed about the outside situation. “Situational awareness” of humans is mainly powered by their visual channel. Therefore, display systems which are able to cross-fade seamless from natural vision to artificial computer vision and vice versa, are of greatest interest within this context. Helmet-mounted displays (HMD) have this property when they apply a head-tracker for measuring the pilot’s head orientation relative to the aircraft reference frame. Together with the aircraft’s position and orientation relative to the world’s reference frame, the on-board graphics computer can generate images which are perfectly aligned with the outside world. We call image elements which match the outside world, “visual-conformal”. Published display formats for helicopter guidance in degraded visual environment apply mostly 2D-symbologies which stay far behind from what is possible. We propose a perspective 3D-symbology for a head-tracked HMD which shows as much as possible visual-conformal elements. We implemented and tested our proposal within our fixed based cockpit simulator as well as in our flying helicopter simulator (FHS). Recently conducted simulation trials with experienced helicopter pilots give some first evaluation results of our proposal.

A concept for a virtual flight deck shown on an HMD

A combination of see-through head-worn or helmet-mounted displays (HMDs) and imaging sensors is frequently used to overcome the limitations of the human visual system in degraded visual environments (DVE). A visual-conformal symbology displayed on the HMD allows the pilots to see objects such as the landing site or obstacles being invisible otherwise. These HMDs are worn by pilots sitting in a conventional cockpit, which provides a direct view of the external scene through the cockpit windows and a user interface with head-down displays and buttons. In a previous publication, we presented the advantages of replacing the conventional head-down display hardware by virtual instruments. These virtual aircraft-fixed cockpit instruments were displayed on the Elbit JEDEYE system, a binocular, see-through HMD. The idea of our current work is to not only virtualize the display hardware of the flight deck, but also to replace the direct view of the out-the-window scene by a virtual view of the surroundings. This imagery is derived from various sensors and rendered on an HMD, however without see-through capability. This approach promises many advantages over conventional cockpit designs. Besides potential weight savings, this future flight deck can provide a less restricted outside view as the pilots are able to virtually see through the airframe. The paper presents a concept for the realization of such a virtual flight deck and states the expected benefits as well as the challenges to be met.

Developing an obstacle display for helicopter brownout situations

Project ALLFlight is DLRs initiative to diminish the problem of piloting helicopters in degraded visual conditions. The problem arises whenever dust or snow is stirred up during landing (brownout/whiteout), eectively blocking the crews vision of the landing site. A possible solution comprises the use of sensors that are able to look through the dust cloud. As part of the project display symbologies are being developed to enable the pilot to make use of the rather abstract and noisy sensor data. In a rst stage sensor data from very dierent sensors is fused. This step contains a classication of points into ground points and obstacle points. In a second step the result is augmented with ground data bases and depicted in a synthetic head-down display. Regarding the design, several variations in symbology are considered, including variations in color coding, continuous or non-continuous terrain displays and dierent obstacle representations. In this paper we present the basic techniques used for obstacle and ground separation. We choose a set of possibilities for the pilot display and detail the implementation. Furthermore, we present a pilot study, including human factors assessment with focus on usability and pilot acceptance.

Design of a pathway display for a retinal scanning HMD

During approach and landing the pilot performs a high-workload task of switching the attention between instrument in-formation and the outside scene. Superimposing both visual domains in head-up (HUD) or head-mounted displays (HMD) reduces the visual scanning load of this task. These displays are collimated at optical infinity; therefore, prevent the pilots eye from permanent accommodation between both visual domains. Besides these performance benefits, visual clutter and attention fixation, i.e. inattentiveness to outside scene events while attending on HUD symbologies, are found to be performance cost factors. Conformal symbology and flight-phase adapted de-cluttering has been found to be prom-ising approaches to overcome these problems. In pursuit of these two approaches, the current paper describes the design of a new pathway display on a monocular head-mounted retinal scanning display and its implementation in DLRs generic cockpit simulator. The pathway can be regarded as a means of linking an instrument symbology (the tunnel) with a virtual element of the outside scene (the in-tended flight path). Scene-linked symbology appear to be part of the outside world, e.g. an instrument reading like air-speed, heading, or altitude that is changing its display location conformal with the gate element of the tunnel symbology moving towards the pilot. Examples of flight phase-adaptive de-cluttering is to successively reduce or remove symbol-ogy when the conformal outside element becomes visible (e.g. the runway). In addition the display includes a conformal presentation of the terrain. A checker board pattern representing the terrain is dynamically generated from worldwide available SRTM-3 data.

Conformal Displays: Human Factors Analysis of Innovative Landing Aids

Abstract. In the past couple of years, research on display content for helicopter operations headed in a new direction. The already reached goals could evolve into a paradigm change for information visualization. Technology advancements allow implementing three-dimensional and conformal content on a helmet-mounted see-through device. This superimposed imagery inherits the same optical flow as the environment. It is supposed to ease switching between display information and environmental cues. The concept is neither pathbreaking nor new, but it has not been successfully established in aviation yet. Nevertheless, there are certainly some advantages to expect—at least from perspective of a human-centered system design. Within the following pages, the next generation displays will be presented and discussed with a focus on human factors. Beginning with recalling some human factor related research facts, an experiment comparing the former two-dimensional research displays will be presented. Before introducing the DLR conformal symbol set and the three experiments about an innovative drift, indication related research activities toward conformal symbol sets will be addressed.

Helmet mounted display supporting helicopter missions during en route flight and landing

Degraded visual environment is still a major problem for helicopter pilots especially during approach and landing. Particularly with regard to the landing phase, pilot’s eyes must be directed outward in order to find visual cues as indicators for drift estimation. If lateral speed exceeds the limits it can damage the airframe or in extreme cases lead to a rollover. Since poor visibility can contribute to a loss of situation awareness and spatial disorientation, it is crucial to intuitively provide the pilot with the essential visual information he needs for a safe landing. With continuous technology advancement helmet-mounted displays (HMD) will soon become a spreading technology, because look through capability is an enabler to offer monitoring the outside view while presenting flight phase depending symbologies on the helmet display. Besides presenting primary flight information, additional information for obstacle accentuation or terrain visualization can be displayed on the visor. Virtual conformal elements like 3D pathway depiction or a 3D landing zone representation can help the pilot to maintain control until touchdown even during poor visual conditions. This paper describes first investigations in terms of both en route and landing symbology presented on a helmet mounted display system in the scope of helicopter flight trials with DLR’s flying helicopter simulator ACT/FHS.

Integration of a 3D perspective view in the navigation display: featuring pilot's mental model

Synthetic vision systems (SVS) appear as spreading technology in the avionic domain. Several studies prove enhanced situational awareness when using synthetic vision. Since the introduction of synthetic vision a steady change and evolution started concerning the primary flight display (PFD) and the navigation display (ND). The main improvements of the ND comprise the representation of colored ground proximity warning systems (EGPWS), weather radar, and TCAS information. Synthetic vision seems to offer high potential to further enhance cockpit display systems. Especially, concerning the current trend having a 3D perspective view in a SVS-PFD while leaving the navigational content as well as methods of interaction unchanged the question arouses if and how the gap between both displays might evolve to a serious problem. This issue becomes important in relation to the transition and combination of strategic and tactical flight guidance. Hence, pros and cons of 2D and 3D views generally as well as the gap between the egocentric perspective 3D view of the PFD and the exocentric 2D top and side view of the ND will be discussed. Further a concept for the integration of a 3D perspective view, i.e., bird’s eye view, in synthetic vision ND will be presented. The combination of 2D and 3D views in the ND enables a better correlation of the ND and the PFD. Additionally, this supports the building of pilot’s mental model. The authors believe it will improve the situational and spatial awareness. It might prove to further raise the safety margin when operating in mountainous areas.

An evaluation environment for a helmet-mounted synthetic degraded visual environment display

Degraded visual environment (DVE) is a term coined for environmental conditions that impair the visual orientation of a helicopter pilot during flight or landing. These conditions include brown-out, but also night, glare, fog and mist, as well as combinations of those. In order to assist pilots under DVE conditions DLR, the German Aerospace Center, initiated project ALLFlight. Using a combination of multi-sensor fusion, specialized symbologies for head-down and helmet-mounted display, and database augmentation the pilot should be enabled to safely operate the helicopter. Based on an earlier implementation of a synthetic head-down display an implementation for a head-tracked, helmet-mounted system is implemented. Since color is presently not an option for head-up displays, alternative symbologies have to be considered, including variations in transparency, density of terrain displays and size and shape variations for obstacle representation. In this paper we present a simulator setup for evaluating the display concepts developed. This includes various simulation stages for visual and sensor input and a flexible simulator cockpit for testing variations of display concepts. We detail the implementation architecture and present first evaluation details from a recent pilot study.

Millimeter-wave data acquisition for terrain mapping, obstacle detection, and dust penetrating capability testing

The DLR project ALLFlight (Assisted Low Level Flight and Landing on Unprepared Landing Sites) is devoted to demonstrating and evaluating the characteristics of sensors for helicopter operations in degraded visual environments. Millimeter wave radar is one of the many sensors considered for use in brown-out. It delivers a lower angular resolution compared to other sensors, however it may provide the best dust penetration capabilities. In cooperation with the NRC, flight tests on a Bell 205 were conducted to gather sensor data from a 35 GHz pencil beam radar for terrain mapping, obstacle detection and dust penetration. In this paper preliminary results from the flight trials at NRC are presented and a description of the radars general capability is shown. Furthermore, insight is provided into the concept of multi-sensor fusion as attempted in the ALLFlight project.

Evaluation of a “Stereo” Radar approach for terrain reconstruction using synthetic data

Enhanced vision systems (EVS) are a possibility to improve the situation awareness of an aircrew during poor visibility conditions. EVS are based on sensor data which might be difficult to interpret especially for radar data. Fast scanning radar systems in the millimeter waveband (35 or 94 GHz) are commonly unable to measure the elevation of a target. Nevertheless these elevation data can sometimes be reconstructed from a series of images or from images taken from different viewpoints or bank angles. In case of forward looking millimeter wave radar it is more promising to use different bank angles. The authors have detailed these ideas using the term ldquoStereo Radarrdquo in previous publications. In this paper we take a closer look at the accuracy and the resolution of the algorithm. For this a series of experiments using synthetic data is performed. Furthermore we show the influence of size, elevation, shape, and different (radar) textures of a target on the reconstructed elevation. Finally some tests are carried out to demonstrate the robustness against different kinds of noise.