Jonathan Andersh

Autonomous altitude estimation of a UAV using a single onboard camera

Autonomous estimation of the altitude of an Unmanned Aerial Vehicle (UAV) is extremely important when dealing with flight maneuvers like landing, steady flight, etc. Vision based techniques for solving this problem have been underutilized. In this paper, we propose a new algorithm to estimate the altitude of a UAV from top-down aerial images taken from a single on-board camera. We use a semi-supervised machine learning approach to solve the problem. The basic idea of our technique is to learn the mapping between the texture information contained in an image to a possible altitude value. We learn an over complete sparse basis set from a corpus of unlabeled images capturing the texture variations. This is followed by regression of this basis set against a training set of altitudes. Finally, a spatio-temporal Markov Random Field is modeled over the altitudes in test images, which is maximized over the posterior distribution using the MAP estimate by solving a quadratic optimization problem with L1 regularity constraints. The method is evaluated in a laboratory setting with a real helicopter and is found to provide promising results with sufficiently fast turnaround time.

Research Infrastructure for Interactive Human- and Autonomous Guidance

This paper describes a research infrastructure setup to exercise and investigate guidance and control capabilities under human and autonomous control modalities. The lab facility is designed to implement tasks that emphasize agent-environment interactions. The overall goal is to characterize these interactions and to apply the gained knowledge to determine interaction models. These can then be used to design guidance and control algorithms as well as human–machine systems. The facility uses miniature rotorcraft as test vehicles with a Vicon motion tracking system and SensoMotoric gaze tracking system. The facility also includes a high-fidelity simulation system to support larger scale autonomy and teleoperation experiments. The simulation incorporates the software components and models of the key flight hardware and sensors. The software system was integrated around the Robotics Operating System (ROS) to support the heterogenous processes and data and allow easy system reconfiguration. The paper describes the research objectives, details of the hardware and software components and their integration, and concludes with a summary of the ongoing research enabled by the lab facility including.

A First Investigation into the Teleoperation of a Miniature Rotorcraft

This paper presents preliminary results for research on the development of a systematic approach for the analysis and design of teleoperation systems for miniature rotorcrafts. Through teleoperation it is possible to take advantage of the human cognitive abilities and control skills when performing precision flight tasks remotely. Miniature rotorcrafts provide unique capabilities for operation indoors or near buildings and infrastructure. However, they also present some unique challenges due to their limited payload and challenging dynamics. Successful aerial teleoperation will depend on our ability to implement control augmentations, and in parallel, relay the necessary cues to the operator to perform the task. To optimize the design of such a system, we need to be able to quantify the operator workload and task performance and have a systematic way to use those metrics to determine the effectiveness of the control augmentation and operator cueing being tested. The paper introduces the key aerial teleoperation challenges and uses an example flight task to illustrate our design and analysis framework.

Experimental investigation of teleoperation performance for miniature rotorcraft

This paper identifies a baseline level of performance that will be used for assessing teleoperation systems for miniature rotorcraft. The experimental results show the performance measured while operating a miniature rotorcraft in a typical RC mode (the pilot directly observes the helicopter) as well as a simple teleoperation mode (the pilot operates the vehicle using video from an onboard camera). The importance of this work is to create a performance benchmark that can be used as a reference point when testing more advanced teleoperation systems. The paper will give a brief overview of the experimental architecture, define the performance metrics being used, and describe the flight tasks being investigated. The performance metrics calculated from experimental data will then be presented along with some conclusions about the findings. This paper is the beginning of a process to systematically investigate different visual cues and control augmentations that can simplify the teleoperation task for an operator. The goal of this research is to gain a detailed understanding of how the human pilots performance during teleoperation is affected by the system configuration and to identify the bounds on the performance.

Classification of Human Gaze in Spatial Guidance and Control

Visual gaze movement is one of the main processes in human interaction with the physical environment, and is a manifestation of the visual perception process. In humans, the interaction of gaze with environment involves mainly three patterns including fixation, smooth pursuit and saccade. This paper investigates two classification techniques to identify gaze patterns: an empirical threshold method and a Hidden Markov Model (HMM). A registration process was initially performed to transform the gaze and environmental visual cues into a common coordinate system, which provides an additional feature to construct gaze models. With the gaze models, the HMM method accurately identifies gaze patterns in gaze trajectories, and in addition enables three types of applications. One is the detailed analysis of visuo-motor control behavior. The second is gaze-mediated teleoperation, showing the real-time capability of the gaze classification method. The third is the application to surgery tasks, enabling surgical skill analysis.

A vision based ensemble approach to velocity estimation for miniature rotorcraft

Successful operation of a miniature rotorcraft relies on capabilities including automated guidance, trajectory following, and teleoperation; all of which require accurate estimates of the vehicle’s body velocities and Euler angles. For larger rotorcraft that operate outdoors, the traditional approach is to combine a highly accurate IMU with GPS measurements. However, for small scale rotorcraft that operate indoors, lower quality MEMS IMUs are used because of limited payload. In indoor applications GPS is usually not available, and state estimates based on IMU measurements drift over time. In this paper, we propose a novel framework for state estimation that combines a dynamic flight model, IMU measurements, and 3D velocity estimates computed from an onboard monocular camera using computer vision. Our work differs from existing approaches in that, rather than using a single vision algorithm to update the vehicle’s state, we capitalize on the strengths of multiple vision algorithms by integrating them into a meta-algorithm for 3D motion estimation. Experiments are conducted on two real helicopter platforms in a laboratory environment for different motion types to demonstrate and evaluate the effectiveness of our approach.

Modeling visuo-motor control and guidance functions in remote-control operation

A large class of human movements rely on the so-called hand-eye coordination for precise and versatile performance. Teleoperation of agile robotic systems in three dimensional environments would benefit from a detailed understanding of the perceptual control mechanisms used by the operator both for the design of operator interfaces and potentially for the use of gaze information as part of the control mechanism. The objective of this work is to model the role and contribution of the operators gaze motion in remote control operation of an agile vehicle. The experiments were conducted using a miniature remote controlled helicopter. The overall human-machine system is described using a multi-loop manual control model. Experiments were designed and conducted to exercise different aspects of this control hierarchy, encompassing stabilization and regulation as well as trajectory tracking and goal directed guidance. The sensing requirements for each loop are established by investigating the relationship between the operators visual gaze trajectories, the vehicle trajectories, and the control actions. Visual gaze data is classified according to the typical smooth pursuit, saccades and fixations and then incorporated into an estimation strategy.

Modeling the Human Visuo-Motor System to Support Remote-Control Operation

The working hypothesis in this project is that gaze interactions play a central role in structuring the joint control and guidance strategy of the human operator performing spatial tasks. Perceptual guidance and control is the idea that the visual and motor systems form a unified perceptuo-motor system where necessary information is naturally extracted by the visual system. As a consequence, the response of this system is constrained by the visual and motor mechanisms and these effects should manifest in the behavioral data. Modeling the perceptual processes of the human operator provides the foundation necessary for a systems-based approach to the design of control and display systems used by remotely operated vehicles. This paper investigates this hypothesis using flight tasks conducted with remotely controlled miniature rotorcraft, taking place in indoor settings that provide rich environments to investigate the key processes supporting spatial interactions. This work also applies to spatial control tasks in a range of application domains that include tele-operation, gaming, and virtual reality. The human-in-the-loop system combines the dynamics of the vehicle, environment, and human perception–action with the response of the overall system emerging from the interplay of perception and action. The main questions to be answered in this work are as follows: (i) what is the general control and guidance strategy of the human operator, and (ii) how is information about the vehicle and environment extracted visually by the operator. The general approach uses gaze as the primary sensory mechanism by decoding the gaze patterns of the pilot to provide information for estimation, control, and guidance. This work differs from existing research by taking what have largely been conceptual ideas on action–perception and structuring them to be implemented for a real-world problem. The paper proposes a system model that captures the human pilot’s perception–action loop; the loop that delineates the main components of the pilot’s perceptuo-motor system, including estimation of the vehicle state and task elements based on operator gaze patterns, trajectory planning, and tracking control. The identified human visuo-motor model is then exploited to demonstrate how the perceptual and control functions system can be augmented to reduce the operator workload.

Fundamental control characteristics of curvilinear motion in human and automatic path tracking tasks

This paper describes fundamental path tracking control performance characteristics for curvilinear motion based on the nonlinear dynamics. Curvilinear motion is ubiquitous to many human behaviors and engineering applications. The results provide essential insights about vehicle control requirements, aerial and ground robots, as well as for understanding human guidance and control skills. The paper describes the theoretical analysis and illustrates the fundamental properties through simulations and follows with two examples based on empirical data. The first example is based on the design specifications for different alpine skiing disciplines. The second example is based on results from closed-loop identification of a miniature helicopter path tracking experiment under both human and automatic control. The results validate the path tracking adaptation strategy suggested by analysis and demonstrate the general significance of the results.