Eric S. Missimer

Blink and wink detection for mouse pointer control

A Human-Computer Interaction (HCI) system that is designed for individuals with severe disabilities to simulate control of a traditional computer mouse is introduced. The camera-based system monitors a users eyes and allows the user to simulate clicking the mouse using voluntary blinks and winks. For users who can control head movements and can wink with one eye while keeping their other eye visibly open, the system allows complete use of a typical mouse, including moving the pointer, left and right clicking, double clicking, and click-and-dragging. For users who cannot wink but can blink voluntarily the system allows the user to perform left clicks, the most common and useful mouse action. The system does not require any training data to distinguish open eyes versus closed eyes. Eye classification is accomplished online during real-time interactions. The system had an accuracy of 8027/8306 = 96.6% in classifying sub-images with open or closed eyes and successfully allows the users to simulate a traditional computer mouse.

Customizable keyboard

Customizable Keyboard is an on-screen keyboard designed to be flexible and expandable. Instead of giving the user a keyboard layout Customizable Keyboard allows the user to create a layout that is accommodating to the users needs. Customizable Keyboard also allows the user to select from a variety of ways to interact with the keyboard including but not limited to using the mouse pointer to select keys and different types of scan based systems. Customizable Keyboard provides more functionality than a typical onscreen keyboard including the ability to control infrared devices such as TVs and send Twitter® Tweets.

Using kernels for a video-based mouse-replacement interface

Some people cannot use their hands to control a computer mouse due to conditions such as cerebral palsy or multiple sclerosis. For these individuals, there are various mouse-replacement solutions. One approach is to enable them to control the mouse pointer using head motions captured with a web camera. One such system, the Camera Mouse, uses an optical flow approach to track a manually-selected small patch of the subject’s face, such as the nostril or the edge of the eyebrow. The optical flow tracker may lose the facial feature when the tracked image patch drifts away from the initially-selected feature or when a user makes a rapid head movement. To address the problem of feature loss, we developed and incorporated the Kernel-Subset-Tracker into the Camera Mouse. The Kernel-Subset-Tracker is an exemplar-based method that uses a training set of representative images to produce online templates for positional tracking. We designed the augmented Camera Mouse so that it can compute these templates in real time, employing kernel techniques traditionally used for classification. We propose three versions of the Kernel-Subset-Tracker, each using a different kernel, and compared their performance to the optical-flow tracker under five different experimental conditions. Our experiments with test subjects show that augmenting the Camera Mouse with the Kernel-Subset-Tracker improves communication bandwidth statistically significantly. Tracking of facial features was accurate, without feature drift, even during rapid head movements and extreme head orientations. We conclude by describing how the Camera Mouse augmented with the Kernel-Subset-Tracker enabled a stroke-victim with severe motion impairments to communicate via an on-screen keyboard.

Mixed-criticality scheduling with I/O

This paper addresses the problem of scheduling tasks with differentcriticality levels in the presence of I/O requests. In mixed-criticalityscheduling, higher criticality tasks are given precedence over those of lowercriticality when it is impossible to guarantee the schedulability of alltasks. While mixed-criticality scheduling has gained attention in recentyears, most approaches typically assume a periodic task model. Thisassumption does not always hold in practice, especially for real-time andembedded systems that perform I/O. In prior work, we developed ascheduling technique in the Quest real-time operating system, which integratesthe time-budgeted management of I/O operations with Sporadic Server schedulingof tasks. This paper extends our previous scheduling approach with support formixed-criticality tasks and I/O requests on the same processing core. Resultsshow that in a real implementation the mixed-criticality scheduling methodintroduced in this paper outperforms a scheduling approach consisting of onlySporadic Servers.

A Virtualized Separation Kernel for Mixed-Criticality Systems

Multi- and many-core processors are becoming increasingly popular in embedded systems. Many of these processors now feature hardware virtualization capabilities, as found on the ARM Cortex A15 and x86 architectures with Intel VT-x or AMD-V support. Hardware virtualization provides a way to partition physical resources, including processor cores, memory, and I/O devices, among guest virtual machines (VMs). Each VM is then able to host tasks of a specific criticality level, as part of a mixed-criticality system with different timing and safety requirements. However, traditional virtual machine systems are inappropriate for mixed-criticality computing. They use hypervisors to schedule separate VMs on physical processor cores. The costs of trapping into hypervisors to multiplex and manage machine physical resources on behalf of separate guests are too expensive for many time-critical tasks. Additionally, traditional hypervisors have memory footprints that are often too large for many embedded computing systems. In this article, we discuss the design of the Quest-V separation kernel, which partitions services of different criticality levels across separate VMs, or sandboxes. Each sandbox encapsulates a subset of machine physical resources that it manages without requiring intervention from a hypervisor. In Quest-V, a hypervisor is only needed to bootstrap the system, recover from certain faults, and establish communication channels between sandboxes. This not only reduces the memory footprint of the most privileged protection domain but also removes it from the control path during normal system operation, thereby heightening security.

Adaptive mouse-replacement interface control functions for users with disabilities

We discuss experiences employing a video-based mouse-replacement interface system, the Camera Mouse, at care facilities for individuals with severe motion impairments and propose adaptations of the system. Traditional approaches to assistive technology are often inflexible, requiring users to adapt their limited motions to the requirements of the system. Such systems may have static or difficult-to-change configurations that make it challenging for multiple users to share the same system or for users whose motion abilities slowly degenerate. As users fatigue, they may experience more limited motion ability or additional unintended motions. To address these challenges, we propose adaptive mouse-control functions to be used in our mouse-replacement system. These functions can be changed to adapt the technology to the needs of the user, rather than making the user adapt to the technology. We present observations of an individual with severe cerebral palsy using our system.

Adaptive mappings for mouse-replacement interfaces

Users of mouse-replacement interfaces may have difficulty conforming to the motion requirements of their interface system. We have observed users with severe motor disabilities who controlled the mouse pointer with a head tracking interface. Our analysis shows that some users may be able to move in some directions easier than other directions. We propose several mouse pointer mappings that adapt to the users movement abilities. These mappings will take into account the users motions in two-or three-dimensions to move the mouse pointer in the intended direction.

Predictable Communication and Migration in the Quest-V Separation Kernel

Quest-V is a separation kernel, which partitions a system into a collection of sandboxes. Each sandbox encapsulates one or more processing cores, a region of machine physical memory, and a subset of I/O devices. Quest-V behaves like a distributed system on a chip, using explicit communication channels to exchange data and migrate addresses spaces between sandboxes, which operate like traditional hosts. This design has benefits in safety-critical systems, which require continued availability in the presence of failures. Additionally, online faults can be recovered without rebooting an entire system. However, the programming model for such a system is more complicated. Each sandbox has its own local scheduler, and threads must communicate using message passing with those in remote sandboxes. Similarly, address spaces may need to be migrated between sandboxes, to ensure newly forked processes do not violate the feasibility of existing local task schedules. Migration may also be needed to move a thread closer to its required resources, such as I/O devices that are not directly available in the local sandbox. This paper describes how Quest-V performs real-time communication and migration without violating service guarantees for existing threads.

Real-time USB communication in the Quest operating system

This paper describes a real-time USB 2 subsystem for the Quest operating system. Quest is designed for real-time embedded systems. Such systems need to interact with their environment using sensors and actuators. On many embedded platforms today there is support for basic serial, USB 2.0 and 100 Mbps Ethernet. Of these, USB 2.0 supports the highest throughput, while also supporting real-time communication. We show how the Quest USB 2.0 sub-system improves upon some of the deficiencies in USB software stacks in systems such as Linux through experimental evaluation. We demonstrate that the Quest USB sub-system is capable of predictable bandwidth allocation and increased overall performance. By dynamically reordering transaction requests, Quests USB sub-system is able to avoid unnecessary admission control rejections. Additionally, we are able to provide real-time guarantees for asynchronous USB transactions such as bulk transfers, which are typically treated in a best-effort manner. Real-time guarantees for bulk transactions are necessary for any system interacting with devices that implement bulk endpoints such as in a real-time file system. The paper also introduces an algorithm for USB scheduling that accepts more requests and provides bulk transfer guarantees, for cases where Linux fails.

An information theoretic mouse trajectory measure

In this paper, we propose the Relative Trajectory Information (RTI) measure, an information theoretic measure to evaluate mouse pointer trajectories. The measure is used to score the level of smoothness of mouse pointer trajectories. We show that, by leveraging Gaussian processes and information theory, RTI accounts for relative differences in timestamps of the mouse pointer trajectories. RTI also does not require explicit descriptions of targets, in either their location or size. Our experimental analysis shows how RTI can capture the motion signature of a user with severe motion disabilities and distinguish it from the motion signature of smooth trajectories obtained in a control experiment.