John J. Magee

A Human–Computer Interface Using Symmetry Between Eyes to Detect Gaze Direction

In the cases of paralysis so severe that a persons ability to control movement is limited to the muscles around the eyes, eye movements or blinks are the only way for the person to communicate. Interfaces that assist in such communication are often intrusive, require special hardware, or rely on active infrared illumination. A nonintrusive communication interface system called EyeKeys was therefore developed, which runs on a consumer-grade computer with video input from an inexpensive Universal Serial Bus camera and works without special lighting. The system detects and tracks the persons face using multiscale template correlation. The symmetry between left and right eyes is exploited to detect if the person is looking at the camera or to the left or right side. The detected eye direction can then be used to control applications such as spelling programs or games. The game ldquoBlockEscaperdquo was developed to evaluate the performance of EyeKeys and compare it to a mouse substitution interface. Experiments with EyeKeys have shown that it is an easily used computer input and control device for able-bodied people and has the potential to become a practical tool for people with severe paralysis.

Customizable keyboard

Customizable Keyboard is an on-screen keyboard designed to be flexible and expandable. Instead of giving the user a keyboard layout Customizable Keyboard allows the user to create a layout that is accommodating to the users needs. Customizable Keyboard also allows the user to select from a variety of ways to interact with the keyboard including but not limited to using the mouse pointer to select keys and different types of scan based systems. Customizable Keyboard provides more functionality than a typical onscreen keyboard including the ability to control infrared devices such as TVs and send Twitter® Tweets.

HAIL: hierarchical adaptive interface layout

We present a framework to adapt software to the needs of individuals with severe motion disabilities who use mouse substitution interfaces. Typically, users are required to adapt to the interfaces that they wish to use. We propose interfaces that change and adapt to the user and their individual abilities. The Hierarchical Adaptive Interface Layout (HAIL) model is a set of specifications for the design of user interface applications that adapt to the user. In HAIL applications, all of the interactive components take place on configurable toolbars along the edge of the screen. We show two HAIL-based applications: a general purpose web browser and a Twitter client.

Tracking, analysis, and recognition of human gestures in video

An overview of research in automated gesture spotting, tracking and recognition by the Image and Video Computing Group at Boston University is given. Approaches for localization and tracking human hands in video, estimation of hand shape and upper body pose; tracking head and facial motion, as well as efficient spotting and recognition of specific gestures in video streams are summarized. Methods for efficient dimensionality reduction of gesture time series, boosting of classifiers for nearest neighbor search in pose space, and model-based pruning of gesture alignment hypotheses are described. Algorithms are demonstrated in three domains: American sign language, hand signals like those employed by flight-directors on airport runways, and gesture-based interfaces for severely disabled users. The methods described are general and can be applied in other domains that require efficient detection and analysis of patterns in time-series, images or video.

Movement and Recovery Analysis of a Mouse-Replacement Interface for Users with Severe Disabilities

The Camera Mouse is a mouse-replacement interface for users with movement impairments. It tracks a selected body feature, such as the nose, eyebrow or finger, through a web camera and translates the users movements to movements of the mouse pointer. Occasionally, the Camera Mouse loses the feature being tracked, when the user moves quickly or out of frame, or when the feature is occluded from view of the web camera. A new system has been developed to recognize when the tracked feature has been lost and to locate and resume tracking of the originally selected feature. In order to better understand the directions of movement which are most and least comfortable for users with disabilities, a game interface was developed to test the accuracy and speed of users across different trajectories. The experiments revealed that trajectories most comfortable for a user with severe cerebral palsy were along diagonal axes.

Web mediators for accessible browsing

We present a highly accurate method for classifying web pages based on link percentage, which is the percentage of text characters that are parts of links normalized by the number of all text characters on a web page. We also present a novel link grouping algorithm using agglomerative hierarchical clustering that groups links in the same spatial neighborhood together while preserving link structure. Grouping allows users with severe disabilities to use a scan-based mechanism to tab through a web page and select items. In experiments, we saw up to a 40-fold reduction in the number of commands needed to click on a link with a scan-based interface. Our classification method consistently outperformed a baseline classifier even when using minimal data to generate article and index clusters, and achieved classification accuracy of 94.0% on web sites with well-formed or slightly malformed HTML, compared with 80.1% accuracy for the baseline classifier.

Fast head tilt detection for human-computer interaction

Accurate head tilt detection has a large potential to aid people with disabilities in the use of human-computer interfaces and provide universal access to communication software. We show how it can be utilized to tab through links on a web page or control a video game with head motions. It may also be useful as a correction method for currently available video-based assistive technology that requires upright facial poses. Few of the existing computer vision methods that detect head rotations in and out of the image plane with reasonable accuracy can operate within the context of a real-time communication interface because the computational expense that they incur is too great. Our method uses a variety of metrics to obtain a robust head tilt estimate without incurring the computational cost of previous methods. Our system runs in real time on a computer with a 2.53 GHz processor, 256 MB of RAM and an inexpensive webcam, using only 55% of the processor cycles.

Camera Mouse + ClickerAID: Dwell vs. Single-Muscle Click Actuation in Mouse-Replacement Interfaces

Point-and-click interface modalities are a pervasive method of interacting with graphical user interfaces. Users of mouse-replacement interfaces use alternative input devices to replace the mouse for pointing and clicking. We present a comparison of click actuation modalities with users of the Camera Mouse, a motion-tracking mouse interface. We compare dwell-time click generation against detecting a single intentional muscle contraction with an attached sensor (ClickerAID). A preliminary evaluation was conducted as well as an in-depth case study with a participant with the neuromuscular disease Friedreich’s Ataxia. The case study shows modest temporal differences among the test conditions in movement time and throughput, though the participant subjectively favored the ClickerAID interface.

Adaptive mouse-replacement interface control functions for users with disabilities

We discuss experiences employing a video-based mouse-replacement interface system, the Camera Mouse, at care facilities for individuals with severe motion impairments and propose adaptations of the system. Traditional approaches to assistive technology are often inflexible, requiring users to adapt their limited motions to the requirements of the system. Such systems may have static or difficult-to-change configurations that make it challenging for multiple users to share the same system or for users whose motion abilities slowly degenerate. As users fatigue, they may experience more limited motion ability or additional unintended motions. To address these challenges, we propose adaptive mouse-control functions to be used in our mouse-replacement system. These functions can be changed to adapt the technology to the needs of the user, rather than making the user adapt to the technology. We present observations of an individual with severe cerebral palsy using our system.

Click control: improving mouse interaction for people with motor impairments

Camera-based mouse-replacement systems allow people with motor impairments to control the mouse pointer with head movements if they are unable to use their hands. To address the difficulties of accidental clicking and usable simulation of a real computer mouse, we developed Click Control, a tool to augment the functionality of these systems. When a user attempts to click, Click Control displays a form that allows him or her to cancel the click if it was accidental, or send different types of clicks with an easy-to-use gesture interface. Initial studies of a prototype with users with motor impairments showed that Click Control improved their mouse control experiences.