Francisco J. Perales

Hands-free vision-based interface for computer accessibility

Physically disabled and mentally challenged people are an important part of our society that has not yet received the same opportunities as others in their inclusion in the Information Society. Therefore, it is necessary to develop easily accessible systems for computers to achieve their inclusion within the new technologies. This paper presents a project whose objective is to draw disabled people nearer to new technologies. It presents a vision-based user interface designed to achieve computer accessibility for disabled users with motor impairments. The interface automatically finds the users face and tracks it through time to recognize gestures within the face region in real time. Subsequently, a new information fusion procedure is proposed to acquire data from computer vision algorithms and its results are used to carry out a robust recognition process. Finally, we show how the system is used to replace a conventional mouse device for computer interaction and as a communication system for non-verbal children.

Experiences using a hands-free interface

Hands-free interfaces could be the best choice for Human-Computer Interaction (HCI) for people with physical disabilities that are not capable of using traditional input devices. Once a first prototype is developed in the laboratory taking into account design and usability requirements, real users is what finally categorize an interface as useful or not. Therefore, an evaluation of our interface with users with cerebral palsy and multiple sclerosis has been carried out during a project of 9 months long. This paper presents a vision-based user interface designed to achieve computer accessibility together with the validation and evaluation of its human computer interaction issues such as usability and accessibility.

Toward natural interaction through visual recognition of body gestures in real-time

In most of the existing human-computer interfaces, enactive knowledge as new natural interaction paradigm has not been fully exploited yet. Recent technological advances have created the possibility to enhance naturally and significantly the interface perception by means of visual inputs, the so-called Vision-Based Interfaces (VBI). In the present paper, we explore the recovery of the users body posture by means of combining robust computer vision techniques and a well known inverse kinematics algorithm in real-time. Specifically, we focus on recognizing the users motions with a particular mean, that is, a body gesture. Defining an appropriate representation of the users body posture based on a temporal parameterization, we apply non-parametric techniques to learn and recognize the users body gestures. This scheme of recognition has been applied to control a computer videogame in real-time to show the viability of the presented approach.

User experience to improve the usability of a vision-based interface

When we develop an input device for users to communicate with computers, we have to take into account that end-users must consider the utilization of the device to be effective, efficient and satisfactory. Users whose expectations are unmet by the interface will tend to abandon it. In this paper we present a vision-based interface for motor-impaired users; a multidisciplinary group developed this interface. The users preferences are a critical issue when selecting an access device; therefore, user requirements should be included in the design. Usability evaluation should be integrated into relevant phases of software development. In order to evaluate the design, we present a process with multiple user studies at different development stages. We describe the combination of a development project and its implementation, with user experience considerations embedded in the process. Finally, we studied the performance of the interface through several tests, paying special attention to satisfaction and fatigue. From our results we observed that although several users found the interface tiring, their satisfaction level was encouraging, suggesting the interface is usable.

Evaluation of on-line analytic and numeric inverse kinematics approaches driven by partial vision input

Despite its central role in the constitution of a truly enactive interface, 3D interaction through human full body movement has been hindered by a number of technological and algorithmic factors. Let us mention the cumbersome magnetic equipments, or the underdetermined data set provided by less invasive video-based approaches. In the present paper, we explore the recovery of the full body posture of a standing subject in front of a stereo camera system. The 3D position of the hands, the head and the center of the trunk segment are extracted in real-time and provided to the body posture recovery algorithmic layer. We focus on the comparison between numeric and analytic inverse kinematics approaches in terms of performances and overall quality of the reconstructed body posture. Algorithmic issues arise from the very partial and noisy input and the singularity of the human standing posture. Despite stability concerns, results confirm the pertinence of this approach in this demanding context.

Hands and face tracking for VR applications

In this paper, we present a robust real-time 3D tracking system of human hands and face. This system can be used as a perceptual interface for virtual reality activities in a workbench environment. The main advantage of our system is that the human, placed in front of the virtual reality device, does not need any type of marker or special suit. The system includes a colour segmentation module to detect in real-time the skin-colour pixels present in the images. The results of this skin-colour segmentation will be skin-colour blobs, these are the inputs of a data association module. This module labels the blobs pixels using a set of hypothesis from previous frames. The 2D-tracking results are used for the 3D reconstruction of hands and face in order to obtain the 3D positions of these limbs. Finally, we present several results using the H-ANIM standard to show the systems output performance.

Design recommendations for camera-based head-controlled interfaces that replace the mouse for motion-impaired users

Abstract This work focuses on camera-based systems that are designed for mouse replacement. Usually, these interfaces are based on computer vision techniques that capture the user’s face or head movements and are specifically designed for users with disabilities. The work identifies and reviews the key factors of these interfaces based on the lessons learnt by the authors’ experience and by a comprehensive analysis of the literature to describe the specific points to consider in their design. These factors are as follows: user features to track, initial user detection (calibration), position mapping, feedback, error recovery, event execution, profiles and ergonomics of the system. The work compiles the solutions offered by different systems to help new designers avoid problems already discussed by the others.

The Intelligent Butler: A Virtual Agent for Disabled and Elderly People Assistance

Social assistance constitutes an increasing problem in developed countries, which can be considered from two dimensions: the home and the hospital frameworks. Anyway, most of the tasks have to do with aiding people with limitations in complex environments as a hospital or a house. Intelligent agents constitute a powerful approach in designing computer systems making possible the interaction of the users (elderly and disabled people) with the elements of a domotics environment. Such a purpose can be achieved through the unification of artificial intelligence techniques, virtual reality, multimodal interfaces and digital nets with domotics services. This paper describes the design and implementation of the prototype for a virtual agent capable of attending disabled people in a home environment. The results are shown through a computer program that simulates the behaviour of the agent in developing some typical functions.

Real Time Segmentation and Tracking of Face and Hands in VR Applications

We describe a robust real-time 3D tracking system of the extreme limbs of the upper human body, i.e., the hands and the face. The goal of the system is that it can be used as a perceptual interface for virtual reality activities in a workbench environment. The whole system includes an input capture and calibration module, a real time color segmentation module, a data association and tracking module and finally a visualization VRML and H-ANIM procedure. The results of our probabilistically skin-color segmentation are skin-color blobs. Then, for each frame of the sequence our algorithm labels the blobs’ pixels using a set of object state hypothesis. This set of hypothesis is built from the results of previous frames. The 2D tracking results are used for the 3D reconstruction of limbs position in order to obtain the H-ANIM visualization results. Several results are presented to show the algorithm performance.

Observing the use of an input device for rehabilitation purposes

We designed and developed a vision-based computer interface which works with head movements. The system was implemented in a centre for users with cerebral palsy and they used it in contexts related with recreation or with education. During this process, it was observed that the continued use of the interface with a set of training tasks may act as a physical and cognitive rehabilitation tool and complement users’ rehabilitation therapy. We comment on five case studies of users who have worked with the interface for five months and whose qualitative outcomes, observed by the therapists who accompanied them, were positive; specifically there was improvement in work posture, head control, increased endurance, decreased involuntary movements and improved spatial orientation. The case studies also showed the need to supervise the users’ work in order to achieve these aims, along with the importance of motivation and active, voluntary participation of users in the rehabilitation process.