Sigal Berman

Sensors for Gesture Recognition Systems

A gesture recognition system (GRS) is comprised of a gesture, gesture-capture device (sensor), tracking algorithm (for motion capture), feature extraction, and classification algorithm. With the impending movement toward natural communication with mechanical and software systems, it is important to examine the first apparatus that separates the human communicator and the device being controlled. Although there are numerous reviews of GRSs, a comprehensive analysis of the integration of sensors into GRSs and their impact on system performance is lacking in the professional literature. Thus, we have undertaken this effort. Determination of the sensor stimulus, context of use, and sensor platform are major preliminary design issues in GRSs. Thus, these three components form the basic structure of our taxonomy. We emphasize the relationship between these critical components and the design of the GRS in terms of its architectural functions and computational requirements. In this treatise, we consider sensors that are capable of capturing dynamic and static arm and hand gestures. Although we discuss various sensor types, our main focus is on visual sensors as we expect these to become the sensor of choice in the foreseeable future. We delineate the challenges ahead for their increased effectiveness in this application domain. We note as a special challenge, the development of sensors that take over many of the functions the GRS designer struggles with today. We believe our contribution, in this first survey on sensors for GRSs, can give valuable insights into this important research and development topic, and encourage advanced research directions and new approaches.

Navigation of decentralized autonomous automatic guided vehicles in material handling

This paper presents a navigation methodology for decentralized autonomous automated guided vehicles used for material handling. The navigation methodology is based on behavior-based control augmented with multirobot coordination behaviors and a priori waypoint determination. Results indicate that the developed methodology fuses well between the desires for optimal vehicle routes on the one hand and decentralized reactive operation on the other.

Fuzzy behavior hierarchies for multi-robot control

Hierarchical approaches and methodologies are commonly used for control system design and synthesis. Well‐known model‐based techniques are often applied to solve problems of complex and large‐scale control systems. The general philosophy of decomposing control problems into modular and more manageable subsystem control problems applies equally to the growing domain of intelligent and autonomous systems. However, for this class of systems, new techniques for subsystem coordination and overall system control are often required. This article presents an approach to hierarchical control design and synthesis for the case where the collection of subsystems is comprised of fuzzy logic controllers and fuzzy knowledge‐based decision systems. The approach is used to implement hierarchical behavior‐based controllers for autonomous navigation of one or more mobile robots. Theoretical details of the approach are presented, followed by discussions of practical design and implementation issues. Example implementations realized on various physical mobile robots are described to demonstrate how the techniques may be applied in practical applications involving homogeneous and heterogeneous robot teams.

Decentralized autonomous AGV system for material handling

Decentralized control of an autonomous automated guided vehicle system (AGVS) used for material handling is expected to lead to high system flexibility and robustness. A complete control methodology for decentralized autonomous AGVS control was developed and implemented in a computer-integrated manufacturing environment. The methodology addresses all aspects of AGVS functionality: system management, navigation and load transfer. Hierarchical fuzzy behaviour-based control, a reactive navigation scheme, was expanded to multirobot control in semistructured environments by incorporating a priori path optimization and right-of-the-way determination.

Most Probable Longest Common Subsequence for Recognition of Gesture Character Input

This paper presents a technique for trajectory classification with applications to dynamic free-air hand gesture recognition. Such gestures are unencumbered and drawn in free air. Our approach is an extension to the longest common subsequence (LCS) classification algorithm. A learning preprocessing stage is performed to create a probabilistic 2-D template for each gesture, which allows taking into account different trajectory distortions with different probabilities. The modified LCS, termed the most probable LCS (MPLCS), is developed to measure the similarity between the probabilistic template and the hand gesture sample. The final decision is based on the length and probability of the extracted subsequence. Validation tests using a cohort of gesture digits from video-based capture show that the approach is promising with a recognition rate of more than 98 % for video stream preisolated digits. The MPLCS algorithm can be integrated into a gesture recognition interface to facilitate gesture character input. This can greatly enhance the usability of such interfaces.

Kinematics of Reaching Movements in a 2-D Virtual Environment in Adults With and Without Stroke

Virtual reality environments are increasingly being used for upper limb rehabilitation in poststroke patients. Our goal was to determine if arm reaching movements made in a 2-D video-capture virtual reality environment are similar to those made in a comparable physical environment. We compared arm and trunk kinematics for reaches made with the right, dominant arm to three targets (14 trials per target) in both environments by 16 adults with right poststroke hemiparesis and by eight healthy age-matched controls. Movement kinematics were recorded with a three-camera optoelectronic system at 100 samples/s. Reaching movements made by both control and stroke subjects were affected by viewing the targets in the video-capture 2-D virtual environment. Movements were slower, shorter, less straight, less accurate and involved smaller ranges of shoulder and elbow joint excursions for target reaches in the virtual environment compared to the physical environment in all subjects. Thus, there was a decrease in the overall movement quality for movements made in the 2-D virtual environment. This suggests that 2-D video-capture virtual reality environments should be used with caution when the goal of the rehabilitation program is to improve the quality of movement patterns of the upper limb.

Most discriminating segment - Longest common subsequence (MDSLCS) algorithm for dynamic hand gesture classification

In this work, we consider the recognition of dynamic gestures based on representative sub-segments of a gesture, which are denoted as most discriminating segments (MDSs). The automatic extraction and recognition of such small representative segments, rather than extracting and recognizing the full gestures themselves, allows for a more discriminative classifier. A MDS is a sub-segment of a gesture that is most dissimilar to all other gesture sub-segments. Gestures are classified using a MDSLCS algorithm, which recognizes the MDSs using a modified longest common subsequence (LCS) measure. The extraction of MDSs from a data stream uses adaptive window parameters, which are driven by the successive results of multiple calls to the LCS classifier. In a preprocessing stage, gestures that have large motion variations are replaced by several forms of lesser variation. We learn these forms by adaptive clustering of a training set of gestures, where we reemploy the LCS to determine similarity between gesture trajectories. The MDSLCS classifier achieved a gesture recognition rate of 92.6% when tested using a set of pre-cut free hand digit (0-9) gestures, while hidden Markov models (HMMs) achieved an accuracy of 89.5%. When the MDSLCS was tested against a set of streamed digit gestures, an accuracy of 89.6% was obtained. At present the HMMs method is considered the state-of-the-art method for classifying motion trajectories. The MDSLCS algorithm had a higher accuracy rate for pre-cut gestures, and is also more suitable for streamed gestures. MDSLCS provides a significant advantage over HMMs by not requiring data re-sampling during run-time and performing well with small training sets.

Compensatory Versus Noncompensatory Shoulder Movements Used for Reaching in Stroke

Background. The extent to which the upper-limb flexor synergy constrains or compensates for arm motor impairment during reaching is controversial. This synergy can be quantified with a minimal marker set describing movements of the arm-plane. Objectives. To determine whether and how (a) upper-limb flexor synergy in patients with chronic stroke contributes to reaching movements to different arm workspace locations and (b) reaching deficits can be characterized by arm-plane motion. Methods. Sixteen post-stroke and 8 healthy control subjects made unrestrained reaching movements to targets located in ipsilateral, central, and contralateral arm workspaces. Arm-plane, arm, and trunk motion, and their temporal and spatial linkages were analyzed. Results. Individuals with moderate/severe stroke used greater arm-plane movement and compensatory trunk movement compared to those with mild stroke and control subjects. Arm-plane and trunk movements were more temporally coupled in stroke compared with controls. Reaching accuracy was related to different segment and joint combinations for each target and group: arm-plane movement in controls and mild stroke subjects, and trunk and elbow movements in moderate/severe stroke subjects. Arm-plane movement increased with time since stroke and when combined with trunk rotation, discriminated between different subject groups for reaching the central and contralateral targets. Trunk movement and arm-plane angle during target reaches predicted the subject group. Conclusions. The upper-limb flexor synergy was used adaptively for reaching accuracy by patients with mild, but not moderate/severe stroke. The flexor synergy, as parameterized by the amount of arm-plane motion, can be used by clinicians to identify levels of motor recovery in patients with stroke.

Arm-plane representation of shoulder compensation during pointing movements in patients with stroke.

Improvements in functional motor activities are often accompanied by motor compensations to overcome persistent motor impairment in the upper limb. Kinematic analysis is used to objectively quantify movement patterns including common motor compensations such as excessive trunk displacement during reaching. However, a common motor compensation to assist reaching, shoulder abduction, is not adequately characterized by current motion analysis approaches. We apply the arm-plane representation that accounts for the co-variation between movements of the whole arm, and investigate its ability to identify and quantify compensatory arm movements in stroke subjects when making forward arm reaches. This method has not been previously applied to the analysis of motion deficits. Sixteen adults with right post-stroke hemiparesis and eight healthy age-matched controls reached in three target directions (14 trials/target; sampling rate: 100Hz). Arm-plane movement was validated against endpoint, joint, and trunk kinematics and compared between groups. In stroke subjects, arm-plane measures were correlated with arm impairment (Fugl-Meyer Assessment) and ability (Box and Blocks) scores and were more sensitive than clinical measures to detect mild motor impairment. Arm-plane motion analysis provides new information about motor compensations involving the co-variation of shoulder and elbow movements that may help to understand the underlying motor deficits in patients with stroke.

Application of motor algebra to the analysis of human arm movements

Motor algebra, a 4D degenerate geometric algebra, offers a rigorous yet simple representation of the 3D velocity of a rigid body. Using this representation, we study 3D extended arm pointing and reaching movements. We analyze the choice of arm orientation about the vector connecting the shoulder and the wrist, in cases for which this orientation is not prescribed by the task. Our findings show that the changes in this orientation throughout the movement were very small, possibly indicating an underlying motion planning strategy. We additionally examine the decomposition of movements into submovements and reconstruct the motion by assuming superposition of the velocity profiles of the underlying submovements by analyzing both the translational and rotational components of the 3D spatial velocity. This movement decomposition method reveals a larger number of submovement than is found using previously applied submovement extraction methods that are based only on the analysis of the hand tangential velocity. The reconstructed velocity profiles and final orientations are relatively close to the actual values, indicating that single-axis submovements may be the basic building blocks underlying 3D movement construction.