Vuokko Lantz

A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors

This paper presents a framework for hand gesture recognition based on the information fusion of a three-axis accelerometer (ACC) and multichannel electromyography (EMG) sensors. In our framework, the start and end points of meaningful gesture segments are detected automatically by the intensity of the EMG signals. A decision tree and multistream hidden Markov models are utilized as decision-level fusion to get the final results. For sign language recognition (SLR), experimental results on the classification of 72 Chinese Sign Language (CSL) words demonstrate the complementary functionality of the ACC and EMG sensors and the effectiveness of our framework. Additionally, the recognition of 40 CSL sentences is implemented to evaluate our framework for continuous SLR. For gesture-based control, a real-time interactive system is built as a virtual Rubiks cube game using 18 kinds of hand gestures as control commands. While ten subjects play the game, the performance is also examined in user-specific and user-independent classification. Our proposed framework facilitates intelligent and natural control in gesture-based interaction.

Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors

This paper describes a novel hand gesture recognition system that utilizes both multi-channel surface electromyogram (EMG) sensors and 3D accelerometer (ACC) to realize user-friendly interaction between human and computers. Signal segments of meaningful gestures are determined from the continuous EMG signal inputs. Multi-stream Hidden Markov Models consisting of EMG and ACC streams are utilized as decision fusion method to recognize hand gestures. This paper also presents a virtual Rubiks Cube game that is controlled by the hand gestures and is used for evaluating the performance of our hand gesture recognition system. For a set of 18 kinds of gestures, each trained with 10 repetitions, the average recognition accuracy was about 91.7% in real application. The proposed method facilitates intelligent and natural control based on gesture interaction.

Hand Gesture Recognition Research Based on Surface EMG Sensors and 2D-accelerometers

For realizing multi-DOF interfaces in wearable computer system, accelerometers and surface EMG sensors are used synchronously to detect hand movement information for multiple hand gesture recognition. Experiments were designed to collect gesture data with both sensing techniques to compare their performance in the recognition of various wrist and finger gestures. Recognition tests were run using different subsets of information: accelerometer and sEMG data separately and combined sensor data. Experimental results show that the combination of sEMG sensors and accelerometers achieved 5-10% improvement in the recognition accuracies for hand gestures when compared to that obtained using sEMG sensors solely.

Multiple Hand Gesture Recognition Based on Surface EMG Signal

For realizing a multi-DOF myoelectric control system with a minimal number of sensors, research work on the recognition of twenty-four hand gestures based on two-channel surface EMG signal measured from human forearm muscles has been carried out. Third-order AR model coefficients, Mean Absolute Value and Mean Absolute Value ratio of the sEMG signal segments were used as features and the recognition of gestures was performed with a linear Bayesian classifier. Our experimental results show that the proposed two sensors setup and the sEMG signal processing and recognition methods are well suited for distinguishing hand gestures consisting of various wrist motions and single finger extension.

Pressages: augmenting phone calls with non-verbal messages

ForcePhone is a mobile synchronous haptic communication system. During phone calls, users can squeeze the side of the device and the pressure level is mapped to vibrations on the recipients device. The pressure/vibrotactile messages supported by ForcePhone are called pressages. Using a lab-based study and a small field study, this paper addresses the following questions: how can haptic interpersonal communication be integrated into a standard mobile device? What is the most appropriate feedback design for pressages? What types of non-verbal cues can be represented by pressages? Do users make use of pressages during their conversations? The results of this research indicate that such a system has value as a communication channel in real-world settings with users expressing greetings, presence and emotions through pressages.

Design of Dynamic Vibrotactile Textures

This paper describes a method for creating virtual textures without force feedback by using a simple motion sensor and a single vibrotactile actuator. It is based on wavetable synthesis driven by the users hand movements. The output of the synthesis is rendered with the tactile actuator attached in a hand-held box together with the motion sensor. The method provides a solution for creating tangible properties for virtual objects which can be explored by pointing at them with the sensor-actuator device. The study introduces 12 virtual textures which were based on three different envelope ridge lengths, two spatial densities, and were either regularly or irregularly organized. To evaluate the role of each design parameter in the perception of the texture, a series of experiments was conducted. The perceived similarity was assessed in a pairwise comparison test and the outcome was analyzed by using multidimensional scaling. The analysis revealed that envelope ridge length and spatial density were distinguishable design parameters while regularity was not. The textures were also rated according to five attribute scales previously determined in the pilot experiment. The results show that ridge length and spatial density influence perceived roughness and flatness similarly as with real textures.

Towards the Temporally Perfect Virtual Button: Touch-Feedback Simultaneity and Perceived Quality in Mobile Touchscreen Press Interactions

Pressing a virtual button is still the major interaction method in touchscreen mobile phones. Although phones are becoming more and more powerful, operating system software is getting more and more complex, causing latency in interaction. We were interested in gaining insight into touch-feedback simultaneity and the effects of latency on the perceived quality of touchscreen buttons. In an experiment, we varied the latency between touch and feedback between 0 and 300 ms for tactile, audio, and visual feedback modalities. We modelled the proportion of simultaneity perception as a function of latency for each modality condition. We used a Gaussian model fitted with the maximum likelihood estimation method to the observations. These models showed that the point of subjective simultaneity (PSS) was 5ms for tactile, 19ms for audio, and 32ms for visual feedback. Our study included the scoring of perceived quality for all of the different latency conditions. The perceived quality dropped significantly between latency conditions 70 and 100 ms when the feedback modality was tactile or audio, and between 100 and 150 ms when the feedback modality was visual. When the latency was 300ms for all feedback modalities, the quality of the buttons was rated significantly lower than in all of the other latency conditions, suggesting that a long latency between a touch on the screen and feedback is problematic for users. Together with PSS and these quality ratings, a 75% threshold was established to define a guideline for the recommended latency range between touch and feedback. Our guideline suggests that tactile feedback latency should be between 5 and 50 ms, audio feedback latency between 20 and 70 ms, and visual feedback latency between 30 and 85 ms. Using these values will ensure that users will perceive the feedback as simultaneous with the fingers touch. These values also ensure that the users do not perceive reduced quality. These results will guide engineers and designers of touchscreen interactions by showing the trade-offs between latency and user preference and the effects that their choices might have on the quality of the interactions and feedback they design.

Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions

Above-device gesture interfaces let people interact in the space above mobile devices using hand and finger movements. For example, users could gesture over a mobile phone or wearable without having to use the touchscreen. We look at how above-device interfaces can also give feedback in the space over the device. Recent haptic and wearable technologies give new ways to provide tactile feedback while gesturing, letting touchless gesture interfaces give touch feedback. In this paper we take a first detailed look at how tactile feedback can be given during above-device interaction. We compare approaches for giving feedback (ultrasound haptics, wearables and direct feedback) and also look at feedback design. Our findings show that tactile feedback can enhance above-device gesture interfaces.

Haptically augmented remote speech communication: a study of user practices and experiences

Haptic technology provides a channel for interpersonal communication through the sense of touch. In the development of novel haptic communication devices, it is essential to explore peoples use behaviors and perceptions of such a communication channel. To this end, we conducted a laboratory study on haptically augmented remote interpersonal communication. Participant pairs tested a communication system that allowed them to send squeezing and thermal feedback to each others forearm during speech discussion. We explored the use practices and user experience of this setup and compared it to traditional speech-only communication. The findings indicate that squeezing was experienced as a more versatile and immediate type of feedback than thermal feedback. Warm and cold were on the other hand useful for communicating positive and negative meanings. Compared to speech-only communication, the added haptic modality allowed conveying emphases, emotions, and touches related to the discussion, and increased the feeling of closeness between the pairs.

Squeezy bracelet: designing a wearable communication device for tactile interaction

While smartphones are increasing in size and complex features, new form factors for simple communication devices are emerging. In this paper, we present the design process for a wrist worn communication device, which enables the user to send text messages over a paired mobile phone. The process includes concept design, user evaluation, design iteration, prototype implementation, and evaluation of alternative interaction techniques. Our particular focus is towards the use of naturally tactile interfaces in a wearable wristband form factor. We present how users perceive deformable communication device concepts and two alternative squeeze based interaction techniques.