Jani Mäntyjärvi

Activity classification using realistic data from wearable sensors

Automatic classification of everyday activities can be used for promotion of health-enhancing physical activities and a healthier lifestyle. In this paper, methods used for classification of everyday activities like walking, running, and cycling are described. The aim of the study was to find out how to recognize activities, which sensors are useful and what kind of signal processing and classification is required. A large and realistic data library of sensor data was collected. Sixteen test persons took part in the data collection, resulting in approximately 31 h of annotated, 35-channel data recorded in an everyday environment. The test persons carried a set of wearable sensors while performing several activities during the 2-h measurement session. Classification results of three classifiers are shown: custom decision tree, automatically generated decision tree, and artificial neural network. The classification accuracies using leave-one-subject-out cross validation range from 58 to 97% for custom decision tree classifier, from 56 to 97% for automatically generated decision tree, and from 22 to 96% for artificial neural network. Total classification accuracy is 82% for custom decision tree classifier, 86% for automatically generated decision tree, and 82% for artificial neural network

Detection of Daily Activities and Sports With Wearable Sensors in Controlled and Uncontrolled Conditions

Physical activity has a positive impact on peoples well-being, and it may also decrease the occurrence of chronic diseases. Activity recognition with wearable sensors can provide feedback to the user about his/her lifestyle regarding physical activity and sports, and thus, promote a more active lifestyle. So far, activity recognition has mostly been studied in supervised laboratory settings. The aim of this study was to examine how well the daily activities and sports performed by the subjects in unsupervised settings can be recognized compared to supervised settings. The activities were recognized by using a hybrid classifier combining a tree structure containing a priori knowledge and artificial neural networks, and also by using three reference classifiers. Activity data were collected for 68 h from 12 subjects, out of which the activity was supervised for 21 h and unsupervised for 47 h. Activities were recognized based on signal features from 3-D accelerometers on hip and wrist and GPS information. The activities included lying down, sitting and standing, walking, running, cycling with an exercise bike, rowing with a rowing machine, playing football, Nordic walking, and cycling with a regular bike. The total accuracy of the activity recognition using both supervised and unsupervised data was 89% that was only 1% unit lower than the accuracy of activity recognition using only supervised data. However, the accuracy decreased by 17% unit when only supervised data were used for training and only unsupervised data for validation, which emphasizes the need for out-of-laboratory data in the development of activity-recognition systems. The results support a vision of recognizing a wider spectrum, and more complex activities in real life settings.

Managing context information in mobile devices

We present a uniform mobile terminal software framework that provides systematic methods for acquiring and processing useful context information from a users surroundings and giving it to applications. The framework simplifies the development of context-aware mobile applications by managing raw context information gained from multiple sources and enabling higher-level context abstractions.

Identifying users of portable devices from gait pattern with accelerometers

Identifying users of portable devices from gait signals acquired with three-dimensional accelerometers was studied. Three approaches, correlation, frequency domain and data distribution statistics, were used. Test subjects (N=36) walked with fast, normal and slow walking speeds in enrolment and test sessions on separate days wearing the accelerometer device on their belt, at back. It was shown to be possible to identify users with this novel gait recognition method. Best equal error rate (EER=7%) was achieved with the signal correlation method, while the frequency domain method and two variations of the data distribution statistics method produced EER of 10%, 18% and 19%, respectively.

Accelerometer-based gesture control for a design environment

Accelerometer-based gesture control is studied as a supplementary or an alternative interaction modality. Gesture commands freely trainable by the user can be used for controlling external devices with handheld wireless sensor unit. Two user studies are presented. The first study concerns finding gestures for controlling a design environment (Smart Design Studio), TV, VCR, and lighting. The results indicate that different people usually prefer different gestures for the same task, and hence it should be possible to personalise them. The second user study concerns evaluating the usefulness of the gesture modality compared to other interaction modalities for controlling a design environment. The other modalities were speech, RFID-based physical tangible objects, laser-tracked pen, and PDA stylus. The results suggest that gestures are a natural modality for certain tasks, and can augment other modalities. Gesture commands were found to be natural, especially for commands with spatial association in design environment control.

Identifying people from gait pattern with accelerometers

Protecting portable devices is becoming more important, not only because of the value of the devices themselves, but for the value of the data in them and their capability for transactions, including m-commerce and m-banking. An unobtrusive and natural method for identifying the carrier of portable devices is presented. The method uses acceleration signals produced by sensors embedded in the portable device. When the user carries the device, the acceleration signal is compared with the stored template signal. The method consists of finding individual steps, normalizing and averaging them, aligning them with the template and computing cross-correlation, which is used as a measure of similarity. Equal Error Rate of 6.4% is achieved in tentative experiments with 36 test subjects.

Enabling fast and effortless customisation in accelerometer based gesture interaction

Accelerometer based gesture control is proposed as a complementary interaction modality for handheld devices. Predetermined gesture commands or freely trainable by the user can be used for controlling functions also in other devices. To support versatility of gesture commands in various types of personal device applications gestures should be customisable, easy and quick to train. In this paper we experiment with a procedure for training/recognizing customised accelerometer based gestures with minimum amount of user effort in training. Discrete Hidden Markov Models (HMM) are applied. Recognition results are presented for an external device, a DVD player gesture commands. A procedure based on adding noise-distorted signal duplicates to training set is applied and it is shown to increase the recognition accuracy while decreasing user effort in training. For a set of eight gestures, each trained with two original gestures and with two Gaussian noise-distorted duplicates, the average recognition accuracy was 97%, and with two original gestures and with four noise-distorted duplicates, the average recognition accuracy was 98%, cross-validated from a total data set of 240 gestures. Use of procedure facilitates quick and effortless customisation in accelerometer based interaction.

Adapting applications in handheld devices using fuzzy context information

Abstract Context-aware devices are able to take advantage of fusing sensory and application specific information to provide proper information on a situation, for more flexible services, and adaptive user interfaces (UI). It is characteristic for handheld devices and their users that they are continuously moving in several simultaneous fuzzy contexts. The dynamic environment sets special requirements for usability and acceptance of context-aware applications. Context-aware applications must be able to operate sensibly even if the context recognition is not 100% reliable and there are multiple contexts present at the same time. We present an approach for controlling context-aware applications in the case of multiple fuzzy contexts. This work has several potential applications in the area of adaptive UI application control. Our study is focused on the adaptation of applications representing information in handheld devices. The design of controllers and experiments with real context data from user scenarios are presented. Experimental results show that the proposed approach enhances the capability of adapting information representation in a handheld device. User reactions indicate that they accept application adaptation in many situations while insisting on retaining the most control over their device. Moreover, user feedback indicates that abrupt adaptations and instability should be avoided in the application control.

Online gesture recognition system for mobile interaction

This paper introduces an accelerometer-based online gesture recognition system. Recognition of gestures can be utilised as a part of a human computer interaction for mobile devices, e.g. cell phones, PDAs and remote controllers. Gestures are captured with a small wireless sensor-box that produces three dimensional acceleration signal. Acceleration signal is preprocessed, vector quantised and finally classified using Hidden Markov Models. The design of online gesture recognition for mobile devices sets requirements for data processing. Thus, the system uses a small size codebook and simple preprocessing methods. The recognition accuracy of system is tested with gestures of four degrees of complexity. Experimental results show great potential for recognising simple and even more complex gestures with good accuracy.

On-line personalization of a touch screen based keyboard

The user expectations for usability and personalization along with decreasing size of handheld devices challenge traditional keypad layout design. We have developed a method for on-line adaptation of a touch pad keyboard layout. The method starts from an original layout and monitors the usage of the keyboard by recording and analyzing the keystrokes. An on-line learning algorithm subtly moves the keys according to the spatial distribution of keystrokes. In consequence, the keyboard matches better to the users physical extensions and grasp of the device, and makes the physical trajectories during typing more comfortable. We present two implementations that apply different vector quantization algorithms to produce an adaptive keyboard with visual on-line feedback. Both qualitative and quantitative results show that the changes in the keyboard are consistent, and related to the users handedness and hand extensions. The testees found the on-line personalization positive. The method can either be applied for on-line personalization of keyboards or for ergonomics research