Domen Novak

A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing

Physiological computing represents a mode of human-computer interaction where the computer monitors, analyzes and responds to the users psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subjects psychological state.

Real-Time Closed-Loop Control of Cognitive Load in Neurological Patients During Robot-Assisted Gait Training

Cognitively challenging training sessions during robot-assisted gait training after stroke were shown to be key requirements for the success of rehabilitation. Despite a broad variability of cognitive impairments amongst the stroke population, current rehabilitation environments do not adapt to the cognitive capabilities of the patient, as cognitive load cannot be objectively assessed in real-time. We provided healthy subjects and stroke patients with a virtual task during robot-assisted gait training, which allowed modulating cognitive load by adapting the difficulty level of the task. We quantified the cognitive load of stroke patients by using psychophysiological measurements and performance data. In open-loop experiments with healthy subjects and stroke patients, we obtained training data for a linear, adaptive classifier that estimated the current cognitive load of patients in real-time. We verified our classification results via questionnaires and obtained 88% correct classification in healthy subjects and 75% in patients. Using the pre-trained, adaptive classifier, we closed the cognitive control loop around healthy subjects and stroke patients by automatically adapting the difficulty level of the virtual task in real-time such that patients were neither cognitively overloaded nor under-challenged.

A survey of sensor fusion methods in wearable robotics

Modern wearable robots are not yet intelligent enough to fully satisfy the demands of end-users, as they lack the sensor fusion algorithms needed to provide optimal assistance and react quickly to perturbations or changes in user intentions. Sensor fusion applications such as intention detection have been emphasized as a major challenge for both robotic orthoses and prostheses. In order to better examine the strengths and shortcomings of the field, this paper presents a review of existing sensor fusion methods for wearable robots, both stationary ones such as rehabilitation exoskeletons and portable ones such as active prostheses and full-body exoskeletons. Fusion methods are first presented as applied to individual sensing modalities (primarily electromyography, electroencephalography and mechanical sensors), and then four approaches to combining multiple modalities are presented. The strengths and weaknesses of the different methods are compared, and recommendations are made for future sensor fusion research. Overview of sensor fusion in wearable robots like prostheses and exoskeletons.Main sensors: electromyography, electroencephalography, and mechanical sensors.Emphasizes multimodality, adaptation and switching between sensor fusion schemes.Online evaluation of sensor fusion methods is crucial.

Automated detection of gait initiation and termination using wearable sensors

This paper presents algorithms for detection of gait initiation and termination using wearable inertial measurement units and pressure-sensitive insoles. Body joint angles, joint angular velocities, ground reaction force and center of plantar pressure of each foot are obtained from these sensors and input into supervised machine learning algorithms. The proposed initiation detection method recognizes two events: gait onset (an anticipatory movement preceding foot lifting) and toe-off. The termination detection algorithm segments gait into steps, measures the signals over a buffer at the beginning of each step, and determines whether this measurement belongs to the final step. The approach is validated with 10 subjects at two gait speeds, using within-subject and subject-independent cross-validation. Results show that gait initiation can be detected timely and accurately, with few errors in the case of within-subject cross-validation and overall good performance in subject-independent cross-validation. Gait termination can be predicted in over 80% of trials well before the subject comes to a complete stop. Results also show that the two sensor types are equivalent in predicting gait initiation while inertial measurement units are generally superior in predicting gait termination. Potential use of the algorithms is foreseen primarily with assistive devices such as prostheses and exoskeletons.

Psychophysiological responses to different levels of cognitive and physical workload in haptic interaction

Psychophysiological measurements, which serve as objective indicators of psychological state, have recently been introduced into human–robot interaction. However, their usefulness in haptic interaction is uncertain, since they are influenced by physical workload. This study analyses psychophysiological responses to a haptic task with three different difficulty levels and two different levels of physical load. Four physiological responses were recorded: heart rate, skin conductance, respiratory rate and skin temperature. Results show that mean respiratory rate, respiratory rate variability and skin temperature show significant differences between difficulty levels regardless of physical load and can be used to estimate cognitive workload in haptic interaction.

Emotion-aware system for upper extremity rehabilitation

Immersive and multimodal sensory feedback was implemented to improve neurorehabilitation movement training. A major aspect of feedback is to reflect back the patients psychophysiological state into the environment, and also to use this as a guidance mechanism as to how events within the virtual environment unfold. The virtual environment was constructed using haptic, visual and acoustic primitives (basic sets of changes applied to the multimodal virtual environment that are expected to change the psychophysiological state of the patient). State transitions between primitives are defined as a response to changes in the users psychological state and motor performance. The mapping of biomechanical and physiological measurements to motor performance and psychological state and then to changes in action primitives was implemented using a fuzzy-logic system.

Predicting Targets of Human Reaching Motions Using Different Sensing Technologies

Rapid recognition of voluntary motions is crucial in human-computer interaction, but few studies compare the predictive abilities of different sensing technologies. This paper thus compares performances of different technologies when predicting targets of human reaching motions: electroencephalography (EEG), electrooculography, camera-based eye tracking, electromyography (EMG), hand position, and the users preferences. Supervised machine learning is used to make predictions at different points in time (before and during limb motion) with each individual sensing modality. Different modalities are then combined using an algorithm that takes into account the different times at which modalities provide useful information. Results show that EEG can make predictions before limb motion onset, but requires subject-specific training and exhibits decreased performance as the number of possible targets increases. EMG and hand position give high accuracy, but only once the motion has begun. Eye tracking is robust and exhibits high accuracy at the very onset of limb motion. Several advantages of combining different modalities are also shown, including advantages of combining measurements with contextual data. Finally, some recommendations are given for sensing modalities with regard to different criteria and applications. The information could aid human-computer interaction designers in selecting and evaluating appropriate equipment for their applications.

Development and validation of a wearable inertial measurement system for use with lower limb exoskeletons

This paper presents a system of inertial measurement units, each consisting of an accelerometer, gyroscope and magnetometer. They are characterized by a small size, wireless transmission, and open architecture, with the purpose of either integration into lower limb exoskeletons or general human movement analysis. Kalman filtering and the factored quaternion algorithm are used to track the orientation of each unit, and angles of the human joints are calculated from multiple units. After calibration, the system was tested with a wooden leg mockup and an actual human. The Optotrak optical measurement system was used as a reference. Differences between the inertial measurement system and the Optotrak were less than 2 degrees for the wooden leg and less than 5 degrees for the human leg, suggesting that the system represents a promising possibility for wearable movement tracking and analysis.

Development of gait segmentation methods for wearable foot pressure sensors

We present an automated segmentation method based on the analysis of plantar pressure signals recorded from two synchronized wireless foot insoles. Given the strict limits on computational power and power consumption typical of wearable electronic components, our aim is to investigate the capability of a Hidden Markov Model machine-learning method, to detect gait phases with different levels of complexity in the processing of the wearable pressure sensors signals. Therefore three different datasets are developed: raw voltage values, calibrated sensor signals and a calibrated estimation of total ground reaction force and position of the plantar center of pressure. The method is tested on a pool of 5 healthy subjects, through a leave-one-out cross validation. The results show high classification performances achieved using estimated biomechanical variables, being on average the 96%. Calibrated signals and raw voltage values show higher delays and dispersions in phase transition detection, suggesting a lower reliability for online applications.

Toward Real-Time Automated Detection of Turns during Gait Using Wearable Inertial Measurement Units

Previous studies have presented algorithms for detection of turns during gait using wearable sensors, but those algorithms were not built for real-time use. This paper therefore investigates the optimal approach for real-time detection of planned turns during gait using wearable inertial measurement units. Several different sensor positions (head, back and legs) and three different detection criteria (orientation, angular velocity and both) are compared with regard to their ability to correctly detect turn onset. Furthermore, the different sensor positions are compared with regard to their ability to predict the turn direction and amplitude. The evaluation was performed on ten healthy subjects who performed left/right turns at three amplitudes (22, 45 and 90 degrees). Results showed that turn onset can be most accurately detected with sensors on the back and using a combination of orientation and angular velocity. The same setup also gives the best prediction of turn direction and amplitude. Preliminary measurements with a single amputee were also performed and highlighted important differences such as slower turning that need to be taken into account.