Matjaž Mihelj

A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing

Physiological computing represents a mode of human-computer interaction where the computer monitors, analyzes and responds to the users psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subjects psychological state.

Psychophysiological responses to different levels of cognitive and physical workload in haptic interaction

Psychophysiological measurements, which serve as objective indicators of psychological state, have recently been introduced into human–robot interaction. However, their usefulness in haptic interaction is uncertain, since they are influenced by physical workload. This study analyses psychophysiological responses to a haptic task with three different difficulty levels and two different levels of physical load. Four physiological responses were recorded: heart rate, skin conductance, respiratory rate and skin temperature. Results show that mean respiratory rate, respiratory rate variability and skin temperature show significant differences between difficulty levels regardless of physical load and can be used to estimate cognitive workload in haptic interaction.

Human arm kinematics for robot based rehabilitation

The paper considers a technique for computation of the inverse kinematic model of the human arm for robot based rehabilitation that uses measurements of the hand position and orientation and radial acceleration of the upper arm. Analytical analysis and empirical validation of the method are presented. The algorithm enables estimation of human arm angles, which can be used in trajectory planning for rehabilitation robots, evaluation of motion of patients with movement disorders, and generation of virtual reality environments.

Inverse Kinematics of Human Arm Based on Multisensor Data Integration

The paper considers a technique for computation of the inverse kinematic model of the human arm. The approach is based on measurements of the hand position and orientation as well as acceleration and angular rate of the upper arm segment. A quaternion description of orientation is used to avoid singularities in representations with Euler angles. A Kalman filter is designed to integrate sensory data from three different types of sensors. The algorithm enables estimation of human arm posture, which can be used in trajectory planning for rehabilitation robots, evaluation of motion of patients with movement disorders, and generation of virtual reality environments.

Skill transfer from symmetric and asymmetric bimanual training using a robotic system to single limb performance

BackgroundHumans are capable of fast adaptation to new unknown dynamics that affect their movements. Such motor learning is also believed to be an important part of motor rehabilitation. Bimanual training can improve post-stroke rehabilitation outcome and is associated with interlimb coordination between both limbs. Some studies indicate partial transfer of skills among limbs of healthy individuals. Another aspect of bimanual training is the (a)symmetry of bimanual movements and how these affect motor learning and possibly post-stroke rehabilitation.MethodsA novel bimanual 2-DOF robotic system was used for both bimanual and unimanual reaching movements. 35 young healthy adults participated in the study. They were divided into 5 test groups that performed movements under different conditions (bimanual or unimanual movements and symmetric or asymmetric bimanual arm loads). The subjects performed a simple tracking exercise with the bimanual system. The exercise was developed to stimulate motor learning by applying a velocity-dependent disturbance torque to the handlebar. Each subject performed 255 trials divided into three phases: baseline without disturbance torque, training phase with disturbance torque and evaluation phase with disturbance torque.ResultsPerformance was assessed with the maximal values of rotation errors of the handlebar. After exposure to disturbance torque, the errors decreased for both unimanual and bimanual training. Errors in unimanual evaluation following the bimanual training phase were not significantly different from errors in unimanual evaluation following unimanual training. There was no difference in performance following symmetric or asymmetric training. Changing the arm force symmetry during bimanual movements from asymmetric to symmetric had little influence on performance.ConclusionsSubjects could adapt to an unknown disturbance torque that was changing the dynamics of the movements. The learning effect was present during both unimanual and bimanual training. Transfer of learned skills from bimanual training to unimanual movements was also observed, as bimanual training also improved single limb performance with the dominant arm. Changes of force symmetry did not have an effect on motor learning. As motor learning is believed to be an important mechanism of rehabilitation, our findings could be tested for future post-stroke rehabilitation systems.

Infant trunk posture and arm movement assessment using pressure mattress, inertial and magnetic measurement units (IMUs)

BackgroundExisting motor pattern assessment methods, such as digital cameras and optoelectronic systems, suffer from object obstruction and require complex setups. To overcome these drawbacks, this paper presents a novel approach for biomechanical evaluation of newborn motor skills development. Multi-sensor measurement system comprising pressure mattress and IMUs fixed on trunk and arms is proposed and used as alternative to existing methods. Observed advantages seem appealing for the focused field and in general. Combined use of pressure distribution data and kinematic information is important also for posture assessment, ulcer prevention, and non-invasive sleep pattern analysis of adults.MethodsArm kinematic parameters, such as root-mean-square acceleration, spectral arc length of hand velocity profile, including arm workspace surface area, and travelled hand path are obtained with the multi-sensor measurement system and compared to normative motion capture data for evaluation of adequacy. Two IMUs per arm, only one IMU on upper arm, and only one IMU on forearm sensor placement options are studied to assess influence of system configuration on method precision. Combination of pressure mattress and IMU fixed on the trunk is used to measure trunk position (obtained from mat), rotation (from IMUs) and associated movements on surface (from both). Measurement system is first validated on spontaneous arm and trunk movements of a dedicated baby doll having realistic anthropometric characteristics of newborns. Next, parameters of movements in a healthy infant are obtained with pressure mattress, along with trunk and forearm IMU sensors to verify appropriateness of method and parameters.ResultsEvaluation results confirm that full sensor set, comprising pressure mattress and two IMUs per arm is a reliable substitution to optoelectronic systems. Motor pattern parameter errors are under 10% and kinematic estimation error is in range of 2 cm. Although, use of only forearm IMU is not providing best possible kinematic precision, the simplicity of use and still acceptable accuracy are convincing for frequent practical use. Measurements demonstrated system high mobility and usability.ConclusionsStudy results confirm adequacy of the proposed multi-sensor measurement system, indicating its enviable potential for accurate infant trunk posture and arm movement assessment.

Dual-task performance in multimodal human-computer interaction: a psychophysiological perspective

This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and concurrently on a computer using verbal and haptic input devices. Heart rate, skin conductance, respiration and peripheral skin temperature were recorded in addition to objective performance measures and self-report questionnaires. Analysis of psychophysiological responses found significant changes from baseline for both single-task and dual-task conditions. There were also significant psychophysiological differences between the mental arithmetic task and the manual error correction task, but no differences in questionnaire results. Additionally, there was no significant psychophysiological difference between performing only the mental arithmetic task and performing both tasks at once. These findings suggest that psychophysiological measures respond differently to different types of tasks and that they do not always agree with performance or with participants’ subjective feelings.

Using Psychophysiological Measurements in Physically Demanding Virtual Environments

Psychophysiological evaluation of mental workload in human-computer interaction has generally been limited to situations with little physical load. This paper examines the viability of using heart rate, skin conductance, respiration and peripheral skin temperature as psychophysiological indicators in a physically demanding task performed in a simple virtual environment. Respiratory rate was found to be a good indicator of arousal while respiratory rate variability and skin temperature indicated changes in valence.

Grasping and manipulation in virtual environment using 3By6 finger device

Realistic simulation of grasping requires accurate modeling of forces and torques on the virtual object produced by fingers in contact. We present isometric 3By6 finger device for multi-fingered grasping in virtual environment (VE). The finger device was designed to measure forces of three fingers. Mathematical model of grasping adopted from the analysis of multi-fingered robot hands was used. The dynamics of the virtual object corresponds to the forces and torques applied by the three fingers. The multi-fingered grasping is demonstrated in four tasks aimed at the rehabilitation of the upper extremities of stroke patients. The tasks include opening of a safe, filling and pouring water from a glass, training of muscle strength with an elastic torus and force-tracking task.

CareToy: Stimulation and Assessment of Preterm Infant's Activity Using a Novel Sensorized System.

Early intervention programs aim at improving cognitive and motor outcomes of preterm infants. Intensive custom-tailored training activities are usually accompanied by assessment procedures, which have shortcomings, such as subjectivity, complex setups, and need for structured environments. A novel sensorized system, called CareToy, was designed to provide stimulation in the form of goal-directed activity training scenarios and motor pattern assessment of main developmental milestones, such as rolling activity, grasping, and postural stability. A group of 28 differently skilled preterm infants were enrolled. Acquired measurement data were analysed with dedicated sensor data processing algorithms, along with clinical evaluation of motor ability. High correlation among technically determined parameters and Alberta Infant Motor Scale values was determined by Pearson correlation coefficients. Due to good accuracy and possibility of single motor skill subfield analysis, results confirm system suitability for motor ability assessment. Statistical analysis of inter-motor ability group and inter-training goal data comparisons demonstrate system’s appropriateness for goal-directed activity stimulation. The proposed system has evident potential of being an important contribution to the field of infant motor development assessment, expanding accessibility of early intervention programs and affecting rehabilitation effectiveness of preterm infants.