Dana Kulic

Incremental Learning, Clustering and Hierarchy Formation of Whole Body Motion Patterns using Adaptive Hidden Markov Chains

This paper describes a novel approach for autonomous and incremental learning of motion pattern primitives by observation of human motion. Human motion patterns are abstracted into a dynamic stochastic model, which can be used for both subsequent motion recognition and generation, analogous to the mirror neuron hypothesis in primates. The model size is adaptable based on the discrimination requirements in the associated region of the current knowledge base. A new algorithm for sequentially training the Markov chains is developed, to reduce the computation cost during model adaptation. As new motion patterns are observed, they are incrementally grouped together using hierarchical agglomerative clustering based on their relative distance in the model space. The clustering algorithm forms a tree structure, with specialized motions at the tree leaves, and generalized motions closer to the root. The generated tree structure will depend on the type of training data provided, so that the most specialized motions will be those for which the most training has been received. Tests with motion capture data for a variety of motion primitives demonstrate the efficacy of the algorithm.

Pre-collision safety strategies for human-robot interaction

Safe planning and control is essential to bringing human-robot interaction into common experience. This paper presents an integrated human−robot interaction strategy that ensures the safety of the human participant through a coordinated suite of safety strategies that are selected and implemented to anticipate and respond to varying time horizons for potential hazards and varying expected levels of interaction with the user. The proposed planning and control strategies are based on explicit measures of danger during interaction. The level of danger is estimated based on factors influencing the impact force during a human-robot collision, such as the effective robot inertia, the relative velocity and the distance between the robot and the human.A second key requirement for improving safety is the ability of the robot to perceive its environment, and more specifically, human behavior and reaction to robot movements. This paper also proposes and demonstrates the use of human monitoring information based on vision and physiological sensors to further improve the safety of the human robot interaction. A methodology for integrating sensor-based information about the users position and physiological reaction to the robot into medium and short-term safety strategies is presented. This methodology is verified through a series of experimental test cases where a human and an articulated robot respond to each other based on the humans physical and physiological behavior.

Incremental learning of full body motion primitives and their sequencing through human motion observation

In this paper we describe an approach for on-line, incremental learning of full body motion primitives from observation of human motion. The continuous observation sequence is first partitioned into motion segments, using stochastic segmentation. Next, motion segments are incrementally clustered and organized into a hierarchical tree structure representing the known motion primitives. Motion primitives are encoded using hidden Markov models, so that the same model can be used for both motion recognition and motion generation. At the same time, the temporal relationship between motion primitives is learned via the construction of a motion primitive graph. The motion primitive graph can then be used to construct motions, consisting of sequences of motion primitives. The approach is implemented and tested during on-line observation and on the IRT humanoid robot.

Online Segmentation and Clustering From Continuous Observation of Whole Body Motions

This paper describes a novel approach for incremental learning of human motion pattern primitives through online observation of human motion. The observed time series data stream is first stochastically segmented into potential motion primitive segments, based on the assumption that data belonging to the same motion primitive will have the same underlying distribution. The motion segments are then abstracted into a stochastic model representation and automatically clustered and organized. As new motion patterns are observed, they are incrementally grouped together into a tree structure, based on their relative distance in the model space. The tree leaves, which represent the most specialized learned motion primitives, are then passed back to the segmentation algorithm so that as the number of known motion primitives increases, the accuracy of the segmentation can also be improved. The combined algorithm is tested on a sequence of continuous human motion data that are obtained through motion capture, and demonstrates the performance of the proposed approach.

Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation

Body movements communicate affective expressions and, in recent years, computational models have been developed to recognize affective expressions from body movements or to generate movements for virtual agents or robots which convey affective expressions. This survey summarizes the state of the art on automatic recognition and generation of such movements. For both automatic recognition and generation, important aspects such as the movements analyzed, the affective state representation used, and the use of notation systems is discussed. The survey concludes with an outline of open problems and directions for future work.

Real-time safety for human–robot interaction☆

This paper presents a strategy for ensuring safety during human-robot interaction in real time. A measure of danger during the interaction is explicitly computed, based on factors affecting the impact force during a potential collision between the human and the robot This danger index is then used as an input to real-time trajectory generation when the index exceeds a predefined threshold. The danger index is formulated to produce stable motion in the presence of multiple surrounding obstacles. A motion strategy to minimize the danger index is developed for articulated multi degree of freedom robots. Simulations and experiments demonstrate the efficacy of this approach

Online Segmentation of Human Motion for Automated Rehabilitation Exercise Analysis

To enable automated analysis of rehabilitation movements, an approach for accurately identifying and segmenting movement repetitions is required. This paper proposes an approach for online, automated segmentation and identification of movement segments from continuous time-series data of human movement, obtained from body-mounted inertial measurement units or from motion capture data. The proposed approach uses a two-stage identification and recognition process, based on velocity features and stochastic modeling of each motion to be identified. In the first stage, motion segment candidates are identified based on a characteristic sequence of velocity features such as velocity peaks and zero velocity crossings. In the second stage, hidden Markov models are used to accurately identify segment locations from the identified candidates. The proposed approach is capable of online segmentation and identification, enabling interactive feedback in rehabilitation applications. The approach is validated on 20 healthy subjects and four rehabilitation patients performing rehabilitation movements, achieving segmentation accuracy of 87% with user specific templates and 79%-83% accuracy with user-independent templates.

Human pose recovery using wireless inertial measurement units

Many applications in rehabilitation and sports training require the assessment of the patients status based on observation of their movement. Small wireless sensors, such as accelerometers and gyroscopes, can be utilized to provide a quantitative measure of the human movement for assessment. In this paper, a kinematics-based approach is developed to estimate human leg posture and velocity from wearable sensors during the performance of typical physiotherapy and training exercises. The proposed approach uses an extended Kalman filter to estimate joint angles from accelerometer and gyroscopic data and is capable of recovering joint angles from arbitrary 3D motion. Additional joint limit constraints are implemented to reduce drift, and an automated approach is developed for estimating and adapting the process noise during online estimation. The approach is validated through a user study consisting of 20 subjects performing knee and hip rehabilitation exercises. When compared to motion capture, the approach achieves an average root-mean-square error of 4.27 cm for unconstrained motion, with an average joint error of 6.5°. The average root-mean-square error is 3.31 cm for sagittal planar motion, with an average joint error of 4.3°.

Representability of human motions by factorial hidden Markov models

This paper describes an improved methodology for human motion recognition and imitation based on factorial hidden Markov models (FHMM). Unlike conventional hidden Markov models (HMMs), FHMMs use a distributed state representation, which allows for more efficient representation of each time sequence. Once the FHMMs are trained with exemplar motion data, they can be used to generate sample trajectories for motion production, and produce significantly more accurate trajectories compared to single Hidden Markov chain models. Due to the additional information encoded in FHMMs models, FHMM models have a higher Kullback-Leibler distance compared to single Markov chain models, making it easier to distinguish between similar models. The efficacy of using FHMMs is tested on a database of human motions obtained through motion capture. The results show that FHMMs provide better generalization to new data when compared to conventional HMMs during motion recognition, as well as providing a better fit for generated data.

Anxiety detection during human-robot interaction

This paper describes an experiment to determine the feasibility of using physiological signals to determine the human response to robot motions during direct human-robot interaction. A robot manipulator is used to generate common interaction motions, and human subjects are asked to report their response to the motions. The human physiological response is also measured. Motion paths are generated using a classic potential field planner and a safe motion planner, which minimizes the potential collision force along the path. A fuzzy inference engine is developed to estimate the human response based on the physiological measures. Results show that emotional arousal can be detected using physiological signals and the inference engine. Comparison of initial results between the two planners shows that subjects report less anxiety and surprise with the safe planner for high planner speeds.