Liliana Lo Presti

3D skeleton-based human action classification

In recent years, there has been a proliferation of works on human action classification from depth sequences. These works generally present methods and/or feature representations for the classification of actions from sequences of 3D locations of human body joints and/or other sources of data, such as depth maps and RGB videos.This survey highlights motivations and challenges of this very recent research area by presenting technologies and approaches for 3D skeleton-based action classification. The work focuses on aspects such as data pre-processing, publicly available benchmarks and commonly used accuracy measurements. Furthermore, this survey introduces a categorization of the most recent works in 3D skeleton-based action classification according to the adopted feature representation.This paper aims at being a starting point for practitioners who wish to approach the study of 3D action classification and gather insights on the main challenges to solve in this emerging field. HighlightsState of the art 3D skeleton-based action classification methods are reviewed.Methods are categorized based on the adopted feature representation.Motivations and challenges for skeleton-based action recognition are highlighted.Data pre-processing, public benchmarks and validation protocols are discussed.Comparison of renowned methods, open problems and future work are presented.

Gesture Modeling by Hanklet-Based Hidden Markov Model

In this paper we propose a novel approach for gesture modeling. We aim at decomposing a gesture into sub-trajectories that are the output of a sequence of atomic linear time invariant (LTI) systems, and we use a Hidden Markov Model to model the transitions from the LTI system to another. For this purpose, we represent the human body motion in a temporal window as a set of body joint trajectories that we assume are the output of an LTI system. We describe the set of trajectories in a temporal window by the corresponding Hankel matrix (Hanklet), which embeds the observability matrix of the LTI system that produced it. We train a set of HMMs (one for each gesture class) with a discriminative approach. To account for the sharing of body motion templates we allow the HMMs to share the same state space. We demonstrate by means of experiments on two publicly available datasets that, even with just considering the trajectories of the 3D joints, our method achieves state-of-the-art accuracy while competing well with methods that employ more complex models and feature representations.

Hankelet-based dynamical systems modeling for 3D action recognition

This paper proposes to model an action as the output of a sequence of atomic Linear Time Invariant (LTI) systems. The sequence of LTI systems generating the action is modeled as a Markov chain, where a Hidden Markov Model (HMM) is used to model the transition from one atomic LTI system to another. In turn, the LTI systems are represented in terms of their Hankel matrices. For classification purposes, the parameters of a set of HMMs (one for each action class) are learned via a discriminative approach. This work proposes a novel method to learn the atomic LTI systems from training data, and analyzes in detail the action representation in terms of a sequence of Hankel matrices. Extensive evaluation of the proposed approach on two publicly available datasets demonstrates that the proposed method attains state-of-the-art accuracy in action classification from the 3D locations of body joints (skeleton). Display Omitted We model an action as sequence of outputs of linear time invariant (LTI) systems.We represent the outputs of LTI systems by means of Hankelets.We adopt an HMM to model the transitions from one LTI system to another.We formulate an inference and supervised learning formulation for our model.We also present a deep analysis of the parameter settings for our action representation.

Using Hankel matrices for dynamics-based facial emotion recognition and pain detection

This paper proposes a new approach to model the temporal dynamics of a sequence of facial expressions. To this purpose, a sequence of Face Image Descriptors (FID) is regarded as the output of a Linear Time Invariant (LTI) system. The temporal dynamics of such sequence of descriptors are represented by means of a Hankel matrix. The paper presents different strategies to compute dynamics-based representation of a sequence of FID, and reports classification accuracy values of the proposed representations within different standard classification frameworks. The representations have been validated in two very challenging application domains: emotion recognition and pain detection. Experiments on two publicly available benchmarks and comparison with state-of-the-art approaches demonstrate that the dynamics-based FID representation attains competitive performance when off-the-shelf classification tools are adopted.

Enabling technologies on hybrid camera networks for behavioral analysis of unattended indoor environments and their surroundings

This paper presents a layered network architecture and the enabling technologies for accomplishing vision-based behavioral analysis of unattended environments. Specifically the vision network covers both the attended environment and its surroundings by means of hybrid cameras. The layer overlooking at the surroundings is laid outdoor and tracks people, monitoring entrance/exit points. It recovers the geometry of the site under surveillance and communicates people positions to a higher level layer. The layer monitoring the unattended environment undertakes similar goals, with the addition of maintaining a global mosaic of the observed scene for further understanding. Moreover, it merges information coming from sensors beyond the vision to deepen the understanding or increase the reliability of the system. The behavioral analysis is demanded to a third layer that merges the information received from the two other layers and infers knowledge about what happened, happens and will be likely happening in the environment. The paper also describes a case study that was implemented in the Engineering Campus of the University of Modena and Reggio Emilia, where our surveillance system has been deployed in a computer laboratory which was often unaccessible due to lack of attendance.

Ensemble of Hankel Matrices for Face Emotion Recognition

In this paper, a face emotion is considered as the result of the composition of multiple concurrent signals, each corresponding to the movements of a specific facial muscle. These concurrent signals are represented by means of a set of multi-scale appearance features that might be correlated with one or more concurrent signals. The extraction of these appearance features from a sequence of face images yields to a set of time series. This paper proposes to use the dynamics regulating each appearance feature time series to recognize among different face emotions. To this purpose, an ensemble of Hankel matrices corresponding to the extracted time series is used for emotion classification within a framework that combines nearest neighbor and a majority vote schema. Experimental results on a public available dataset show that the adopted representation is promising and yields state-of-the-art accuracy in emotion classification.

A data association approach to detect and organize people in personal photo collections

In this paper we present a method to automatically segment a photo sequence in groups containing the same persons. Many methods in literature accomplish to this task by adopting clustering techniques. We model the problem as the search for probable associations between faces detected in subsequent photos considering the mutual exclusivity constraint: a person can not be in a photo two times, nor two faces in the same photo can be assigned to the same group. Associations have been found considering face and clothing descriptions. In particular, a two level architecture has been adopted: at the first level, associations are computed within meaningful temporal windows (situations); at the second level, the resulting clusters are re-processed to find associations across situations. Experiments confirm our technique generally outperforms clustering methods. We present an analysis of the results on a public dataset, enabling future comparison, and on private collections.

Hankelet-based action classification for motor intention recognition

Powered lower-limb prostheses require a natural, and an easy-to-use, interface for communicating amputee’s motor intention in order to select the appropriate motor program in any given context, or simply to commute from active (powered) to passive mode of functioning. To be widely accepted, such an interface should not put additional cognitive load at the end-user, it should be reliable and minimally invasive. In this paper we present a one such interface based on a robust method for detecting and recognizing motor actions from a low-cost wearable sensor network mounted on a sound leg providing inertial (accelerometer, gyrometer and magnetometer) data in real-time. We assume that the sensor measurement trajectories – in a given temporal window – can be represented as the output of a linear time invariant system. We describe such set of trajectories via a Hankel matrix, which embeds the observability matrix of the LTI system generating the set of trajectories. The use of Hankel matrices (known as Hankelets) avoids the burden of performing system identification while providing a computationally convenient descriptor for the dynamics of a time-series. For the recognition of actions, we use two off-the-shelf classifiers, nearest neighbor (NN) and support vector machines (SVM), in cross-subject validation. We present results using either the joint angles or the raw sensor data showing a net improvement of the Hankelet-based approach against a baseline method. In addition, we compare results on action recognition using joint angles provided by trakSTAR, a high-accuracy motion tracking unit, demonstrating – somewhat surprisingly – that best results (in terms of average recognition accuracy over different actions) are provided by raw inertial data, paving the way towards a wider usage of our method in the field of active prosthetics, in particular, and motor intention recognition, in general.

Boosting Hankel matrices for face emotion recognition and pain detection

HighligthsDynamics of face expression descriptors are modeled for emotion recognition.A set of Hankel matrices is built upon several multi-scale face representations.Boosting and random subspace projection are used for dynamics selection.Dynamics of Haar-like features and Gabor Energies are compared.Fine-grained dynamics of subtle expressions can be modeled at small spatial scales. Studies in psychology have shown that the dynamics of emotional expressions play an important role in face emotion recognition in humans. Motivated by these studies, in this paper the dynamics of face expressions are modeled and used for automatic emotion recognition and pain detection.Given a temporal sequence of face images, several appearance-based descriptors are computed at each frame. Over the sequence, the descriptors corresponding to the same feature type and spatial scale define a time series. The Hankel matrix built upon each time series is used to represent the dynamics of face expressions with respect to the used feature-scale pair.The set of Hankel matrices obtained by varying the feature type and the scale is used within a boosting approach to train a strong classifier. During training, random subspace projection is adopted for feature and scale selection.Experiments on two challenging publicly available datasets show that the dynamics of appearance-based face expression representations can be used to discriminate among different emotion classes and, within a boosting approach, attain state-of-the-art average accuracy values in classification.

A Decisional Multi-Agent Framework for Automatic Supply Chain Arrangement

In this work, a multi-agent system (MAS) for supply chain dynamic configuration is proposed. The brain of each agent is composed of a Bayesian Decision Network (BDN); this choice allows the agent for taking the best decisions estimating benefits and potential risks of different strategies, analyzing and managing uncertain information about the collaborating companies. Each agent collects information about customer’s orders and current market prices, and analyzes previous experiences of collaborations with trading partners. The agent therefore performs a probabilistic inferential reasoning to filter information modeled in its knowledge base in order to achieve the best performance in the supply chain organization.