Ahmad Jalal

A Depth Video Sensor-Based Life-Logging Human Activity Recognition System for Elderly Care in Smart Indoor Environments

Recent advancements in depth video sensors technologies have made human activity recognition (HAR) realizable for elderly monitoring applications. Although conventional HAR utilizes RGB video sensors, HAR could be greatly improved with depth video sensors which produce depth or distance information. In this paper, a depth-based life logging HAR system is designed to recognize the daily activities of elderly people and turn these environments into an intelligent living space. Initially, a depth imaging sensor is used to capture depth silhouettes. Based on these silhouettes, human skeletons with joint information are produced which are further used for activity recognition and generating their life logs. The life-logging system is divided into two processes. Firstly, the training system includes data collection using a depth camera, feature extraction and training for each activity via Hidden Markov Models. Secondly, after training, the recognition engine starts to recognize the learned activities and produces life logs. The system was evaluated using life logging features against principal component and independent component features and achieved satisfactory recognition rates against the conventional approaches. Experiments conducted on the smart indoor activity datasets and the MSRDailyActivity3D dataset show promising results. The proposed system is directly applicable to any elderly monitoring system, such as monitoring healthcare problems for elderly people, or examining the indoor activities of people at home, office or hospital.

Real-time life logging via a depth silhouette-based human activity recognition system for smart home services

A real-time life logging system that provide monitoring, recording and recognition of daily human activities using video cameras offers life-care or health-care services at smart homes. Such a vision-based life logging system can provide continuous monitoring and recording of a residents daily activities from which one can obtain behavior patterns of daily life events and improve the quality of life especially for the elderly. This paper presents a real-time life logging system via depth imaging-based human activity recognition. A depth imaging device is utilized to obtain depth silhouettes of human activities. Then from the silhouettes, human body points information gets extracted and used in activity recognition, producing life logs. The system is composed of two key processes; one is training of the life logging system, and the other is running the trained life-logging system to record life logs. In the training process, the system includes the data collection from a depth camera, extraction of body points features from each depth silhouette and finally training of the activity recognizer (i.e., Hidden Markov Models). Then, after training, one can run the trained system which recognizes learned activities and store life logs in real-time. The proposed approach is evaluated against the life logging system that uses the conventional principal components (PC) and Radon transform features of depth silhouettes and achieves superior recognition rate. Real-time experimental results show the feasibility and functionality of the implemented system which could be used to generate life logs of human activities at smart homes.

Robust human activity recognition from depth video using spatiotemporal multi-fused features

Abstract The recently developed depth imaging technologies have provided new directions for human activity recognition (HAR) without attaching optical markers or any other motion sensors to human body parts. In this paper, we propose novel multi-fused features for online human activity recognition (HAR) system that recognizes human activities from continuous sequences of depth map. The proposed online HAR system segments human depth silhouettes using temporal human motion information as well as it obtains human skeleton joints using spatiotemporal human body information. Then, it extracts the spatiotemporal multi-fused features that concatenate four skeleton joint features and one body shape feature. Skeleton joint features include the torso-based distance feature (DT), the key joint-based distance feature (DK), the spatiotemporal magnitude feature (M) and the spatiotemporal directional angle feature (θ). The body shape feature called HOG-DDS represents the projections of the depth differential silhouettes (DDS) between two consecutive frames onto three orthogonal planes by the histogram of oriented gradients (HOG) format. The size of the proposed spatiotemporal multi-fused feature is reduced by a code vector in the code book which is generated by vector quantization method. Then, it trains the hidden Markov model (HMM) with the code vectors of the multi-fused features and recognizes the segmented human activity by the forward spotting scheme using the trained HMM-based human activity classifiers. The experimental results on three challenging depth video datasets such as IM-DailyDepthActivity, MSRAction3D and MSRDailyActivity3D demonstrate that the proposed online HAR method using the proposed multi-fused features outperforms the state-of-the-art HAR methods in terms of recognition accuracy.

Ridge body parts features for human pose estimation and recognition from RGB-D video data

This paper addresses the issues of 3D human pose estimation, tracking and recognition from RGB-D video sequences using a generative structured framework. Most existing approaches focus on these issues using discriminative models. However, a discriminative model has certain drawbacks: a) it requires expensive training steps and large amount of training samples for covering inherently wide pose space, and (b) not suitable for real-time applications due to its slow algorithmic inferences. In this work, a real-time tracking system has been proposed for human pose recognition utilizing ridge body parts features. Initially, depth silhouettes extract ridge data inside the binary edges and initialize each body joints information using predefined pose. Then, body parts tracking incorporates appearance learning to handle occlusions and manage body joints features. Lastly, Support Vector Machine is used to recognize different poses. Experimental results demonstrate that the proposed method is reliable and efficient in recognizing human poses at different realistic scenes.

Dense RGB-D Map-Based Human Tracking and Activity Recognition using Skin Joints Features and Self-Organizing Map

This paper addresses the issues of 3D human activity detection, tracking and recognition from RGB-D video sequences using a feature structured framework. During human tracking and activity recognition, initially, dense depth images are captured using depth camera. In order to track human silhouettes, we considered spatial/temporal continuity, constraints of human motion information and compute centroids of each activity based on chain coding mechanism and centroids point extraction. In body skin joints features, we estimate human body skin color to identify human body parts (i.e., head, hands, and feet) likely to extract joint points information. These joints points are further processed as feature extraction process including distance position features and centroid distance features. Lastly, self-organized maps are used to recognize different activities. Experimental results demonstrate that the proposed method is reliable and efficient in recognizing human poses at different realistic scenes. The proposed system should be applicable to different consumer application systems such as healthcare system, video surveillance system and indoor monitoring systems which track and recognize different activities of multiple users.

Depth map-based human activity tracking and recognition using body joints features and Self-Organized Map

In this paper, we implement human activity tracking and recognition system utilizing body joints features using depth maps. During HAR settings, depth maps are processed to track human silhouettes by considering temporal continuity constraints of human motion information and compute centroids for each activity based on contour generation. In body joints features, depth silhouettes are computed first through geodesic distance to identify anatomical landmarks which produce joint points information from specific body parts. Then, body joints are processed to produce centroid distance features and key joints distance features. Finally, Self-Organized Map (SOM) is employed to train and recognize different human activities from the features. Experimental results show that body joints features achieved high recognition rate over the conventional features. The proposed system should be applicable as e-healthcare systems for monitoring elderly people, surveillance systems for observing pedestrian traffic areas and indoor environment systems which recognize activities of multiple users.

Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data.

This paper addresses the problem of automatic detection, tracking and recognition of three-dimensional human poses from monocular depth video sequences for machine vision applications. In this paper, we present a real-time tracking system for body parts pose recognition utilizing ridge data of depth maps. At first, the depth maps are processed to extract features by considering ridge data surrounded by binary edges silhouettes acting as skeleton shape of human body. Then, the pose estimation is applied to initialize each body parts having joint points information using predefined pose. For body part tracking, all features (i.e., ridge data or depth values) are extracted according to a continuously updated torso-center, head and body part joint points. This help to provide the estimation of 3D body joint angles using the forward kinematic analysis. Our experimental results believe that the proposed method is reliable and efficient for tracking and recognizing the exact skeleton for even dynamic scenes and complex human pose.

Shape and Motion Features Approach for Activity Tracking and Recognition from Kinect Video Camera

Recent development in depth sensors opens up new challenging task in the field of computer vision research areas, including human-computer interaction, computer games and surveillance systems. This paper addresses shape and motion features approach to observe, track and recognize human silhouettes using a sequence of RGB-D images. Under our proposed activity recognition framework, the required procedure includes: detecting human silhouettes from the image sequence, we remove noisy effects from background and track human silhouettes using temporal continuity constraints of human motion information for each activity, extracting the shape and motion features to identify richer motion information and then these features are clustered and fed into Hidden Markov Model (HMM) to train, model and recognize human activities based on transition and emission probabilities values. During experimental results, we demonstrate this approach on two challenging depth video datasets: one based on our own annotated database and other based on public database (i.e., MSRAction3D). Our approach shows significant recognition results over the state of the art algorithms.

Depth Images-based Human Detection, Tracking and Activity Recognition Using Spatiotemporal Features and Modified HMM

Human activity recognition using depth information is an emerging and challenging technology in computer vision due to its considerable attention by many practical applications such as smart home/office system, personal health care and 3D video games. This paper presents a novel framework of 3D human body detection, tracking and recognition from depth video sequences using spatiotemporal features and modified HMM. To detect human silhouette, raw depth data is examined to extract human silhouette by considering spatial continuity and constraints of human motion information. While, frame differentiation is used to track human movements. Features extraction mechanism consists of spatial depth shape features and temporal joints features are used to improve classification performance. Both of these features are fused together to recognize different activities using the modified hidden Markov model (M-HMM). The proposed approach is evaluated on two challenging depth video datasets. Moreover, our system has significant abilities to handle subjects body parts rotation and body parts missing which provide major contributions in human activity recognition.

Human daily activity recognition with joints plus body features representation using Kinect sensor

Human activity recognition has been studied actively from decades using a sequence of 2D images/video. With the development of depth sensors, new opportunities arise to improve and advance this field. This study presents a depth imaging activity recognition system to monitor and recognize daily activities of the human without attaching optical markers or motion sensors. In this paper, we proposed a new feature representation and extraction method using a sequence of depth silhouettes. Particularly, we first extract the depth silhouette by removing background from noisy effects and then extract the joints plus body features as skin color detection from joint information and multi-view body shape from depth silhouettes (i.e., front and side views). We combine the joints plus body shape features to make feature vector. These features have two nice properties including invariant with respect to body shape or size and insensitive to small noise. Self-Organized Map (SOM) is then used to train and test the feature vectors. Experimental results regarding our proposed human activity dataset and publically available dataset demonstrate that our feature extraction method is more promising and outperforms the state-of-the-art feature extraction methods.