Recognizing Human-Object Interaction (HOI) Using Wrist-Mounted Inertial Sensors

Abstract

Detecting and recognizing objects and human-object interactions have great importance in many application areas, such as security and surveillance, robotics, and health monitoring. Over the past few years, many methods have been introduced for activity and object recognition with the help of image and video processing techniques. However, these techniques are very costly and subjected to privacy challenges. Ubiquitous sensing based on wearable sensors (i.e., accelerometer and gyroscope) provides a viable solution to these challenges. Thus, in this paper, we use the wrist-mounted wearable sensors to propose a new method for recognizing human-object interactions, which entails identifying an object, object interactions, and object users. In this aspect, we chose three objects (book, mobile, and laptop) with overall fourteen different interactions performed by 21 users and recorded their data using the wrist-mounted inertial sensors. Various time-domain features are extracted from the data and classified into objects in the first stage. After that, object-dependent interaction and user recognition are carried in the later stage. Three different classifiers are used for recognition experiments (including random forest (RF), J48, and support vector machine (SVM)). The highest F1-scores of 0.872, 0.888, and 0.876 are achieved using RF classifier for recognizing book, mobile, and laptop respectively. The best-case results demonstrate the significance of our proposed method.