Fuzzy Integral-Based CNN Classifier Fusion for 3D Skeleton Action Recognition

Abstract

Action recognition based on skeleton key joints has gained popularity due to its cost effectiveness and low complexity. Existing Convolutional Neural Network (CNN) based models mostly fail to capture various aspects of the skeleton sequence. To this end, four feature representations, which capture complementary characteristics of the sequence of key joints, are extracted with novel contribution of features estimated from angular information, and kinematics of the human actions. Single channel grayscale images are used to encode these features for classification using four CNNs, with the complementary nature verified through Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences. As opposed to straightforward classifier combination generally used in existing literature, fuzzy fusion through the Choquet integral leverages the degree of uncertainty of decision scores obtained from four CNNs. Experimental results support the efficacy of fuzzy combination of CNNs to adaptively generate final decision score based upon confidence of each information source. Impressive results on the challenging UTD-MHAD, HDM05, G3D, and NTU RGB+D 60 and 120 datasets demonstrate the effectiveness of the proposed method. The source code for our method is available at https://github.com/theavicaster/fuzzy-integral-cnn-fusion-3d-har