Multiple Time Scale Motion Images for Action Recognition

Abstract

This paper proposes a simple and effective approach for RGB-based action recognition using multiple streams Convolutional Neural Networks (ConvNets). We utilize a new method to represent temporal structure of RGB videos, named Motion Image (MI), which is constructed from the difference between frames with a certain time scale. Considering the different duration of different actions, multiple time scale sampling MIs can obtain more temporal information. Furthermore, we adopt multiple streams ConvNets, including MIs and RGB streams, to learn spatial-temporal features for action recognition. Our approach has been evaluated on UCF-101 and HMDB-51, and the experimental results demonstrate the effectiveness and significantly improve action recognition rate at a small computational cost.