Francisco Imai

Leveraging Knowledge-based Inference for Material Classification

Material classification is one of the fundamental problems for multimedia content analysis, computer vision and graphics. Existing efforts mostly focus on extracting representative visual features and training a classifier to recognize unknown materials. Compared with human visual recognition, automatic recognition cannot leverage common sense knowledge regarding material categories and contextual information such as object and scene. In this paper, we propose to first extract such knowledge on material, object and scene from heterogeneous sources, i.e. a public data set of 100 million Flickr images [13] and Bing search results. To improve the material classification task, the knowledge information is further exploited in a probabilistic inference framework. Our method is evaluated on OpenSurfaces [10], the largest public material data set which contains both visual features of physical properties as well as image context information. The quantitative evaluation demonstrates the superior performance of our proposed method.

Towards temporal adaptive representation for video action recognition

Action recognition has been one of the challenging problems in the computer vision community. Most of the recent research work in this area exploits the motion features captured by dense trajectory descriptors. On the other hand, static image classification has seen the rise of deep learning architectures, with evidence that the output of intermediate layers could be successfully employed as a low level descriptor for new learning tasks. However, the same level of classification success has not yet translated to the video domain. In this paper, we investigate to jointly combine dynamic trajectory features and static deep features that enhance the distinctiveness of the classifiers. We also propose a Temporal-Pyramid-Pooling strategy with intermediate layer deep features for improving action classification performance. Extensive empirical evaluations are provided to corroborate the effectiveness of the proposed framework on real-world untrimmed video datasets.