Shayok Chakraborty

Multimodal emotion recognition using deep learning architectures

Emotion analysis and recognition has become an interesting topic of research among the computer vision research community. In this paper, we first present the emoF-BVP database of multimodal (face, body gesture, voice and physiological signals) recordings of actors enacting various expressions of emotions. The database consists of audio and video sequences of actors displaying three different intensities of expressions of 23 different emotions along with facial feature tracking, skeletal tracking and the corresponding physiological data. Next, we describe four deep belief network (DBN) models and show that these models generate robust multimodal features for emotion classification in an unsupervised manner. Our experimental results show that the DBN models perform better than the state of the art methods for emotion recognition. Finally, we propose convolutional deep belief network (CDBN) models that learn salient multimodal features of expressions of emotions. Our CDBN models give better recognition accuracies when recognizing low intensity or subtle expressions of emotions when compared to state of the art methods.

Generalized batch mode active learning for face-based biometric recognition

Against the backdrop of growing concerns about security, face-based biometrics has emerged as a methodology to reliably infer human identity. Active learning algorithms automatically select appropriate data samples to train a classifier and reduce human effort in annotating data instances. In this work, a novel optimization based batch mode active learning strategy has been applied to face recognition. The flexibility of the framework is corroborated by its ability to incorporate additional available information. Our results on the VidTIMIT and the NIST MBGC datasets certify the potential of this method in being used for real world biometric applications. Highlights? A generalized scheme for Batch Mode Active Learning in face-based biometrics is presented. ? The framework can incorporate additional information available in biometric contexts. ? Additional variants of the algorithm are described. ? Results show the merit of the framework in selecting instances for manual annotation.

Deep Hashing Network for Unsupervised Domain Adaptation

In recent years, deep neural networks have emerged as a dominant machine learning tool for a wide variety of application domains. However, training a deep neural network requires a large amount of labeled data, which is an expensive process in terms of time, labor and human expertise. Domain adaptation or transfer learning algorithms address this challenge by leveraging labeled data in a different, but related source domain, to develop a model for the target domain. Further, the explosive growth of digital data has posed a fundamental challenge concerning its storage and retrieval. Due to its storage and retrieval efficiency, recent years have witnessed a wide application of hashing in a variety of computer vision applications. In this paper, we first introduce a new dataset, Office-Home, to evaluate domain adaptation algorithms. The dataset contains images of a variety of everyday objects from multiple domains. We then propose a novel deep learning framework that can exploit labeled source data and unlabeled target data to learn informative hash codes, to accurately classify unseen target data. To the best of our knowledge, this is the first research effort to exploit the feature learning capabilities of deep neural networks to learn representative hash codes to address the domain adaptation problem. Our extensive empirical studies on multiple transfer tasks corroborate the usefulness of the framework in learning efficient hash codes which outperform existing competitive baselines for unsupervised domain adaptation.

Generalized Query by Transduction for online active learning

Transductive inference has gained popularity in recent years as a means to develop pattern classification approaches that address the specific issue of predicting the class label of a given data point, instead of the more general problem of inferring the ideal classifier function. In this paper, we propose a Generalized Query by Transduction (GQBT) approach for active learning in the online setting. This approach is based on the theory of conformal predictions, which has recently been proposed based on principles of algorithmic randomness, transductive inference and hypothesis testing. The proposed GQBT approach can be used along with any existing pattern classification algorithm, and can also be used to combine multiple criteria in selecting an unlabeled example appropriately in the active learning process. The results of our experiments with different datasets from the UCI Machine Learning repository demonstrate high promise in the proposed approach, with significantly lower label complexities than other existing online active learning approaches. The GQBT approach was also evaluated on face recognition using videos from the VidTIMIT dataset, and the observed superior performance supports the potential of applicability of the proposed approach in real-world problems.

Adaptive Batch Mode Active Learning

Active learning techniques have gained popularity to reduce human effort in labeling data instances for inducing a classifier. When faced with large amounts of unlabeled data, such algorithms automatically identify the exemplar and representative instances to be selected for manual annotation. More recently, there have been attempts toward a batch mode form of active learning, where a batch of data points is simultaneously selected from an unlabeled set. Real-world applications require adaptive approaches for batch selection in active learning, depending on the complexity of the data stream in question. However, the existing work in this field has primarily focused on static or heuristic batch size selection. In this paper, we propose two novel optimization-based frameworks for adaptive batch mode active learning (BMAL), where the batch size as well as the selection criteria are combined in a single formulation. We exploit gradient-descent-based optimization strategies as well as properties of submodular functions to derive the adaptive BMAL algorithms. The solution procedures have the same computational complexity as existing state-of-the-art static BMAL techniques. Our empirical results on the widely used VidTIMIT and the mobile biometric (MOBIO) data sets portray the efficacy of the proposed frameworks and also certify the potential of these approaches in being used for real-world biometric recognition applications.

Optimal batch selection for active learning in multi-label classification

Multi-label classification is a generalization of conventional classification, where it is possible for a single data point to have multiple labels. Manual annotation of a multi-label data point requires a human oracle to consider the presence/absence of every possible class separately, which involves significant labor. Active learning techniques are effective in reducing human labeling effort to induce a classification model. When exposed to large quantities of unlabeled data, such algorithms automatically select the salient and representative instances for manual annotation. Further, to address the high redundancy in data such as image or video sequences as well as the availability of multiple labeling agents, there have been recent attempts towards a batch mode form of active learning, where a batch of data points is selected simultaneously from an unlabeled set. In this work, we propose a novel optimization based batch mode active learning strategy to minimize human labeling effort in multi-label classification problems. To the best of our knowledge, this is the first attempt to develop such a scheme primarily intended for the multi-label context. The proposed framework is computationally simple, easy to implement and can be suitably modified to perform batch mode active learning in other formulations, such as single-label classification or problems involving hierarchical label spaces. Our results corroborate the efficacy of the proposed algorithm and certify the potential of the framework in being used for real world applications.

Social Interaction Assistant: A Person-Centered Approach to Enrich Social Interactions for Individuals With Visual Impairments

Social interaction is a central component of human experience. The ability to interact with others and communicate effectively within an interactive context is a fundamental necessity for professional success as well as personal fulfillment. Individuals with visual impairment face significant challenges in social communication, which if unmitigated, may lead to lifelong needs for extensive social and economic support. Unfortunately, todays multimedia technologies largely cater to the needs of the “able” population, resulting in solutions that mostly meet the needs of that community. Individuals with disabilities (such as visual impairment) have largely been absent in the design process, and have to adapt themselves (often unsuccessfully) to available solutions. In this paper, we propose a social interaction assistant for individuals who are blind or visually impaired, incorporating novel contributions in: 1) person recognition through batch mode active learning; 2) reliable multimodal person recognition through the conformal predictions framework; and 3) facial expression recognition through topic models. Moreover, individuals with visual impairments often have specific requirements that necessitate a personalized, adaptive approach to multimedia computing. To address this challenge, our proposed solutions place emphasis on understanding the individual users needs, expectations and adaptations toward designing, and developing and deploying effective multimedia solutions. Our empirical results demonstrate the significant potential in using person centered multimedia solutions to enrich the lives of individuals with disabilities.

Dynamic batch mode active learning

Active learning techniques have gained popularity to reduce human effort in labeling data instances for inducing a classifier. When faced with large amounts of unlabeled data, such algorithms automatically identify the exemplar and representative instances to be selected for manual annotation. More recently, there have been attempts towards a batch mode form of active learning, where a batch of data points is simultaneously selected from an unlabeled set. Real-world applications require adaptive approaches for batch selection in active learning. However, existing work in this field has primarily been heuristic and static. In this work, we propose a novel optimization-based framework for dynamic batch mode active learning, where the batch size as well as the selection criteria are combined in a single formulation. The solution procedure has the same computational complexity as existing state-of-the-art static batch mode active learning techniques. Our results on four challenging biometric datasets portray the efficacy of the proposed framework and also certify the potential of this approach in being used for real world biometric recognition applications.

Kernel Learning for Efficiency Maximization in the Conformal Predictions Framework

The Conformal Predictions framework is a recent development in machine learning to associate reliable measures of confidence with results in classification and regression. This framework is founded on the principles of algorithmic randomness (Kolmogorov complexity), transductive inference and hypothesis testing. While the formulation of the framework guarantees validity, the efficiency of the framework depends greatly on the choice of the classifier and appropriate kernel functions or parameters. While this framework has extensive potential to be useful in several applications, the lack of efficiency can limit its usability. In this paper, we propose a novel kernel learning methodology to maximize efficiency in the CP framework. This method is validated using the k-Nearest Neighbors classifier on three different datasets, and our results show immense promise in applying this method to obtain efficient conformal predictors that can be practically useful.

Dynamic Batch Size Selection for Batch Mode Active Learning in Biometrics

Robust biometric recognition is of paramount importance in security and surveillance applications. In face based biometric systems, data is usually collected using a video camera with high frame rate and thus the captured data has high redundancy. Selecting the appropriate instances from this data to update a classification model, is a significant, yet valuable challenge. Active learning methods have gained popularity in identifying the salient and exemplar data instances from superfluous sets. Batch mode active learning schemes attempt to select a batch of samples simultaneously rather than updating the model after selecting every single data point. Existing work on batch mode active learning assume a fixed batch size, which is not a practical assumption in biometric recognition applications. In this paper, we propose a novel framework to dynamically select the batch size using clustering based unsupervised learning techniques. We also present a batch mode active learning strategy specially suited to handle the high redundancy in biometric datasets. The results obtained on the challenging VidTIMIT and MOBIO datasets corroborate the superiority of dynamic batch size selection over static batch size and also certify the potential of the proposed active learning scheme in being used for real world biometric recognition applications.