Vasant Manohar

Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol

Common benchmark data sets, standardized performance metrics, and baseline algorithms have demonstrated considerable impact on research and development in a variety of application domains. These resources provide both consumers and developers of technology with a common framework to objectively compare the performance of different algorithms and algorithmic improvements. In this paper, we present such a framework for evaluating object detection and tracking in video: specifically for face, text, and vehicle objects. This framework includes the source video data, ground-truth annotations (along with guidelines for annotation), performance metrics, evaluation protocols, and tools including scoring software and baseline algorithms. For each detection and tracking task and supported domain, we developed a 50-clip training set and a 50-clip test set. Each data clip is approximately 2.5 minutes long and has been completely spatially/temporally annotated at the I-frame level. Each task/domain, therefore, has an associated annotated corpus of approximately 450,000 frames. The scope of such annotation is unprecedented and was designed to begin to support the necessary quantities of data for robust machine learning approaches, as well as a statistically significant comparison of the performance of algorithms. The goal of this work was to systematically address the challenges of object detection and tracking through a common evaluation framework that permits a meaningful objective comparison of techniques, provides the research community with sufficient data for the exploration of automatic modeling techniques, encourages the incorporation of objective evaluation into the development process, and contributes useful lasting resources of a scale and magnitude that will prove to be extremely useful to the computer vision research community for years to come.

Towards macro- and micro-expression spotting in video using strain patterns

This paper presents a novel method for automatic spotting (temporal segmentation) of facial expressions in long videos comprising of continuous and changing expressions. The method utilizes the strain impacted on the facial skin due to the non-rigid motion caused during expressions. The strain magnitude is calculated using the central difference method over the robust and dense optical flow field of each subjects face. Testing has been done on 2 datasets (which includes 100 macro-expressions) and promising results have been obtained. The method is robust to several common drawbacks found in automatic facial expression segmentation including moderate in-plane and out-of-plane motion. Additionally, the method has also been modified to work with videos containing micro-expressions. Micro-expressions are detected utilizing their smaller spatial and temporal extent. A subjects face is divided in to sub-regions (mouth, cheeks, forehead, and eyes) and facial strain is calculated for each of these regions. Strain patterns in individual regions are used to identify subtle changes which facilitate the detection of micro-expressions.

Performance evaluation of text detection and tracking in video

Text detection and tracking is an important step in a video content analysis system as it brings important semantic clues which is a vital supplemental source of index information. While there has been a significant amount of research done on video text detection and tracking, there are very few works on performance evaluation of such systems. Evaluations of this nature have not been attempted because of the extensive effort required to establish a reliable ground truth even for a moderate video dataset. However, such ventures are gaining importance now. In this paper, we propose a generic method for evaluation of object detection and tracking systems in video domains where ground truth objects can be bounded by simple geometric shapes (polygons, ellipses). Two comprehensive measures, one each for detection and tracking, are proposed and substantiated to capture different aspects of the task in a single score. We choose text detection and tracking tasks to show the effectiveness of our evaluation framework. Results are presented from evaluations of existing algorithms using real world data and the metrics are shown to be effective in measuring the total accuracy of these detection and tracking algorithms.

Performance evaluation of object detection and tracking in video

The need for empirical evaluation metrics and algorithms is well acknowledged in the field of computer vision. The process leads to precise insights to understanding current technological capabilities and also helps in measuring progress. Hence designing good and meaningful performance measures is very critical. In this paper, we propose two comprehensive measures, one each for detection and tracking, for video domains where an object bounding approach to ground truthing can be followed. Thorough analysis explaining the behavior of the measures for different types of detection and tracking errors are discussed. Face detection and tracking is chosen as a prototype task where such an evaluation is relevant. Results on real data comparing existing algorithms are presented and the measures are shown to be effective in capturing the accuracy of the detection/tracking systems.

Graph Clustering-Based Ensemble Method for Handwritten Text Line Segmentation

Handwritten text line segmentation on real-world data presents significant challenges that cannot be overcome by any single technique. Given the diversity of approaches and the recent advances in ensemble-based combination for pattern recognition problems, it is possible to improve the segmentation performance by combining the outputs from different line finding methods. In this paper, we propose a novel graph clustering-based approach to combine the output of an ensemble of text line segmentation algorithms. A weighted undirected graph is constructed with nodes corresponding to connected components and edge connecting pairs of connected components. Text line segmentation is then posed as the problem of minimum cost partitioning of the nodes in the graph such that each cluster corresponds to a unique line in the document image. Experimental results on a challenging Arabic field dataset using the ensemble method shows a relative gain of 18% in the F1 score over the best individual method within the ensemble.

Facial Strain Pattern as a Soft Forensic Evidence

The success of forensic identification largely depends on the availability of strong evidence or traces that substantiate the prosecution hypothesis that a certain person is guilty of crime. In light of this, extracting subtle evidences which the criminals leave behind at the crime scene will be of valuable help to investigators. We propose a novel method of using strain pattern extracted from changing facial expressions in video as an auxiliary evidence for person identification. The strength of strain evidence is analyzed based on the increase in likelihood ratio it provides in a suspect population. Results show that strain pattern can be used as a supplementary biometric evidence in adverse operational conditions such as shadow lighting and face camouflage where pure intensity-based face recognition algorithms will fail

Finite element modeling of facial deformation in videos for computing strain pattern

We present a finite element modeling based approach to compute strain patterns caused by facial deformation during expressions in videos. A sparse motion field computed through a robust optical flow method drives the FE model. While the geometry of the model is generic, the material constants associated with an individualpsilas facial skin are learned at a coarse level sufficient for accurate strain map computation. Experimental results using the computational strategy presented in this paper emphasize the uniqueness and stability of strain maps across adverse data conditions (shadow lighting and face camouflage) making it a promising feature for image analysis tasks that can benefit from such auxiliary information.

Towards registration of temporal mammograms by finite element simulation of MR breast volumes

Performing regular mammographic screening and comparing corresponding mammograms taken from multiple views or at different times are necessary for early detection and treatment evaluation of breast cancer, which is key to successful treatment. However, mammograms taken at different times are often obtained under different compression, orientation, or body position. A temporal pair of mammograms may vary significantly due to the spatial disparities caused by the variety in acquisition environments, including 3D position of the breast, the amount of pressure applied, etc. Such disparities can be corrected through the process of temporal registration. We propose to use a 3D finite element model for temporal registration of digital mammography. In this paper, we apply patient specific 3D breast model constructed from MRI data of the patient, for cases where lesions are detectable in multiple mammographic views across time. The 3D location of the lesion in the breast model is computed through a breast deformation simulation step presented in our earlier work. Lesion correspondence is established by using a nearest neighbor approach in the uncompressed breast volume. Our experiments show that the use of a 3D finite element model for simulating and analyzing breast deformation contributes to good accuracy when matching suspicious regions in temporal mammograms.

Face recognition under camouflage and adverse illumination

This paper presents a method for face identification under adverse conditions by combining regular, frontal face images with facial strain maps using score-level fusion. Strain maps are generated by calculating the central difference method of the optical flow field obtained from each subjects face during the open mouth expression. Subjects were recorded with and without camouflage under three lighting conditions: normal lighting, low lighting, and strong shadow. Experimental results demonstrate that strain maps are a useful supplemental biométrie in all three adverse conditions, especially in the camouflage condition, where a 30% increase in rank 1 recognition is observed over a baseline PCA-based algorithm.

Audio-visual fusion using bayesian model combination for web video retrieval

Combining features from multiple, heterogeneous, audio visual sources can significantly improve retrieval performance in consumer domain videos. However, such videos often contain unrelated overlaid audio content, or have significant camera motion to reliably extract visual features. We present an approach, which overcomes errors in individual feature streams by combining classifiers trained on multiple, heterogeneous feature streams using Bayesian model combination (BAYCOM). We demonstrate our method, by combining low-level audio and visual features, for classification of a large 200 hour web video corpus. The combined models outperform any of the individual features by 10%. Further, BAYCOM consistently outperforms traditional early and late fusion methods.