gki Byun

Explicit performance metric optimization for fusion-based video retrieval

We present a learning framework for fusion-based video retrieval system, which explicitly optimizes given performance metrics. Real-world computer vision systems serve sophisticated user needs, and domain-specific performance metrics are used to monitor the success of such systems. However, the conventional approach for learning under such circumstances is to blindly minimize standard error rates and hope the targeted performance metrics improve, which is clearly suboptimal. In this work, a novel scheme to directly optimize such targeted performance metrics during learning is developed and presented. Our experimental results on two large consumer video archives are promising and showcase the benefits of the proposed approach.

A detection-based approach to broadcast news video story segmentation

A detection-based paradigm decomposes a complex system into small pieces, solves each subproblem one by one, and combines the collected evidence to obtain a final solution. In this study of video story segmentation, a set of key events are first detected from heterogeneous multimedia signal sources, including a large scale concept ontology for images, text generated from automatic speech recognition systems, features extracted from audio track, and high-level video transcriptions. Then a discriminative evidence fusion scheme is investigated. We use the maximum figure-of-merit learning approach to directly optimize the performance metrics used in system evaluation, such as precision, recall, and F1 measure. Some experimental evaluations conducted on the TRECVID 2003 dataset demonstrate the effectiveness of the proposed detection-based paradigm. The proposed framework facilitates flexible combination and extensions of event detector design and evidence fusion to enable other related video applications.

Honest signals in video conferencing

We propose a novel system to analyze gestural and nonverbal cues of participants in video conferencing. These cues have previously been referred to as “honest signals” and are usually associated with the underlying cognitive state of the participants. The presented system analyzes a set of audio-visual, non-linguistic features in real time from the audio and video streams of two participants in a video conference. We show how these features can be used to compute indicators of the overall quality and type of conversation being held. The system also provides visual feedback to the participants, who then have the choice of modifying their conversational style in order to achieve the desired outcome of the video conference. Experiments on real-life data show that the system can predict the type of conversation with high accuracy using the non-linguistic signals only. Qualitative user studies highlight the positive effects of increased awareness amongst the participants about their own gestural and non-verbal cues.

An experimental study on discriminative concept classifier combination for TRECVID high-level feature extraction

In this paper, we present an experimental study on using high-dimensional image features to perform discriminative classifier combination for TRECVID concept detection. We combine a multi-class classifier with binary-class classifiers. After training a multi-class classifier, we train binary-class classifiers by decomposing a multi-class problem into several binary-class classification problems, and fuse them together using a discriminative classifier combination approach. This idea leverages on each classifiers properties; multi-class classifiers emphasize on segmenting a decision space optimally in terms of some overall performance criteria whereas binary classifiers focus on detecting corresponding positive samples locally. Testing on the TRECVID2005 development set with 39 LSCOM-Lite concepts by adding an additional set of 39 pairs of binary concept classifiers, the mean average precision was improved by 34.1% over our baseline system with only 39 multi-class concept classifiers. When compared with state-of-the-art systems our proposed method is quite competitive especially for concepts with a relatively small number of positive samples.

An incremental learning framework combining sample confidence and discrimination with an application to automatic image annotation

We propose an incremental classifier learning framework that starts with a small amount of labeled training data to create an initial set of classifiers, and gradually incorporates unlabeled data into the incremental learning process to improve the models. A key to the effectiveness of the proposed framework is to judicially select a good incremental learning subset from all remaining unlabeled samples by computing a confidence measure and a margin-like discrimination score that measures potential contributions of the selection set to enhancing the existing models. To further refine the above selection set, class prior densities were also exploited. The proposed framework was tested on an automatic image annotation application using a subset of the Corel image set. When all data, including both initially labeled and incrementally learned samples, were used, the final model was shown to achieve a significant improvement over the initial set of classifiers in terms of micro-averaging F1 even when only a small number of images were initially labeled. Furthermore, when 30% of the images were initially labeled the incrementally learned models achieved comparable results to the case when models were created with all training data labeled.

A kernelized maximal-figure-of-merit learning approach based on subspace distance minimization

We propose a kernelized maximal-figure-of-merit (MFoM) learning approach to efficiently training a nonlinear model using subspace distance minimization. In particular, a fixed, small number of training samples are chosen in a way that the distance between function spaces constructed with a subset of training samples and with the entire training data set is minimized. This construction of the subset enables us to learn a nonlinear model efficiently while keeping the resulting model nearly optimal compared to the model from the whole training data set. We show that the subspace distance can be minimized through the Nyström extension. Experimental results on various machine learning problems demonstrate clear advantages of the proposed technique over the case where the function space is built with randomly selected training samples. Additional comparisons with the model trained with the entire training samples show that the proposed technique achieves comparable results while reducing training time tremendously.