Reede Ren

Integrating facial expressions into user profiling for the improvement of a multimodal recommender system

Over the years, recommender systems have been systematically applied in both industry and academia to assist users in dealing with information overload. One of the factors that determine the performance of a recommender system is user feedback, which has been traditionally communicated through the application of explicit and implicit feedback techniques. In this paper, we propose a novel video search interface that predicts the topical relevance of a video by analysing affective aspects of user behaviour. We, furthermore, present a method for incorporating such affective features into user profiling, to facilitate the generation of meaningful recommendations, of unseen videos. Our experiment shows that multimodal interaction feature is a promising way to improve the performance of recommendation.

Attention-based video summarisation in rushes collection

This paper presents the framework of a general video summarisation system on the rushes collection, which formalises the summarisation process as an 0-1 Knapsack optimisation problem. Three stages are included, namely content analysis, content selection and summary composition. Content analysis is the pre-processing step, consisting of shot segmentation, feature extraction, raw video discrimination and shot clustering. Content selection weights the importance of video segments by an attention model. A greedy approximation approach is employed in the composition of summary video with the cost function, which balances the video importance gain and the duration cost. The average content coverage achieved on the rushes test collection is about 29%, while the average qualification score on readability is 3.13 with the redundancy credit at 4.08.

Visual Sentences for Pose Retrieval Over Low-Resolution Cross-Media Dance Collections

We describe a system for matching human posture (pose) across a large cross-media archive of dance footage spanning nearly 100 years, comprising digitized photographs and videos of rehearsals and performances. This footage presents unique challenges due to its age, quality and diversity. We propose a forest-like pose representation combining visual structure (self-similarity) descriptors over multiple scales, without explicitly detecting limb positions which would be infeasible for our data. We explore two complementary multi-scale representations, applying passage retrieval and latent Dirichlet allocation (LDA) techniques inspired by the text retrieval domain, to the problem of pose matching. The result is a robust system capable of quickly searching large cross-media collections for similarity to a visually specified query pose. We evaluate over a cross-section of the UK National Research Centre for Dances (UK-NRCD), and the Siobhan Davies Replays (SDR) digital dance archives, using visual queries supplied by dance professionals. We demonstrate significant performance improvements over two base-lines: classical single and multi-scale bag of visual words (BoVW) and spatial pyramid kernel (SPK) matching .

A BOVW based query generative model

Bag-of-visual words (BOVW) is a local feature based framework for content-based image and video retrieval. Its performance relies on the discriminative power of visual vocabulary, i.e. the cluster set on local features. However, the optimisation of visual vocabulary is of a high complexity in a large collection. This paper aims to relax such a dependence by adapting the query generative model to BOVW based retrieval. Local features are directly projected onto latent content topics to create effective visual queries; visual word distributions are learnt around local features to estimate the contribution of a visual word to a query topic; the relevance is justified by considering concept distributions on visual words as well as on local features. Massive experiments are carried out the TRECVid 2009 collection. The notable improvement on retrieval performance shows that this probabilistic framework alleviates the problem of visual ambiguity and is able to afford visual vocabulary with relatively low discriminative power.

General Highlight Detection in Sport Videos

Attention is a psychological measurement of human reflection against stimulus. We propose a general framework of highlight detection by comparing attention intensity during the watching of sports videos. Three steps are involved: adaptive selection on salient features, unified attention estimation and highlight identification. Adaptive selection computes feature correlation to decide an optimal set of salient features. Unified estimation combines these features by the technique of multi-resolution auto-regressive (MAR) and thus creates a temporal curve of attention intensity. We rank the intensity of attention to discriminate boundaries of highlights. Such a framework alleviates semantic uncertainty around sport highlights and leads to an efficient and effective highlight detection. The advantages are as follows: (1) the capability of using data at coarse temporal resolutions; (2) the robustness against noise caused by modality asynchronism, perception uncertainty and feature mismatch; (3) the employment of Markovian constrains on content presentation, and (4) multi-resolution estimation on attention intensity, which enables the precise allocation of event boundaries.

Video redundancy detection in rushes collection

The rushes is a collection of raw material videos. There are various redundancies, such as rainbow screen, clipboard shot, white/black view, and unnecessary re-take. This paper develops a set of solutions to remove these video redundancies as well as an effective system for video summarisation. We regard manual editing effects, e.g. clipboard shots, as differentiators in the visual language. A rushes video is therefore divided into a group of subsequences, each of which stands for a re-take instance. A graph matching algorithm is proposed to estimate the similarity between re-takes and suggests the best instance for content presentation. The experiments on the Rushes 2008 collection show that a video can be shortened to 4%-16% of the original size by redundancy detection. This significantly reduces the complexity in content selection and leads to an effective and efficient video summarisation system.

Semantic based adaptive movie summarisation

This paper proposes a framework for automatic video summarization by exploiting internal and external textual descriptions. The web knowledge base Wikipedia is used as a middle media layer, which bridges the gap between general user descriptions and exact film subtitles. Latent Dirichlet Allocation (LDA) detects as well as matches the distribution of content topics in Wikipedia items and movie subtitles. A saliency based summarization system then selects perceptually attractive segments from each content topic for summary composition. The evaluation collection consists of six English movies and a high topic coverage is shown over official trails from the Internet Movie Database.

Feature subspace selection for efficient video retrieval

The curse of dimensionality is a major issue in video indexing. Extremely high dimensional feature space seriously degrades the efficiency and the effectiveness of video retrieval. In this paper, we exploit the characteristics of document relevance and propose a statistical approach to learn an effective sub feature space from a multimedia document collection. This involves four steps: (1) density based feature term extraction, (2) factor analysis, (3) bi-clustering and (4) communality based component selection. Discrete feature terms are a set of feature clusters which smooth feature distribution in order to enhance the discrimination power; factor analysis tries to depict correlation between different feature dimensions in a loading matrix; bi-clustering groups both components and factors in the factor loading matrix and selects feature components from each bi-cluster according to the communality. We have conducted extensive comparative video retrieval experiments on the TRECVid 2006 collection. Significant performance improvements are shown over the baseline, PCA based K-mean clustering.

Temporal salient graph for sports event detection

A graphic representation is proposed to facilitate the analysis of temporal saliency in a video document. This representation visualizes temporal saliency at multiple resolutions and introduces efficient image operations to the computationally intensive task of saliency analysis. Algorithms for graphic-based saliency fusion and attended area detection are presented. We evaluate the effectiveness by the application of general highlight detection in football videos. The experimental collection includes six full games from FIFA World Cup and European Champion.

Query Generation from Multiple Media Examples

This paper exploits a media document representation called feature terms to generate a query from multiple media examples, e.g. images. A feature term denotes a continuous interval of a media feature dimension. This approach (1) helps feature accumulation from multiple examples; (2) enables the exploration of text-based retrieval models for multimedia retrieval. Three criteria, minimised χ2, minimised AC/DC and maximised entropy, are proposed to optimise feature term selection. Two ranking functions, KL divergence and BM25, are used for relevance estimation. Experiments on Corel photo collection and TRECVid 2006 collection show the effectiveness in image/video retrieval.