Andre F. de Araújo

Efficient video search using image queries

We study the challenges of image-based retrieval when the database consists of videos. This variation of visual search is important for a broad range of applications that require indexing video databases based on their visual contents. We present new solutions to reduce storage requirements, while at the same time improving video search quality. The video database is preprocessed to find different appearances of the same visual elements, and build robust descriptors. Compression algorithms are developed to reduce systems storage requirements. We introduce a dataset of CNN broadcasts and queries that include photos taken with mobile phones and images of objects. Our experiments include pairwise matching and retrieval scenarios. We demonstrate one order of magnitude storage reduction and search quality improvements of up to 12% in mean average precision, compared to a baseline system that does not make use of our techniques.

Temporal aggregation for large-scale query-by-image video retrieval

We address the challenge of using image queries to retrieve video clips from a large database. Using binarized Fisher Vectors as global signatures, we present three novel contributions. First, an asymmetric comparison scheme for binarized Fisher Vectors is shown to boost retrieval performance by 0.27 mean Average Precision, exploiting the fact that query images contain much less clutter than database videos. Second, aggregation of frame-based local features over shots is shown to achieve retrieval performance comparable to aggregation of those local features over single frames, while reducing retrieval latency and memory requirements by more than 3X. Several shot aggregation strategies are compared and results indicate that most perform equally well. Third, aggregation over scenes, in combination with shot signatures, is shown to achieve one order of magnitude faster retrieval at comparable performance. Scene aggregation also outperforms the recently proposed aggregation in random groups.

Stanford I2V: a news video dataset for query-by-image experiments

Reproducible research in the area of visual search depends on the availability of large annotated datasets. In this paper, we address the problem of querying a video database by images that might share some contents with one or more video clips. We present a new large dataset, called Stanford I2V. We have collected more than 3; 800 hours of newscast videos and annotated more than 200 ground-truth queries. In the following, the dataset is described in detail, the collection methodology is outlined and retrieval performance for a benchmark algorithm is presented. These results may serve as a baseline for future research and provide an example of the intended use of the Stanford I2V dataset. The dataset can be downloaded at http://purl.stanford.edu/zx935qw7203.

Interframe Coding of Global Image Signatures for Mobile Augmented Reality

For mobile augmented reality, an image captured by a mobile devices camera is often compared against a database hosted on a remote server to recognize objects in the image. It is critically important that the amount of data transmitted over the network is as small as possible to reduce the system latency. A low bitrate global signature for still images has been previously shown to achieve high-accuracy image retrieval. In this paper, we develop new methods for interframe coding of a continuous stream of global signatures that can reduce the bitrate by nearly two orders of magnitude compared to independent coding of these global signatures, while achieving the same or better image retrieval accuracy. The global signatures are constructed in an embedded data structure that offers rate scalability. The usage of these new coding methods and the embedded data structure allows the streaming of high-quality global signatures at a bitrate that is less than 2 kbps. Furthermore, a statistical analysis of the retrieval and coding performance is performed to understand the trade off between bitrate and image retrieval accuracy and explain why interframe coding of global signatures substantially outperforms independent coding.

Large-Scale Video Retrieval Using Image Queries

Retrieving videos from large repositories using image queries is important for many applications, such as brand monitoring or content linking. We introduce a new retrieval architecture, in which the image query can be compared directly with database videos—significantly improving retrieval scalability compared with a baseline system that searches the database on a video frame level. Matching an image to a video is an inherently asymmetric problem. We propose an asymmetric comparison technique for Fisher vectors and systematically explore query or database items with varying amounts of clutter, showing the benefits of the proposed technique. We then propose novel video descriptors that can be compared directly with image descriptors. We start by constructing Fisher vectors for video segments, by exploring different aggregation techniques. For a database of lecture videos, such methods obtain a two orders of magnitude compression gain with respect to a frame-based scheme, with no loss in retrieval accuracy. Then, we consider the design of video descriptors, which combine Fisher embedding with hashing techniques, in a flexible framework based on Bloom filters. Large-scale experiments using three datasets show that this technique enables faster and more memory-efficient retrieval, compared with a frame-based method, with similar accuracy. The proposed techniques are further compared against pre-trained convolutional neural network features, outperforming them on three datasets by a substantial margin.

Eigennews: Generating and delivering personalized news video

Next-generation news video consumption will be more personalized, device agnostic, and pooled from many different sources. The EigenNews system provides each viewer with a personalized newscast filled with stories that matter most to them. While personalized text-based news services already exist, personalized video news streams are still a critically missing technology. Our system records multiple channels, segments each news stream into individual stories, discovers links between the stories themselves and links between the stories and online articles, and generates a personalized playlist for each user. The dynamic mixing and aggregation of broadcast news videos from multiple sources greatly enriches the news watching experience by providing more comprehensive coverage and varying perspectives. Over time, the website collects click and view histories and uses this information to continuously optimize the personalization.

Compression of VQM features for low bit-rate video quality monitoring

Reduced reference video quality assessment techniques provide a practical and convenient way of evaluating the quality of a processed video. In this paper, we propose a method to efficiently compress standardized VQM (Video Quality Model) [1] features to bit-rates that are small relative to the transmitted video. This is achieved through two stages of compression. In the first stage, we remove the redundancy in the features by only transmitting the necessary original video features at the lowest acceptable resolution for the calculation of the final VQM value. The second stage involves using the features of the processed video at the receiver as side-information for efficient entropy coding and reconstruction of the original video features. Experimental results demonstrate that our approach achieves high compression ratios of more than 30× with small error in the final VQM values.

Real-time query-by-image video search system

We demonstrate a novel multimedia system that continuously indexes videos and enables real-time search using images, with a broad range of potential applications. Television shows are recorded and indexed continuously, and iconic images from recent events are discovered automatically. Users can query an uploaded image or an image in the web. When a result is served, the user can play the video clip from the beginning or from the point in time where the retrieved image was found.

Analysis of visual similarity in news videos with robust and memory-efficient image retrieval

Many large collections of news videos dating back several decades can now be accessed online. For users to easily retrieve a compilation of stories on a particular event/topic and to quickly sample each story clip, all the news videos must be precisely segmented into stories and a representative video summary must be generated for each story. In this paper, we demonstrate that effectively exploiting the visual similarities pervasive in all news videos can greatly help to fulfill these technical requirements and thus enable the dynamic retrieval and mixing of small news video fragments. Two new algorithms are developed to accurately detect two important sources of visual similarity: (1) similar preview and story frames, and (2) repeated appearances of a news anchor. As a result, valuable sources of preview clips and informative clues about story boundaries are obtained from identification of these visual similarities. The retrieval engine implemented in both algorithms employs compact global image signatures and requires a small memory footprint, so that many instances of the detection algorithms can run concurrently on the same server for fast processing of a large collection of news videos. At the same time, the retrieval engine is robust to the large appearance variations encountered in the preview matching and anchor detection problems. In addition, since the video frames color information is not required in our algorithms, both modern color and vintage black-and-white news footage can be processed in the same framework.

EigenNews: a personalized news video delivery platform

We demonstrate EigenNews, a personalized television news system. Upon visiting the EigenNews website, a user is shown a variety of news videos which have been automatically selected based on her individual preferences. These videos are extracted from 16 continually recorded television programs using a multimodal segmentation algorithm. Relevant metadata for each video are generated by linking videos to online news articles. Selected news videos can be watched in three different layouts and on various devices.