Kai Uwe Barthel

Interactive video search tools: a detailed analysis of the video browser showdown 2015

Interactive video retrieval tools developed over the past few years are emerging as powerful alternatives to automatic retrieval approaches by giving the user more control as well as more responsibilities. Current research tries to identify the best combinations of image, audio and text features that combined with innovative UI design maximize the tools performance. We present the last installment of the Video Browser Showdown 2015 which was held in conjunction with the International Conference on MultiMedia Modeling 2015 (MMM 2015) and has the stated aim of pushing for a better integration of the user into the search process. The setup of the competition including the used dataset and the presented tasks as well as the participating tools will be introduced . The performance of those tools will be thoroughly presented and analyzed. Interesting highlights will be marked and some predictions regarding the research focus within the field for the near future will be made.

Navigating a Graph of Scenes for Exploring Large Video Collections

We present a novel approach to browse huge sets of video scenes using a hierarchical graph and visually sorted image maps allowing the user to explore the graph similar to navigation services. In a previous paper [1] we proposed a scheme to generate such a graph of video scenes and investigated several browsing and visualization concepts. In this paper we extend our work by adding semantic features learned from a convolutional neural network. In combination with visual features we constructed an improved graph where related images (video scenes) are connected with each other. Different images or areas in the graph may be reached by following the most promising path of edges. For efficient navigation we propose a method which projects images onto a 2D plane preserving their complex inter-image relationships. To start a search process, the user may either choose from a selection of typical videos scenes or use tools such as search by sketch or category. The retrieved video frames are arranged on a canvas and the view of the graph is directed to a location where matching frames can be found.

Graph-Based Browsing for Large Video Collections

We present a graph-based browsing system for visually searching video clips in large collections. It is an extension of a previously proposed system ImageMap which allows visual browsing in millions of images using a hierarchical pyramid structure of images sorted by their similarities. Image subsets can be explored through a viewport at different pyramid levels, however, due to the underlying 2D-organization the high dimensional relationships between all images could not be represented. In order to preserve the complex inter-image relationships we propose to use a hierarchical graph where edges connect related images. By traversing this graph the users may navigate to other similar images. Different visualization and navigation modes are available. Various filters and search tools such as search by example, color, or sketch may be applied. These tools help to narrow down the amount of video frames to be inspected or to direct the view to regions of the graph where matching frames are located.

ImageMap - Visually Browsing Millions of Images

In this paper we showcase ImageMap - an image browsing system to visually explore and search millions of images from stock photo agencies and the like. Similar to map services like Google Maps users may navigate through multiple image layers by zooming and dragging. Zooming in (or out) shows more (or less) similar images from lower (or higher) levels. Dragging the view shows related images from the same level. Layers are organized as an image pyramid which is build using image sorting and clustering techniques. Easy image navigation is achieved because the placement of the images in the pyramid is based on an improved fused similarity calculation using visual and semantic image information. Our system also allows to perform searches. After starting an image search the user is automatically directed to a region with suiting results. This paper describes how to efficiently construct an easily navigable image pyramid even if the total number of images is huge.

Visually Browsing Millions of Images Using Image Graphs

We present a new approach to visually browse very large sets of untagged images. High quality image features are generated using transformed activations of a convolutional neural network. These features are used to model image similarities, from which a hierarchical image graph is build. We show how such a graph can be constructed efficiently. In our experiments we found best user experience for navigating the graph is achieved by projecting sub-graphs onto a regular 2D image map. This allows users to explore the image collection like an interactive map.

Fusing Keyword Search and Visual Exploration for Untagged Videos

Video collections often cannot be searched by keywords because most videos are poorly annotated. We present a system that allows to search untagged videos by sketches, example images and keywords. Having analyzed the most frequent search terms and the corresponding images from the Pixabay stock photo agency we derived visual features that allow to search for 20000 keywords. For each keyword we use several image features to be able to cope with large visual and conceptual variations. As the intention of a user searching for an image is unknown, we retrieve thousands of result images (video scenes), which are shown as a visually sorted hierarchical image map. The user can easily find images of interest by dragging and zooming. The visual arrangement of the images is performed with an improved version of a self-sorting map, which allows organizing thousands of images in fractions of a second. If an image similar to the search query has been found, further zooming will show more related images, retrieved from a precomputed image graph. The new approach helps to find untagged images very quickly in an exploratory, incremental way.

ImageX - Explore and Search Local/Private Images

In this paper we present a system to visually explore and search large sets of untagged images, running on common operating systems and consumer hardware. High quality image descriptors are computed using activations of a convolutional neural network. By applying normalization and a principal component analysis of the activations compact feature vectors of only 64 bytes are generated. The L1-distances for these feature vectors can be calculated very fast using a novel computation approach and allows search-by-example queries to be processed in fractions of a second. We further show how entire image collections can be transferred into hierarchical image graphs and describe a scheme to explore this complex data structure in an intuitive way. To enable keyword search for untagged images, reference features for common keywords are generated. These features are constructed by collecting and clustering examples images from the web.

Image retrieval using collaborative filtering and visual navigation

Internet image search systems mostly use words from the context of the web page containing the image as keywords. The performance of these search systems is rather poor, as the search systems neither know the intention of the searching user nor the semantic relationships of these images. Content-based image retrieval (CBIR) systems rely on the assumption that similar images share similar visual features. Despite intense research efforts, the results of CBIR systems have not reached the performance of text based search engines. The main problem of CBIR systems is the semantic gap between the content that can be described with low-level visual features and the description of image content that humans use with high-level semantic concepts. Some image retrieval systems have combined the keyword and the content-based visual search approach. However with this approach many images may be found that semantically do not match. In addition semantically similar images that visually look different cannot be found at all.

Dynamic Construction and Manipulation of Hierarchical Quartic Image Graphs

Over the last years, we have published papers about intuitive image graph navigation and showed how to build static hierarchical image graphs efficiently. In this paper, we showcase new results and present techniques to dynamically construct and manipulate these kinds of graphs. They connect similar images and perform well in retrieving tasks regardless of the number of nodes. By applying an improved fast self-sorting map algorithm, entire image collections (structured in a graph) can be explored with a user interface resembling common navigation services.

Graph Navigation for Exploring Very Large Image Collections.

We present a new approach to visually browse very large sets of untagged images. In this paper we describe how to generate high quality image descriptors/features using transformed activations of a convolutional neural network. These features are used to model image similarities, which again are used to build a hierarchical image graph. We show how such an image graph can be constructed efficiently. After investigating several browsing and visualization concepts, we found best user experience and ease of usage is achieved by projecting sub-graphs onto a regular 2D-image map. This allows users to explore the image graph similar to navigation services.