Gunther Heidemann

State of the Art Report on Video-Based Graphics and Video Visualization

In recent years, a collection of new techniques which deal with video as input data, emerged in computer graphics and visualization. In this survey, we report the state of the art in video‐based graphics and video visualization. We provide a review of techniques for making photo‐realistic or artistic computer‐generated imagery from videos, as well as methods for creating summary and/or abstract visual representations to reveal important features and events in videos. We provide a new taxonomy to categorize the concepts and techniques in this newly emerged body of knowledge. To support this review, we also give a concise overview of the major advances in automated video analysis, as some techniques in this field (e.g. feature extraction, detection, tracking and so on) have been featured in video‐based modelling and rendering pipelines for graphics and visualization.

Inter-active learning of ad-hoc classifiers for video visual analytics

Learning of classifiers to be used as filters within the analytical reasoning process leads to new and aggravates existing challenges. Such classifiers are typically trained ad-hoc, with tight time constraints that affect the amount and the quality of annotation data and, thus, also the users trust in the classifier trained. We approach the challenges of ad-hoc training by inter-active learning, which extends active learning by integrating human experts background knowledge to greater extent. In contrast to active learning, not only does inter-active learning include the users expertise by posing queries of data instances for labeling, but it also supports the users in comprehending the classifier model by visualization. Besides the annotation of manually or automatically selected data instances, users are empowered to directly adjust complex classifier models. Therefore, our model visualization facilitates the detection and correction of inconsistencies between the classifier model trained by examples and the users mental model of the class definition. Visual feedback of the training process helps the users assess the performance of the classifier and, thus, build up trust in the filter created. We demonstrate the capabilities of inter-active learning in the domain of video visual analytics and compare its performance with the results of random sampling and uncertainty sampling of training sets.

Evaluation of Multi-view 3D Reconstruction Software

A number of software solutions for reconstructing 3D models from multi-view image sets have been released in recent years. Based on an unordered collection of photographs, most of these solutions extract 3D models using structure-from-motion SFM algorithms. In this work, we compare the resulting 3D models qualitatively and quantitatively. To achieve these objectives, we have developed different methods of comparison for all software solutions. We discuss the perfomance and existing drawbacks. Particular attention is paid to the ability to create printable 3D models or 3D models usable for other applications.

A Survey on Video-based Graphics and Video Visualization

In recent years, a collection of new techniques, which deal with videos a s the input data, emerged in computer graphics and visualization. In this survey, we report the state of the art in video-based graphics and video visualization. We provide a comprehensive review of techniques for making photo- realistic or artistic computer-generated imagery from videos, as well as methods for creating summary and/or abs tract visual representations to reveal important features and events in videos. We propose a new taxonomy to ca tegorize the concepts and techniques in this newly-emerged body of knowledge. To support this review, we als o give a concise overview of the major advances in automated video analysis, as some techniques in this field (e.g ., feature extraction, detection, tracking and so on) have been featured in video-based modeling and rendering p ipelines for graphics and visualization.

Uncertainty-aware video visual analytics of tracked moving objects

Vast amounts of video data render manual video analysis useless while recent automatic video analytics techniques suffer from insufficient performance. To alleviate these issues, we present a scalable and reliable approach exploiting the visual analytics methodology. This involves the user in the iterative process of exploration, hypotheses generation, and their verification. Scalability is achieved by interactive filter definitions on trajectory features extracted by the automatic computer vision stage. We establish the interface between user and machine adopting the VideoPerpetuoGram (VPG) for visual- ization and enable users to provide filter-based relevance feedback. Additionally, users are supported in deriving hypotheses by context-sensitive statistical graphics. To allow for reliable decision making, we gather uncertainties introduced by the computer vision step, communicate these information to users through uncertainty visualization, and grant fuzzy hypothesis formulation to interact with the machine. Finally, we demonstrate the effectiveness of our approach by the video analysis mini challenge which was part of the IEEE Symposium on Visual Analytics Science and Technology 2009.

Interactive Schematic Summaries for Faceted Exploration of Surveillance Video

We present a scalable technique to explore surveillance videos by scatter/gather browsing of trajectories of moving objects. Trajectories are clustered according to a variety of properties, such as location, orientation, and velocity that can be selected by the users. These properties allow for faceted video exploration and refinement of previous browsing steps. The proposed approach facilitates interactive clustering of trajectories by an effective way of cluster visualization that we term schematic summaries. This novel visualization illustrates cluster summaries in a schematic, nonphotorealistic style. To reduce visual clutter, we introduce the trajectory bundling technique. Further, schematic summaries include a timeline view and a showcase view to represent the facets present in a cluster. The fusion of schematic summaries, a variety of facets, and user interaction lead to efficient hierarchical exploration of video data. Examples of different browsing scenarios and initial user feedback demonstrate the potentials of our method.

Efficient annotation of image data sets for computer vision applications

High quality ground truth data sets are crucial for the development of image recognition systems. However, the task of annotating large image data sets manually takes a lot of time and effort. In order to lower the burden for the development of application-specific image recognition systems, we developed an advanced user interface. This interface is especially designed for non-expert users with little-to-no knowledge of computer vision techniques. The interface presents images clustered by similarity and allows for an efficient and simple annotation of large data sets. The integration of overview+detail concepts allows the precise navigation inside large data sets. The interface can be used without prior instructions on the underlying concepts, like self-organizing maps, image features or visualization techniques.

Evaluation of Fast-Forward Video Visualization

We evaluate and compare video visualization techniques based on fast-forward. A controlled laboratory user study (n = 24) was conducted to determine the trade-off between support of object identification and motion perception, two properties that have to be considered when choosing a particular fast-forward visualization. We compare four different visualizations: two representing the state-of-the-art and two new variants of visualization introduced in this paper. The two state-of-the-art methods we consider are frame-skipping and temporal blending of successive frames. Our object trail visualization leverages a combination of frame-skipping and temporal blending, whereas predictive trajectory visualization supports motion perception by augmenting the video frames with an arrow that indicates the future object trajectory. Our hypothesis was that each of the state-of-the-art methods satisfies just one of the goals: support of object identification or motion perception. Thus, they represent both ends of the visualization design. The key findings of the evaluation are that object trail visualization supports object identification, whereas predictive trajectory visualization is most useful for motion perception. However, frame-skipping surprisingly exhibits reasonable performance for both tasks. Furthermore, we evaluate the subjective performance of three different playback speed visualizations for adaptive fast-forward, a subdomain of video fast-forward.

Semi-automatic ground truth annotation in videos: An interactive tool for polygon-based object annotation and segmentation

Knowledge extraction from video data is challenging due to its high complexity in both the spatial and temporal domain. Ground truth is crucial for the evaluation and the adaptation of algorithms to new domains. Unfortunately, ground truth annotation is inconvenient and time consuming. Common annotation tools mostly rely on simple geometric primitives such as rectangles or ellipses. Here we propose a novel, interactive and semi-automatic process, which actively asks for user input if the result of the automatic annotation appears to be incorrect. After a brief review of related tools for video annotation, we explain our proposed semi-automatic method iSeg using a prototype implementation. iSeg has been tested on two visual stimulus datasets for eye tracking experiments and on two surveillance datasets. The experimental results and the usability are compared to existing annotation tools. Finally, we discuss the properties and opportunities of polygon-based video annotation.

Scalable video visual analytics

Video visual analytics is the research field that addresses scalable and reliable analysis of video data. The vast amount of video data in typical analysis tasks renders manual analysis by watching the video data impractical. However, automatic evaluation of video material is not reliable enough, especially when it comes to semantic abstraction from the video signal. In this article, we describe the video visual analytics method that combines the complementary strengths of human recognition and machine processing. After inspecting the challenges of scalable video analysis, we derive the main components of visual analytics for video data. Based on these components, we present our video visual analytics system that has its origins in our IEEE VAST Challenge 2009 participation.