Hannes Fassold

A Perceptual Image Sharpness Metric Based on Local Edge Gradient Analysis

In this letter, a no-reference perceptual sharpness metric based on a statistical analysis of local edge gradients is presented. The method takes properties of the human visual system into account. Based on perceptual properties, a relationship between the extracted statistical features and the metric score is established to form a Perceptual Sharpness Index (PSI). A comparison with state-of-the-art metrics shows that the proposed method correlates highly with human perception and exhibits low computational complexity. In contrast to existing metrics, the PSI performs well for a wide range of blurriness and shows a high degree of invariance for different image contents.

Real-time Person Tracking in High-resolution Panoramic Video for Automated Broadcast Production

For enabling immersive user experiences for interactive TV services and automating camera view selection and framing, knowledge of the location of persons in a scene is essential. We describe an architecture for detecting and tracking persons in high-resolution panoramic video streams, obtained from the Omni Cam, a panoramic camera stitching video streams from 6 HD resolution tiles. We use a CUDA accelerated feature point tracker, a blob detector and a CUDA HOG person detector, which are used for region tracking in each of the tiles before fusing the results for the entire panorama. In this paper we focus on the application of the HOG person detector in real-time and the speedup of the feature point tracker by porting it to NVIDIAs Fermi architecture. Evaluations indicate significant speedup for our feature point tracker implementation, enabling the entire process in a real-time system.

A real-time GPU implementation of the SIFT algorithm for large-scale video analysis tasks

The SIFT algorithm is one of the most popular feature extraction methods and therefore widely used in all sort of video analysis tasks like instance search and duplicate/ near-duplicate detection. We present an efficient GPU implementation of the SIFT descriptor extraction algorithm using CUDA. The major steps of the algorithm are presented and for each step we describe how to efficiently parallelize it massively, how to take advantage of the unique capabilities of the GPU like shared memory / texture memory and how to avoid or minimize common GPU performance pitfalls. We compare the GPU implementation with the reference CPU implementation in terms of runtime and quality and achieve a speedup factor of approximately 3 - 5 for SD and 5 - 6 for Full HD video with respect to a multi-threaded CPU implementation, allowing us to run the SIFT descriptor extraction algorithm in real-time on SD video. Furthermore, quality tests show that the GPU implementation gives the same quality as the reference CPU implementation from the HessSIFT library. We further describe the benefits of GPU-accelerated SIFT descriptor calculation for video analysis applications such as near-duplicate video detection.

Automatic freeze frame detection for video preservation

A significant amount of work in film and video preservation is dedicated to quality assessment of the content to be archived or reused out of the archive. This paper proposes automatic content analysis algorithms which reduce manual inspection time in software based preservation environments. We list the requirements for such algorithms and tools and show exemplarily on a freeze frame impairment detector how analysis algorithms need to be designed for meeting the requirements. The evaluation has shown that robustness against other impairments (noise and flickering) is an essential part of the algorithm. Successful detection of freeze frame impairments with a minimum length of three frames is achieved. The consideration of human perception is important to achieve low false detection rates. Analysis results are represented in a MPEG-7 standard compliant way. The proposed defect summary visualization tool enables efficient human exploration of visually impaired content.

Quality Analysis on Mobile Devices for Real-Time Feedback

Media capture of live events such as concerts can be improved by including user generated content, adding more perspectives and possibly covering scenes outside the scope of professional coverage. In this paper we propose methods for visual quality analysis on mobile devices, in order to provide direct feedback to the contributing user about the quality of the captured content. Thus, wasting bandwidth and battery for uploading/streaming low-quality content can be avoided. We focus on real-time quality analysis that complements information that can be obtained from other sensors e.g., stability. The proposed methods include real-time capable algorithms for sharpness, noise and over-/ underexposure which are integrated in a capture app for Android. Objective evaluation results show that our algorithms are competitive to state-of-the art quality algorithms while enabling real-time quality feedback on mobile devices.

Automated Visual Quality Analysis for Media Production

Automatic quality control for audiovisual media is an important tool in the media production process. In this paper we present tools for assessing the quality of audiovisual content in order to decide about the reusability of archive content. We first discuss automatic detectors for the common impairments noise and grain, video breakups, sharpness, image dynamics and blocking. For the efficient viewing and verification of the automatic results by an operator, three approaches for user interfaces are presented. Finally, we discuss the integration of the tools into a service oriented architecture, focusing on the recent standardization efforts by EBU and AMWAs Joint Task Force on a Framework for Interoperability of Media Services in TV Production (FIMS).

Tracking and Clustering Salient Features in Image Sequences

Many applications in media production need information about moving objects in the scene, e.g. insertion of computer-generated objects, association of sound sources to these objects or visualization of object trajectories in broadcasting. We present a GPU accelerated approach for detecting and tracking salient features in image sequences and we propose an algorithm for clustering the obtained feature point trajectories in order to obtain a motion segmentation of the set of feature trajectories. Evaluation results for both the tracking and clustering steps are presented. Finally we discuss the application of the proposed approach for associating audio sources to objects to support audio rendering for virtual sets.

A hybrid wavelet and temporal fusion algorithm for film and video denoising

Noise is an impairment which often occurs in both film and digital video and severely degrades the viewing experience of the content. In this work, we propose a two-phase algorithm for film an video denoising. In the first phase, the concept of semi-local shrinkage functions is used to effectively separate noise from image structure. In the second phase, we show how to fuse the result images of the first phase in a way which is robust to motion-compensation errors. A quantitative evaluation of the proposed algorithm on a realistic test dataset with noise of different coarseness and magnitude shows that the proposed method delivers better results than the video denoising method CVBM3D.

A GPU-accelerated two stage visual matching pipeline for image and video retrieval

We propose a two stage visual matching pipeline including a first step using VLAD signatures for filtering results, and a second step which reranks the top results using raw matching of SIFT descriptors. This enables adjusting the tradeoff between high computational cost of matching local descriptors and the insufficient accuracy of compact signatures in many application scenarios. We describe GPU accelerated extraction and matching algorithms for SIFT, which result in a speedup factor of at least 4. The VLAD filtering step reduces the number of images/frames for which the local descriptors need to be matched, thus speeding up retrieval by an additional factor of 9-10 without sacrificing mean average precision over full raw descriptor matching.

Real-time video quality analysis on mobile devices

Media capture of live events such as concerts can be improved by including user generated content. In this paper we propose methods for visual quality analysis on mobile devices, in order to provide direct feedback to the contributing user about the quality of the captured content and avoid wasting bandwidth and battery for uploading/streaming low-quality content. We focus on quality analysis that complements information we can obtain from other sensors. The proposed methods include real-time algorithms for sharpness, noise and over-/underexposure.