Hugh Denman

Joint audio visual retrieval for tennis broadcasts

In recent years, there has been increasing work in the area of content retrieval for sports. The idea is generally to extract important events or create summaries to allow personalisation of the media stream. While previous work in sports analysis has employed either the audio or video stream to achieve some goal, there is little work that explores how much can be achieved by combining the two streams. This paper combines both audio and image features to identify the key episode in tennis broadcasts. The image feature is based on image moments and is able to capture the essence of scene geometry without recourse to 3D modelling. The audio feature uses PCA to identify the sound of the ball hitting the racket. The features are modelled as stochastic processes and the work combines the features using a likelihood approach. The results show that combining the features yields a much more robust system than using the features separately.

Content-based analysis for video from snooker broadcasts

This paper presents three new tools appropriate for content analysis of sports, applied to footage from snooker broadcasts in particular. The first tool is a new feature for parsing a sequence based on geometry without the need for deriving 3D information. The second tool allows events to be detected where an event is characterised by an object leaving the scene at a particular location. The final tool is a mechanism for summarising motion in a shot for use in a content-based summary. As a matter of course, the paper considers a number of enabling techniques such as the removal of irrelevant objects and object tracking using a particle filter. The paper shows that by exploiting context, a convincing summary can be made for snooker footage.

Gradient based dominant motion estimation with integral projections for real time video stabilisation

This paper presents a new expression of the relationship between integral projections and motion in an image pair. The resulting new multiresolution gradient based approach is used to estimate dominant motion in image sequences degraded by random shake. The paper also describes an implementation using the GPU as a coprocessor for the CPU that allows, for the first time, real time video stabilisation in software on broadcast standard definition television images.

Content Based Analysis for Video from Snooker Broadcasts

This paper presents the tools in a system for creating a semantically meaningful summary of broadcast Snooker footage. The system parses the sequence, identifies relevant camera views and tracks ball movement. Two new algorithms for video retrieval are presented. The first is the use of the 2nd order moment of the Hough Transform of the image for sequence parsing. The second method is the use of enclosed regions for detecting the disappearance of objects. The moment feature captures the geometry of the scene without extracting the 3D scene geometry. It is expected that this new feature is applicable to most sports in which a playing area is delineated by field lines.

Multipass encoding for reducing pulsing artifacts in cloud based video transcoding

Low latency video transcoding is an important feature in video sharing platforms. Typically this is achieved by splitting a clip into short segments, followed by parallel encoding of the segments. However, this introduces quality artifacts due to transients in encoder rate control. We present a model for more effectively predicting the behavior of the encoder in each encoding pass, which minimizes this problem. We learn the model using measurements from more than 500 video clips, hence ensuring reliable estimation. Finally, we show that the proposed multipass strategy delivers more stable reconstructed quality for parallel video transcoding frameworks.

Temporal synchronization of multiple audio signals

Given the proliferation of consumer media recording devices, events often give rise to a large number of recordings. These recordings are taken from different spatial positions and do not have reliable timestamp information. In this paper, we present two robust graph-based approaches for synchronizing multiple audio signals. The graphs are constructed atop the over-determined system resulting from pairwise signal comparison using cross-correlation of audio features. The first approach uses a Minimum Spanning Tree (MST) technique, while the second uses Belief Propagation (BP) to solve the system. Both approaches can provide excellent solutions and robustness to pairwise outliers, however the MST approach is much less complex than BP. In addition, an experimental comparison of audio features-based synchronization shows that spectral flatness outperforms the zero-crossing rate and signal energy.

Content based access for a massive database of human observation video

We present in this paper a CBIR system for use in a psychological study of the relationship between human movement and Dyslexia. The system allows access to up to 500 hours of video and is an example of a scientific user context. This user context requires 100% accurate indexing and retrieval for a set of specific queries. This paper presents a novel use of interactive visual and audio cues for attaining this level of indexing performance. Furthermore, the issue of motion estimation accuracy in the presence of compression artifacts is explored as part of the data integrity storage problem. In addition, content based motion analysis techniques accurate enough to parse sequences on the basis of motion and objectively evaluate that motion are investigated. The tool allows Psychologists to undertake a study that would previously be impractical and the paper presents a number of lessons gained from the ongoing work

Measuring noise correlation for improved video denoising

The vast majority of previous work in noise reduction for visual media has assumed uncorrelated, white, noise sources. In practice this is almost always violated by real media. Film grain noise is never white, and this paper highlights that the same applies to almost all consumer video content. We therefore present an algorithm for measuring the spatial and temporal spectral density of noise in archived video content, be it consumer digital camera or film orginated. As an example of how this information can be used for video denoising, the spectral density is then used for spatio-temporal noise reduction in the Fourier frequency domain. Results show improved performance for noise reduction in an easily pipelined system.

Exploiting temporal discontinuities for event detection and manipulation in video streams

Discontinuities in any information bearing signal serve to represent much of the vital or interesting content in that signal. A sharp loud noise in a movie could be a gun, or something breaking. In sports like tennis, cricket or snooker/pool it would indicate a point scoring event. In both cases the discontinuity is likely to be semantically relevant without further inference being necessary, once a particular domain is adopted. This paper discusses the importance of temporal motion discontinuities in inferring events in visual media. Two particular application domains are considered: content based audio/video synchronisation and event spotting in observational Psychology.

The subjective importance of noise spectral content

This paper presents secondary Standard Quality Scale (SQS2) rankings in overall quality JNDs for a subjective analysis of the 3 axes of noise, amplitude, spectral content, and noise type, based on the ISO 20462 softcopy ruler protocol. For the initial pilot study, a Python noise simulation model was created to generate the matrix of noise masks for the softcopy ruler base images with different levels of noise, different low pass filter noise bandwidths and different band pass filter center frequencies, and 3 different types of noise: luma only, chroma only, and luma and chroma combined. Based on the lessons learned, the full subjective experiment, involving 27 observers from Google, NVIDIA and STMicroelectronics was modified to incorporate a wider set of base image scenes, and the removal of band pass filtered noise masks to ease observer fatigue. Good correlation was observed with the Aptina subjective noise study. The absence of tone mapping in the noise simulation model visibly reduced the contrast at high levels of noise, due to the clipping of the high levels of noise near black and white. Under the 34-inch viewing distance, no significant difference was found between the luma only noise masks and the combined luma and chroma noise masks. This was not the intuitive expectation. Two of the base images with large uniform areas, ‘restaurant’ and ‘no parking’, were found to be consistently more sensitive to noise than the texture rich scenes. Two key conclusions are (1) there are fundamentally different sensitivities to noise on a flat patch versus noise in real images and (2) magnification of an image accentuates visual noise in a way that is non-representative of typical noise reduction algorithms generating the same output frequency. Analysis of our experimental noise masks applied to a synthetic Macbeth ColorChecker Chart confirmed the color-dependent nature of the visibility of luma and chroma noise.