Joe Kilner

Objective Quality Assessment in Free-Viewpoint Video Production

This paper addresses the problem of objectively measuring quality in free-viewpoint video production. The accuracy of scene reconstruction is typically limited and an evaluation of free-viewpoint video should explicitly consider the quality of image production. A simple objective measure of accuracy is presented in terms of structural registration error in view synthesis. This technique can be applied as a full-reference metric to measure the fidelity of view synthesis to a ground truth image or as a no-reference metric to measure the error in registering scene appearance in image-based rendering. The metric is applied to a data-set with known geometric accuracy and a comparison is also demonstrated between two free-viewpoint video techniques across two prototype production studios.

Robust graph-cut scene segmentation and reconstruction for free-viewpoint video of complex dynamic scenes

Current state-of-the-art image-based scene reconstruction techniques are capable of generating high-fidelity 3D models when used under controlled capture conditions. However, they are often inadequate when used in more challenging outdoor environments with moving cameras. In this case, algorithms must be able to cope with relatively large calibration and segmentation errors as well as input images separated by a wide-baseline and possibly captured at different resolutions. In this paper, we propose a technique which, under these challenging conditions, is able to efficiently compute a high-quality scene representation via graph-cut optimisation of an energy function combining multiple image cues with strong priors. Robustness is achieved by jointly optimising scene segmentation and multiple view reconstruction in a view-dependent manner with respect to each input camera. Joint optimisation prevents propagation of errors from segmentation to reconstruction as is often the case with sequential approaches. View-dependent processing increases tolerance to errors in on-the-fly calibration compared to global approaches. We evaluate our technique in the case of challenging outdoor sports scenes captured with manually operated broadcast cameras and demonstrate its suitability for high-quality free-viewpoint video.

A Robust Free-Viewpoint Video System for Sport Scenes

This contribution describes robust methods to provide a free-viewpoint video visualisation of sport scenes using a multi-camera set-up. This allows generation of novel views of actions from any angle and is of interest for visualisation in TV productions. The system utilises 3D reconstruction techniques previously developed for studio use. This paper discusses some experiences found while applying these techniques for an uncontrolled outdoor environment and addresses robustness issues. This includes segmentation, camera calibration and 3D reconstruction. A number of different 3D representations, including billboards, visual hulls and view-dependent geometry are evaluated for the purpose.

A Free-Viewpoint Video Renderer

Free-viewpoint video allows interactive control of the camera viewpoint in video playback using a three-dimensional (3D) scene representation. This paper describes the challenges and the state-of-the-art techniques adopted in developing a free-viewpoint renderer for 3D video production of people. The system requires video streams from a set of fixed or dynamic real-world video cameras and synthesizes arbitrary viewpoints on the GPU using a dynamic 3D scene model. We have released the renderer as an open source application for the research community (see the web information at the end of this paper). The renderer provides (i) the facility to simulate camera viewpoints from publicly available multiple-view video datasets and (ii) a baseline technique for free-viewpoint video synthesis in the development of interactive 3D video and 3DTV applications.

A Bayesian Framework for Simultaneous Matting and 3D Reconstruction

Conventional approaches to 3D scene reconstruction often treat matting and reconstruction as two separate problems, with matting a prerequisite to reconstruction. The problem with such an approach is that it requires taking irreversible decisions at the first stage, which may translate into reconstruction errors at the second stage. In this paper, we propose an approach which attempts to solve both problems jointly, thereby avoiding this limitation. A general Bayesian formulation for estimating opacity and depth with respect to a reference camera is developed. In addition, it is demonstrated that in the special case of binary opacity values (background/foreground) and discrete depth values, a global solution can be obtained via a single graph-cut computation. We demonstrate the application of the method to novel view synthesis in the case of a large-scale outdoor scene. An experimental comparison with a two-stage approach based on chroma-keying and shape-from-silhouette illustrates the advantages of the new method.

Objective quality assessment in free-viewpoint video production

This paper addresses the problem of objectively measuring quality in free-viewpoint video production. The accuracy of scene reconstruction is typically limited and an evaluation of free-viewpoint video should explicitly consider the quality of image production. A simple objective measure of accuracy is presented in terms of structural registration error in view synthesis. This technique can be applied as a full-reference metric to measure the fidelity of view synthesis to a ground truth image or as a no-reference metric to measure the error in registering scene appearance in image-based rendering. The metric is applied to a data-set with known geometric accuracy and a comparison is also demonstrated between two free-viewpoint video techniques across two prototype production studios.

3D action matching with key-pose detection

This paper addresses the problem of human action matching in outdoor sports broadcast environments, by analysing 3D data from a recorded human activity and retrieving the most appropriate proxy action from a motion capture library. Typically pose recognition is carried out using images from a single camera, however this approach is sensitive to occlusions and restricted fields of view, both of which are common in the outdoor sports environment. This paper presents a novel technique for the automatic matching of human activities which operates on the 3D data available in a multi-camera broadcast environment. Shape is retrieved using multi-camera techniques to generate a 3D representation of the scene. Use of 3D data renders the system camera-pose-invariant and allows it to work while cameras are moving and zooming. By comparing the reconstructions to an appropriate 3D library, action matching can be achieved in the presence of significant calibration and matting errors which cause traditional pose detection schemes to fail. An appropriate feature descriptor and distance metric are presented as well as a technique to use these features for key-pose detection and action matching. The technique is then applied to real footage captured at an outdoor sporting event.

Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events

Generating free-viewpoint video in outdoor sports environments is currently an unsolved problem due to difficulties in obtaining accurate background segmentation and camera calibration. This paper introduces a technique for the reconstruction of a scene in the presence of these errors. We tackle the issues of reconstruction completeness, and accuracy of surface shape and appearance. We introduce the concept of the conservative visual hull as a technique to improve reconstruction completeness. We then present a view-dependent surface optimisation technique using deformable models to improve surface shape and appearance. We contribute a novel dual-mode snake algorithm that is robust to noise and demonstrates reduced dependence on parameterisation by separating the search of the solution space from the data fitting. We conclude by presenting results of this technique along with a quantitative evaluation against other reconstruction techniques using a leave-one- out data set.

A free-viewpoint video system for visualization of sport scenes

This paper introduces a new approach to using multicamera images of an event to provide a free-viewpoint video visualization of sport scenes. This allows a virtual camera to be moved freely around, and to view the action from any angle. The system is based on 3-D reconstruction techniques previously developed for studio use. A number of different 3-D representations, including billboards, visual hulls, and view-dependent geometry are evaluated for the purpose. First results show the potential of the new approach for action replay and strategy analysis of sport scenes. Current limitations of the 3-D representations are discussed in the context of a practical use of the system.

Free-Viewpoint Video for TV Sport Production

Free-viewpoint video in sports TV production presents a challenging problem involving the conflicting requirements of broadcast picture quality with video-rate generation of novel views, together with practical problems in developing robust systems for cost effective deployment at live events.To date most multiple view video systems have been developed for studio applications with a fixed capture volume, controlled illumination and backgrounds. Live outdoor events such as sports present a number of additional challenges for both acquisition and processing. Multiple view capture systems in sports such as football must cover the action taking place over an entire pitch with video acquisition at sufficient resolution for analysis and production of desired virtual camera views.In this chapter we identify the requirements for broadcast production of free-viewpoint video in sports and review the state-of-the-art. We present the iview free-viewpoint video system which enables production of novel camera views of sports action for use in match commentary, for example the referees viewpoint. Automatic online calibration, segmentation and reconstruction is performed to allow rendering of novel viewpoints from the moving match cameras.Results are reported of production trials in football (soccer) and rugby which demonstrate free-viewpoint video with a visual quality comparable to the broadcast footage.