Toby Howard

Accurate camera calibration for off-line, video-based augmented reality

Camera tracking is a fundamental requirement for video-based augmented reality applications. The ability to accurately calculate the intrinsic and extrinsic camera parameters for each frame of a video sequence is essential if synthetic objects are to be integrated into the image data in a believable way. In this paper, we present an accurate and reliable approach to camera calibration for off-line video-based augmented reality applications. We first describe an improved feature tracking algorithm, based on the widely used Kanade-Lucas-Tomasi tracker. Estimates of inter-frame camera motion are used to guide tracking, greatly reducing the number of incorrectly tracked features. We then present a robust hierarchical scheme that merges sub-sequences together to form a complete projective reconstruction. Finally, we describe how RANSAC-based random sampling can be applied to the problem of self-calibration, allowing for more reliable upgrades to metric geometry. Results of applying our calibration algorithms are given for both synthetic and real data.

Rapid shadow generation in real-world lighting environments

We propose a new algorithm that uses consumer-level graphics hardware to render shadows cast by synthetic objects and a real lighting environment. This has immediate benefit for interactive Augmented Reality applications, where synthetic objects must be accurately merged with real images. We show how soft shadows cast by direct and indirect illumination sources may be generated and composited into a background image at interactive rates. We describe how the sources of light (and hence shadow) affecting each point in an image can be efficiently encoded using a hierarchical shaft-based subdivision of line-space. This subdivision is then used to determine the sources of light that are occluded by synthetic objects, and we show how the contributions from these sources may be removed from a background image using facilities available on modern graphics hardware. A trade-off may be made at run-time between shadow accuracy and rendering cost, converging towards a result that is subjectively similar to that obtained using ray-tracing based differential rendering algorithms. Examples of the proposed technique are given for a variety of different lighting environments, and the visual fidelity of images generated by our algorithm is compared to both real photographs and synthetic images generated using non-real-time techniques.

Interactive reconstruction of virtual environments from video sequences

Abstract There are many real-world applications of Virtual Reality requiring the construction of complex and accurate three-dimensional models that represent real environments. In this paper, we describe a rapid and robust semi-automatic system that allows such environments to be quickly and easily built from video sequences captured with standard consumer-level digital cameras. The system combines an automatic camera calibration algorithm with an interactive model-building phase, followed by automatic extraction and synthesis of surface textures from frames of the video sequence. The capabilities of the system are illustrated using a variety of example reconstructions.

Real-time 3-D human body tracking using learnt models of behaviour

In this paper, we introduce a 3-D human-body tracker capable of handling fast and complex motions in real-time. We build upon the Monte-Carlo Bayesian framework, and propose novel prediction and evaluation methods improving the robustness and efficiency of the tracker. The parameter space, augmented with first order derivatives, is automatically partitioned into Gaussian clusters each representing an elementary motion: hypothesis propagation inside each cluster is therefore accurate and efficient. The transitions between clusters use the predictions of a variable length Markov model which can explain high-level behaviours over a long history. Using Monte-Carlo methods, evaluation of model candidates is critical for both speed and robustness. We present a new evaluation scheme based on hierarchical 3-D reconstruction and blob-fitting, where appearance models and image evidences are represented by mixtures of Gaussian blobs. Our tracker is also capable of automatic-initialisation and self-recovery. We demonstrate the application of our tracker to long video sequences exhibiting rapid and diverse movements.

Virtual environments for scene of crime reconstruction and analysis

This paper describes research conducted in collaboration with Greater Manchester Police (UK), to evalute the utility of Virtual Environments for scene of crime analysis, forensic investigation, and law enforcement briefing and training. We present an illustrated case study of the construction of a high-fidelity virtual environment, intended to match a particular real-life crime scene as closely as possible. We describe and evaluate the combination of several approaches including: the use of the Manchester Scene Description Language for constructing complex geometrical models; the application of a radiosity rendering algorithm with several novel features based on human perceptual consideration; texture extraction from forensic photography; and experiments with interactive walkthroughs and large-screen stereoscopic display of the virtual environment implemented using the MAVERIK system. We also discuss the potential applications of Virtual Environment techniques in the Law Enforcement and Forensic communities.

AVIARY – A Generic Virtual Reality Interface for Real Applications

This paper introduces the work of the Advanced Interfaces Group at the University of Manchester, which is applying recent innovations in the field of human–computer interaction to important real-world applications, whose present human–computer interfaces are difficult and unnatural. We begin with an analysis of the problems of existing interfaces, and present an overview of our proposed solution – AVIARY, the generic, hierarchical, extensible virtual world model. We describe a users’ conceptual model for AVIARY, implementation strategies for software and hardware, and the application of the model to specific real-world problems.

Real-Time 3-D Human Body Tracking using Variable Length Markov Models.

In this paper, we introduce a 3-D human-body tracker capable of handling fast and complex motions in real-time. The parameter space, augmented with first order derivatives, is automatically partitioned into Gaussian clusters each representing an elementary motion: hypothesis propagation inside each cluster is therefore accurate and efficient. The transitions between clusters use the predictions of a Variable Length Markov Model which can explain highlevel behaviours over a long history. Using Monte-Carlo methods, evaluation of model candidates is critical for both speed and robustness. We present a new evaluation scheme based on volumetric reconstruction and blobs-fitting, where appearance models and image evidences are represented by Gaussian mixtures. We demonstrate the application of our tracker to long video sequences exhibiting rapid and diverse movements.

Interactive reconstruction of virtual environments from photographs, with application to scene-of-crime analysis

There are many real-world applications of Virtual Reality that require the construction of complex and accurate three-dimensional models, suitably structured for interactive manipulation. In this paper, we present semi-automatic methods that allow such environments to be quickly and easily built from photographs taken with uncalibrated cameras, and illustrate the techniques by application to the real-world problem of scene-of-crime reconstruction.

Flexible Image-Based Photometric Reconstruction using Virtual Light Sources

Photometric reconstruction is the process of estimating the illumination and surface reflectance properties of an environment, given a geometric model of the scene and a set of photographs of its surfaces. For mixed‐reality applications, such data is required if synthetic objects are to be correctly illuminated or if synthetic light sources are to be used to re‐light the scene. Current methods of estimating such data are limited in the practical situations in which they can be applied, due to the fact that the geometric and radiometric models of the scene which are provided by the user must be complete, and that the position (and in some cases, intensity) of the light sources must also be specified a‐priori. In this paper, a novel algorithm is presented which overcomes these constraints, and allows photometric data to be reconstructed in less restricted situations. This is achieved through the use of virtual light sources which mimic the effect of direct illumination from unknown luminaires, and indirect illumination reflected off unknown geometry. The intensity of these virtual light sources and the surface material properties are estimated using an iterative algorithm which attempts to match calculated radiance values to those observed in photographs. Results are presented for both synthetic and real scenes that show the quality of the reconstructed data and its use in off‐line mixed‐reality applications.

Design issues for virtual reality systems

In this paper we describe a number of issues which are central to the design of a software architecture for a distributed, generic, virtual reality system. These include support for diverse and demanding applications, the management of time to provide high-quality interaction with tightly controlled closed-loop feedback, and the need for continuity of the experience presented to the user. These issues are being addressed in the design of a generic VR system called AVIARY.