Christopher G. Harris

A COMBINED CORNER AND EDGE DETECTOR

The problem we are addressing in Alvey Project MMI149 is that of using computer vision to understand the unconstrained 3D world, in which the viewed scenes will in general contain too wide a diversity of objects for topdown recognition techniques to work. For example, we desire to obtain an understanding of natural scenes, containing roads, buildings, trees, bushes, etc., as typified by the two frames from a sequence illustrated in Figure 1. The solution to this problem that we are pursuing is to use a computer vision system based upon motion analysis of a monocular image sequence from a mobile camera. By extraction and tracking of image features, representations of the 3D analogues of these features can be constructed.

3D positional integration from image sequences

Abstract An explicit three-dimensional (3D) representation is constructed from feature points extracted from a sequence of images taken by a moving camera. The points are tracked through the sequence, and their 3D locations are accurately determined by use of Kalman filters. The egomotion of the camera is also determined.

Development and preliminary evaluation of VISLAN, a surgical planning and guidance system using intra-operative video imaging

VISLAN is an integrated neurosurgical planning and guidance system. New segmentation and rendering techniques have been incorporated. A stereo video system is used intra-operatively and fulfils four roles. First, the video display is overlaid with graphical outlines showing the position of the planned craniotomy or the target (enhanced reality displays). Second, a skin surface patch is reconstructed from the stereo video images using patterned light (mean errors of surface point location are < 0.15 mm). Third, a freely mobile, hand-held localizer is tracked in real time (position errors are < 0.5 mm and with improved calibration < 0.2 mm), with its position superimposed on the pre-operative patient representation to assist surgical guidance. Fourth, markers fixed to the skull bone next to the cranial opening are used to detect intra-operative movement and to update registration. Initial results from phantom experiments show an overall system accuracy of better than 0.9 mm for intra-operative localization of features defined in pre-operative images. The prototype system has been tested during six neurosurgical operations with very good results.

Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation

Dynamic movements of the hand, fingers, and thumb are difficult to measure due to the versatility and complexity of movement inherent in function. An innovative approach to measuring hand kinematics is proposed and validated. The proposed system utilizes the Microsoft Kinect and goes beyond gesture recognition to develop a validated measurement technique of finger kinematics. The proposed system adopted landmark definition (validated through ground truth estimation against assessors) and grip classification algorithms, including kinematic definitions (validated against a laboratory-based motion capture system). The results of the validation show 78% accuracy when identifying specific markerless landmarks. In addition, comparative data with a previously validated kinematic measurement technique show accuracy of MCP ± 10° (average absolute error (AAE) = 2.4°), PIP ± 12° (AAE = 4.8°), and DIP ± 11° (AAE = 4.8°). These results are notably better than clinically based alternative manual measurement techniques. The ability to measure hand movements, and therefore functional dexterity, without interfering with underlying composite movements, is the paramount objective to any bespoke measurement system. The proposed system is the first validated markerless measurement system using the Microsoft Kinect that is capable of measuring finger joint kinematics. It is suitable for home-based motion capture for the hand and, therefore, achieves this objective.

Structure-from-motion under orthographic projection

Abstract Structure-from-motion algorithms based on matched point-like features under orthographic projection are explored, for use in analysing image motion from small rigid moving objects. For two-frame analysis, closed-form n-point algorithms are devised that minimize image-plane positional errors. The bas-relief ambiguity is shown to exist for arbitrary object rotations. The algorithm is applied to real images, and good estimates of the projection of the axis of rotation on to the imageplane are obtained.

Structure-from-Motion under Orthographic Projection

Structure-from-motion algorithms based on matched point-like features under orthographic projection are explored, for use in analysing image motion from small rigid moving objects. For two-frame analysis, closed-from n-point algorithms are devised that minimise image-plane positional errors. The bas-relief ambiguity is shown to exist for arbitrary object rotations. The algorithm is applied to real images, and good estimates of the projection of the axis of rotation onto the image-plane are obtained.

3D wire-frame integration from image sequences

Abstract When integrating visual features into 3D for a structure from motion algorithm, the connectivity and relationships of features are an important adjunct to any quantitative 3D geometry. This paper describes a general purpose vision system which aims to perceive and refine this topology in conjunction with geometry, using edges, vertices and isolated corners extracted from a sequence of monocular images of an unconstrained scene. Rules which tackle the practical difficulties of imperfect image processing and obscuration features are defined. Results are shown for a rotating view of a polyhedral object and for an outdoor scene viewed from a moving vehicle.

A parallel implementation of a structure-from-motion algorithm

This paper describes the implementation of a 3D vision algorithm, Droid, on the Oxford parallel vision architecture, PARADOX, and the results of experiments to gauge the algorithms effectiveness in providing navigation data for an autonomous guided vehicle. The algorithm reconstructs 3D structure by analysing image sequences obtained from a moving camera. In this application, the architecture delivers a performance of greater than 1 frame per second — 17 times the performance of a Sun-4 alone.

A video based tracker for use in computer aided surgery

Recent improvements in medical equipment and data visualisation tools have made it possible for surgeons to use medical images of patients during operations as well as during the preparation phase. In order to use the data effectively in the theatre the surgeon must be given live information about his current location inside the patient relative to the pre-operative images. This paper describes a vision-based system that can locate and track a passive hand-held instrument in the operating theatre to provide the real-time information required. A target acquisition system is described which locates a pre-defined object in an arbitrary scene and calculates its 3D location and attitude. Enhancements to the RAPiD model-based tracking system are also detailed including multiple camera operation, improvements to edge identification in the image and combining robust tracking with high accuracy. Experimental results are also provided.

Building aerial mosaics for visual MTI

This paper addresses the task of making a mosaic from images gathered by a down-looking camera on an airborne platform. This is in the context of a system to detect and map the positions of moving objects. We present three mosaicing approaches based on integrating together sets of measured pairwise homographies, i.e. geometric relationships, between overlapping image frames. The methods are simple chaining, consensus placement and bundle adjustment. We have demonstrated all the approaches with simulated data whilst the simplest way of using pairwise links, simple one-dimensional chaining, has been demonstrated with real data. In our bundle adjustment method, we use a two-dimensional network of pairwise links; when each frame is added to the mosaic, all the constituent frames are adjusted with respect to each other so that the consistency over the entire network is optimised. We have successfully shown, in simulation, that the bundle adjustment technique results in much more consistent, undistorted maps.