Simon Stent

Understanding RealWorld Indoor Scenes with Synthetic Data

Scene understanding is a prerequisite to many high level tasks for any automated intelligent machine operating in real world environments. Recent attempts with supervised learning have shown promise in this direction but also highlighted the need for enormous quantity of supervised data- performance increases in proportion to the amount of data used. However, this quickly becomes prohibitive when considering the manual labour needed to collect such data. In this work, we focus our attention on depth based semantic per-pixel labelling as a scene understanding problem and show the potential of computer graphics to generate virtually unlimited labelled data from synthetic 3D scenes. By carefully synthesizing training data with appropriate noise models we show comparable performance to state-of-the-art RGBD systems on NYUv2 dataset despite using only depth data as input and set a benchmark on depth-based segmentation on SUN RGB-D dataset.

gvnn: neural network library for geometric computer vision

We introduce gvnn, a neural network library in Torch aimed towards bridging the gap between classic geometric computer vision and deep learning. Inspired by the recent success of Spatial Transformer Networks, we propose several new layers which are often used as parametric transformations on the data in geometric computer vision. These layers can be inserted within a neural network much in the spirit of the original spatial transformers and allow backpropagation to enable end-to-end learning of a network involving any domain knowledge in geometric computer vision. This opens up applications in learning invariance to 3D geometric transformation for place recognition, end-to-end visual odometry, depth estimation and unsupervised learning through warping with a parametric transformation for image reconstruction error.

SceneNet: An annotated model generator for indoor scene understanding

We introduce SceneNet, a framework for generating high-quality annotated 3D scenes to aid indoor scene understanding. SceneNet leverages manually-annotated datasets of real world scenes such as NYUv2 to learn statistics about object co-occurrences and their spatial relationships. Using a hierarchical simulated annealing optimisation, these statistics are exploited to generate a potentially unlimited number of new annotated scenes, by sampling objects from various existing databases of 3D objects such as ModelNet, and textures such as OpenSurfaces and ArchiveTextures. Depending on the task, SceneNet can be used directly in the form of annotated 3D models for supervised training and 3D reconstruction benchmarking, or in the form of rendered annotated sequences of RGB-D frames or videos.

Visual change detection on tunnel linings

We describe an automated system for detecting, localising, clustering and ranking visual changes on tunnel surfaces. The system is designed to provide assistance to expert human inspectors carrying out structural health monitoring and maintenance on ageing tunnel networks. A three-dimensional tunnel surface model is first recovered from a set of reference images using Structure from Motion techniques. New images are localised accurately within the model and changes are detected versus the reference images and model geometry. We formulate the problem of detecting changes probabilistically and evaluate the use of different feature maps and a novel geometric prior to achieve invariance to noise and nuisance sources such as parallax and lighting changes. A clustering and ranking method is proposed which efficiently presents detected changes and further improves the inspection efficiency. System performance is assessed on a real data set collected using a low-cost prototype capture device and labelled with ground truth. Results demonstrate that our system is a step towards higher frequency visual inspection at a reduced cost.

Detecting Change for Multi-View, Long-Term Surface Inspection.

We describe a system for the detection of changes in multiple views of a tunnel surface. From data gathered by a robotic inspection rig, we use a structure-from-motion pipeline to build panoramas of the surface and register images from different time instances. Reliably detecting changes such as hairline cracks, water ingress and other surface damage between the registered images is a challenging problem: achieving the best possible performance for a given set of data requires sub-pixel precision and careful modelling of the noise sources. The task is further complicated by factors such as unavoidable registration error and changes in image sensors, capture settings and lighting. Our contribution is a novel approach to change detection using a two-channel convolutional neural network. The network accepts pairs of approximately registered image patches taken at different times and classifies them to detect anomalous changes. To train the network, we take advantage of synthetically generated training examples and the homogeneity of the tunnel surfaces to eliminate most of the manual labelling effort. We evaluate our method on field data gathered from a live tunnel over several months, demonstrating it to outperform existing approaches from recent literature and industrial practice.

SynthCam3D: Semantic Understanding With Synthetic Indoor Scenes.

We are interested in automatic scene understanding from geometric cues. To this end, we aim to bring semantic segmentation in the loop of real-time reconstruction. Our semantic segmentation is built on a deep autoencoder stack trained exclusively on synthetic depth data generated from our novel 3D scene library, SynthCam3D. Importantly, our network is able to segment real world scenes without any noise modelling. We present encouraging preliminary results.

Precise deterministic change detection for smooth surfaces

We introduce a precise deterministic approach for pixel-wise change detection in images taken of a scene of interest over time. Our motivation is for applications such as artefact condition monitoring and structural inspection, where a common problem is the need to efficiently and accurately identify subtle signs of damage and deterioration. The approach we describe is designed to compensate for the three most common sources of nuisance variation encountered when tackling the problem of change detection, namely: viewpoint variation due to camera motion between images, photometric variation due to lighting differences, and changes in image resolution/focal settings. To tackle viewpoint variation, particularly in areas of low texture, we propose the use of the generalised PatchMatch (PM) correspondence algorithm to compute a dense flow field. The flow field is regularized using a Thin Plate Spline (TPS) model which assumes a smooth underlying geometry and allows registration to be interpolated precisely through areas of low texture or uncertain flow. To compensate for low-frequency lighting variation, we fit a second TPS model to the photometric differences between registered images. Finally, to account for changes in focal settings, we estimate and apply a blurring kernel via optimisation over image differences. We provide a thorough evaluation of the performance of our method on an illustrative toy dataset and on two recent, real-world inspection datasets. Our approach performs favourably versus state-of-the-art baselines in both cases, while remaining relatively transparent to understand and simple to compute.

A low-cost robotic system for the efficient visual inspection of tunnels

We introduce a low-cost robotic system designed to enable the safe, objective and efficient visual inspection of tunnels. The system captures high resolution images and processes them to produce maps of tunnel linings that are suitable for detailed inspection. It is unique in that the total cost of hardware is an order of magnitude less than most existing systems while producing an equivalent or higher quality of output. The device makes use of consumer-grade digital cameras and high-power LEDs in a rotating rig, carried by a lightweight aluminium frame which is designed to reduce vibrations during data capture. It is portable and installable by hand and has a modular design, making it possible to adapt to different types of carriage units, tunnels and sensors. Within the paper, we share insight into features of the devices design, including lessons learned from trials of earlier prototypes and comparisons with alternative systems. Using field data gathered from a 2km utility tunnel, we demonstrate the use of our system as a means of visualising tunnel conditions through image mosaicing, cataloguing tunnel segments using barcode detection and improving the objectivity of visual condition surveys over time by the detection of sub-mm crack growth. We believe that our device is the first to provide comprehensive survey-quality data at such a low cost, making it very attractive as a tool for the improved visual monitoring of tunnels.

Pedestrian monitoring techniques for crowd-flow prediction

The high concentration and flow rate of people in train stations during rush hours can pose a prominent risk to passenger safety and comfort. In situ counting systems are a critical element for predicting pedestrian flows in real time, and their capabilities must be rigorously tested in live environments. The focus of this paper is on evaluating the reliability of two alternative counting systems, the first using an array of infrared depth sensors and the second a visible light (RGB) camera. Both proposed systems were installed at a busy walkway in London Bridge station. The data were collected over a period of 2 months, after which, portions of the data set were labelled for quantitative evaluation against ground truth. In this paper, the implementation of the two different counting technologies is described, and the accuracy and limitations of both approaches under different conditions are discussed. The results show that the developed RGB-based system performs reliably across a wide range of conditions, while the depth-based approach proves to be a useful complement in conditions without significant ambient sunlight, such as underground passageways.