Sergio Orts-Escolano

Holoportation: Virtual 3D Teleportation in Real-time

We present an end-to-end system for augmented and virtual reality telepresence, called Holoportation. Our system demonstrates high-quality, real-time 3D reconstructions of an entire space, including people, furniture and objects, using a set of new depth cameras. These 3D models can also be transmitted in real-time to remote users. This allows users wearing virtual or augmented reality displays to see, hear and interact with remote participants in 3D, almost as if they were present in the same physical space. From an audio-visual perspective, communicating and interacting with remote users edges closer to face-to-face communication. This paper describes the Holoportation technical system in full, its key interactive capabilities, the application scenarios it enables, and an initial qualitative study of using this new communication medium.

PointNet: A 3D Convolutional Neural Network for real-time object class recognition

During the last few years, Convolutional Neural Networks are slowly but surely becoming the default method solve many computer vision related problems. This is mainly due to the continuous success that they have achieved when applied to certain tasks such as image, speech, or object recognition. Despite all the efforts, object class recognition methods based on deep learning techniques still have room for improvement. Most of the current approaches do not fully exploit 3D information, which has been proven to effectively improve the performance of other traditional object recognition methods. In this work, we propose PointNet, a new approach inspired by VoxNet and 3D ShapeNets, as an improvement over the existing methods by using density occupancy grids representations for the input data, and integrating them into a supervised Convolutional Neural Network architecture. An extensive experimentation was carried out, using ModelNet - a large-scale 3D CAD models dataset - to train and test the system, to prove that our approach is on par with state-of-the-art methods in terms of accuracy while being able to perform recognition under real-time constraints.

A Comparative Study of Registration Methods for RGB-D Video of Static Scenes

The use of RGB-D sensors for mapping and recognition tasks in robotics or, in general, for virtual reconstruction has increased in recent years. The key aspect of these kinds of sensors is that they provide both depth and color information using the same device. In this paper, we present a comparative analysis of the most important methods used in the literature for the registration of subsequent RGB-D video frames in static scenarios. The analysis begins by explaining the characteristics of the registration problem, dividing it into two representative applications: scene modeling and object reconstruction. Then, a detailed experimentation is carried out to determine the behavior of the different methods depending on the application. For both applications, we used standard datasets and a new one built for object reconstruction.

Point Light Source Estimation based on Scenes Recorded by a RGB-D camera.

Bastiaan J. Boom1 http://homepages.inf.ed.ac.uk/bboom/ Sergio Orts-Escolano2 http://www.dtic.ua.es/~sorts/ Xin X. Ning1 avener1446@hotmail.com Steven McDonagh1 http://homepages.inf.ed.ac.uk/s0458953/ Peter Sandilands1 http://homepages.inf.ed.ac.uk/s0569500/ Robert B. Fisher1 http://homepages.inf.ed.ac.uk/rbf/ 1 Institute of Perception, Action and Behaviour University of Edinburgh Edinburgh, UK 2 Computer Technology Department University of Alicante Alicante, Spain

Point cloud data filtering and downsampling using growing neural gas

3D sensors provide valuable information for mobile robotic tasks like scene classification or object recognition, but these sensors often produce noisy data that makes impossible applying classical keypoint detection and feature extraction techniques. Therefore, noise removal and downsampling have become essential steps in 3D data processing. In this work, we propose the use of a 3D filtering and downsampling technique based on a Growing Neural Gas (GNG) network. GNG method is able to deal with outliers presents in the input data. These features allows to represent 3D spaces, obtaining an induced Delaunay Triangulation of the input space. Experiments show how GNG method yields better input space adaptation to noisy data than other filtering and downsampling methods like Voxel Grid. It is also demonstrated how the state-of-the-art keypoint detectors improve their performance using filtered data with GNG network. Descriptors extracted on improved keypoints perform better matching in robotics applications as 3D scene registration.

Real time motion estimation using a neural architecture implemented on GPUs

Abstract This work describes a neural network based architecture that represents and estimates object motion in videos. This architecture addresses multiple computer vision tasks such as image segmentation, object representation or characterization, motion analysis and tracking. The use of a neural network architecture allows for the simultaneous estimation of global and local motion and the representation of deformable objects. This architecture also avoids the problem of finding corresponding features while tracking moving objects. Due to the parallel nature of neural networks, the architecture has been implemented on GPUs that allows the system to meet a set of requirements such as: time constraints management, robustness, high processing speed and re-configurability. Experiments are presented that demonstrate the validity of our architecture to solve problems of mobile agents tracking and motion analysis.

3D colour object reconstruction based on Growing Neural Gas

With the advent of low-cost 3D sensors and 3D printers, surface reconstruction has become an important research topic in the last years. In this work, we propose an automatic method for 3D surface reconstruction from raw unorganized point clouds acquired using low-cost sensors. We have modified the Growing Neural Gas (GNG) network, which is a suitable model because of its flexibility, rapid adaptation and excellent quality of representation, to perform 3D surface reconstruction of different real-world objects. Some improvements have been made on the original algorithm considering colour information during the learning stage and creating complete triangular meshes instead of basic wire-frame representations. The proposed method is able to create 3D faces online, whereas existing 3D reconstruction methods based on Self-Organizing Maps (SOMs) required post-processing steps to close gaps and holes produced during the 3D reconstruction process. Performed experiments validated how the proposed method improves existing techniques removing post-processing steps and including colour information in the final triangular mesh.

3D reconstruction of medical images from slices automatically landmarked with growing neural models

In this study, we utilise a novel approach to segment out the ventricular system in a series of high resolution T1-weighted MR images. We present a brain ventricles fast reconstruction method. The method is based on the processing of brain sections and establishing a fixed number of landmarks onto those sections to reconstruct the ventricles 3D surface. Automated landmark extraction is accomplished through the use of the self-organising network, the growing neural gas (GNG), which is able to topographically map the low dimensionality of the network to the high dimensionality of the contour manifold without requiring a priori knowledge of the input space structure. Moreover, our GNG landmark method is tolerant to noise and eliminates outliers. Our method accelerates the classical surface reconstruction and filtering processes. The proposed method offers higher accuracy compared to methods with similar efficiency as Voxel Grid.

Improving 3D keypoint detection from noisy data using growing neural gas

3D sensors provides valuable information for mobile robotic tasks like scene classification or object recognition, but these sensors often produce noisy data that makes impossible applying classical keypoint detection and feature extraction techniques. Therefore, noise removal and downsampling have become essential steps in 3D data processing. In this work, we propose the use of a 3D filtering and down-sampling technique based on a Growing Neural Gas (GNG) network. GNG method is able to deal with outliers presents in the input data. These features allows to represent 3D spaces, obtaining an induced Delaunay Triangulation of the input space. Experiments show how the state-of-the-art keypoint detectors improve their performance using GNG output representation as input data. Descriptors extracted on improved keypoints perform better matching in robotics applications as 3D scene registration.

Fast Autonomous Growing Neural Gas

This paper aims to address the ability of self-organizing neural network models to manage real-time applications. Specifically, we introduce fAGNG (fast Autonomous Growing Neural Gas), a modified learning algorithm for the incremental model Growing Neural Gas (GNG) network. The Growing Neural Gas network with its attributes of growth, flexibility, rapid adaptation, and excellent quality of representation of the input space makes it a suitable model for real time applications. However, under time constraints GNG fails to produce the optimal topological map for any input data set. In contrast to existing algorithms the proposed fAGNG algorithm introduces multiple neurons per iteration. The number of neurons inserted and input data generated is controlled autonomous and dynamically based on a priory learnt model. Comparative experiments using topological preservation measures are carried out to demonstrate the effectiveness of the new algorithm to represent linear and non-linear input spaces under time restrictions.