Francisco Gomez-Donoso

PointNet: A 3D Convolutional Neural Network for real-time object class recognition

During the last few years, Convolutional Neural Networks are slowly but surely becoming the default method solve many computer vision related problems. This is mainly due to the continuous success that they have achieved when applied to certain tasks such as image, speech, or object recognition. Despite all the efforts, object class recognition methods based on deep learning techniques still have room for improvement. Most of the current approaches do not fully exploit 3D information, which has been proven to effectively improve the performance of other traditional object recognition methods. In this work, we propose PointNet, a new approach inspired by VoxNet and 3D ShapeNets, as an improvement over the existing methods by using density occupancy grids representations for the input data, and integrating them into a supervised Convolutional Neural Network architecture. An extensive experimentation was carried out, using ModelNet - a large-scale 3D CAD models dataset - to train and test the system, to prove that our approach is on par with state-of-the-art methods in terms of accuracy while being able to perform recognition under real-time constraints.

LonchaNet: A sliced-based CNN architecture for real-time 3D object recognition

In the last few years, Convolutional Neural Networks (CNNs) had become the default paradigm to address classification problems, specially, but not only, in image recognition. This is mainly due to the high success rate that they provide. Despite there currently exist approaches that apply deep learning to the 3D recognition problem, they are either too slow for online uses or too error prone. To fill this gap, we propose LonchaNet, a deep learning architecture for point clouds classification. Our system successfully achieves a high accuracy yet providing a low computation cost. A dense set of experiments were carried out in order to validate our system in the frame of the ModelNet — a large-scale 3D CAD models dataset — challenge. Our proposal achieves a success rate of 94.37% in the ModelNet-10 classification task, the second place in the leaderboard as of today (November, 2016).

A study of the effect of noise and occlusion on the accuracy of convolutional neural networks applied to 3D object recognition

Abstract In this work, we carry out a study of the effect of adverse conditions, which characterize real-world scenes, on the accuracy of a Convolutional Neural Network applied to 3D object class recognition. Firstly, we discuss possible ways of representing 3D data to feed the network. In addition, we propose a set of representations to be tested. Those representations consist of a grid-like structure (fixed and adaptive) and a measure for the occupancy of each cell of the grid (binary and normalized point density). After that, we propose and implement a Convolutional Neural Network for 3D object recognition using Caffe. At last, we carry out an in-depth study of the performance of the network over a 3D CAD model dataset, the Princeton ModelNet project, synthetically simulating occlusions and noise models featured by common RGB-D sensors. The results show that the volumetric representations for 3D data play a key role on the recognition process and Convolutional Neural Network can be considerably robust to noise and occlusions if a proper representation is chosen.

Automatic Schaeffer's gestures recognition system

Schaeffers sign language consists of a reduced set of gestures designed to help children with autism or cognitive learning disabilities to develop adequate communication skills. Our automatic recognition system for Schaeffers gesture language uses the information provided by an RGB-D camera to capture body motion and recognize gestures using dynamic time warping combined with k-nearest neighbors methods. The learning process is reinforced by the interaction with the proposed system that accelerates learning itself thus helping both children and educators. To demonstrate the validity of the system, a set of qualitative experiments with children were carried out. As a result, a system which is able to recognize a subset of 11 gestures of Schaeffers sign language online was achieved.

A robotic platform for customized and interactive rehabilitation of persons with disabilities

Abstract In this work, we have developed a multisensor system for rehabilitation and interaction with persons with motor and cognitive disabilities. The system enables them to perform different therapies using multiple modes of interaction (head and body pose, hand gestures, voice, touch and gaze) depending on the type and degree of disability. Through a training process, the system can be customized enabling the definition of patients’ own gestures for each sensor. The system is integrated with a range of applications for rehabilitation. Examples of these applications are puzzle solving, mazes and text writing using predictive text tools. The system also provides a flexible and modular framework for the development of new applications oriented towards novel rehabilitation therapies. The proposed system has been integrated in a mobile robotic platform and uses low-cost sensors allowing to perform non-intrusive rehabilitation therapies at home. Videos showing the proposed system and users interacting in different ways (multimodal) are available on our project website www.rovit.ua.es/patente/ .

An augmented reality application for improving shopping experience in large retail stores

In several large retail stores, such as malls, sport or food stores, the customer often feels lost due to the difficulty in finding a product. Although these large stores usually have visual signs to guide customers toward specific products, sometimes these signs are also hard to find and are not updated. In this paper, we propose a system that jointly combines deep learning and augmented reality techniques to provide the customer with useful information. First, the proposed system learns the visual appearance of different areas in the store using a deep learning architecture. Then, customers can use their mobile devices to take a picture of the area where they are located within the store. Uploading this image to the system trained for image classification, we are able to identify the area where the customer is located. Then, using this information and novel augmented reality techniques, we provide information about the area where the customer is located: route to another area where a product is available, 3D product visualization, user location, analytics, etc. The system developed is able to successfully locate a user in an example store with 98% accuracy. The combination of deep learning systems together with augmented reality techniques shows promising results toward improving user experience in retail/commerce applications: branding, advance visualization, personalization, enhanced customer experience, etc.

Robust Hand Pose Regression Using Convolutional Neural Networks

Hand pose estimation is useful for several human-computer interaction applications, like sign language recognition, the identification of more complex behaviors such as hand gestures and interaction in virtual reality applications. In this work, we propose a system which is able to predict the 2D hand joints using a monocular color camera. To do that, we propose to use a 3D hand tracking sensor for collecting ground truth information that is projected to the camera image plane. We present a novel pipeline that leverages deep learning techniques for hand pose estimation. The proposed Convolutional Neural Networks (CNN) is able to infer the joints of the hand from an image without the need of any additional sensor.

3D Object Mapping Using a Labelling System

3D data has arisen as the most used information for environment representation thanks to the advent of low cost RGB-D cameras. We propose a 3D map representation that uses not only depth information but the information provided by an expert system. This expert consists on a Convolutional Neural Network trained with deep learning techniques for scene labelling purposes. For every partial 3D map captured, we receive a set of labels with their associated probability of presence in that scene. The final map is obtained by registering and merging all these partial maps. The semantic labels from the expert system are used to recognise and locate objects in the environment.