Jose Martinez-Carranza

Unifying Planar and Point Mapping in Monocular SLAM

Planar features in filter-based Visual SLAM systems require an initialisation stage that delays their use within the estimation. In this stage, surface and pose are initialised either by using an already generated map of point features [2, 3] or by using visual clues from frames [4]. This delay is unsatisfactory specially in scenarios where the camera moves rapidly such that visual features are observed for a very limited period. In this paper we present a unified approach to mapping in which points and planes are initialised alongside each other within the same framework. The best structure emerges according to what the camera observes, thus avoiding delayed initialisation for planar features. To do this we use a similar parameterisation to the one used for planar features in [3, 4]. The Inverse Depth Planar Parameterisation (IDPP), as we call it, is used to represent both planes and points. This IDPP is also combined with a point based measurement model where the planar constraint is introduced. The latter allows us to estimate and grow a planar structure if suitable, or to estimate a 3-D point if visual measurements do not support the constraint. The IDPP contains three main components: (1) A reference camera (RC); (2) the depth w.r.t. the RC of a seed 3-D point on the plane; (3) the normal of the plane.

Efficient visual odometry using a structure-driven temporal map

We describe a method for visual odometry using a single camera based on an EKF framework. Previous work has shown that filtering based approaches can achieve accuracy performance comparable to that of optimisation methods providing that large numbers of features are used. However, computational requirements are significantly increased and frame rates are low. We address this by employing higher level structure - in the form of planes - to efficiently parameterise features and so reduce the filter state size and computational load. Moreover, we extend a 1-point RANSAC outlier rejection method to the case of features lying on planes. Results of experiments with both simulated and real-world data demonstrate that the method is effective, achieving comparable accuracy whilst running at significantly higher frame rates.

Enhancing 6D visual relocalisation with depth cameras

Relocalisation in 6D is relevant to a variety of Robotics applications and in particular to agile cameras exploring a 3D environment. While the use of geometry has commonly helped to validate appearance as a back-end process in several relocalisation systems before, we are interested in using 3D information to assist fast pose relocalisation computation as part of a front-end task. Our approach rapidly searches for a reduced number of visual descriptors, previously observed and stored in a database, that can be used to effectively compute the camera pose corresponding to the current view. We guide the search by means of constructing validated candidate sets using a 3D test involving the depth information obtained with an RGB-D camera (e.g. stereo of with structured light). Our experiments demonstrate that this process returns a compact quality set that works better for the pose estimation stage than when using a typical Nearest-Neighbor search over appearance only. The improvements are observed in terms of percentage of relocalised frames and speed, where the latter goes up to two orders of magnitude w.r.t. the conventional search.

Efficiently Increasing Map Density in Visual SLAM Using Planar Features with Adaptive Measurement

The visual simultaneous localisation and mapping (SLAM) systems now in widespread use are based on localised point features [2, 4, 5]. Although effective in many respects, the approach has limitations when considering the density and efficiency of map representation. With a dense population of features, camera tracking can be robust, able to withstand significant occlusion and large changes in camera viewpoint. But this comes at a high computational cost, typically increasing quadratically with the number of features. In this work we propose increasing map density by building in higherorder structure in the form of planar features. An important and novel aspect of the work is the manner in which the planar features are updated and used to localise the camera. We base our approach on an extended Kalman filter (EKF) monocular SLAM system developed by Chekhlov et al. [3]. This provides real-time estimates of the 3-D pose of a calibrated camera whilst simultaneously mapping the scene in terms of point based features. In order to incorporate planar structure into the real-time monocular SLAM we carry out three steps: detection of planar structure in the scene; insertion of planar features into the map; and adaptive measurement of the features. To apply the principle of adaptive measurement it is essential that planar features inserted into the map correspond to actual planar structure in the scene. For this we employ the method proposed by Martinez-Carranza and Calway [6], which uses an appearance model to detect planes defined by subsets of mapped point features (at least three points). Having detected planar features in the scene these are inserted into the map using a suitable representation within the filter state. This has two components: plane parameterisation and the reference camera. The plane is defined by yp = (θ ,φ ,ρ), where (θ ,φ) defines the unit normal of the plane in polar coordinates in the reference camera and ρ is the inverse depth of the plane centre along the ray defined by uo, with the latter being stored at initialisation of the plane, see figure 1a. Insertion of the reference camera is done by augmenting the state with a copy of the current pose, i.e. vp = v, and with initialised plane parameters yp derived from the pose and the subset of mapped points which define the plane. The reference camera serves two purposes: it references the plane in the SLAM coordinate system (with the associated uncertainties) and enables subsequent measurement of the planar feature using region based matching with respect to the current frame (key frame). To facilitate the latter the key frame image is also stored in the system. As illustrated in figure 1b, measurements for a planar feature are therefore assumed to take the following form:

Appearance Based Extraction of Planar Structure in Monocular SLAM

This paper concerns the building of enhanced scene maps during real-time monocular SLAM. Specifically, we present a novel algorithm for detecting and estimating planar structure in a scene based on both geometric and appearance and information. We adopt a hypothesis testing framework, in which the validity of planar patches within a triangulation of the point based scene map are assessed against an appearance metric. A key contribution is that the metric incorporates the uncertainties available within the SLAM filter through the use of a test statistic assessing error distribution against predicted covariances, hence maintaining a coherent probabilistic formulation. Experimental results indicate that the approach is effective, having good detection and discrimination properties, and leading to convincing planar feature representations.

Visual mapping using learned structural priors

We investigate a new approach to vision based mapping, in which single image structure recognition is used to derive strong priors for initialisation of higher-level primitives in the map. This can reduce state size and speed up the building of more meaningful maps. We focus on plane mapping and use a recognition algorithm to detect and estimate the 3D orientation of planar structures in key frames, which are then used as priors for initialising planes in the map. The recognition algorithm learns the relationship between such structure and appearance from training examples offline. We demonstrate the approach in the context of an EKF based visual odometry system. Preliminary results of experiments in urban environments show that the system is able to build large maps with significant planar structure at average frames rates of around 60 fps whilst maintaining good trajectory estimation. The results suggest that the approach has considerable potential.

Accelerating the construction of BRIEF descriptors using an FPGA-based architecture

BRIEF emerged as a novel alternative to conventional floating-point-based descriptors such as SIFT or SURF. In contrast to these descriptors, BRIEF is a descriptor represented by a binary number offering two main advantages: low memory footprint and fast descriptor comparison. These qualities make it a suitable descriptor to be implemented on a hardware architecture, where the comparison operation can be implemented efficiently via a parallel scheme. However, the construction of BRIEF involves a sequential operation in the form of a set of pairwise tests on the image intensities, and as consequence, sequential memory access is necessary. In this paper, we propose a novel way to construct the BRIEF descriptor by arranging the pairwise tests such that data retrieval from memory is exploited, thus accelerating the descriptor construction up to 4 times when compared to the sequential way.

On combining wearable sensors and visual SLAM for remote controlling of low-cost micro aerial vehicles

In this work we present initial results of a system that combines wearable technology and monocular simultaneous localisation and mapping (SLAM) for remote controlling of a low-cost micro aerial vehicle (MAV) that flies beyond the visual line-of-sight. To this purpose, as a first step, we use a state-of-the-art visual SLAM system, called ORB-SLAM, to create a 3D map of the scene. The visual data feeding ORB-SLAM is obtained from imagery transmitted from the on-board camera of our low-cost vehicle. This vehicle can not process data on board, however, it can transmit images at a rate of 15-20 Hz, which we found sufficient to carry out the visual localisation and mapping. The second step in our system is to replace the conventional controller with a pair of wearable-sensor-based gloves worn by the user so he/she can command the MAV by only performing hand gestures. Our goal is to show that the user can fly the vehicle beyond the line-of-sight by only using the vehicles pose and map estimates in real time and that commanding the MAV with hand gestures will enable him/her to focus more on the flight task. Our preliminary results indicate the feasibility of our approach.

Towards autonomous flight of micro aerial vehicles using ORB-SLAM

In the last couple of years a novel visual simultaneous localisation and mapping (SLAM) system, based on visual features, has emerged as one of the best, if not the best, systems for estimating the 6D camera pose whilst building a 3D map of the observed scene. This method is called ORB-SLAM and one of its key ideas is to use the same visual descriptor, a binary descriptor called ORB, for all the visual tasks, this is, for feature matching, relocalisation and loop closure. On the top of this, ORB-SLAM combines local and graph-based global bundle adjustment, which enables a scalable map generation whilst keeping real-time performance. Therefore, motivated by its performance in terms of processing speed, robustness against erratic motion and scalability, in this paper we present an implementation of autonomous flight for a low-cost micro aerial vehicle (MAV), where ORB-SLAM is used as a visual positioning system that feeds a PD controller that controls pitch, roll and yaw. Our results indicate that our implementation has potential and could soon be implemented on a bigger aerial platform with more complex trajectories to be flown autonomously.

Indoor MAV Auto-Retrieval Using Fast 6D Relocalisation

This paper develops and evaluates methods for performing auto-retrieval of a micro aerial vehicle (MAV) using fast 6D relocalisation from visual features. Auto-retrieval involves a combination of guided operation to direct the vehicle through obstacles using a human pilot and autonomous operation to navigate the vehicle on its return or during re-exploration. This approach is useful in tasks such as industrial inspection and monitoring, and in particular to operate indoors in GPS-denied environments. Our relocalisation methodology contrasts two sources of information: depth data and feature co-visibility, but in a novel manner that validates matches before a RANSAC procedure. The result is the ability of performing 6D relocalisation at an average of 50 Hz on individual maps containing 120 K features. The use of feature co-visibility reduces memory footprint as well as removes the need to employ depth data as used in previous work. This paper concludes with an example of an industrial application involving visual monitoring from a MAV aided by autonomous navigation. Graphical Abstract