Adel H. Fakih

3D Pose tracking of walker users' lower limb with a structured-light camera on a moving platform

Tracking and understanding human gait is an important step towards improving elderly mobility and safety. Our research team is developing a vision-based tracking system that estimates the 3D pose of a wheeled walker users lower limbs with a depth sensor, Kinect, mounted on the moving walker. Our tracker estimates 3D poses from depth images of the lower limbs in the coronal plane in a dynamic, uncontrolled environment. We employ a probabilistic approach based on particle filtering, with a measurement model that works directly in the 3D space and another measurement model that works in the projected image space. Empirical results show that combining both measurements, assuming independence between them, yields tracking results that are better than with either one alone. Experiments are conducted to evaluate the performance of the tracking system with different users. We demonstrate that the tracker is robust against unfavorable conditions such as partial occlusion, missing observations, and deformable tracking target. Also, our tracker does not require user intervention or manual initialization commonly required in most trackers.

A vison-based system for mapping the inside of a pipe

Underground pipes constitute the backbone of the infrastructure of a country. Dirty, broken, or clogged pipes have direct implications on the health hazards of humans. It is therefore no surprise that fault assessment of pipes is an important topic, which has received considerable attention in the past. While most pipe analysis systems rely on active sensors such as laser or sonar, the use of passive vision sensors has advantages in terms of cost and safety. This paper presents an automated 3D pipe reconstruction system using a single monocular camera as the only sensor. The contribution of our work is threefold. Firstly, the paper analyzes the implications of different environmental conditions on the result of the 3D reconstruction. Issues like different texture, diameter size, and lighting conditions are addressed. Secondly, while previous vision-based techniques use a special type of fisheye camera to perform the reconstruction, the method presented here is implemented using a regular off-the-shelf camera. Thirdly and finally, the 3D reconstruction system is the first to be able to detect and localize obstructions inside a pipe. Experiments are performed inside real pipes and results prove the success of our techniques.

Structure from Motion: Combining features correspondences and optical flow

This paper suggests using discrete feature displacements and optical flow simultaneously to determine the camera motion and its velocity. This is advantageous when the number of feature correspondences is low or when the feature correspondences are noisy. The reason is that usually the available optical flow data largely outnumbers the available feature correspondences data. It is also advantageous from the perspective of the instantaneous motion estimation because it gives better estimates for the camera velocity than those obtained from optical flow by itself. We propose a probabilistic framework capitalizing on the this idea. Monte-Carlo filtering is employed due to the non-linearities involved in the problem and to the non-Gaussianity of the measurements¿ probability distributions.

Scalable Near-Optimal Recursive Structure from Motion

Rao-BlackWellized particle filters have achieved a breakthrough in the scalability of filters used for Structure from Motion (SFM) and Simultaneous Localization and Mapping (SLAM). The new generations of these filters employ as proposal distribution the optimal \emph{importance function} i.e, the one taking into consideration not only the previous motion of the camera, but also the most recent measurement. However the way they sample from this importance function is not optimal since the locations of 3-d features are updated using a motion predicted only from the previous state. This results in a performance lower than the Extended Kalman Filters (EKF)s. We propose in this paper an approach that bears similarity with the Random Sample Consensus (RANSAC) paradigm and that enables us to sample more efficiently from the optimal importance function. It allows us to update the depth based on a motion updated using information from the most recent image and hence the updated samples would have a higher chance to be in regions corresponding to high posterior probability. This results in a performance equal to the performance of the EKF with much higher scalability. Also, our samples being generated and updated based on random sampling of the features, this provides an improved robustness to outliers.

Filtering 3D Keypoints Using GIST For Accurate Image-Based Localization.

Image-Based Localization (IBL) is the problem of estimating the 3D pose of a camera with respect to a 3D representation of the scene. IBL is quite challenging in largescale environments spanning a wide variety of viewpoints, illumination, and areas where matching a query image against hundreds of thousands of 3D points becomes prone to a large number of outliers and ambiguous situations. The current state of the art IBL solutions attempted to address the problem using paradigms such as bag-of-words, features co-occurrence, deep learning and others, with varying degrees of success. This paper presents GIST-based Search Space Reduction (GSSR) for indoor and large scale Image-Based Localization applications such as relocalization, loop closure and location recognition. GSSR explores the use of global descriptors, in particular GIST, to introduce a new similarity measure for keyframes that combines the GIST descriptor scores of all neighboring frames to qualify a limited number of 3D points for the matching process, hence reducing the problem to its small size counterpart. Our results on standard datasets show that our system can achieve better localization accuracy and speed than the main state of the art. It obtains approximately 0.24m and 0.3◦ in less than 0.1 seconds.

Computer vision geo-location, awareness & detail

Computer vision (i.e., image understanding) involves understanding the 3D scene creating the image. Computer vision is challenging because it is the computer that decides how to act based on an understanding of the image. Key image understanding tasks include depth computation, as well as object detection, localization, recognition and tracking. Techniques up to now have not been able to perform any of these tasks robustly with the precision and accuracy demanded by many real-world applications. Additional complications include operational and environmental factors. For humans, visual recognition is fast and accurate, yet robust against occlusion, clutter, viewpoint variations, and changes in lighting conditions. Moreover, learning new categories requires minimal supervision and a very small set of exemplars. Achieving this level of performance in a wearable portable system would enable a great number of useful applications especially for enhancing mobile cell phone and camera operation. We demonstrate some of the computer vision techniques that we have developed and tested in real environments for applications in the field of automotive navigation, personal navigation, assistive devices and augmented reality. Some of the techniques include object detection and recognition, depth from motion, context recognition and the general task of mapping and localization. Our object detection techniques have shown to have performance close to 100%. We have actually shown that we can triangulate based on objects in the environment using only a camera; which can aid when GPS drops out such as in urban canyons and indoor environments. We argue that all of this potential can be packaged within a smart phone like an iphone. A category with the (minimum) three required fields

A Factorized Recursive Estimation of Structure and Motion from Image Velocities

We propose a new approach for the recursive estimation of structure and motion from image velocities. The estimation of structure and motion from image velocities is preferred to the estimation from pixel correspondences when the image displacements are small, since the former approach provides a stronger constraint being based on the instantaneous equation of rigid bodies motion. However the recursive estimation when dealing with image velocities is harder than its counterpart (in the case of pixel correspondences) since the number of points is usually larger and the equations are more involved. For this reason, in contrast to the case of point correspondences, the approaches presented so far are mostly limited to assuming a known 3D motion, or estimating the motion and structure independently. The approach presented in this paper introduces a factorized particle filter for estimating simultaneously the 3D motion and depth. Each particle consists of a 3D motion and a set of probability distributions of the depths of the pixels. The recursive estimation is done in three stages. (1) a resampling and a prediction of new samples; (2) a recursive filtering of the individual depths distributions performed using Extended Kalman Filters; and (3)finally a reweighting of the particles based on the image measurement. Results on simulation data show the efficiency of the approach. Future work will focus on incorporating an estimation of object boundaries to be used in a following regularization step.

Towards Global Localization Using Global Descriptors

3D pose of a camera with respect to a 3D representation of the scene. IBL, despite being a trivial problem for small scenes, becomes quite challenging as the size of the scene grows. Aside from the computational burden, matching against a very large number of 3D keypoints spanning a wide variety of viewpoints, illumination, and areas is a very unreliable process that results in a large number of outliers and ambiguous situations. In recent years, a number of approaches have attempted to address the problem using paradigms such as bag-of-words, features co-occurrence and others, with varying degrees of success. This paper explores the use of global descriptors, in particular GIST, to tackle this problem. We present a system that relies on a similarity measure derived from GIST to qualify a limited number of 3D points for the matching process, hence reducing the problem to its small size counterpart. Our results on a standard dataset show that our system can achieve better localization accuracy than the state of the art at a fraction of the computational cost, which can used towards global localization.

Augmenting analytic SFM filters with frame-to-frame features ☆

Abstract In Structure From Motion (SFM), image features are matched in either an extended number of frames or only in pairs of consecutive frames. Traditionally, SFM filters have been applied using only one of the two matching paradigms, with the Long Range (LR) feature technique being more popular because of the fact that features that are matched across multiple frames provide stronger constraints on structure and motion. Nevertheless, Frame-to-Frame (F2F) features possess the desirable property of being abundant because of the large similarity that exists between closely spaced frames. Although the use of such features has been limited mostly to the determination of inter-frame camera motion, we argue that significant improvements can be attained in online filter-based SFM by integrating the F2F features into filters that use LR features. The main contributions of this paper are twofold. First, it presents a new method that enables the incorporation of F2F information in any analytical filter in a fashion that requires minimal change to the existing filter. Our results show that by doing so, large increases in accuracy are achieved in both the structure and motion estimates. Second, thanks to mathematical simplifications we realize in the filter, we minimize the computational burden of F2F integration by two orders of magnitude, thereby enabling its real-time implementation. Experimental results on real and simulated data prove the success of the proposed approach.

Extending Filter-based Structure from Motion to Large Baselines

Filter-based Structure from Motion (SfM) approaches work usually in two steps: prediction and update. Prediction is the process of determining a prior distribution of the state vector at time t + 1 from the previous distribution at time t. Update is the process of adjusting the predicted distribution so it complies with the new received measurements at time t + 1. A key issue in those two steps is that the prediction and update should use statistically independent data and hence the same data can not be used in both of them. In Bayesian SfM filters that maintain a state vector composed of a set of 3D features and of the camera motion, and that use the projections of the 3D features in the images as measurements for the filter, this two step process faces a serious problem in the case where the baseline between successive frames (i.e. the displacement between the camera centers) is wide. This is because the previous estimate of the state vector at time t does not allow to solely determine an estimate of the motion at t+1 accurate enough for the filtering as there would be a significant change of motion between t and t + 1. In this paper, we provide a probabilistic solution to this problem by using features that are matched in the last three frames only. We show that this solution provides reliable prediction of the motion across large baselines.