Pierre Fite Georgel

Estimation of Location Uncertainty for Scale Invariant Feature Points

Image feature points are the basis for numerous computer vision tasks, such as pose estimation or object detection. State of the art algorithms detect features that are invariant to scale and orientation changes. While feature detectors and descriptors have been widely studied in terms of stability and repeatability, their localisation error has often been assumed to be uniform and insignificant. We argue that this assumption does not hold for scale-invariant feature detectors and demonstrate that the detection of features at different image scales actually has an influence on the localisation accuracy. A general framework to determine the uncertainty of multi-scale image features is introduced. This uncertainty is represented via anisotropic covariances with varying orientation and magnitude. We apply our framework to the well-known SIFT and SURF algorithms, detail its implementation and make it available 1 . Finally, the usefulness of such covariance estimates for bundle adjustment and homography computation is illustrated.

Personal Photograph Enhancement Using Internet Photo Collections

Given the growth of Internet photo collections, we now have a visual index of all major cities and tourist sites in the world. However, it is still a difficult task to capture that perfect shot with your own camera when visiting these places, especially when your camera itself has limitations, such as a limited field of view. In this paper, we propose a framework to overcome the imperfections of personal photographs of tourist sites using the rich information provided by large-scale Internet photo collections. Our method deploys state-of-the-art techniques for constructing initial 3D models from photo collections. The same techniques are then used to register personal photographs to these models, allowing us to augment personal 2D images with 3D information. This strong available scene prior allows us to address a number of traditionally challenging image enhancement techniques and achieve high-quality results using simple and robust algorithms. Specifically, we demonstrate automatic foreground segmentation, mono-to-stereo conversion, field-of-view expansion, photometric enhancement, and additionally automatic annotation with geolocation and tags. Our method clearly demonstrates some possible benefits of employing the rich information contained in online photo databases to efficiently enhance and augment ones own personal photographs.

Navigation Tools for Viewing Augmented CAD Models

During the production of an object, design modifications can occur because of, for example, corrections of design flaws, third-party components that must be replaced but are unavailable, or simply mistakes. So, undocumented discrepancies might exist between the CAD model and the final object. Although such discrepancies might not have consequences for the objects functionality, this gap between the virtual model and the real object could become problematic when the model is used for maintenance and repair. To solve these problems, we first developed a zoom-and-pan user interface for navigating within a mixed view. We then created tools that let users change direction and focus. These two tools allow navigation within a set of images using virtual 3D points, thus letting users intuitively access other mixed views.

A Unified Approach Combining Photometric and Geometric Information for Pose Estimation

In this paper, we present a novel approach for the relative pose estimation problem from point correspondences extracted from image pairs. Unlike classical algorithms, such as the Gold Standard algorithm, the proposed approach ensures that the matched points are photo-consistent throughout the pose estimation process. In fact, common algorithms use the photometric information to extract the feature points and to establish the 2D point correspondences. Then, they focus on minimizing, in a non-linear scheme, geometric distances between the projection of reconstructed 3D points and the coordinates of the extracted image points without taking the photometric information into account. The approach we propose in this paper merges geometric and photometric information in a unified cost function for the final non-linear minimization. This allows us to achieve results with higher precision and also with higher convergence frequency. Extensive experiments with ground truth on synthetic data show the superiority of the proposed approach in terms of robustness and precision. The simulation results have been confirmed by several tests on real image data.

How to augment the second image? Recovery of the translation scale in image to image registration

In this paper, we present an automatic pose estimation (6 DoF) technique to augment images using keyframes pre-registered to a CAD model. State of the art techniques recover the essential matrix (5 DoF) in an automatic manner, but include a manual step to align the image with the CAD reference system because the essential matrix does not provide the scale of the translation. We propose using planar structures to recover this scale automatically and to offer immediate augmentation. These techniques have been implemented in our augmented reality software. Qualitative tests are performed in an industrial environment.

On the Privacy Risks of Virtual Keyboards: Automatic Reconstruction of Typed Input from Compromising Reflections

We investigate the implications of the ubiquity of personal mobile devices and reveal new techniques for compromising the privacy of users typing on virtual keyboards. Specifically, we show that so-called compromising reflections (in, for example, a victims sunglasses) of a devices screen are sufficient to enable automated reconstruction, from video, of text typed on a virtual keyboard. Through the use of advanced computer vision and machine learning techniques, we are able to operate under extremely realistic threat models, in real-world operating conditions, which are far beyond the range of more traditional OCR-based attacks. In particular, our system does not require expensive and bulky telescopic lenses: rather, we make use of off-the-shelf, handheld video cameras. In addition, we make no limiting assumptions about the motion of the phone or of the camera, nor the typing style of the user, and are able to reconstruct accurate transcripts of recorded input, even when using footage captured in challenging environments (e.g., on a moving bus). To further underscore the extent of this threat, our system is able to achieve accurate results even at very large distances-up to 61 m for direct surveillance, and 12 m for sunglass reflections. We believe these results highlight the importance of adjusting privacy expectations in response to emerging technologies.

Closed-form solutions to multiple-view homography estimation

The quality of a mosaic depends on the projective alignment of the images involved. After point-correspondences between the images have been established, bundle adjustment finds an alignment considered optimal under certain hypotheses. This procedure minimizes a nonlinear cost and has to be initialized with care. It is very common to compose inter-frame homographies which have been computed with standard methods in order to get an initial global alignment. This technique is suboptimal if there is noise or missing ho-mographies as it typically uses a small part of the available data. We propose four new closed-form solutions. They all provide non-heuristic initial alignments using all the known inter-frame homographies. Our methods are tested with synthetic and real data and are compared to the standard method. These experiments reveal that our methods are more accurate, taking advantage of the redundant information available in the set of inter-frame homographies.

Photo-based Industrial Augmented Reality application using a single keyframe registration procedure

In the recent years, many Industrial Augmented Reality (IAR) applications are shifting from video to still images to create a mixed view. This new type of application is called Photo-based Augmented Reality. In order to guarantee the success of these applications, a simple and efficient registration method is required. We present a new method to register an image to a CAD model using a single keyframe. This registration is based on sparse 3D information from the model linked to the keyframe during its offline registration. We demonstrate this method in our in-house IAR software for Visual Inspection and Documentation: VID.

Navigation Tools for Augmented CAD Viewing

The creation of a computer aided design (CAD) model is the first step in the development of any modern physical product. This model will be used during the complete life cycle of the product: prototyping, fabrication, maintenance and upgrade. During the construction, a discrepancy between the model and the object can occur. In order to maintain and upgrade the object it is mandatory to have a model that represents the reality. So that one can have an up-to-date model one has to verify it and sometimes update it. We propose a scalable solution where CAD software has been augmented with pictures of the object. Still images have been aligned to the model allowing visualization of the model and the object at the same time. This creates what can be called a mixed view. The virtual camera that renders the model in a mixed view is restricted by the still image because the alignment between the image and the model has to be maintained. We developed tools to navigate in this mixed world. We transposed the zoom and pan from 2D user interfaces in order to navigate in the mixed view. Additionally we introduced tools for intuitive navigation within a set of mixed views.

Recovering the full pose from a single keyframe

Photo-based augmentation is a growing field in particular for industrial augmented reality (IAR) applications. Registration is at the core of every photo-based AR software. This alignment of the image to the 3D model coordinate system is usually achieved with fiducial markers. When a single keyframe is used, the unknown baseline length has to be estimated in order to superimpose virtual models onto the image. In this paper, we develop an automatic algorithm to augment the relative pose, estimated using a single keyframe, into a full pose that will permit superimposition. This is performed by propagating known 2D-3D correspondences to the target image using perspectively corrected template matching and followed by a refinement of the estimated full pose that combines geometric and photometric information. The performance and the stability of the proposed method is extensively demonstrated on synthetic data and its applicability is shown within an industrial AR software for visual inspection and documentation.