Mirko Appel

Scene augmentation via the fusion of industrial drawings and uncalibrated images with a view to marker-less calibration

The application presented augments uncalibrated images of factories with industrial drawings. Industrial drawings are among the most important documents used during the lifetime of industrial environments. They are the only common documents used during design, installation, monitoring and control, maintenance, update and finally dismantling of industrial units. Leading traditional industries towards the full use of virtual and augmented reality technology is impossible unless industrial drawings are integrated into our systems. We provide the missing link between industrial drawings and digital images of industrial sites. On one hand, this could enable us to calibrate cameras and build a 3D model of the scene without using any calibration markers. On the other hand it brings industrial drawings, floor map, images and 3D models into one unified framework. This provides a solid foundation for building efficient enhanced virtual industrial environments. The augmented scene is obtained by perspective warping of an industrial drawing of the factory onto its floor, wherever the floor is visible. The visibility of the floor is determined using probabilistic reasoning over a set of clues including (1) floor color/intensity (2) image warping and differencing between an uncalibrated stereoscopic image pair using the ground plane homography. Experimental results illustrate the approach.

Registration of technical drawings and calibrated images for industrial augmented reality

Abstract. Despite tremendous progress in 3D modelling technology, most sites in traditional industries do not have a computer model of their facilities at their disposal. In these industries, 2D technical drawings are typically the most commonly used documents. In many cases, a database of fully calibrated and oriented photogrammetric images of parts of the plant is also available. These images are often used for metric measurement and 3D as-built modelling. For planning revamps and maintenance, it is necessary to use industrial drawings as well as images and 3D models represented in a common “world” coordinate system. This paper proposes a method for full integration of technical drawings, calibrated images and as-built 3D models. A new algorithm is developed in order to use only a few correspondences between points on a technical drawing and multiple images to estimate a metric planar transformation between the drawing and the world coordinate system. The paper describes the mathematical relationship between this transformation and the set of homographies needed for merging the technical drawing with all the calibrated images. The method is implemented and fully integrated into an industrial software we developed for 3D as-built reconstruction. We present examples of a real application, in which the method is successfully applied to create an augmented reality representation of a waste water plant.

Canonical Representation and Multi-View Geometry of Cylinders

This paper first introduces a canonical representation for cylinders. The canonical representation introduced here is closely related to the Plücker line representation. In this paper, we show that this representation is an appropriate one for computer vision applications. In particular, it allows us to easily develop a series of mathematical methods for pose estimation, 3D reconstruction, and motion estimation.One of the major novelties in this paper is the introduction of the main equations dominating the three view geometry of cylinders. We show the relationship between cylinders’ three-view geometry and that of lines (Spetsakis and Aloimonos, 1990; Weng et al., 1993) and points (Shashua, 1995) defined by the trilinear tensor (Hartley, 1997), and propose a linear method, which uses the correspondences between six cylinders across three views in order to recover the motion and structure.Cylindrical pipes and containers are the main components in the majority of chemical, water treatment and power plants, oil platforms, refineries and many other industrial installations. We have developed a professional software, called CyliCon, which allows efficient as-built reconstruction of such installations from a series of pre-calibrated images. Markers are used for this pre-calibration process. The theoretical and practical results in this paper represent the first steps towards marker-less calibration and reconstruction of such industrial sites. Here, the experimental results take advantage of the two-view and three-view geometry of cylinders introduced in this paper to provide initial camera calibration results.

3D reconstruction from co-registered orthographic and perspective images: theoretical framework and applications

In this paper a complete framework for 3D reconstruction is proposed which is based on co-registered technical drawings and images. Instead of gaining 3D information from multiple perspective images of a scene, we use only a single perspective image which is augmented by a registered 2D drawing of the scene. By superimposing images with drawings a new type of document is created which we call co-registered orthographic and perspective (COP) image. A COP image contains two views, whose spatial relationship in regard to each other is known. The corresponding features within the COP image, i.e. between the perspective view and the orthographic view, provides the same information as if two separate images were used. However, correspondence identification becomes much simpler and less erroneous, since the two views are overlaid. We provide a complete framework for registration of calibrated images with drawings as well as methods for reconstruction of primitive objects in industrial environment. Finally, results on scenes in industrial environment are presented to demonstrate the feasibility of the proposed method.

How to augment the second image? Recovery of the translation scale in image to image registration

In this paper, we present an automatic pose estimation (6 DoF) technique to augment images using keyframes pre-registered to a CAD model. State of the art techniques recover the essential matrix (5 DoF) in an automatic manner, but include a manual step to align the image with the CAD reference system because the essential matrix does not provide the scale of the translation. We propose using planar structures to recover this scale automatically and to offer immediate augmentation. These techniques have been implemented in our augmented reality software. Qualitative tests are performed in an industrial environment.

As-Built Reconstruction Using Images and Industrial Drawings

Despite the advanced 3D technology, majority of large industrial sites do not have access to a SD model of their facilities. These industries often use printed 2D drawings for almost all engineering designs and Updates. Here we introduce a new framework to combine different types of images available for an industrial site to recover the 3D structure. In particular, we consider two types of images: industrial drawings and photogrammetric images.