Øystein Skotheim

A flexible 3D object localization system for industrial part handling

We present a flexible system that can scan and localize workpieces in 3D for assembly and pick and place operations. The system contains a vision robot that acquires 3D point clouds by performing sweeps with a laser triangulation sensor. Because the sensor is mounted on a robot, we can choose a viewpoint and a scanner trajectory that is optimal for a given task. For example, we can sweep along a semicircular trajectory in order to recognize and grasp parts from a pallet. With the same vision robot, we can perform a sweep of the packaging container in order to recognize and manipulate elements of cardboard. The system also incorporates software that recognizes and localizes objects based on an acquired 3D point cloud and a CAD model of the object to search for. The core matching algorithm is based on oriented point pairs and a Hough-like voting scheme. The method has been improved with a robust clustering algorithm as well as methods for pose verification and pose refinement that significantly increase the accuracy and robustness of the system. As an application of the system, an industrial prototype workcell is presented. The task is to recognize, grasp and transfer parts of office chairs, such as seats, seat backs and armrests, from a pallet to a cardboard container. It demonstrates how the vision system is easily set up to recognize three different components by using CAD models. The prototype workcell further demonstrates how the pose estimation results can be fed directly to a separate handling robot in order to grasp a chair seat. A series of ten experiments was performed where the chair seat was placed in arbitrary poses in the pallet. Pose estimations were performed in just over one second per experiment, and the obtained accuracy was well within the tolerances for the grasp operation in all ten cases.

A robotic concept for remote maintenance operations: A robust 3D object detection and pose estimation method and a novel robot tool

Future normally-unmanned oil platforms offer potentially significantly lower commissioning and operation costs than their current manned counterparts. The ability to initiate and perform remote inspection and maintenance (I&M) operations is crucial for maintaining such platforms. This paper presents a system solution, including key components such as a 3D robot vision system, a robot tool and a control architecture for remote I&M operations on processes similar to those on topside oil platforms. In particular, a case study on how to automatically replace a battery in a wireless process sensor is investigated. A novel robot tool for removing and re-attaching the sensor lid has been designed. Moreover, a robot control architecture for remote control of industrial-type robot manipulators is presented. A 3D robot vision system for localizing the sensor lid and the battery has been developed. The system utilizes structured light, using an off-the-shelf projector and a standard machine vision camera. A novel, robust and fast vision algorithm called 3D-MaMa has been adapted to work for object localization and pose estimation in complex scenes, in our case the process equipment in our lab facility. Experimental results from our lab facility are presented which describe a series of battery replacement operations for various unknown positions of the wireless sensor, and we report on accuracies and success ratios. The experiments demonstrate that the described vision system is able to recover the full pose and orientation of an object, and that the results are directly applicable for controlling advanced robot contact operations. Moreover, the custom-built lid operation tool demonstrates successful results.

HoloVision — A software package for reconstruction and analysis of digitally sampled holograms

HoloVision is a software package for performing digital holography on the Microsoft Windows platform. Basic theory for reconstruction of digitally sampled holograms is presented along with some more specific software implementation details. This includes the Fresnel method, the Convolution method and the Fourier method. A method involving a tilt of the reference wave and magnification through a numerical lens is presented to enlarge the visible region of the reconstructed image. Two different approaches for suppressing the undesired zero-order components are investigated. Examples are included for ordinary intensity images as well as for phase difference images from digital holographic interferometry.

Robust 3D object localization and pose estimation for random bin picking with the 3DMaMa algorithm

Enabling robots to automatically locate and pick up randomly placed and oriented objects from a bin is an important challenge in factory automation, replacing tedious and heavy manual labor. A system should be able to recognize and locate objects with a predefined shape and estimate the position with the precision necessary for a gripping robot to pick it up. We describe a system that consists of a structured light instrument for capturing 3D data and a robust approach for object location and pose estimation. The method does not depend on segmentation of range images, but instead searches through pairs of 2D manifolds to localize candidates for object match. This leads to an algorithm that is not very sensitive to scene complexity or the number of objects in the scene. Furthermore, the strategy for candidate search is easily reconfigurable to arbitrary objects. Experiments reported in this paper show the utility of the method on a general random bin picking problem, in this paper exemplified by localization of car parts with random position and orientation. Full pose estimation is done in less than 380 ms per image. We believe that the method is applicable for a wide range of industrial automation problems where precise localization of 3D objects in a scene is needed.

A flexible 3D vision system based on structured light for in-line product inspection

A flexible and highly configurable 3D vision system targeted for in-line product inspection is presented. The system includes a low cost 3D camera based on structured light and a set of flexible software tools that automate the measurement process. The specification of the measurement tasks is done in a first manual step. The user selects regions of the point cloud to analyze and specifies primitives to be characterized within these regions. After all measurement tasks have been specified, measurements can be carried out on successive parts automatically and without supervision. As a test case, a measurement cell for inspection of a V-shaped car component has been developed. The car component consists of two steel tubes attached to a central hub. Each of the tubes has an additional bushing clamped to its end. A measurement is performed in a few seconds and results in an ordered point cloud with 1.2 million points. The software is configured to fit cylinders to each of the steel tubes as well as to the inside of the bushings of the car part. The size, position and orientation of the fitted cylinders allow us to measure and verify a series of dimensions specified on the CAD drawing of the component with sub-millimetre accuracy.

A motion based real-time foveation control loop for rapid and relevant 3D laser scanning

We present an implementation of a novel foveating 3D sensor concept, inspired by the human eye, which intends to allow future robots to better interact with their surroundings. The sensor is based on a time-of-flight laser scanning technology, where each range distance measurement is performed individually for increased quality. Micro-mirrors enable detailed control on where and when each sample point is acquired in the scene. By finding regions-of-interest (ROIs) and mainly concentrating the data acquisition here, the spatial resolution or frame rate of these ROIs can be significantly increased compared to a non-foveating system. Foveation is enabled through a real-time implementation of a feed-back control loop for the sensor hardware, based on vision algorithms for 3D scene analysis. In this paper, we describe and apply an algorithm for detecting ROIs based on motion detection in range data using background modeling. Heuristics are incorporated to cope with camera motion. We report first results applying this algorithm to scenes with moving objects, and show that the foveation capability allows the frame rate to be increased by up to 8.2 compared to a non-foveating sensor, utilizing up to 99% of the potential frame rate increase. The incorporated heuristics significantly improves the foveations performance for moving camera scenes.

System for conveyor belt part picking using structured light and 3D pose estimation

Automatic picking of parts is an important challenge to solve within factory automation, because it can remove tedious manual work and save labor costs. One such application involves parts that arrive with random position and orientation on a conveyor belt. The parts should be picked off the conveyor belt and placed systematically into bins. We describe a system that consists of a structured light instrument for capturing 3D data and robust methods for aligning an input 3D template with a 3D image of the scene. The method uses general and robust pre-processing steps based on geometric primitives that allow the well-known Iterative Closest Point algorithm to converge quickly and robustly to the correct solution. The method has been demonstrated for localization of car parts with random position and orientation. We believe that the method is applicable for a wide range of industrial automation problems where precise localization of 3D objects in a scene is needed.

Fast high resolution 3D laser scanning by real-time object tracking and segmentation

This paper presents a real-time contour tracking and object segmentation algorithm for 3D range images. The algorithm is used to control a novel micro-mirror based imaging laser scanner, which provides a dynamic trade-off between resolution and frame rate. The micro-mirrors are controllable, enabling us to speed up acquisition significantly by only sampling on the object that is tracked and of interest. As the hardware is under development, we benchmark our algorithms on data from a SICK LMS100-10000 laser scanner mounted on a tilting platform. We find that objects are tracked and segmented well on pixel-level; that frame rate/resolution can be increased 3-4 times through our approach compared to scanners having static scan trajectories, and that the algorithm runs in 30 ms/image on a Intel Core i7 CPU using a single core.

Multi-technique platform for dynamic and static MEMS characterisation

MEMS characterisation is an important application area for interferometry. In this paper a Mach-Zehnder interferometer configuration is presented that combines both coherent and low coherent techniques in one setup. It incorporates the application of classical Laser Interferometry (LI) and Electronic Speckle Pattern Interferometry as well as classical Low Coherence Interferometry (LCI), full-field Optical Coherence Tomography and Low Coherence Speckle Interferometry. Digital Holography can be applied by minor modifications of the setup. The setup, working principle, and applications of the interferometer will be described. Measurements on a MEMS-based pressure sensor are presented. The sensor consists of a glass wafer attached to a silicon membrane. A cavity is etched into the glass wafer. The wafers are bonded and form a vacuum cavity. Membrane deformations are measured through the window using LI and LCI. LCI provides information about the shape of the glass window. Results from speckle techniques are compared with similar results from plane wave techniques. The influence of the glass window and the illumination of the object are investigated.

A real-time 3D range image sensor based on a novel tip-tilt-piston micromirror and dual frequency phase shifting

Structured light is a robust and accurate method for 3D range imaging in which one or more light patterns are projected onto the scene and observed with an off-axis camera. Commercial sensors typically utilize DMD- or LCD-based LED projectors, which produce good results but have a number of drawbacks, e.g. limited speed, limited depth of focus, large sensitivity to ambient light and somewhat low light efficiency. We present a 3D imaging system based on a laser light source and a novel tip-tilt-piston micro-mirror. Optical interference is utilized to create sinusoidal fringe patterns. The setup allows fast and easy control of both the frequency and the phase of the fringe patterns by altering the axes of the micro-mirror. For 3D reconstruction we have adapted a Dual Frequency Phase Shifting method which gives robust range measurements with sub-millimeter accuracy. The use of interference for generating sine patterns provides high light efficiency and good focusing properties. The use of a laser and a bandpass filter allows easy removal of ambient light. The fast response of the micro-mirror in combination with a high-speed camera and real-time processing on the GPU allows highly accurate 3D range image acquisition at video rates.