Christian Dahmen

Focus-based depth estimation in the SEM

Depth estimation in the scanning electron microscope (SEM) is an important topic especially for automation purposes. The SEM only delivers two-dimensional (2D) images, which makes manipulation processes difficult. In spite of the high depth of focus in the SEM, it is still possible to use depth from focus as a depth estimation technique for nanomanipulation applications. This article deals with the extraction of depth information from SEM images using focus-based methods, and possibilities to improve the performance of these algorithms. A new approach is presented, combining 2D object tracking with focus-based depth estimation methods in order to obtain a possibility for limited three-dimensional tracking.

Vision-Based Haptic Feedback for Remote Micromanipulation in–SEM Environment

This article presents an intuitive environment for remote micromanipulation composed of both haptic feedback and virtual reconstruction of the scene. To enable nonexpert users to perform complex teleoperated micromanipulation tasks, it is of utmost importance to provide them with information about the 3-D relative positions of the objects and the tools. Haptic feedback is an intuitive way to transmit such information. Since position sensors are not available at this scale, visual feedback is used to derive information about the scene. In this work, three different techniques are implemented, evaluated, and compared to derive the object positions from scanning electron microscope images. The modified correlation matching with generated template algorithm is accurate and provides reliable detection of objects. To track the tool, a marker-based approach is chosen since fast detection is required for stable haptic feedback. Information derived from these algorithms is used to propose an intuitive remote manipulation system that enables users situated in geographically distant sites to benefit from specific equipments, such as SEMs. Stability of the haptic feedback is ensured by the minimization of the delays, the computational efficiency of vision algorithms, and the proper tuning of the haptic coupling. Virtual guides are proposed to avoid any involuntary collisions between the tool and the objects. This approach is validated by a teleoperation involving melamine microspheres with a diameter of less than 2 μ m between Paris, France and Oldenburg, Germany.

MRI magnetic signature imaging, tracking and navigation for targeted micro/nano-capsule therapeutics

The propulsion of nano-ferromagnetic objects by means of MRI gradients is a promising approach to enable new forms of therapy. In this work, necessary techniques are presented to make this approach work. This includes path planning algorithms working on MRI data, ferromagnetic artifact imaging and a tracking algorithm which delivers position feedback for the microdevice and a propulsion sequence to enable interleaved magnetic propulsion and imaging. Using a dedicated software environment integrating path-planning methods and real-time tracking, a clinical MRI system is adapted to provide this new functionality for potential controlled interventional targeted therapeutic applications. Through MRI-based sensing analysis, this paper aims to propose a framework to plan a robust pathway to enhance the navigation ability to reach deep locations in human body. The proposed approaches are validated with different experiments.

VISUAL FEEDBACK METHODS FOR NANOHANDLING AUTOMATION

This paper presents different image processing methods and algorithms, which are needed to enable the reliable automation of nanohandling processes. These applications use the scanning electron microscope (SEM) as a visual sensor. SEMs are widespread and powerful tools for manipulations on the nanoscale. Due to the timing constraints in automated setups, the trade-off between SEM scanning speed and image quality is a concern for algorithm development. Tasks to be fulfilled on image data provided by the SEM include object recognition, object tracking and depth estimation. A selection of algorithms that have been applied in automated setups for nanomanipulation is discussed and validated.

Recognition and tracking of magnetic nanobots using MRI

By switching the gradient fields of a clinical magnetic resonance imaging (MRI) scanner, magnetic objects may be moved inside the cardiovascular system of the human body. The main field of application is seen in targeted drug therapy or embolization. A successful navigation of such devices requires continuous position determination. The occurrence of magnetic susceptibility artifacts can be exploited for this purpose. This article studies the effect of magnetic microscopic objects and nanoparticles on the process of MRI image formation in several imaging sequences. An MRI simulator based on evaluation of the Bloch equation is presented and applied for the simulation of artifact formation. Also, artifact properties are studied by experiments carried out on clinical MRI scanners, using magnetic objects placed into an agarose gel phantom. The transferability of the results from the gel phantom to a real tissue environment is proven. Based on the results, a two-stage procedure for visual servoing is proposed. It is initialized by object detection, carried out in a 3D scan. Object tracking is performed on fast 2D scans by template matching. The slice position is adjusted automatically in a feedback loop in order to follow object movements perpendicular to the image plane.

Tracking algorithms for closed-loop positioning of mobile microrobots

Visual servoing of tools and object detection are major tasks of serial micro-assembly, as the visual feedback is often the only way to track positions. High speed cameras can be used for high speed automated micromanipulation. This paper analyzes different object tracking approaches for their feasibility of high speed tracking. First, state of the art software algorithms are analyzed. Second, a hardware approach is analyzed. All algorithms are compared in terms of high-speed feasibility, movement speed and limitations. The comparison shows that hardware-based approaches are needed for update frequencies beyond 100 Hz.

Vision feedback for automated nanohandling

The handling of nanoscale objects is a field with high prospects and good perspectives. This paper presents different necessary image processing methods and algorithms, which are needed to enable the reliable automation of nanohandling processes. The imaging sensor used to gain access to the nanoscale world is the scanning electron microscope (SEM). Tasks to be fulfilled on image data from the SEM include object recognition, object tracking and depth estimation. All these algorithms are discussed and validated.

Actuation and Tracking of Ferromagnetic Objects Using MRI

The MRI is a common medical imaging modality with powerful properties for diagnostic imaging. In order to enable robotic approaches for medical therapies like targeted drug delivery and others, the MRI systems suitability for actuation has been evaluated and exploited recently. The technologies used for imaging like coils, etc., can also be used to exert forces on ferromagnetic objects. In this article, a tracking algorithm for the three-dimensional tracking of ferromagnetic objects inside the MRI is presented. It only relies on the acquisition of single slices, which intersect the generated artifact volume. Additionally, some factors influencing the imaging and tracking are discussed, as well as the generated force measured and compared to theoretical values.

MRI-based dynamic tracking of an untethered ferromagnetic microcapsule navigating in liquid

ABSTRACT The propulsion of ferromagnetic objects by means of MRI gradients is a promising approach to enable new forms of therapy. In this work, necessary techniques are presented to make this approach work. This includes path planning algorithms working on MRI data, ferromagnetic artifact imaging and a tracking algorithm which delivers position feedback for the ferromagnetic objects, and a propulsion sequence to enable interleaved magnetic propulsion and imaging. Using a dedicated software environment, integrating path-planning methods and real-time tracking, a clinical MRI system is adapted to provide this new functionality for controlled interventional targeted therapeutic applications. Through MRI–based sensing analysis, this article aims to propose a framework to plan a robust pathway to enhance the navigation ability to reach deep locations in the human body. The proposed approaches are validated with different experiments.

Tracking of objects in motion-distorted scanning electron microscope images

Microrobotics is a field of high interest in the recent past. Commonly used imaging modalities in this field include optical microscopes and scanning electron microscopes (SEM). Due to the scanning principle of the SEMs, objects in motion may not be represented in the SEM images the same way as they really are. Movement leads to movement artifacts. This effect strongly limits the possibilities for the use of image based methods for tracking and recognition and it also limits the possible speeds achievable for automated procedures. In this work, a tracking algorithm is proposed which is robust to artifacts and distortions generated by object motion. The algorithm is tested and validated in experiments.