Jakob Wasza

Segmentation of kidneys using a new active shape model generation technique based on non-rigid image registration

Active shape models (ASMs) are widely used for applications in the field of image segmentation. Building an ASM requires to determine point correspondences for input training data, which usually results in a set of landmarks distributed according to the statistical variations. State-of-the-art methods solve this problem by minimizing the description length of all landmarks using a parametric mapping of the target shape (e.g. a sphere). In case of models composed of multiple sub-parts or highly non-convex shapes, these techniques feature substantial drawbacks. This article proposes a novel technique for solving the crucial correspondence problem using non-rigid image registration. Unlike existing approaches the new method yields more detailed ASMs and does not require explicit or parametric formulations of the problem. Compared to other methods, the already built ASM can be updated with additional prior knowledge in a very efficient manner. For this work, a training set of 3-D kidney pairs has been manually segmented from 41 CT images of different patients and forms the basis for a clinical evaluation. The novel registration based approach is compared to an already established algorithm that uses a minimum description length (MDL) formulation. The presented results indicate that the use of non-rigid image registration to solve the point correspondence problem leads to improved ASMs and more accurate segmentation results. The sensitivity could be increased by approximately 10%. Experiments to analyze the dependency on the user initialization also show a higher sensitivity of 5-15%. The mean squared error of the segmentation results and the ground truth manually classified data could also be reduced by 20-34% with respect to varying numbers of training samples.

Real-time preprocessing for dense 3-D range imaging on the GPU: Defect interpolation, bilateral temporal averaging and guided filtering

Recent advances in range imaging (RI) have enabled dense 3-D scene acquisition in real-time. However, due to physical limitations and the underlying range sampling principles, range data are subject to noise and may contain invalid measurements. Hence, data preprocessing is a prerequisite for practical applications but poses a challenge with respect to real-time constraints. In this paper, we propose a generic and modality-independent pipeline for efficient RI data preprocessing on the graphics processing unit (GPU). The contributions of this work are efficient GPU implementations of normalized convolution for the restoration of invalid measurements, bilateral temporal averaging for dynamic scenes, and guided filtering for edge-preserving denoising. Furthermore, we show that the transformation from range measurements to 3-D world coordinates can be computed efficiently on the GPU. The pipeline has been evaluated on real data from a Time-of-Flight sensor and Microsofts Kinect. In a run-time performance study, we show that for VGA-resolution data, our preprocessing pipeline runs at ∼100 fps on an off-the-shelf consumer GPU.

Multi-modal surface registration for markerless initial patient setup in radiation therapy using microsoft's Kinect sensor

In radiation therapy, prior to each treatment fraction, the patient must be aligned to computed tomography (CT) data. Patient setup verification systems based on range imaging (RI) can accurately verify the patient position and adjust the treatment table at a fine scale, but require an initial manual setup using lasers and skin markers. We propose a novel markerless solution that enables a fully-automatic initial coarse patient setup. The table transformation that brings template and reference data in congruence is estimated from point correspondences based on matching local surface descriptors. Inherently, this point-based registration approach is capable of coping with gross initial misalignments and partial matching. Facing the challenge of multi-modal surface registration (RI/CT), we have adapted state-of-the-art descriptors to achieve invariance to mesh resolution and robustness to variations in topology. In a case study on real data from a low-cost RI device (Microsoft Kinect), the performance of different descriptors is evaluated on anthropomorphic phantoms. Furthermore, we have investigated the systems resilience to deformations for mono-modal RI/RI registration of data from healthy volunteers. Under gross initial misalignments, our method resulted in an average angular error of 1.5° and an average translational error of 13.4 mm in RI/CT registration. This coarse patient setup provides a feasible initialization for subsequent refinement with verification systems.

Real-Time Range Imaging in Health Care: A Survey

The recent availability of dynamic, dense, and low-cost range imaging has gained widespread interest in health care. It opens up new opportunities and has an increasing impact on both research and commercial activities. This chapter presents a state-of-the-art survey on the integration of modern range imaging sensors into medical applications. The scope is to identify promising applications and methods, and to provide an overview of recent developments in this rapidly evolving domain. The survey covers a broad range of topics, including guidance in computer-assisted interventions, operation room monitoring and workflow analysis, touch-less interaction and on-patient visualization, as well as prevention and support in elderly care and rehabilitation. We put emphasis on dynamic and interactive tasks where real-time and dense 3-D imaging forms the key aspect. While considering different range imaging modalities that fulfill these requirements, we particularly investigate the impact of Time-of-Flight imaging in this domain. Eventually, we discuss practical demands and limitations, and open research issues and challenges that are of fundamental importance for the progression of the field.

Real-time RGB-D mapping and 3-D modeling on the GPU using the random ball cover data structure

The modeling of three-dimensional scene geometry from temporal point cloud streams is of particular interest for a variety of computer vision applications. With the advent of RGB-D imaging devices that deliver dense, metric and textured 6-D data in real-time, on-the-fly reconstruction of static environments has come into reach. In this paper, we propose a system for real-time point cloud mapping based on an efficient implementation of the iterative closest point (ICP) algorithm on the graphics processing unit (GPU). In order to achieve robust mappings at real-time performance, our nearest neighbor search evaluates both geometric and photometric information in a direct manner. For acceleration of the search space traversal, we exploit the inherent computing parallelism of GPUs. In this work, we have investigated the fitness of the random ball cover (RBC) data structure and search algorithm, originally proposed for high-dimensional problems, for 6-D data. In particular, we introduce a scheme that enables both fast RBC construction and queries. The proposed system is validated on an indoor scene modeling scenario. For dense data from the Microsoft Kinect sensor (640×480 px), our implementation achieved ICP runtimes of < 20 ms on an off-the-shelf consumer GPU.

Laparoscopic instrument localization using a 3-D Time-of-Flight/RGB endoscope

Minimally invasive procedures are of importance in modern surgery due to reduced operative trauma and recovery time. To enable robot assisted interventions, automatic tracking of endoscopie tools is an essential task. State-of-the-art techniques rely on 2-D color information only which is error prone for varying illumination and unpredictable color distribution within the human body. In this paper, we use a novel 3-D Time-of-Flight/RGB endoscope that allows to use both color and range information to locate laparoscopic instruments in 3-D. Regarding color and range information the proposed technique calculates a score to indicate which information is more reliable and adopts the next steps of the localization procedure based on this reliability. In experiments on real data the tool tip is located with an average 3-D distance error of less than 4 mm compared to manually labeled ground truth data with a frame-rate of 10 fps.

ToF meets RGB: novel multi-sensor super-resolution for hybrid 3-D endoscopy.

3-D endoscopy is an evolving field of research with the intention to improve safety and efficiency of minimally invasive surgeries. Time-of-Flight (ToF) imaging allows to acquire range data in real-time and has been engineered into a 3-D endoscope in combination with an RGB sensor (640x480px) as a hybrid imaging system, recently. However, the ToF sensor suffers from a low spatial resolution (640 x 480 px) and a poor signal-to-noise ratio. In this paper, we propose a novel multi-frame super-resolution framework to improve range images in a ToF/RGB multi-sensor setup. Our approach exploits high-resolution RGB data to estimate subpixel motion used as a cue for range super-resolution. The underlying non-parametric motion model based on optical flow makes the method applicable to endoscopic scenes with arbitrary endoscope movements. The proposed method was evaluated on synthetic and real images. Our approach improves the peak-signal-to-noise ratio by 1.6 dB and structural similarity by 0.02 compared to single-sensor super-resolution.

Real-Time motion compensated patient positioning and non-rigid deformation estimation using 4-d shape priors

Over the last years, range imaging (RI) techniques have been proposed for patient positioning and respiration analysis in motion compensation. Yet, current RI based approaches for patient positioning employ rigid-body transformations, thus neglecting free-form deformations induced by respiratory motion. Furthermore, RI based respiration analysis relies on non-rigid registration techniques with run-times of several seconds. In this paper we propose a real-time framework based on RI to perform respiratory motion compensated positioning and non-rigid surface deformation estimation in a joint manner. The core of our method are pre-procedurally obtained 4-D shape priors that drive the intra-procedural alignment of the patient to the reference state, simultaneously yielding a rigid-body table transformation and a free-form deformation accounting for respiratory motion. We show that our method outperforms conventional alignment strategies by a factor of 3.0 and 2.3 in the rotation and translation accuracy, respectively. Using a GPU based implementation, we achieve run-times of 40 ms.

Outlier Detection for Multi-Sensor Super-Resolution in Hybrid 3D Endoscopy

In hybrid 3D endoscopy, range data is used to augment pho- tometric information for minimally invasive surgery. As range sensors suffer from a rough spatial resolution and a low signal-to-noise ratio, subpixel motion between multiple range images is used as a cue for super- resolution to obtain reliable range data. Unfortunately, this method is sensitive to outliers in range images and the estimated subpixel displace- ments. In this paper, we propose an outlier detection scheme for robust super-resolution. First, we derive confidence maps to identify outliers in the displacement fields by correlation analysis of photometric data. Second, we apply an iteratively re-weighted least squares algorithm to obtain the associated range confidence maps. The joint confidence map is used to obtain super-resolved range data. We evaluate our approach on synthetic images and phantom data acquired by a Time-of-Flight/RGB endoscope. Our outlire detection improves the median peak-signal-to- noise ratio by 1.1 dB.

RITK: The Range Imaging Toolkit - A Framework for 3-D Range Image Stream Processing

The recent introduction of low-cost devices for real-time acquisition of dense 3-D range imaging (RI) streams has attracted a great deal of attention. However, to date, there exists no open source framework that is explicitly dedicated to real-time processing of RI streams. In this paper, we present the Range Imaging Toolkit (RITK). The goal is to provide a powerful yet intuitive software platform that facilitates the development of range image stream applications. RITK puts emphasis on real-time processing of range image streams and proposes the use of a dedicated pipeline mechanism. Furthermore, we introduce a powerful and convenient interface for range image processing on the graphics processing unit (GPU). Being designed thoroughly and in a generic manner, the toolkit is able to cope with the broad diversity of data streams provided by available RI devices and can easily be extended by custom range imaging sensors or processing modules. RITK is an open source project and will be made publicly available at http://www5.cs.fau.de/ritk.