Sven Mersmann

Can Masses of Non-Experts Train Highly Accurate Image Classifiers?

Machine learning algorithms are gaining increasing interest in the context of computer-assisted interventions. One of the bottlenecks so far, however, has been the availability of training data, typically generated by medical experts with very limited resources. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. In this work, we investigate the potential of crowdsourcing for segmenting medical instruments in endoscopic image data. Our study suggests that (1) segmentations computed from annotations of multiple anonymous non-experts are comparable to those made by medical experts and (2) training data generated by the crowd is of the same quality as that annotated by medical experts. Given the speed of annotation, scalability and low costs, this implies that the scientific community might no longer need to rely on experts to generate reference or training data for certain applications. To trigger further research in endoscopic image processing, the data used in this study will be made publicly available.

Crowdsourcing for Reference Correspondence Generation in Endoscopic Images

Computer-assisted minimally-invasive surgery (MIS) is often based on algorithms that require establishing correspondences between endoscopic images. However, reference annotations frequently required to train or validate a method are extremely difficult to obtain because they are typically made by a medical expert with very limited resources, and publicly available data sets are still far too small to capture the wide range of anatomical/scene variance. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. To our knowledge, this paper is the first to investigate the concept of crowdsourcing in the context of endoscopic video image annotation for computer-assisted MIS. According to our study on publicly available in vivo data with manual reference annotations, anonymous non-experts obtain a median annotation error of 2 px (n = 10,000). By applying cluster analysis to multiple annotations per correspondence, this error can be reduced to about 1 px, which is comparable to that obtained by medical experts (n = 500). We conclude that crowdsourcing is a viable method for generating high quality reference correspondences in endoscopic video images.

Towards mobile augmented reality for on-patient visualization of medical images

Despite considerable technical and algorithmic developments related to the fields of medical image acquisition and processing in the past decade, the devices used for visualization of medical images have undergone rather minor changes. As anatomical information is typically shown on monitors provided by a radiological work station, the physician has to mentally transfer internal structures shown on the screen to the patient. In this work, we present a new approach to on-patient visualization of 3D medical images, which combines the concept of augmented reality (AR) with an intuitive interaction scheme. The method requires mounting a Time-of-Flight (ToF) camera to a portable display (e.g., a tablet PC). During the visualization process, the pose of the camera and thus the viewing direction of the user is continuously determined with a surface matching algorithm. By moving the device along the body of the patient, the physician gets the impression of being able to look directly into the human body. The concept can be used for intervention planning, anatomy teaching and various other applications that require intuitive visualization of 3D data.

3D surface reconstruction for laparoscopic computer-assisted interventions: comparison of state-of-the-art methods

One of the main challenges related to computer-assisted laparoscopic surgery is the accurate registration of pre-operative planning images with patients anatomy. One popular approach for achieving this involves intraoperative 3D reconstruction of the target organs surface with methods based on multiple view geometry. The latter, however, require robust and fast algorithms for establishing correspondences between multiple images of the same scene. Recently, the first endoscope based on Time-of-Flight (ToF) camera technique was introduced. It generates dense range images with high update rates by continuously measuring the run-time of intensity modulated light. While this approach yielded promising results in initial experiments, the endoscopic ToF camera has not yet been evaluated in the context of related work. The aim of this paper was therefore to compare its performance with different state-of-the-art surface reconstruction methods on identical objects. For this purpose, surface data from a set of porcine organs as well as organ phantoms was acquired with four different cameras: a novel Time-of-Flight (ToF) endoscope, a standard ToF camera, a stereoscope, and a High Definition Television (HDTV) endoscope. The resulting reconstructed partial organ surfaces were then compared to corresponding ground truth shapes extracted from computed tomography (CT) data using a set of local and global distance metrics. The evaluation suggests that the ToF technique has high potential as means for intraoperative endoscopic surface registration.

MITK-ToF—Range data within MITK

PurposeThe time-of-flight (ToF) technique is an emerging technique for rapidly acquiring distance information and is becoming increasingly popular for intra-operative surface acquisition. Using the ToF technique as an intra-operative imaging modality requires seamless integration into the clinical workflow. We thus aim to integrate ToF support in an existing framework for medical image processing.MethodsMITK-ToF was implemented as an extension of the open-source C++ Medical Imaging Interaction Toolkit (MITK) and provides the basic functionality needed for rapid prototyping and development of image-guided therapy (IGT) applications that utilize range data for intra-operative surface acquisition. This framework was designed with a module-based architecture separating the hardware-dependent image acquisition task from the processing of the range data.ResultsThe first version of MITK-ToF has been released as an open-source toolkit and supports several ToF cameras and basic processing algorithms. The toolkit, a sample application, and a tutorial are available from http://mitk.org.ConclusionsWith the increased popularity of time-of-flight cameras for intra-operative surface acquisition, integration of range data supports into medical image processing toolkits such as MITK is a necessary step. Handling acquisition of range data from different cameras and processing of the data requires the establishment and use of software design principles that emphasize flexibility, extendibility, robustness, performance, and portability. The open-source toolkit MITK-ToF satisfies these requirements for the image-guided therapy community and was already used in several research projects.

Adaptive bilateral filter for image denoising and its application to in-vitro Time-of-Flight data

Image-guided therapy systems generally require registration of pre-operative planning data with the patients anatomy. One common approach to achieve this is to acquire intra-operative surface data and match it to surfaces extracted from the planning image. Although increasingly popular for surface generation in general, the novel Time-of-Flight (ToF) technology has not yet been applied in this context. This may be attributed to the fact that the ToF range images are subject to considerable noise. The contribution of this study is two-fold. Firstly, we present an adaption of the well-known bilateral filter for denoising ToF range images based on the noise characteristics of the camera. Secondly, we assess the quality of organ surfaces generated from ToF range data with and without bilateral smoothing using corresponding high resolution CT data as ground truth. According to an evaluation on five porcine organs, the root mean squared (RMS) distance between the denoised ToF data points and the reference computed tomography (CT) surfaces ranged from 3.0 mm (lung) to 9.0 mm (kidney). This corresponds to an error-reduction of up to 36% compared to the error of the original ToF surfaces.

Time-of-flight camera technique for augmented reality in computer-assisted interventions

Augmented reality (AR) for enhancement of intra-operative images is gaining increasing interest in the field of navigated medical interventions. In this context, various imaging modalities such as ultrasound (US), C-Arm computed tomography (CT) and endoscopic images have been applied to acquire intra-operative information about the patients anatomy. The aim of this paper was to evaluate the potential of the novel Time-of-Flight (ToF) camera technique as means for markerless intra-operative registration. For this purpose, ToF range data and corresponding CT images were acquired from a set of explanted non-transplantable human and porcine organs equipped with a set of marker that served as targets. Based on a rigid matching of the surfaces generated from the ToF images with the organ surfaces generated from the CT data, the targets extracted from the planning images were superimposed on the 2D ToF intensity images, and the target visualization error (TVE) was computed as quality measure. Color video data of the same organs were further used to assess the TVE of a previously proposed marker-based registration method. The ToF-based registration showed promising accuracy yielding a mean TVE of 2.5±1.1 mm compared to 0.7±0.4 mm with the marker-based approach. Furthermore, the target registration error (TRE) was assessed to determine the anisotropy in the localization error of ToF image data. The TRE was 8.9± 4.7 mm on average indicating a high localization error in the viewing direction of the camera. Nevertheless, the young ToF technique may become a valuable means for intra-operative surface acquisition. Future work should focus on the calibration of systematic distance errors.

Calibration of time-of-flight cameras for accurate intraoperative surface reconstruction.

PURPOSE In image-guided surgery (IGS) intraoperative image acquisition of tissue shape, motion, and morphology is one of the main challenges. Recently, time-of-flight (ToF) cameras have emerged as a new means for fast range image acquisition that can be used for multimodal registration of the patient anatomy during surgery. The major drawbacks of ToF cameras are systematic errors in the image acquisition technique that compromise the quality of the measured range images. In this paper, we propose a calibration concept that, for the first time, accounts for all known systematic errors affecting the quality of ToF range images. Laboratory and in vitro experiments assess its performance in the context of IGS. METHODS For calibration the camera-related error sources depending on the sensor, the sensor temperature and the set integration time are corrected first, followed by the scene-specific errors, which are modeled as function of the measured distance, the amplitude and the radial distance to the principal point of the camera. Accounting for the high accuracy demands in IGS, we use a custom-made calibration device to provide reference distance data, the cameras are calibrated too. To evaluate the mitigation of the error, the remaining residual error after ToF depth calibration was compared with that arising from using the manufacturer routines for several state-of-the-art ToF cameras. The accuracy of reconstructed ToF surfaces was investigated after multimodal registration with computed tomography (CT) data of liver models by assessment of the target registration error (TRE) of markers introduced in the livers. RESULTS For the inspected distance range of up to 2 m, our calibration approach yielded a mean residual error to reference data ranging from 1.5±4.3 mm for the best camera to 7.2±11.0 mm. When compared to the data obtained from the manufacturer routines, the residual error was reduced by at least 78% from worst calibration result to most accurate manufacturer data. After registration of the CT data with the ToF data, the mean TRE ranged from 3.7±2.1 mm for point-based and 5.7±1.9 mm for surface-based registration for the best camera to 6.2±3.4 and 11.1±2.8 mm, respectively. Compared to data provided by the manufacturer, the mean TRE decreased by 8%-60% for point-based and by 18%-74% for surface-based registration. CONCLUSIONS Using the proposed calibration approach improved the measurement accuracy of all investigated ToF cameras. Although evaluated in the context of intraoperative image acquisition, the proposed calibration procedure can easily be applied to other medical applications using ToF cameras, such as patient positioning or respiratory motion tracking in radiotherapy.

Registration of partially overlapping surfaces for range image based augmented reality on mobile devices

Visualization of anatomical data for disease diagnosis, surgical planning, or orientation during interventional therapy is an integral part of modern health care. However, as anatomical information is typically shown on monitors provided by a radiological work station, the physician has to mentally transfer internal structures shown on the screen to the patient. To address this issue, we recently presented a new approach to on-patient visualization of 3D medical images, which combines the concept of augmented reality (AR) with an intuitive interaction scheme. Our method requires mounting a range imaging device, such as a Time-of-Flight (ToF) camera, to a portable display (e.g. a tablet PC). During the visualization process, the pose of the camera and thus the viewing direction of the user is continuously determined with a surface matching algorithm. By moving the device along the body of the patient, the physician is given the impression of looking directly into the human body. In this paper, we present and evaluate a new method for camera pose estimation based on an anisotropic trimmed variant of the well-known iterative closest point (ICP) algorithm. According to in-silico and in-vivo experiments performed with computed tomography (CT) and ToF data of human faces, knees and abdomens, our new method is better suited for surface registration with ToF data than the established trimmed variant of the ICP, reducing the target registration error (TRE) by more than 60%. The TRE obtained (approx. 4-5 mm) is promising for AR visualization, but clinical applications require maximization of robustness and run-time.

MITK-ToF: Time-of-Flight Kamera-Integration in das Medical Imaging Interaction Toolkit

Time-of-Flight (ToF) Kameras bieten aufgrund derMoglichkeit zur schnellen und robusten Oberflachenerfassung groses Potential fur die intra-interventionelle Akquise von Informationen uber die Patientenanatomie und Organmorphologie. Eine Nutzung der neuen Technik als medizinische Bildgebungsmodalitat erfordert eine nahtlose Integration in die verwendete Softwareinfrastruktur. Nachdem sich das Medical Imaging Interaction Toolkit (MITK) als Framework fur die medizinische Bildverarbeitung etabliert hat, stellen wir in diesem Beitrag eine Erweiterung um die Anbindung von Time-of-Flight Kamerasystemen vor (MITK-ToF). MITK-ToF unterstutzt die Ansteuerung verbreiteter ToF-Kameratypen und stellt die akquirierten Bilddaten in mehreren Dateiformaten bereit. Die durch die Integration in das MITK ermoglichte Verwendung der dort vorhandenen Komponenten zur Bildverabeitung, Visualisierung und Interaktion ermoglichen die Entwicklung komplexer ToF-basierter Anwendungen. Wir zeigen das Potential des vorgestellten Toolkits beispielhaft anhand einer Anwendung zur Darstellung, Aufnahme und Wiedergabe von ToF Daten. Zur BVM 2011 wird eine erste open-source Version des Toolkits veroffentlicht.