Kurt Höller

Time-of-Flight 3-D Endoscopy

This paper describes the first accomplishment of the Time-of-Flight (ToF) measurement principle via endoscope optics. The applicability of the approach is verified by in-vitro experiments. Off-the-shelf ToF camera sensors enable the per-pixel, on-chip, real-time, marker-less acquisition of distance information. The transfer of the emerging ToF measurement technique to endoscope optics is the basis for a new generation of ToF rigid or flexible 3-D endoscopes. No modification of the endoscope optic itself is necessary as only an enhancement of illumination unit and image sensors is necessary. The major contribution of this paper is threefold: First, the accomplishment of the ToF measurement principle via endoscope optics; second, the development and validation of a complete calibration and post-processing routine; third, accomplishment of extensive in-vitro experiments. Currently, a depth measurement precision of 0.89 mm at 20 fps with 3072 3-D points is achieved.

3D surface reconstruction for laparoscopic computer-assisted interventions: comparison of state-of-the-art methods

One of the main challenges related to computer-assisted laparoscopic surgery is the accurate registration of pre-operative planning images with patients anatomy. One popular approach for achieving this involves intraoperative 3D reconstruction of the target organs surface with methods based on multiple view geometry. The latter, however, require robust and fast algorithms for establishing correspondences between multiple images of the same scene. Recently, the first endoscope based on Time-of-Flight (ToF) camera technique was introduced. It generates dense range images with high update rates by continuously measuring the run-time of intensity modulated light. While this approach yielded promising results in initial experiments, the endoscopic ToF camera has not yet been evaluated in the context of related work. The aim of this paper was therefore to compare its performance with different state-of-the-art surface reconstruction methods on identical objects. For this purpose, surface data from a set of porcine organs as well as organ phantoms was acquired with four different cameras: a novel Time-of-Flight (ToF) endoscope, a standard ToF camera, a stereoscope, and a High Definition Television (HDTV) endoscope. The resulting reconstructed partial organ surfaces were then compared to corresponding ground truth shapes extracted from computed tomography (CT) data using a set of local and global distance metrics. The evaluation suggests that the ToF technique has high potential as means for intraoperative endoscopic surface registration.

Spatial orientation in translumenal surgery.

Abstract “Natural Orifice Translumenal Endoscopic Surgery” (NOTES) is assumed to offer significant benefits to patients, such as reduced trauma as well as reduced collateral damage. But the potential advantages of this new technology can only be achieved through safe and standardized operation methods. Several barriers, which have been identified during clinical practice in flexible intra-abdominal endoscopy, can only be solved with computer-assisted surgical (CAS) systems. In order to assist the surgeon during the intervention and enhance his visual possibilities, some of these CAS systems require 3-D information of the intervention site, for others 3-D information is even mandatory. Therefore it is evident that the definition and design of new technologies for CAS systems must be strongly considered. A 3-D endoscope, called “Multisensor-Time-of-Flight” (MUSTOF) endoscope, is actually being developed. Within these developments, an optical 3-D time-of-flight (TOF) sensor is attached to the proximal end of a common endoscope. The 3-D depth information obtained by this enhanced endoscope can furthermore be registered with preoperatively acquired 3-D volumetric datasets such as CT or MRI. These enhanced or augmented 3-D data volumes could then be used to find the transgastric or transcolonic entry point to the abdomen. Furthermore, such acquired endoscopic depth data can be used to provide better orientation within the abdomen. Moreover it can also prevent intra-operative collisions and provide an optimized field of view with the possibility for off-axis viewing. Furthermore, providing a stable horizon on video-endoscopic images, especially within non-rigid endoscopic surgery scenarios (particularly within NOTES), remains an open issue. Hence, our recently presented “endorientation” approach for automated image orientation rectification could turn out as an important contribution. It works with a tiny micro-electro-mechanical systems (MEMS) tri-axial inertial sensor that is placed on the distal tip of an endoscope. By measuring the impact of gravity on each of the three orthogonal axes the rotation angle can be estimated with some calculations out of these three acceleration values, which can be used to automatically rectify the endoscopic images using image processing methods. Using such enhanced, progressive endoscopic system extensions proposed in this article, translumenal surgery could in the future be performed in a safer and more feasible manner.

Clinical evaluation of Endorientation: Gravity related rectification for endoscopic images

Providing a stable horizon on endoscopic images especially in non-rigid endoscopic surgery (particularly NOTES) is still an open issue. Image rectification can be realized with a tiny MEMS tri-axial inertial sensor that is placed on the tip of an endoscope. By measuring the impact of gravity on each of the three orthogonal axes the rotation angle can be estimated with some calculations out of these three acceleration values. Achievable repetition rate for angle termination has to be above the usual endoscopic video frame rate of 25-30 Hz. The accelerometer frame rate can be set up to 400 Hz. Accuracy has to be less than one degree even within periods of high movement and superposed acceleration. Therefore an intelligent downsampling algorithm has to be found. The image rotation is performed by rotating digitally a capture of the endoscopic analog video signal. Improvements and benefits have been evaluated in a clinical evaluation: For different peritoneoscopic tasks time was taken and instrument position was tracked and recorded.

The “Iceberg Phenomenon” As Soon as One Technological Problem in NOTES Is Solved, the Next One Appears!

Purpose. Though already proclaimed about 7 years ago, natural orifice transluminal endoscopic surgery (NOTES) is still in its early stages. A multidisciplinary working team tried to analyze the technical obstacles and identify potential solutions. Methods. After a comprehensive review of the literature, a group of 3 surgeons, 1 gastroenterologist, 10 engineers, and 1 representative of biomedical industry defined the most important deficiencies within the system and then compiled as well as evaluated innovative technologies that could be used to help overcome these problems. These technologies were classified with regard to the time needed for their implementation and associated hindrances, where priority is based on the level of impact and significance that it would make. Results. Both visualization and actuation require significant improvement. Advanced illumination, mist elimination, image stabilization, view extension, 3-dimensional stereoscopy, and augmented reality are feasible options and could optimize visual information. Advanced mechatronic platforms with miniaturized, powerful actuators, and intuitive human–machine interfaces could optimize dexterity, as long as enabling technologies are used. The latter include depth maps in real time, precise navigation, fast pattern recognition, partial autonomy, and cognition systems. Conclusion. The majority of functional deficiencies that still exist in NOTES platforms could be overcome by a broad range of already existing or emerging enabling technologies. To combine them in an optimal manner, a permanent dialogue between researchers and clinicians is mandatory.

In-vitro Evaluation von endoskopischer Oberflächenrekonstruktion mittels Time-of-Flight-Kameratechnik

Eine der grosten Herausforderungen im Kontext von computergestutzten Systemen fur laparoskopische Eingriffe stellt die intraoperative prazise und schnelle Rekonstruktion von Organoberflachen dar. Diese ermoglicht eine Registrierung praoperativer Planungsdaten auf die Patientenanatomie zur Einblendung von Ziel- und Risikostrukturen in das Videobild. Vor diesem Hintergrund eroffnet die Time-of- Flight (ToF)-Kameratechnik aufgrund der schnellen und dichten 3D-Oberflachenvermessung neue Perspektiven fur die laparoskopische computerassistierte Chirurgie. In diesem Beitrag stellen wir die erste invitro Evaluationsstudie zum Vergleich ToF-basierter endoskopischer mit Stereoskopie-basierter Oberflachenrekonstruktion vor.

Orientierung endoskopischer Bilder: Rektifizierung durch Schwerkraft / Orientation of endoscopic images: rectification by gravity

Zusammenfassung Ein nach wie vor ungelöstes Problem in der endoskopischen Chirurgie (insbesondere mit Hilfe von flexiblen Video-Endoskopen) ist das Fehlen eines stabilen Horizonts der endoskopischen Bilder auf dem Monitor. Mit unserem „Endoscopy with the New Dimension of Orientation (ENDOrientation)“-Ansatz kann eine Korrektur der Bildverdrehung, sogar in der nicht-starren interventionellen Endoskopie (insbesondere NOTES) mit einem 3 mm schmalen dreiachsigen MEMS-Beschleunigungssensor realisiert werden, der an der Spitze des Endoskops positioniert wird. Dieser Sensor misst den Anteil der Erdanziehungskraft auf jede der drei orthogonalen Achsen. Nach einer initialen Kalibrierung und zeitlichen Filterung dieser drei Datenströme kann der Rotationswinkel direkt ermittelt werden. Die erreichbare Abtastrate liegt dabei deutlich über der gewöhnlichen Videoframe-Rate von 25 Hz, die Genauigkeit der Rotationserkennung liegt bei etwa einem Grad. Die Aufrichtung des Endoskopbildes in Relation zu einem stabilen Horizont wird in Echtzeit über eine digitale Rotation des analogen Endoskopie-Video-Signals ermöglicht. Die Verbesserungen und Vorteile wurden in Tierversuchen evaluiert. Die Koordination und Führung mehrerer Instrumente zugleich wurde bei endoskopischen Bildern mit stabilisiertem Horizont als deutlich intuitiver beurteilt. Die ermittelten Arbeitsschrittzeiten und Bewegungspfade stützten diese Beobachtung eindeutig. Abstract A known problem in endoscopic surgery (especially with flexible video endoscopes) is the absence of a stable horizon in endoscopic images displayed on a monitor. With our “ENDOrientation” approach, image rectification, even in non-rigid endoscopic surgery (particularly NOTES), can be realized with a tiny MEMS tri-axial inertial sensor placed on the tip of an endoscope. This sensor measures the impact of gravity on each of the three orthogonal accelerometer axes in real time. After an initial calibration and temporal filtering of these three data steams, the rotation angle of an endoscope can be estimated directly. The achievable sampling rate of the inertial sensor is above the usual endoscopic video frame rate of 25 Hz; the rotation accuracy is approximately one degree. The image rectification can be performed in real time by digitally rotating the endoscopic video signal. Improvements and benefits have been evaluated in animal studies: coordination and movement of different instruments was rated to be much more intuitive with a stable horizon on endoscopic images. The recorded time stamps and position tracks clearly support this observation.