Hubertus Feußner

Modeling and segmentation of surgical workflow from laparoscopic video

Modeling and analyzing surgeries based on signals that are obtained automatically from the operating room (OR) is a field of recent interest. It can be valuable for analyzing and understanding surgical workflow, for skills evaluation and developing context-aware ORs. In minimally invasive surgery, laparoscopic video is easy to record but it is challenging to extract meaningful information from it. We propose a method that uses additional information about tool usage to perform a dimensionality reduction on image features. Using Canonical Correlation Analysis (CCA) a projection of a high-dimensional image feature space to a low dimensional space is obtained such that semantic information is extracted from the video. To model a surgery based on the signals in the reduced feature space two different statistical models are compared. The capability of segmenting a new surgery into phases only based on the video is evaluated. Dynamic Time Warping which strongly depends on the temporal order in combination with CCA shows the best results.

Workflow mining for visualization and analysis of surgeries

ObjectiveModeling the workflow of a surgery is a topic of growing interest. Workflow models can be used to analyze statistical properties of a surgery, for intuitive visualization, evaluation and other applications. In most cases, workflow models are created manually, which is a time consuming process that might suffer from a personal bias. In this work, an approach for automatic workflow mining is presented.Materials and methodsTen process logs, each describing a single instance of a laparoscopic cholecystectomy, are used to build a Hidden Markov Model (HMM). Using a merging approach, models at different levels of detail are generated. These embody statistical information concerning aspects like duration of actions or tool usage during the surgery.ResultsA Graphical User Interface (GUI) is presented, that uses a graph representation of the HMM to intuitively visualize surgical workflow. It allows changing the level of detail by expanding and merging nodes. The GUI can also be used to compare videos of surgeries which are synchronized to the model.ConclusionsThe proposed method allows automatic generation and visualization of a statistical model describing the workflow of a surgery.

Time-of-Flight 3-D Endoscopy

This paper describes the first accomplishment of the Time-of-Flight (ToF) measurement principle via endoscope optics. The applicability of the approach is verified by in-vitro experiments. Off-the-shelf ToF camera sensors enable the per-pixel, on-chip, real-time, marker-less acquisition of distance information. The transfer of the emerging ToF measurement technique to endoscope optics is the basis for a new generation of ToF rigid or flexible 3-D endoscopes. No modification of the endoscope optic itself is necessary as only an enhancement of illumination unit and image sensors is necessary. The major contribution of this paper is threefold: First, the accomplishment of the ToF measurement principle via endoscope optics; second, the development and validation of a complete calibration and post-processing routine; third, accomplishment of extensive in-vitro experiments. Currently, a depth measurement precision of 0.89 mm at 20 fps with 3072 3-D points is achieved.

Modeling and Online Recognition of Surgical Phases Using Hidden Markov Models

The amount of signals that can be recorded during a surgery, like tracking data or state of instruments, is constantly growing. These signals can be used to better understand surgical workflow and to build surgical assist systems that are aware of the current state of a surgery. This is a crucial issue for designing future systems that provide context-sensitive information and user interfaces. In this paper, Hidden Markov Models (HMM) are used to model a laparoscopic cholecystectomy. Seventeen signals, representing tool usage, from twelve surgeries are used to train the model. The use of a model merging approach is proposed to build the HMM topology and compared to other methods of initializing a HMM. The merging method allows building a model at a very fine level of detail that also reveals the workflow of a surgery in a human-understandable way. Results for detecting the current phase of a surgery and for predicting the remaining time of the procedure are presented.

Endoskopisches Operieren über natürliche Körperöffnungen (NOTES) in Deutschland: Zusammenfassung der Arbeitsgruppensitzungen der „D-NOTES 2009”

The D-NOTES-group met in June 2009 for an evaluation of ongoing preclinical and clinical activities in natural orifice endoscopic surgery and the further coordination of research in Germany. Different working groups with various topics were formed. Consensus statements among various participants with different scientific and medical background were initiated. In summary, important topics were handled such as the correct handling of bacterial contamination and related complications, the question of the ideal entry point and a secure closure, interdisciplinary cooperation, and matters related to training and education. Furthermore, participants agreed on terminological basics. A to-do-list for medical engineering was formulated.

ToF meets RGB: novel multi-sensor super-resolution for hybrid 3-D endoscopy.

3-D endoscopy is an evolving field of research with the intention to improve safety and efficiency of minimally invasive surgeries. Time-of-Flight (ToF) imaging allows to acquire range data in real-time and has been engineered into a 3-D endoscope in combination with an RGB sensor (640x480px) as a hybrid imaging system, recently. However, the ToF sensor suffers from a low spatial resolution (640 x 480 px) and a poor signal-to-noise ratio. In this paper, we propose a novel multi-frame super-resolution framework to improve range images in a ToF/RGB multi-sensor setup. Our approach exploits high-resolution RGB data to estimate subpixel motion used as a cue for range super-resolution. The underlying non-parametric motion model based on optical flow makes the method applicable to endoscopic scenes with arbitrary endoscope movements. The proposed method was evaluated on synthetic and real images. Our approach improves the peak-signal-to-noise ratio by 1.6 dB and structural similarity by 0.02 compared to single-sensor super-resolution.

Outlier Detection for Multi-Sensor Super-Resolution in Hybrid 3D Endoscopy

In hybrid 3D endoscopy, range data is used to augment pho- tometric information for minimally invasive surgery. As range sensors suffer from a rough spatial resolution and a low signal-to-noise ratio, subpixel motion between multiple range images is used as a cue for super- resolution to obtain reliable range data. Unfortunately, this method is sensitive to outliers in range images and the estimated subpixel displace- ments. In this paper, we propose an outlier detection scheme for robust super-resolution. First, we derive confidence maps to identify outliers in the displacement fields by correlation analysis of photometric data. Second, we apply an iteratively re-weighted least squares algorithm to obtain the associated range confidence maps. The joint confidence map is used to obtain super-resolved range data. We evaluate our approach on synthetic images and phantom data acquired by a Time-of-Flight/RGB endoscope. Our outlire detection improves the median peak-signal-to- noise ratio by 1.1 dB.

Multi-sensor super-resolution for hybrid range imaging with application to 3-D endoscopy and open surgery

In this paper, we propose a multi-sensor super-resolution framework for hybrid imaging to super-resolve data from one modality by taking advantage of additional guidance images of a complementary modality. This concept is applied to hybrid 3-D range imaging in image-guided surgery, where high-quality photometric data is exploited to enhance range images of low spatial resolution. We formulate super-resolution based on the maximum a-posteriori (MAP) principle and reconstruct high-resolution range data from multiple low-resolution frames and complementary photometric information. Robust motion estimation as required for super-resolution is performed on photometric data to derive displacement fields of subpixel accuracy for the associated range images. For improved reconstruction of depth discontinuities, a novel adaptive regularizer exploiting correlations between both modalities is embedded to MAP estimation. We evaluated our method on synthetic data as well as ex-vivo images in open surgery and endoscopy. The proposed multi-sensor framework improves the peak signal-to-noise ratio by 2 dB and structural similarity by 0.03 on average compared to conventional single-sensor approaches. In ex-vivo experiments on porcine organs, our method achieves substantial improvements in terms of depth discontinuity reconstruction.

Femtosecond laser aided processing of optical sensor fibers for 3D medical navigation and tracking (FiberNavi)

A new concept for fiber-optical 3D shape sensing applying femtosecond laser technology for highprecision direct writing of Bragg gratings within the core and the cladding of single core standard telecom fibers is presented. This new technology enables a cost-efficient and real-time 3D shape sensing and navigation of medical catheters or endoscopes only by means of passive optical sensor elements. First prototypes showed the possibility to achieve absolute navigation accuracy of four mm per meter and have successfully been tested in clinical environment.

Standortbestimmung und Entwicklung: NOTES -- Welcher Zugang?

Über einen transvaginalen oder transgastrischen Zugang wurden erste diagnostische und therapeutische NOTESEingriffe am Menschen bereits erfolgreich durchgeführt. Dennoch bleibt NOTES eine noch experimentelle Alternative zur konventionellen offenen und laparoskopischen Chirurgie. Ein idealer, universell anwendbarer Zugangsweg existiert nicht. Entscheidend dafür, welcher Weg gewählt wird, ist daher die jeweilige Indikation. Der transgastrale, der transvesikale, der transvaginale und der transrektale Zugang weisen jeweils spezifische Vor- und Nachteile auf. Nicht alle Ergebnisse aus dem Tiermodell lassen sich auf den Menschen übertragen, das muss bei der klinischen Anwendung berücksichtigt werden. Der klinische Einsatz von NOTES sollte zunächst ausschließlich innerhalb von Studien stattfinden. Langfristig wird eine enge Zusammenarbeit zwischen laparoskopisch tätigen Chirurgen, Gastroenterologen, Gynäkologen und eventuell Urologen notwendig sein, um diese vielversprechende Methode erfolgreich weiterzuentwickeln.