Marie-Odile Berger

Statistical Modeling and Recognition of Surgical Workflow

In this paper, we contribute to the development of context-aware operating rooms by introducing a novel approach to modeling and monitoring the workflow of surgical interventions. We first propose a new representation of interventions in terms of multidimensional time-series formed by synchronized signals acquired over time. We then introduce methods based on Dynamic Time Warping and Hidden Markov Models to analyze and process this data. This results in workflow models combining low-level signals with high-level information such as predefined phases, which can be used to detect actions and trigger an event. Two methods are presented to train these models, using either fully or partially labeled training surgeries. Results are given based on tool usage recordings from sixteen laparoscopic cholecystectomies performed by several surgeons.

Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery

This paper presents a method for real-time augmentation of vascular network and tumors during minimally invasive liver surgery. Internal structures computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to state-of-the-art methods, our method uses a real-time biomechanical model to compute a volumetric displacement field from partial three-dimensional liver surface motion. This permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Real-time augmentation results are presented on in vivo and phantom data and illustrate the benefits of such an approach for minimally invasive surgery.

Reconstructing while registering: a novel approach for markerless augmented reality

This paper addresses the registration problem for unprepared multi-planar scenes. An interactive process is proposed to obtain accurate results using only the texture information of planes. In particular, classical preparation steps (camera calibration, scene acquisition) are greatly simplified, since they are included in the on-line registration process. Results are shown on indoor and outdoor scenes. Videos are available at url http://www.loria.fr//spl tilde/gsimon/Ismar.

A semi-automatic method for resolving occlusion in augmented reality

Realistic merging of virtual and real objects requires that the augmented patterns be correctly occluded by foreground objects. In this paper we propose a semi-automatic method for resolving occlusion in augmented reality which makes use of key-views. Once the user has outlined the occluding objects in the key-views, our system detects automatically these occluding objects in the intermediate views. A region of interest that contains the occluding objects is first computed from the outlined silhouettes. One of the main contribution of this paper is that this region takes into account the uncertainty on the computed interframe motion. Then a deformable region-based approach is used to recover the actual occluding boundary within the region of interest from this prediction.

Towards autonomy in active contour models

The strengths and the drawbacks of active contour models are described, and the absolute necessity of a criterion for assessing the solutions is pointed out. A method called snake growing, based on successive lengthenings of the snake, is proposed. The strength of this approach is that, at each stage, good convergence conditions are realized and initialization problems can be eliminated.<<ETX>>

Workflow monitoring based on 3D motion features

Activity recognition has primarily addressed the identification of either actions or well-defined interactions among objects in a scene. In this work, we extend the scope to the study of workflow monitoring. In a workflow, ordered groups of activities (phases) with different durations take place in a constrained environment and create temporal patterns across the workflow instances. We address the problem of recognizing phases, based on exemplary recordings. We propose to use Workflow-HMMs, a form of HMMs augmented with phase probability variables that model the complete workflow process. This model takes into account the full temporal context which improves on-line recognition of the phases, especially in case of partial labeling. Targeted applications are workflow monitoring in hospitals and factories, where common action recognition approaches are difficult to apply. To avoid interfering with the normal workflow, we capture the activity of a room with a multiple-camera system. Additionally, we propose to rely on real-time low-level features (3D motion flow) to maintain a generic approach. We demonstrate our methods on sequences from medical procedures performed in a mock-up operating room. The sequences follow a complex workflow, containing various alternatives.

A two-stage robust statistical method for temporal registration from features of various type

A model registration system capable of tracking an object, the model of which is known, in an image sequence is presented. It integrates tracking, pose determination and updating of the visible features. The heart of our system is the pose computation method, which handles various features (points, lines and free-form curves) in a very robust way and is able to give a correct estimate of the pose even when tracking errors occur. The reliability of the system is shown on an augmented reality project.

A boosted segmentation method for surgical workflow analysis

As demands on hospital efficiency increase, there is a stronger need for automatic analysis, recovery, and modification of surgical workflows. Even though most of the previous work has dealt with higher level and hospital-wide workflow including issues like document management, workflow is also an important issue within the surgery room. Its study has a high potential, e.g., for building context-sensitive operating rooms, evaluating and training surgical staff, optimizing surgeries and generating automatic reports. In this paper we propose an approach to segment the surgical workflow into phases based on temporal synchronization of multidimensional state vectors. Our method is evaluated on the example of laparoscopic cholecystectomy with state vectors representing tool usage during the surgeries. The discriminative power of each instrument in regard to each phase is estimated using AdaBoost. A boosted version of the Dynamic Time Warping (DTW) algorithm is used to create a surgical reference model and to segment a newly observed surgery. Full cross-validation on ten surgeries is performed and the method is compared to standard DTW and to Hidden Markov Models.

Fully Automatic 3D/2D Subtracted Angiography Registration

Today, 3-D angiography volumes are routinely generated from rotational angiography sequences. In previous work [7], we have studied the precision reached by registering such volumes with classical 2-D angiography images, inferring this matching only from the sensors of the angiography machine. The error led by such a registration can be described as a 3-D rigid motion composed of a large translation and a small rotation.

Model of a vascular C-arm for 3D augmented fluoroscopy in interventional radiology

This paper deals with the modeling of a vascular C-arm to generate 3D augmented fluoroscopic images in an interventional radiology context. A methodology based on the use of a multi-image calibration is proposed to assess the physical behavior of the C-arm. From the knowledge of the main characteristics of the C-arm, realistic models of the acquisition geometry are proposed. Their accuracy was evaluated and experiments showed that the C-arm geometry can be predicted with a mean 2D reprojection error of 0.5 mm. The interest of 3D augmented fluoroscopy is also assessed on a clinical case.