Darko Katic

Context-aware Augmented Reality in laparoscopic surgery

Augmented Reality is a promising paradigm for intraoperative assistance. Yet, apart from technical issues, a major obstacle to its clinical application is the man-machine interaction. Visualization of unnecessary, obsolete or redundant information may cause confusion and distraction, reducing usefulness and acceptance of the assistance system. We propose a system capable of automatically filtering available information based on recognized phases in the operating room. Our system offers a specific selection of available visualizations which suit the surgeons needs best. The system was implemented for use in laparoscopic liver and gallbladder surgery and evaluated in phantom experiments in conjunction with expert interviews.

Knowledge-based situation interpretation for context-aware augmented reality in dental implant surgery

The objective of this research is to develop and evaluate a context-aware Augmented Reality system which filters content based on the local context of the surgical instrument. We optically track positions of the patient and the instrument and interpret this data to recognize the phase of the operation. Depending on the result, an appropriate visualization is generated and displayed. For the interpretation, we combine a rule-based, deductive approach and a case-based, inductive one. Both rely on a description-logic based ontology. In phantom experiments the system was used to support implant positioning in models of the mandible. It recognized the phase correctly and provided an appropriate visualization about 85% of the time. The knowledge-based concept for intraoperative assistance proved capable of generating useful visualizations in a timely manner. However, further work is necessary to improve accuracy and reduce the deviation from the actual and planned implant positions.

Surgical data science for next-generation interventions

Interventional healthcare will evolve from an artisanal craft based on the individual experiences, preferences and traditions of physicians into a discipline that relies on objective decision-making on the basis of large-scale data from heterogeneous sources.Lena Maier-Hein, Swaroop Vedula, Stefanie Speidel, Nassir Navab, Ron Kikinis, Adrian Park, Matthias Eisenmann, Hubertus Feussner, Germain Forestier, Stamatia Giannarou, Makoto Hashizume, Darko Katic, Hannes Kenngott, Michael Kranzfelder, Anand Malpani, Keno März, Thomas Neumuth, Nicolas Padoy, Carla Pugh, Nicolai Schoch, Danail Stoyanov, Russell Taylor, Martin Wagner, Gregory D. Hager, Pierre Jannin

Knowledge-Driven Formalization of Laparoscopic Surgeries for Rule-Based Intraoperative Context-Aware Assistance

The rise of intraoperatively available information threatens to outpace our abilities to process data and thus cause informational overload. Context-aware systems, filtering information to match the current situation in the OR, will be necessary to reap all benefits of integrated and computerized surgery. To interpret surgical situations, such systems need a robust set of knowledge to make sense of intraoperative measurements. Building on our own ontology for laparoscopy, we formalized the workflow of laparoscopic adrenalectomies, cholecystectomies and pancreatic resections and developed a novel, rule-based situation interpretation algorithm based on OWL and SWRL to recognize phases of these surgeries. The system was evaluated on ground truth data from 19 manually annotated surgeries with an average recognition rate of 89%.

Toward cognitive pipelines of medical assistance algorithms

PurposeAssistance algorithms for medical tasks have great potential to support physicians with their daily work. However, medicine is also one of the most demanding domains for computer-based support systems, since medical assistance tasks are complex and the practical experience of the physician is crucial. Recent developments in the area of cognitive computing appear to be well suited to tackle medicine as an application domain.MethodsWe propose a system based on the idea of cognitive computing and consisting of auto-configurable medical assistance algorithms and their self-adapting combination. The system enables automatic execution of new algorithms, given they are made available as Medical Cognitive Apps and are registered in a central semantic repository. Learning components can be added to the system to optimize the results in the cases when numerous Medical Cognitive Apps are available for the same task. Our prototypical implementation is applied to the areas of surgical phase recognition based on sensor data and image progressing for tumor progression mappings.ResultsOur results suggest that such assistance algorithms can be automatically configured in execution pipelines, candidate results can be automatically scored and combined, and the system can learn from experience. Furthermore, our evaluation shows that the Medical Cognitive Apps are providing the correct results as they did for local execution and run in a reasonable amount of time.ConclusionThe proposed solution is applicable to a variety of medical use cases and effectively supports the automated and self-adaptive configuration of cognitive pipelines based on medical interpretation algorithms.

Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy

PurposeComputer assistance is increasingly common in surgery. However, the amount of information is bound to overload processing abilities of surgeons. We propose methods to recognize the current phase of a surgery for context-aware information filtering. The purpose is to select the most suitable subset of information for surgical situations which require special assistance.MethodsWe combine formal knowledge, represented by an ontology, and experience-based knowledge, represented by training samples, to recognize phases. For this purpose, we have developed two different methods. Firstly, we use formal knowledge about possible phase transitions to create a composition of random forests. Secondly, we propose a method based on cultural optimization to infer formal rules from experience to recognize phases.ResultsThe proposed methods are compared with a purely formal knowledge-based approach using rules and a purely experience-based one using regular random forests. The comparative evaluation on laparoscopic pancreas resections and adrenalectomies employs a consistent set of quality criteria on clean and noisy input. The rule-based approaches proved best with noisefree data. The random forest-based ones were more robust in the presence of noise.ConclusionFormal and experience-based knowledge can be successfully combined for robust phase recognition.

Ontology-based prediction of surgical events in laparoscopic surgery

Context-aware technologies have great potential to help surgeons during laparoscopic interventions. Their underlying idea is to create systems which can adapt their assistance functions automatically to the situation in the OR, thus relieving surgeons from the burden of managing computer assisted surgery devices manually. To this purpose, a certain kind of understanding of the current situation in the OR is essential. Beyond that, anticipatory knowledge of incoming events is beneficial, e.g. for early warnings of imminent risk situations. To achieve the goal of predicting surgical events based on previously observed ones, we developed a language to describe surgeries and surgical events using Description Logics and integrated it with methods from computational linguistics. Using n-Grams to compute probabilities of followup events, we are able to make sensible predictions of upcoming events in real-time. The system was evaluated on professionally recorded and labeled surgeries and showed an average prediction rate of 80%.

Towards an open-source semantic data infrastructure for integrating clinical and scientific data in cognition-guided surgery

In the surgical domain, individual clinical experience, which is derived in large part from past clinical cases, plays an important role in the treatment decision process. Simultaneously the surgeon has to keep track of a large amount of clinical data, emerging from a number of heterogeneous systems during all phases of surgical treatment. This is complemented with the constantly growing knowledge derived from clinical studies and literature. To recall this vast amount of information at the right moment poses a growing challenge that should be supported by adequate technology. While many tools and projects aim at sharing or integrating data from various sources or even provide knowledge-based decision support - to our knowledge - no concept has been proposed that addresses the entire surgical pathway by accessing the entire information in order to provide context-aware cognitive assistance. Therefore a semantic representation and central storage of data and knowledge is a fundamental requirement. We present a semantic data infrastructure for integrating heterogeneous surgical data sources based on a common knowledge representation. A combination of the Extensible Neuroimaging Archive Toolkit (XNAT) with semantic web technologies, standardized interfaces and a common application platform enables applications to access and semantically annotate data, perform semantic reasoning and eventually create individual context-aware surgical assistance. The infrastructure meets the requirements of a cognitive surgical assistant system and has been successfully applied in various use cases. The system is based completely on free technologies and is available to the community as an open-source package.

Model-based formalization of medical knowledge for context-aware assistance in laparoscopic surgery

The increase of technological complexity in surgery has created a need for novel man-machine interaction techniques. Specifically, context-aware systems which automatically adapt themselves to the current circumstances in the OR have great potential in this regard. To create such systems, models of surgical procedures are vital, as they allow analyzing the current situation and assessing the context. For this purpose, we have developed a Surgical Process Model based on Description Logics. It incorporates general medical background knowledge as well as intraoperatively observed situational knowledge. The representation consists of three parts: the Background Knowledge Model, the Preoperative Process Model and the Integrated Intraoperative Process Model. All models depend on each other and create a concise view on the surgery. As a proof of concept, we applied the system to a specific intervention, the laparoscopic distal pancreatectomy.

Lightweight distributed computing for intraoperative real-time image guidance

In order to provide real-time intraoperative guidance, computer assisted surgery (CAS) systems often rely on computationally expensive algorithms. The real-time constraint is especially challenging if several components such as intraoperative image processing, soft tissue registration or context aware visualization are combined in a single system. In this paper, we present a lightweight approach to distribute the workload over several workstations based on the OpenIGTLink protocol. We use XML-based message passing for remote procedure calls and native types for transferring data such as images, meshes or point coordinates. Two different, but typical scenarios are considered in order to evaluate the performance of the new system. First, we analyze a real-time soft tissue registration algorithm based on a finite element (FE) model. Here, we use the proposed approach to distribute the computational workload between a primary workstation that handles sensor data processing and visualization and a dedicated workstation that runs the real-time FE algorithm. We show that the additional overhead that is introduced by the technique is small compared to the total execution time. Furthermore, the approach is used to speed up a context aware augmented reality based navigation system for dental implant surgery. In this scenario, the additional delay for running the computationally expensive reasoning server on a separate workstation is less than a millisecond. The results show that the presented approach is a promising strategy to speed up real-time CAS systems.