Real-time clipper tip visibility detection using computer vision

Abstract

\n \n \n Laparoscopic cholecystectomy is one of the most common laparoscopic procedures. The critical phase of this intervention consists in dissecting the hepatocystic triangle and clipping the cystic artery and duct. Poor visibility of the clipper tips can result in unintentional clipping of neighboring tissues (“past-pointing”) or improper enclosing of the artery or duct, leading to hemorrhage or bile leaks. To improve patient safety during this clipping phase, we propose real-time intraoperative feedback to alert a surgeon when departing from safe behavior, i.e., losing visibility of the clipper tip. This is achieved using a deep learning model which classifies the clipper tip visibility in each frame.\n \n \n \n We tailored a dataset for our application by selecting frames containing a clipper that were selected from 300 laparoscopic cholecystectomy videos. These 122k frames were annotated with binary labels: clipper tip visible/invisible. A frame was labelled as tip visible when the tips of both clipper jaws were visible. Frames in which the clipper tip was occluded (e.g. by tissue) or frames with poor image quality (e.g., bad contrast, blurriness/smoke) were labelled as tip invisible. Frames from 29 videos were set aside for a test set; the remaining frames were used for training/validation (80%, 20% resp.).\n Using a 5-fold cross-validation scheme, convolutional neural networks (Resnet50 architecture) were trained to classify the clipper tip visibility in each frame. Finally, 5 neural networks trained in the cross-validation were ensembled into a single model by averaging their predictions.\n \n \n \n On the test set, the ensembled model achieved an AUROC of 0.906 and a specificity of 64.5% at 95% sensitivity. Looking at per video performance, the median specificity across videos raised to 76.6% (at 95% sensitivity). That is, the model would correctly detect 95% of the clipper tip not visible cases; in the majority of the interventions, 7 out of 10 warnings would be justified.\n \n \n \n We propose a novel safety feedback which warns on poor visibility of the clipper while clipping the cystic duct or artery. While being accurate, our technical solution runs in real-time, a requirement for intraoperative use. We believe this feedback can raise surgeons’ attentiveness when departing from safe visibility during this critical phase of laparoscopic cholecystectomy.\n