Paul M. Novotny

GPU Based Real-time Instrument Tracking with Three Dimensional Ultrasound

Real-time 3D ultrasound can enable new image-guided surgical procedures, but high data rates prohibit the use of traditional tracking techniques. We present a new method based on the modified Radon transform that identifies the axis of instrument shafts as bright patterns in planar projections. Instrument rotation and tip location are then determined using fiducial markers. These techniques are amenable to rapid execution on the current generation of personal computer graphics processor units (GPU). Our GPU implementation detected a surgical instrument in 31 ms, sufficient for real-time tracking at the 26 volumes per second rate of the ultrasound machine. A water tank experiment found instrument tip position errors of less than 0.2 mm, and an in vivo study tracked an instrument inside a beating porcine heart. The tracking results showed good correspondence to the actual movements of the instrument.

Robotic Motion Compensation for Beating Heart Intracardiac Surgery

3D ultrasound imaging has enabled minimally invasive, beating heart intracardiac procedures. However, rapid heart motion poses a serious challenge to the surgeon that is compounded by significant time delays and noise in 3D ultrasound. This paper investigates the concept of using a one-degree-of-freedom motion compensation system to synchronize with tissue motions that may be approximated by 1D motion models. We characterize the motion of the mitral valve annulus and show that it is well approximated by a 1D model. The subsequent development of a motion compensation instrument (MCI) is described, as well as an extended Kalman filter (EKF) that compensates for system delays. The benefits and robustness of motion compensation are tested in user trials under a series of non-ideal tracking conditions. Results indicate that the MCI provides an approximately 50% increase in dexterity and 50% decrease in force when compared with a solid tool, but is sensitive to time delays. We demonstrate that the use of the EKF for delay compensation restores performance, even in situations of high heart rate variability. The resulting system is tested in an in vitro 3D ultrasound-guided servoing task, yielding accurate tracking (1.15 mm root mean square) in the presence of noisy, time-delayed 3D ultrasound measurements.

Quasiperiodic predictive filtering for robot-assisted beating heart surgery

Beating heart procedures promise significant health benefits to patients but the fast motion of the heart poses a serious challenge to the surgeon. Robotic motion synchronization to heart movements could facilitate these surgeries, although for intracardiac procedures this requires the development of a predictive filter to compensate for the measurement noise and time delay present in 3D ultrasound imaging. In this paper, we present a quasiperiodic cardiac motion model and apply the extended Kalman filter to estimation of its parameters in real-time. We experimentally demonstrate high accuracy robot tracking to heart motion using this filter.

Real-Time Visual Servoing of a Robot Using Three-Dimensional Ultrasound

This paper presents a robotic system capable of using three-dimensional ultrasound to guide a surgical instrument to a tracked target location. Tracking of both the surgical instrument and target was done using image based algorithms on the real-time 3D ultrasound data. The tracking techniques are shown to be especially amenable for execution on powerful graphics processor units. By harnessing a graphics card, it was possible to detect both a surgical instrument and a surgical target at a rate of 25 Hz. The high update rate permits the use of tracked instrument and target locations for controlling a robot. Validation of the system was done in a water tank, where the robot moved the instrument to the target site with a mean error of 1.2 mm

Tool Localization in 3D Ultrasound Images

Real-time three-dimensional ultrasound has been demonstrated as a viable tool for guiding surgical procedures [1]. This visualization technique may enable a range of new minimally invasive techniques in cardiac and fetal surgery. It is becoming increasingly clear, however, that instruments such as endoscopic graspers when immersed in liquids display artifacts and irregularities that make it difficult to determine the tool’s location, orientation, and geometry (Figure 1a). In addition, the instrument’s shape can appear incomplete due to its orientation or obstacles in the field. Since the geometry and properties of these instruments are known a priori, it is feasible to combine this information with the ultrasound image data to locate and render an enhanced representation of the instrument. This paper introduces a preliminary study of enhancing tool display in three-dimensional ultrasound, focusing on the relationship between the tissue and instrument.

An active motion compensation instrument for beating heart mitral valve surgery

New 3D ultrasound visualization has enabled minimally invasive, beating-heart intracardiac procedures. However, rapid motion of internal heart structures limits the realization of these new procedures. This paper investigates the concept of using a single actuator to compensate for tissue motions which occur largely in one direction. We characterize mitral valve annulus motion and show that it is well approximated by a ID model. The subsequent development of a motion-compensating tool (MCT) is described. The resulting instrument was tested in user trials under a series of positional error and tracking delay conditions. Results indicate that the MCT provides an approximately 50% increase in dexterity and 50% decrease in applied force in comparison to a solid tool. The study also shows that MCT tracking efficacy is highly dependent on tracking delays, indicating the importance of predictive, cyclical control algorithms.

Image Guided Surgical Interventions

urgeons have traditionally performed procedures to treat diseases by aining direct access to the internal structures involved, and using direct isual inspection to diagnose and treat the defects. Much effort has gone nto identifying the most appropriate incisions and approaches to enable ull access inside body cavities, specific organs, or musculoskeletal tructures. Imaging has traditionally been used primarily for preoperative iagnosis and at times for surgical planning. Intraoperative imaging, hen used, was meant to provide further diagnostic information or to ssess adequacy of repair. In most cases radiograph static images or uoroscopy have been used in the operating room. As the application of ess invasive procedures has progressed, other imaging technology has een applied in an effort to address the limitations of simple radiographs r fluoroscopy. Computed tomography (CT), magnetic resonance imagng (MRI), ultrasound, nuclear radiographic imaging, and modified ptical imaging have been introduced to provide the information required o plan and perform complex interventions inside the body without the eed for direct open visual inspection. In parallel with the developments in imaging modalities, endoscopic urgery has advanced with the introduction of rigid and flexible scopes quipped with video cameras to magnify and display the image obtained. dvances in optics and digital electronics have combined to provide nparalleled image quality even with small diameter scopes, resulting in n explosion of endoscopic procedures involving virtually every structure n the body. The only real limitation to imaging has been the inability o see or “image” through opaque structures, since the irradiating or illuminating” energy provided through the scope has been almost xclusively visible light. This limitation has confined endoscopic surgery o areas where a natural body cavity or physical space can be accessed ith a scope and instruments, and filled with nonopaque medium such as gas or clear fluid. Despite these limitations, optical endoscopy has evolutionized the way many surgical procedures are performed, and has pawned a whole industry of instrument manufacturers that, in conjunc-

Stereoscopic vision display technology in real-time three-dimensional echocardiography-guided intracardiac beating-heart surgery

OBJECTIVE Stereoscopic vision display technology has been shown to be a useful tool in image-guided surgical interventions. However, the concept has not been applied to 3-dimensional echocardiography-guided cardiac procedures. We evaluated stereoscopic vision display as an aid for intracardiac navigation during 3-dimensional echocardiography-guided beating-heart surgery in a model of atrial septal defect closure. METHODS An atrial septal defect (6 mm) was created in 6 pigs using 3-dimensional echocardiography guidance. The defect was then closed using a catheter-based patch delivery system, and the patch was attached with tissue mini-anchors. Stereoscopic vision was generated with a high-performance volume renderer with stereoscopic glasses. Three-dimensional echocardiography with stereoscopic vision display was compared with 3-dimensional echocardiography with standard display for guidance of surgical repair. Task performance measures for each anchor placement (N = 32 per group) were completion time, trajectory of the tip of the anchor deployment device, and accuracy of the anchor placement. RESULTS The mean time of the anchor deployment for stereoscopic vision display group was shorter by 44% compared with the standard display group: 9.7 +/- 0.9 seconds versus 17.2 +/- 0.9 seconds (P < .001). Trajectory tracking of the anchor deployment device tip demonstrated greater navigational accuracy measured by trajectory deviation: 3.8 +/- 0.7 mm versus 6.1 +/- 0.3 mm, 38% improvement (P < .01). Accuracy of anchor placement was not significantly different: 2.3 +/- 0.3 mm for the stereoscopic vision display group versus 2.3 +/- 0.3 mm for the standard display group. CONCLUSION Stereoscopic vision display combined with 3-dimensional echocardiography improved the visualization of 3-dimensional echocardiography ultrasound images, decreased the time required for surgical task completion, and increased the precision of instrument navigation, potentially improving the safety of beating-heart intracardiac surgical interventions.

GPU based real-time instrument tracking with three dimensional ultrasound

Real-time three-dimensional ultrasound enables new intracardiac surgical procedures, but the distorted appearance of instruments in ultrasound poses a challenge to surgeons. This paper presents a detection technique that identifies the position of the instrument within the ultrasound volume. The algorithm uses a form of the generalized Radon transform to search for long straight objects in the ultrasound image, a feature characteristic of instruments and not found in cardiac tissue. When combined with passive markers placed on the instrument shaft, the full position and orientation of the instrument is found in 3D space. This detection technique is amenable to rapid execution on the current generation of personal computer graphics processor units (GPU). Our GPU implementation detected a surgical instrument in 31 ms, sufficient for real-time tracking at the 25 volumes per second rate of the ultrasound machine. A water tank experiment found instrument orientation errors of 1.1 degrees and tip position errors of less than 1.8mm. Finally, an in vivo study demonstrated successful instrument tracking inside a beating porcine heart.

Starting on the Right Track: Introducing Students to Mechanical Engineering With a Project-Based Machine Design Course

Over the past four years, we have redesigned Harvard’s introductory mechanical engineering course to introduce the principles, practices, and pleasures of mechanical engineering in an accessible format. The main goals of the course are to provide experience in the design process, demonstrate the connection between engineering science and design early in the curriculum, and build student enthusiasm for engineering, serving to attract and retain students. Unlike most introductory mechanical engineering courses, we cover strength of materials and machine elements, material usually presented much later in the curriculum, in order to provide tools for the students to quantitatively evaluate their designs. By providing just enough of this background knowledge to allow for analysis of designs, we demonstrate the connection between engineering science and design early in curriculum and motivate in-depth coverage of these topics in later courses.