Leonardo S. Mattos

A novel computerized surgeon-machine interface for robot-assisted laser phonomicrosurgery.

To introduce a novel computerized surgical system for improved usability, intuitiveness, accuracy, and controllability in robot‐assisted laser phonomicrosurgery.

New Developments Towards Automated Blastocyst Microinjections

This paper presents results related to our latest semi-automated blastocyst microinjection system. Here, the improvements made to the microinjection system are described and evaluated. First, after replacing the original piezo-electric kinematic stage by a DC motor-based robot manipulator, experimentation showed that the speed and the precise motion control of pipettes were improved. Second, by introducing an X-Y stage into the system, to manipulate the Petri dish around the microscopes field of view, multiple microinjection speed was improved. Third, by using SSD template matching to track the injection pipette, rather than the cross-correlation template matching algorithm used in the original system, improvements were made to pipette localization. Under human control, this new semi-automated system gives improved microinjection performance metrics compared to previously obtained results. The system is also providing implicit human knowledge of the microinjection process via the human-control interface. It is the encoding of this knowledge that will lead to the first fully automated system. The semi-automated microinjection system is being tested and evaluated in the AMC at UNC-Chapel Hill.

Blastocyst Microinjection Automation

Blastocyst microinjections are routinely involved in the process of creating genetically modified mice for biomedical research, but their efficiency is highly dependent on the skills of the operators. As a consequence, much time and resources are required for training microinjection personnel. This situation has been aggravated by the rapid growth of genetic research, which has increased the demand for mutant animals. Therefore, increased productivity and efficiency in this area are highly desired. Here, we pursue these goals through the automation of a previously developed teleoperated blastocyst microinjection system. This included the design of a new system setup to facilitate automation, the definition of rules for automatic microinjections, the implementation of video processing algorithms to extract feedback information from microscope images, and the creation of control algorithms for process automation. Experimentation conducted with this new system and operator assistance during the cells delivery phase demonstrated a 75% microinjection success rate. In addition, implantation of the successfully injected blastocysts resulted in a 53% birth rate and a 20% yield of chimeras. These results proved that the developed system was capable of automatic blastocyst penetration and retraction, demonstrating the success of major steps toward full process automation.

A Fully Automated System for Adherent Cells Microinjection

This paper proposes an automated robotic system to perform cell microinjections to relieve human operators from this highly difficult and tedious manual procedure. The system, which uses commercial equipment currently found on most biomanipulation laboratories, consists of a multitask software framework combining computer vision and robotic control elements. The vision part features an injection pipette tracker and an automatic cell targeting system that is responsible for defining injection points within the contours of adherent cells in culture. The main challenge is the use of bright-field microscopy only, without the need for chemical markers normally employed to highlight the cells. Here, cells are identified and segmented using a threshold-based image processing technique working on defocused images. Fast and precise microinjection pipette positioning over the automatically defined targets is performed by a two-stage robotic system which achieves an average injection rate of 7.6 cells/min with a pipette positioning precision of 0.23 μm. The consistency of these microinjections and the performance of the visual targeting framework were experimentally evaluated using two cell lines (CHO-K1 and HEK) and over 500 cells. In these trials, the cells were automatically targeted and injected with a fluorescent marker, resulting in a correct cell detection rate of 87% and a successful marker delivery rate of 67.5%. These results demonstrate that the new system is capable of better performances than expert operators, highlighting its benefits and potential for large-scale application.

Semi-automated blastocyst microinjection

The focus of this paper is the design and development of a semi-automated system for microinjection of embryonic stem cells into blastocysts. Semi-automation is achieved through treating cell microinjection as a computer game. In this first phase, cell manipulation and microinjection is carried out using a joystick and an interactive graphical user interface (GUI). For this system to be developed further, to achieve full automation, this first phase had to succeed. The interactive GUI records and displays: real-time video images of the cells; processed images of cells; and the microinjection strategies adopted by operators during cell manipulation through a real-time video processing algorithm. Experiments showed that this first phase of the research was successful. Future research develop knowledge-based controllers using machine-learning techniques as the research drives towards its goal of fully automated cell microinjection

A virtual scalpel system for computer-assisted laser microsurgery

A medical robotic system for teleoperated laser microsurgery based on a concept we have called “virtual scalpel” is presented in this paper. This system allows surgeries to be safely and precisely performed using a graphics pen directly over a live video from the surgical site. This is shown to eliminate hand-eye coordination problems that affect other microsurgery systems and to make full use of the operators manual dexterity without requiring extra training. The implementation of this system, which is based on a tablet PC and a new motorized laser micromanipulator offering 1µm aiming accuracy within the traditional line-of-sight 2D operative space, is fully described. This includes details on the systems hardware and software structures and on its calibration process, which is essential for guaranteeing precise matching between a point touched on the live video and the laser aiming point at the surgical site. Together, the new hardware and software structures make both the calibration parameters and the laser aiming accuracy (on any plane orthogonal to the imaging axis) independent of the target distance and of its motions. Automatic laser control based on new intraoperative planning software and safety improvements based on virtual features are also described in this paper, which concludes by presenting results from sets of path following evaluation experiments conducted with 10 different subjects. These demonstrate an error reduction of almost 50% when using the virtual scalpel system versus the traditional laser microsurgery setup, and an 80% error reduction when using the automatic laser control routines, evidencing great improvements in terms of precision and controllability, and suggesting that the technological advances presented herein will lead to a significantly enhanced capacity for treating a variety of internal human pathologies.

A fast and precise micropipette positioning system based on continuous camera-robot recalibration and visual servoing

Micro-biomanipulations rely on fine motion control and precise tool positioning for the execution of delicate operations on biological structures. Appropriate control is typically enabled by the use of micromanipulators; however, manual execution of biomanipulations is difficult, requiring extensive operator training (up to one year) and meticulous work under fatiguing conditions. Automation can be an alternative to improve such operations. In this case, challenges such as interface design, system integration, acquisition of feedback data, and the design of control systems have to be dealt with. This paper concentrates in the later challenge and proposes a new strategy for fast and precise control of the micromanipulation robot. Here, problems related to tool position drift are dealt with using continuous system recalibration. In addition, speed limitations imposed by the vision system are eliminated by the definition of a guaranteed region of interest (gROI) for micropipette localizations. The final system combines the use of mapping functions and visual servoing to achieve a 60% speedup in positioning speed and an 85% reduction in the pipette positioning error. The design and evaluation of this new positioning system is described here, including detail explanations of its modules and their interconnections.

Passive sonar applications: target tracking and navigation of an autonomous robot

This paper demonstrates the use of small area acoustic array technology as passive sonar for an autonomous mobile robot sound localization and direction control. Real-time target tracking is based solely on received audio signals.

Next-generation micromanipulator for computer-assisted laser phonomicrosurgery

This paper describes a new motorized laser micromanipulator created for computer-assisted laser laryngeal microsurgeries. This new device is based on a single tip/tilt fast steering mirror driven by a parallel kinematics mechanism, which is shown to improve previous laser micromanipulator designs by offering highly accurate motorized laser aiming control from a small, simple, and robust system created for real operating settings. The integration of this new device to a previously developed assistive surgical system is also described, including the new calibration model implemented to enable precise laser beam control from the live microscope video displayed on a computer screen. High system accuracy is demonstrated by the results of trajectory following experiments, which reveal an average RMSE of only 0.06 mm over an 8.5 mm diameter circular path. These numbers are also shown to favorably compare to those obtained with a traditional laser micromanipulator, evidencing a 70% error reduction when using the new system. In addition, experiments under higher optical magnification (40× instead of 16×) demonstrate even better accuracy results, proving the new system is highly scalable and indicating it has the potential to greatly improve laser microsurgery quality and safety.

Design and control of a robotic system for assistive laser phonomicrosurgery

This paper presents the design, implementation and control of a novel robotic system for assistive laser phonomicrosurgeries. The goals here are to improve the precision, the controllability, the safety and the ergonomics of traditional transoral laser laryngeal surgeries. Successful steps achieved by the developed system in those directions are presented here, including the design, control and characterization of a novel laser micromanipulator system. Such system is shown to achieve very fast (181mm/s) and accurate (1µm resolution) laser beam aiming within an 11×11mm target area. The design and implementation of a high-level control system for safe operation of the new laser phonomicrosurgery equipment is also described in this paper. This controller runs on a personal computer, offering a comfortable and safe surgical environment for the surgeon. In addition, it implements safe teleoperation by allowing the definition of safe and exclusion zones for surgical laser aiming. Automation of ablation procedures is also implemented on this high-level controller and described here. This feature adds an extra level of safety for phonomicrosurgeries by allowing the precise execution of surgical plans defined by the surgeon.