Brett L. Moore

Reinforcement learning: a novel method for optimal control of propofol-induced hypnosis.

Reinforcement learning (RL) is an intelligent systems technique with a history of success in difficult robotic control problems. Similar machine learning techniques, such as artificial neural networks and fuzzy logic, have been successfully applied to clinical control problems. Although RL presents a mathematically robust method of achieving optimal control in systems challenged with noise, nonlinearity, time delay, and uncertainty, no application of RL in clinical anesthesia has been reported.

Reinforcement Learning Versus Proportional-Integral-Derivative Control of Hypnosis in a Simulated Intraoperative Patient

BACKGROUND:Research has demonstrated the efficacy of closed-loop control of anesthesia using bispectral index (BIS) as the controlled variable. Model-based and proportional–integral– derivative (PID) controllers outperform manual control. We investigated the application of reinforcement learning (RL), an intelligent systems control method, to closed-loop BIS-guided, propofol-induced hypnosis in simulated intraoperative patients. We also compared the performance of the RL agent against that of a conventional PID controller. METHODS:The RL and PID controllers were evaluated during propofol induction and maintenance of hypnosis. The patient-hypnotic episodes were designed to challenge both controllers with varying degrees of interindividual variation and noxious surgical stimulation. Each controller was tested in 1000 simulated patients, and control performance was assessed by calculating the median performance error (MDPE), median absolute performance error (MDAPE), Wobble, and Divergence for each controller group. A separate analysis was performed for the induction and maintenance phases of hypnosis. RESULTS:During maintenance, RL control demonstrated an MDPE of −1% and an MDAPE of 3.75%, with 80% of the time at BIStarget ± 5. The PID controller yielded a MDPE of −8.5% and an MDAPE of 8.6%, with 57% of the time at BIStarget ± 5. In comparison, the MDAPE in the worst-controlled patient of the RL group was observed to be almost half that of the worst-controlled patient in the PID group. CONCLUSIONS:When compared with the PID controller, RL control resulted in slower induction but less overshoot and faster attainment of steady state. No difference in interindividual patient variation and noxious destabilizing challenge on control performance was observed between the 2 patient groups.

Fuzzy control for closed-loop, patient-specific hypnosis in intraoperative patients: A simulation study

Research has demonstrated the efficacy of closed-loop control of anesthesia using bispectral index (BIS) as the controlled variable, and the recent development of modelbased, patient-adaptive systems has considerably improved anesthetic control. To further explore the use of model-based control in anesthesia, we investigated the application of fuzzy control in the delivery of patient-specific propofol-induced hypnosis. In simulated intraoperative patients, the fuzzy controller demonstrated clinically acceptable performance, suggesting that further study is warranted.

SEDATION OF SIMULATED ICU PATIENTS USING REINFORCEMENT LEARNING BASED CONTROL

The Intensive Care Unit (ICU) is a challenging environment to both patient and caregiver. Continued shortages in staffing increase risk to patients. To evaluate the use of intelligent systems in the improvement of patient care, an intelligent agent was developed to regulate ICU patient sedation. A temporal differencing form of reinforcement learning was used to train the agent in the administration of intravenous propofol in simulated ICU patients. The agent utilized a well-studied pharmacokinetic model to calculate the distribution of drug within the patient. Pharmacodynamics were then estimated for the drug effect. A derivative of the electroencephalograms, the bispectral index, served as the system control variable. The agent demonstrated satisfactory control of the simulated patients consciousness level in static and dynamic setpoint conditions. The agent demonstrated superior stability and responsiveness when compared to a well-tuned PID controller, the control method of choice in closed-loop sedation control literature.

An Adaptive Neural Network Filter for Improved Patient State Estimation in Closed-Loop Anesthesia Control

Recent studies in the controlled administration of intravenous propofol favor a robust automated delivery control system in lieu of a manual controller. In previous work, a Reinforcement Learning (RL) controller was successfully tested in silico and in human volunteers with promising results. In this paper, an Adaptive Neural Network Filter (ANNF) is introduced in an effort to improve RL control of propofol hypnosis. The modified controller was tested in silico on simulated intraoperative patients, and its performance was compared against previously published results. Results from the experiments show that the new controller outperformed the previous controller in the maintenance of propofol anesthesia, with modest improvement in performance during anesthetic induction.