S. Syafiie

Model-free control based on reinforcement learning for a wastewater treatment problem

This article presents a proposal, based on the model-free learning control (MFLC) approach, for the control of the advanced oxidation process in wastewater plants. This is prompted by the fact that many organic pollutants in industrial wastewaters are resistant to conventional biological treatments, and the fact that advanced oxidation processes, controlled with learning controllers measuring the oxidation-reduction potential (ORP), give a cost-effective solution. The proposed automation strategy denoted MFLC-MSA is based on the integration of reinforcement learning with multiple step actions. This enables the most adequate control strategy to be learned directly from the process response to selected control inputs. Thus, the proposed methodology is satisfactory for oxidation processes of wastewater treatment plants, where the development of an adequate model for control design is usually too costly. The algorithm proposed has been tested in a lab pilot plant, where phenolic wastewater is oxidized to carboxylic acids and carbon dioxide. The obtained experimental results show that the proposed MFLC-MSA strategy can achieve good performance to guarantee on-specification discharge at maximum degradation rate using readily available measurements such as pH and ORP, inferential measurements of oxidation kinetics and peroxide consumption, respectively.

Learning control for batch thermal sterilization of canned foods

A control technique based on Reinforcement Learning is proposed for the thermal sterilization of canned foods. The proposed controller has the objective of ensuring a given degree of sterilization during Heating (by providing a minimum temperature inside the cans during a given time) and then a smooth Cooling, avoiding sudden pressure variations. For this, three automatic control valves are manipulated by the controller: a valve that regulates the admission of steam during Heating, and a valve that regulate the admission of air, together with a bleeder valve, during Cooling. As dynamical models of this kind of processes are too complex and involve many uncertainties, controllers based on learning are proposed. Thus, based on the control objectives and the constraints on input and output variables, the proposed controllers learn the most adequate control actions by looking up a certain matrix that contains the state-action mapping, starting from a preselected state-action space. This state-action matrix is constantly updated based on the performance obtained with the applied control actions. Experimental results at laboratory scale show the advantages of the proposed technique for this kind of processes.

MODEL-FREE INTELLIGENT CONTROL USING REINFORCEMENT LEARNING AND TEMPORAL ABSTRACTION-APPLIED TO pH CONTROL

Abstract This article presents a solution to pH control based on model-free intelligent control (MFIC) using reinforcement learning. This control technique is proposed because the algorithm gives a general solution for acid-base system, yet simple enough for its implementation in existing control hardware. In standard reinforcement learning, the interaction between an agent and the environment is based on a fixed time scale: during learning, the agent can select several primitive actions depending on the system state. A novel solution is presented, using multi-step actions (MSA): actions on multiple time scales consist of several identical primitive actions. This solves the problem of determining a suitable fixed time scale to select control actions so as to trade off accuracy in control against learning complexity. The application of multi-step actions on a simulated pH process shows that the proposed MFIC learns to control adequately the neutralization process.

Modelling for dynamic simulation of pretreatment in Reverse Osmosis plants

The development of mathematical dynamical models of the pretreatment process in reverse osmosis plants is presented. The objective is to use the models for testing and comparison of control strategies in reverse osmosis plants. Thus, these models have been developed to implement them using off-the-shelf software (EcoSimPro), so that advanced control algorithms can be easily tested. During modeling, the parameters are selected to be simple to obtain from available plant measurements.

Model-Free Learning Control of Chemical Processes

Learning is the nature for human being. For example, a school-student learns a subject by doing exercise and home-work. Then, a school-teacher grades the school-student’s works. From this student and teacher interaction, the ability of the student mastering the subject is a feedback that the previous teaching method is successful or failure. As a result, the teacher will change the teaching method to improve the student ability for mastering the subject. This is a picture that the reinforcement learning (RL) agent learns the environment. Process control mainly focuses on controlling variable such as pressure, level, flow, temperature, pH, level in the process industries. However, the methodologies and principles are the same as in all control fields. The early successful application control strategy in process control is in evolution of the PID controller and Ziegler-Nichols tuning method (Ziegler and Nichols, 1942). Till nowadays, 95% of the controllers implemented in the process industries are PID-type (Chidambaram and See, 2002). However, as (i) the industrial demands (ii) the computational capabilities of controllers and (iii) complexity of systems under control increase, so the challenge is to implement advanced control algorithms. There have been commercial successes of the intelligent control methods, but the dominating controller in process industries is still by far the PID-controller (Chidambaram and See, 2002). This stands to the fact that a simple and general purpose automatic controller (for example PID) is demanded in process industries. Therefore, designing advanced controllers are to address the industrial user demand. This is the reason that a learning method called model-free learning control (MFLC) is introduced. The MFLC algorithm is based on a well known Q-learning algorithm (Watkins, 1989). Successful applications of RL are well documented in the recent literature, including learning to control mobile robots (Bucak and Zohdy, 2001), sustained inverted flight on an autonomous helicopter (Ng et al., 2004), and learning to minimize average wait time in elevators (Crites and Barto, 1996). However, only few articles can be found regarding RL applications for process control: multi-step actions based on RL was fruitfully applied for thermostat control (Schoknecht and Riedmiller, 2003), and one of the authors successfully applied RL for modeling for optimization in bath reactors by making the most effective use of cumulative data and an approximate model (Martinez, 2000). The reason for the difference between robotics and process control is possibly the nature of the control task in

Intelligent Control Based on Reinforcement Learning for Batch Thermal Sterilization of Canned Foods

Abstract A control technique based on Reinforcement Learning is proposed for controlling thermal sterilization of canned food. Without using an a-priori mathematical model of the process, the proposed Model-Free Learning Controller (MFLC) aims to follow a temperature profile during two stages of the process: first heating by manipulating the saturated steam valve and then cooling by opening the water valve) by learning. From the defined state-action space, the MFLC agent learns the environment interacting with the process batch to batch and then using a tabular state-action mapping. The results show the advantages of the proposed technique for this kind of processes.

Macro-Actions in Model-Free Intelligent Control with Application to pH Control

MFIC (Model-Free Intelligent Control) is a technique, based on Reinforcement Learning, previously proposed by the authors to control processes without needing a precalculated model. In standard reinforcement learning algorithms (including MFIC), the interaction between an agent and the environment is based on a fixed time scale: during learning, the agent can select several primitive actions depending on the system state. This creates the problem of selecting a suitable fixed time scale to select control actions, to trade off accuracy in control against learning complexity and flexibility. A novel solution to this problem is presented in this paper: Macro-actions, that incorporate a general closed-loop policy and temporal extended actions. The application of macro actions on a laboratory plant of pH process shows that the proposed MFIC learns to control adequately the neutralization process, with reduced computational effort.

Modelling of the pretreatment section for dynamic simulation of water production using reverse osmosis

Abstract The development of mathematical models for the dynamic simulation of the pretreatment section in Reverse Osmosis plants, used for drinking water production, is presented. The objective is to use these models to reduce energy consumption and exploitation costs by testing and comparing advanced control strategies. Thus, these models have been developed for implementation using off-the-shelf software (EcoSimPro), so that advanced control algorithms can be easily tested, while the parameters are selected to be simple to obtain from available plant measurements, so the models can be easily adapted to specific plants.

Positive infusion of propofol drug during induction

Non-opioid intravenous anesthetic agents, such as propofol, have been used in anesthesia since 1970s for conscious sedation. In this paper, the study aims to regulate the syringe pump of propofol infusion during induction. The syringe pump is regulated by using a linear positive controller. The controller is designed so as to satisfy positivity in state. The control output is positive and bounded. Effect site concentration is used as a feedback to the controller. Simulation results show that the controller regulates propofol very well and the BIS responses of the patient are observed so that there is no overshoot or oscillation.

Q(λ) learning technique for pH control

A simple learning control approach for controlling a pH neutralization process has been developed in this paper. A one-step-ahead Q-learning, namely Q(λ)-learning, using a lookup table is developed and applied to a pH process: weak base - strong acid process. The objective of the controller is to maintain the pH within the goal state. The application at a laboratory pilot plant shows that the proposed Q(λ)-learning regulates the process well, even in the presence of reference changes.