Felix Hernandez-del-Olmo

Autonomous Adaptive and Active Tuning Up of the Dissolved Oxygen Setpoint in a Wastewater Treatment Plant Using Reinforcement Learning

The aim of this paper is to face one of the main problems in the control of wastewater treatment plants (WWTPs). It appears that the control system does not respond as it should because of changes on influent load or flow. In that case, it is required that a plant operator tunes up the parameters of the plant. The dissolved oxygen setpoint is one of those parameters. In this paper, we present a model-free reinforcement learning agent that autonomously learns to actively tune up the oxygen setpoint by itself. By active, we mean continuous, minute after minute, tuning up. By autonomous and adaptive, we mean that the agent learns just by itself from its direct interaction with the WWTP. This agent has been tested with data from the well-known public benchamark simulation model no. 1, and the results that are obtained allow us to conclude that it is possible to build agents that actively and autonomously adapt to each new scenario to control a WWTP.

An emergent approach for the control of wastewater treatment plants by means of reinforcement learning techniques

Highlights? RL agent that supervises the WWTP 24h/day. ? RL agent that adapts to each particular WWTP by itself. ? Better performance than standard BSM1 control strategy. ? Control strategy learned autonomously adapted to different locations. One of the main problems in the automation of the control of wastewater treatment plants (WWTPs) appears when the control system does not respond as it should because of changes on influent load or flow. To tackle this difficult task, the application of Artificial Intelligence is not new, and in fact, currently Expert Systems may supervise the plant 24h/day assisting the plant operators in their daily work. However, the knowledge of the Expert System must be elicited previously from interviews to plant operators and/or extracted from data previously stored in databases. Although this approach still has a place in the control of wastewater treatment plants, it should aim to develop autonomous systems that learn from the direct interaction with the WWTP and that can operate taking into account changing environmental circumstances. In this paper we present an approach based on an agent with learning capabilities. In this approach, the agents knowledge emerges from the interaction with the plant. In order to show the validity of our assertions, we have implemented such an emergent approach for the N-Ammonia removal process in a well established simulated WWTP known as Benchmark Simulation Model No.1 (BSM1).

Reinforcement learning techniques for the control of wastewater treatment plants

Since water pollution is one of the most serious environmental problems today, control of wastewater treatment plants (WWTPs) is a crucial issue nowadays and stricter standards for the operation of WWTPs have been imposed by authorities. One of the main problems in the automation of the control of Wastewater Treatment Plants (WWTPs) appears when the control system does not respond as it should because of changes on influent load or flow. Thus, it is desirable the development of autonomous systems that learn from interaction with a WWTP and that can operate taking into account changing environmental circumstances. In this paper we present an intelligent agent using reinforcement learning for the oxygen control in the N-Ammonia removal process in the well known Benchmark Simulation Model no.1 (BSM1). The aim of the approach presented in this paper is to minimize the operation cost changing the set-points of the control system autonomously.

Tackling the start-up of a reinforcement learning agent for the control of wastewater treatment plants

Abstract Reinforcement learning problems involve learning by doing. Therefore, a reinforcement learning agent will have to fail sometimes (while doing) in order to learn. Nevertheless, even with this starting error, introduced at least during the non-optimal learning stage, reinforcement learning can be affordable in some domains like the control of a wastewater treatment plant. However, in wastewater treatment plants, trying to solve the day-to-day problems, plant operators will usually not risk to leave their plant in the hands of an inexperienced and untrained reinforcement learning agent. In fact, it is somewhat obvious that plant operators will require firstly to check that the agent has been trained and that it works as it should at their particular plant. In this paper, we present a solution to this problem by giving a previous instruction to the reinforcement learning agent before we let it act on the plant. In fact, this previous instruction is the key point of the paper. In addition, this instruction is given effortlessly by the plant operator. As we will see, this solution does not just solve the starting up problem of leaving the plant in the hands of an untrained agent, but it also improves the future performance of the agent.