Damien Ernst

Extremely randomized trees

This paper proposes a new tree-based ensemble method for supervised classification and regression problems. It essentially consists of randomizing strongly both attribute and cut-point choice while splitting a tree node. In the extreme case, it builds totally randomized trees whose structures are independent of the output values of the learning sample. The strength of the randomization can be tuned to problem specifics by the appropriate choice of a parameter. We evaluate the robustness of the default choice of this parameter, and we also provide insight on how to adjust it in particular situations. Besides accuracy, the main strength of the resulting algorithm is computational efficiency. A bias/variance analysis of the Extra-Trees algorithm is also provided as well as a geometrical and a kernel characterization of the models induced.

Power systems stability control: reinforcement learning framework

In this paper, we explore how a computational approach to learning from interactions, called reinforcement learning (RL), can be applied to control power systems. We describe some challenges in power system control and discuss how some of those challenges could be met by using these RL methods. The difficulties associated with their application to control power systems are described and discussed as well as strategies that can be adopted to overcome them. Two reinforcement learning modes are considered: the online mode in which the interaction occurs with the real power system and the offline mode in which the interaction occurs with a simulation model of the real power system. We present two case studies made on a four-machine power system model. The first one concerns the design by means of RL algorithms used in offline mode of a dynamic brake controller. The second concerns RL methods used in online mode when applied to control a thyristor controlled series capacitor (TCSC) aimed to damp power system oscillations.

Contingency Filtering Techniques for Preventive Security-Constrained Optimal Power Flow

This paper focuses on contingency filtering to accelerate the iterative solution of preventive security-constrained optimal power flow (PSCOPF) problems. To this end, we propose two novel filtering techniques relying on the comparison at an intermediate PSCOPF solution of post-contingency constraint violations among postulated contingencies. We assess these techniques by comparing them with severity index-based filtering schemes, on a 60-and a 118-bus system. Our results show that the proposed contingency filtering techniques lead to faster solution of the PSCOPF, while being more robust and meaningful, than severity-index based ones.

Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem

This paper compares reinforcement learning (RL) with model predictive control (MPC) in a unified framework and reports experimental results of their application to the synthesis of a controller for a nonlinear and deterministic electrical power oscillations damping problem. Both families of methods are based on the formulation of the control problem as a discrete-time optimal control problem. The considered MPC approach exploits an analytical model of the system dynamics and cost function and computes open-loop policies by applying an interior-point solver to a minimization problem in which the system dynamics are represented by equality constraints. The considered RL approach infers in a model-free way closed-loop policies from a set of system trajectories and instantaneous cost values by solving a sequence of batch-mode supervised learning problems. The results obtained provide insight into the pros and cons of the two approaches and show that RL may certainly be competitive with MPC even in contexts where a good deterministic system model is available.

Clinical data based optimal STI strategies for HIV: a reinforcement learning approach

This paper addresses the problem of computing optimal structured treatment interruption strategies for HIV infected patients. We show that reinforcement learning may be useful to extract such strategies directly from clinical data, without the need of an accurate mathematical model of HIV infection dynamics. To support our claims, we report simulation results obtained by running a recently proposed batch-mode reinforcement learning algorithm, known as fitted Q iteration, on numerically generated data

Cross-Entropy Optimization of Control Policies With Adaptive Basis Functions

This paper introduces an algorithm for direct search of control policies in continuous-state discrete-action Markov decision processes. The algorithm looks for the best closed-loop policy that can be represented using a given number of basis functions (BFs), where a discrete action is assigned to each BF. The type of the BFs and their number are specified in advance and determine the complexity of the representation. Considerable flexibility is achieved by optimizing the locations and shapes of the BFs, together with the action assignments. The optimization is carried out with the cross-entropy method and evaluates the policies by their empirical return from a representative set of initial states. The return for each representative state is estimated using Monte Carlo simulations. The resulting algorithm for cross-entropy policy search with adaptive BFs is extensively evaluated in problems with two to six state variables, for which it reliably obtains good policies with only a small number of BFs. In these experiments, cross-entropy policy search requires vastly fewer BFs than value-function techniques with equidistant BFs, and outperforms policy search with a competing optimization algorithm called DIRECT.

Estimation of rotor angles of synchronous machines using artificial neural networks and local PMU-based quantities

This paper investigates a possibility for estimating rotor angles in the time frame of transient (angle) stability of electric power systems, for use in real-time. The proposed dynamic state estimation technique is based on the use of voltage and current phasors obtained from a phasor measurement unit supposed to be installed on the extra-high voltage side of the substation of a power plant, together with a multilayer perceptron trained off-line from simulations. We demonstrate that an intuitive approach to directly map phasor measurement inputs to the neural network to generator rotor angle does not offer satisfactory results. We found out that a good way to approach the angle estimation problem is to use two neural networks in order to estimate the sin(@d) and cos(@d) of the angle and recover the latter from these values by simple post-processing. Simulation results on a part of the Mexican interconnected system show that the approach could yield satisfactory accuracy for real-time monitoring and control of transient instability.

Online least-squares policy iteration for reinforcement learning control

Reinforcement learning is a promising paradigm for learning optimal control. We consider policy iteration (PI) algorithms for reinforcement learning, which iteratively evaluate and improve control policies. State-of-the-art, least-squares techniques for policy evaluation are sample-efficient and have relaxed convergence requirements. However, they are typically used in offline PI, whereas a central goal of reinforcement learning is to develop online algorithms. Therefore, we propose an online PI algorithm that evaluates policies with the so-called least-squares temporal difference for Q-functions (LSTD-Q). The crucial difference between this online least-squares policy iteration (LSPI) algorithm and its offline counterpart is that, in the online case, policy improvements must be performed once every few state transitions, using only an incomplete evaluation of the current policy. In an extensive experimental evaluation, online LSPI is found to work well for a wide range of its parameters, and to learn successfully in a real-time example. Online LSPI also compares favorably with offline LSPI and with a different flavor of online PI, which instead of LSTD-Q employs another least-squares method for policy evaluation.

A control strategy for controllable series capacitor in electric power systems

It has been verified that a controllable series capacitor with a suitable control scheme can improve transient stability and help to damp electromechanical oscillations. A question of great importance is the selection of the input signals and a control strategy for this device in order to damp power oscillations in an effective and robust manner. Based on Lyapunov theory a control strategy for damping of electromechanical power oscillations in a multi-machine power system is derived. Lyapunov theory deals with dynamical systems without inputs. For this reason, it has traditionally been applied only to closed-loop control systems, that is, systems for which the input has been eliminated through the substitution of a predetermined feedback control. However, in this paper, we use Lyapunov function candidates in feedback design itself by making the Lyapunov derivative negative when choosing the control. This control strategy is called control Lyapunov function for systems with control inputs. Also, two input signals for this control strategy are used. The first one is based on local information and the second one on remote information derived by the single machine equivalent method.

Active Management of Low-Voltage Networks for Mitigating Overvoltages Due to Photovoltaic Units

In this paper, the overvoltage problems that might arise from the integration of photovoltaic (PV) panels into low-voltage (LV) distribution networks is addressed. A distributed scheme is proposed that adjusts the reactive and active power output of inverters to prevent or alleviate such problems. The proposed scheme is model-free and makes use of limited communication between the controllers in the form of a distress signal only during emergency conditions. It prioritizes the use of reactive power, while active power curtailment is performed only as a last resort. The behavior of the scheme is studied using dynamic simulations on a single LV feeder and on a larger network composed of 14 LV feeders. Its performance is compared with a centralized scheme based on the solution of an optimal power flow (OPF) problem, whose objective function is to minimize the active power curtailment. The proposed scheme successfully mitigates overvoltage situations due to high PV penetration and performs almost as well as the OPF-based solution with significantly less information and communication requirements.