Frederik Ruelens

Self-learning demand side management for a heterogeneous cluster of devices with binary control actions

Finding an optimal planning for a large cluster of devices with binary control actions is a challenging task for both centralized and distributed approaches. This is certainly the case when a significant fraction of devices in the cluster has binary control actions, since the resulting optimization problem belongs to NP-hard integer programming. A distributed approach can be a good solution to address this problem. Good performance however, often relies on the presence of local intelligence, such as planning and prediction at device or household level. In this work we apply a self-learning agent-based demand side management approach to a heterogeneous cluster of devices with binary control actions. The required local intelligence is limited to a state estimation and local comfort and constraint checking. Each device is represented by an individual agent communicating a bid function to a virtual energy market. In the approach the aggregated energy and power constraints of a cluster of devices are learned, independent of the type and number of devices. The aggregated constraints are estimated based upon the aggregated bid functions. These constraints are used to determine an optimal control signal managing the cluster. The approach has been evaluated in two distinct scenarios including devices with binary control actions, showing that the self-learning approach converges within 12 days to obtain 80 % of the maximum optimization potential, with a generic approach that requires limited intelligence at device level.

Demand response of a heterogeneous cluster of electric water heaters using batch reinforcement learning

A demand response aggregator, that manages a large cluster of heterogeneous flexibility carriers, faces a complex optimal control problem. Moreover, in most applications of demand response an exact description of the system dynamics and constraints is unavailable, and information comes mostly from observations of system trajectories. This paper presents a model-free approach for controlling a cluster of domestic electric water heaters. The objective is to schedule the cluster at minimum electricity cost by using the thermal storage of the water tanks. The control scheme applies a model-free batch reinforcement learning (batch RL) algorithm in combination with a market-based heuristic. The considered batch RL technique is tested in a stochastic setting, without prior information or model of the system dynamics of the cluster. The simulation results show that the batch RL technique is able to reduce the daily electricity cost within a reasonable learning period of 40-45 days, compared to a hysteresis controller.

Demand side management of electric vehicles with uncertainty on arrival and departure times

Uncertainty on arrival and departure times makes the scheduling of plug-in hybrid electric vehicles an intrinsically stochastic optimization problem. To take the stochastic nature of this problem into consideration, a scalable stochastic optimization strategy has been formulated. Generally, stochastic programming methods are computationally demanding and become impractical for large-scale problems. This work reduced the dimensionality of the scheduling problem with techniques from approximate dynamic programming. To illustrate the advantage of the stochastic algorithm a deterministic method has been formulated. Compared to the deterministic method, the proposed stochastic method can help an aggregator to reduce its expensive peak charging or avoid penalties for not fully charging the batteries of its clients.

Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning

Driven by recent advances in batch Reinforcement Learning (RL), this paper contributes to the application of batch RL to demand response. In contrast to conventional model-based approaches, batch RL techniques do not require a system identification step, making them more suitable for a large-scale implementation. This paper extends fitted Q-iteration, a standard batch RL technique, to the situation when a forecast of the exogenous data is provided. In general, batch RL techniques do not rely on expert knowledge about the system dynamics or the solution. However, if some expert knowledge is provided, it can be incorporated by using the proposed policy adjustment method. Finally, we tackle the challenge of finding an open-loop schedule required to participate in the day-ahead market. We propose a model-free Monte Carlo method that uses a metric based on the state-action value function or Q-function and we illustrate this method by finding the day-ahead schedule of a heat-pump thermostat. Our experiments show that batch RL techniques provide a valuable alternative to model-based controllers and that they can be used to construct both closed-loop and open-loop policies.

A Flexible Stochastic Optimization Method for Wind Power Balancing With PHEVs

This paper proposes a flexible optimization method, based on state of the art algorithms, for the smart control of plug-in hybrid electric vehicles (PHEVs) to balance wind power production. The problem is approached from the perspective of a balance responsible party (BRP) with a large share of wind power in its portfolio. The BRP uses controllable PHEVs to minimize the imbalance of its portfolio resulting from wind power forecast errors. A Markov Decision Process (MDP) formulation in combination with dynamic programming is used to solve the multistage stochastic problem. The main difficulty for applying MDPs to this problem is to efficiently include time interdependence of the wind power forecast error. In the presented approach, the probability distribution and time interdependence of the forecast error are represented by a scenario tree. Because of the MDP formulation, the algorithm is adaptable to deal with different transition models and constraints. This feature enables to use the algorithm in a dynamic environment such as the future smart grid. To demonstrate this, a generic charging model for PHEVs is used in the BRP wind balancing case. The flexibility of the algorithm is shown by investigating the solution for different degrees of complexity in the charging model.

Cluster Control of Heterogeneous Thermostatically Controlled Loads Using Tracer Devices

Managing the aggregated demand of large heterogeneous clusters of thermostatically controlled loads (TCLs) is considered a sequential decision-making problem under uncertainty. Recent research indicates that using reduced-order models in combination with a broadcasted control signal offers a viable solution to the tradeoff between computational feasibility, and accurately describing the steady-state and transient cluster response. In this paper, we propose a novel control strategy based on tracer devices, which we define as a limited amount of virtual TCLs that represent the entire cluster of heterogeneous TCLs. These second-order model devices are identified in a nonintrusive manner, and capture both steady-state and transient population dynamics, as well as cluster heterogeneity. Additionally, the dispatch mechanism is included in the optimization, further improving the tracking performance. The parameterizable number of tracer devices enables a covering of the tradeoff domain. Both approaches have been evaluated in two scenarios. In the first small-scale scenario, improvements in price and power deviations are evaluated when using increasing numbers of tracer devices and integrating the dispatch dynamics. Results from the second large-scale scenario show that root mean square dispatch errors can be reduced by more than 10% when integrating the dispatch mechanism in the resulting high-fidelity model.

Peak shaving of a heterogeneous cluster of residential flexibility carriers using reinforcement learning

Demand response is often defined as an optimal control problem. However, the practical application is challenged by computational complexity and lack of accurate models and data. In this work we extend upon previous work and combine batch reinforcement learning, using function approximators, with a market-based multi-agent system. The resulting adaptive control strategy is model-free and needs no prior knowledge of the cluster configuration. The strategy is evaluated for two distinct heterogeneous clusters of residential flexibility carriers. The evaluation shows that our self-learning strategy supports effective peak shaving and valley filling within a limited convergence time.

Double-layered control methodology combining price objective and grid constraints

A major challenge consists of considering all stakeholders of the future Smart Grid, each with their specific and possibly opposing objectives. A distribution network operator aims at guaranteeing power quality criteria while consumers aspire lowering their power consumption bill. This fundamental issue currently delays the transition from small-scale research projects to a large-scale all-encompassing smart distribution grid. This paper describes a double-layered control methodology using the available flexibility of the majority of discrete smart appliances currently in use. The effect of striving for the objectives separately as well as in combination is examined. The results show that the targeted objective(s) strongly influence(s) the performance in terms of cost effectiveness as well as number of voltage issues.

Stochastic portfolio management of an electric vehicles aggregator under price uncertainty

This paper considers the portfolio management problem of a flexibility aggregator under uncertainty on real-time prices. Solving this stochastic optimal control problem in a reasonable time, considering overall scalability, comfort settings and grid constraints, is a challenging task. This paper tackles these problems by making use of a Three-Step Approach (TSA). Two control approaches are considered in the second step of the TSA: Model Predictive Control (MPC) and Approximate Dynamic Programming (ADP). The performance of both controllers for different temporal autocorrelated price profiles is illustrated for an aggregator with a fleet of 1000 electric vehicles. The simulations show that the TSA extended with a stochastic controller can reduce the cost of the aggregator compared to a certainty equivalent approach. The paper concludes by discussing the strength and weaknesses of MPC and ADP in a smart grid setting.

Beyond theory: Experimental results of a self-learning air conditioning unit

This paper demonstrates the application of a data-driven approach, based on fitted Q-iteration, in a living lab with an air conditioning unit and a photovoltaic system. More specifically, the objective is to minimize the quadratic difference between the locally produced photovoltaic power and the power consumption of the air conditioning unit. A first simulation-based experiment assesses the performance of the data-driven approach by comparing its performance with the default thermostat and a model-based method. The simulation-based results indicate that the data-driven control method was able to achieve near-optimal policies within approximately 15 days of operation. In a second experiment, the proposed control method is applied to a living lab. The qualitative results indicate that the control method was able to successfully reduce the peak power of the photovoltaic system that is injected into the grid.