Stijn Vandael

A Scalable Three-Step Approach for Demand Side Management of Plug-in Hybrid Vehicles

In this paper, we present a scalable approach for DSM (demand side management) of PHEVs (plug-in hybrid electric vehicles). Essentially, our approach consists of three steps: aggregation, optimization, and control. In the aggregation step, individual PHEV charging constraints are aggregated upwards in a tree structure. In the optimization step, the aggregated constraints are used for scalable computation of a collective charging plan, which minimizes costs for electricity supply. In the real-time control step, this charging plan is used to create an incentive signal for all PHEVs, determined by a market-based priority scheme. These three steps are executed iteratively to cope with uncertainty and dynamism. In simulation experiments, the proposed three-step approach is benchmarked against classic, fully centralized approaches. Results show that our approach is able to charge PHEVs with comparable quality to optimal, centrally computed charging plans, while significantly improving scalability.

An Event-Driven Dual Coordination Mechanism for Demand Side Management of PHEVs

This paper addresses the challenges of integrating existing PHEV charging algorithms, which optimize PHEV charging per market timeslot (e.g., 15 minutes), into an environment with realistic communication conditions. To address this challenge, we propose a dual coordination mechanism, which controls a cluster of devices on two different operation levels: market operation and real-time operation. The market operation level uses an existing timeslot-based algorithm to calculate a charging schedule per timeslot. The real-time operation level translates this schedule into event-based control actions for a realistic communication environment, wherein a limited number of messages can be exchanged. A case study of 1000 PHEVs shows that it is possible to achieve results on par with the timeslot based algorithm but with significantly reduced communication with the PHEVs.

Self-learning demand side management for a heterogeneous cluster of devices with binary control actions

Finding an optimal planning for a large cluster of devices with binary control actions is a challenging task for both centralized and distributed approaches. This is certainly the case when a significant fraction of devices in the cluster has binary control actions, since the resulting optimization problem belongs to NP-hard integer programming. A distributed approach can be a good solution to address this problem. Good performance however, often relies on the presence of local intelligence, such as planning and prediction at device or household level. In this work we apply a self-learning agent-based demand side management approach to a heterogeneous cluster of devices with binary control actions. The required local intelligence is limited to a state estimation and local comfort and constraint checking. Each device is represented by an individual agent communicating a bid function to a virtual energy market. In the approach the aggregated energy and power constraints of a cluster of devices are learned, independent of the type and number of devices. The aggregated constraints are estimated based upon the aggregated bid functions. These constraints are used to determine an optimal control signal managing the cluster. The approach has been evaluated in two distinct scenarios including devices with binary control actions, showing that the self-learning approach converges within 12 days to obtain 80 % of the maximum optimization potential, with a generic approach that requires limited intelligence at device level.

Demand response of a heterogeneous cluster of electric water heaters using batch reinforcement learning

A demand response aggregator, that manages a large cluster of heterogeneous flexibility carriers, faces a complex optimal control problem. Moreover, in most applications of demand response an exact description of the system dynamics and constraints is unavailable, and information comes mostly from observations of system trajectories. This paper presents a model-free approach for controlling a cluster of domestic electric water heaters. The objective is to schedule the cluster at minimum electricity cost by using the thermal storage of the water tanks. The control scheme applies a model-free batch reinforcement learning (batch RL) algorithm in combination with a market-based heuristic. The considered batch RL technique is tested in a stochastic setting, without prior information or model of the system dynamics of the cluster. The simulation results show that the batch RL technique is able to reduce the daily electricity cost within a reasonable learning period of 40-45 days, compared to a hysteresis controller.

Demand side management of electric vehicles with uncertainty on arrival and departure times

Uncertainty on arrival and departure times makes the scheduling of plug-in hybrid electric vehicles an intrinsically stochastic optimization problem. To take the stochastic nature of this problem into consideration, a scalable stochastic optimization strategy has been formulated. Generally, stochastic programming methods are computationally demanding and become impractical for large-scale problems. This work reduced the dimensionality of the scheduling problem with techniques from approximate dynamic programming. To illustrate the advantage of the stochastic algorithm a deterministic method has been formulated. Compared to the deterministic method, the proposed stochastic method can help an aggregator to reduce its expensive peak charging or avoid penalties for not fully charging the batteries of its clients.

Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning

Driven by recent advances in batch Reinforcement Learning (RL), this paper contributes to the application of batch RL to demand response. In contrast to conventional model-based approaches, batch RL techniques do not require a system identification step, making them more suitable for a large-scale implementation. This paper extends fitted Q-iteration, a standard batch RL technique, to the situation when a forecast of the exogenous data is provided. In general, batch RL techniques do not rely on expert knowledge about the system dynamics or the solution. However, if some expert knowledge is provided, it can be incorporated by using the proposed policy adjustment method. Finally, we tackle the challenge of finding an open-loop schedule required to participate in the day-ahead market. We propose a model-free Monte Carlo method that uses a metric based on the state-action value function or Q-function and we illustrate this method by finding the day-ahead schedule of a heat-pump thermostat. Our experiments show that batch RL techniques provide a valuable alternative to model-based controllers and that they can be used to construct both closed-loop and open-loop policies.

Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market

This paper addresses the problem of defining a day-ahead consumption plan for charging a fleet of electric vehicles (EVs), and following this plan during operation. A challenge herein is the beforehand unknown charging flexibility of EVs, which depends on numerous details about each EV (e.g., plug-in times, power limitations, battery size, power curve, etc.). To cope with this challenge, EV charging is controlled during opertion by a heuristic scheme, and the resulting charging behavior of the EV fleet is learned by using batch mode reinforcement learning. Based on this learned behavior, a cost-effective day-ahead consumption plan can be defined. In simulation experiments, our approach is benchmarked against a multistage stochastic programming solution, which uses an exact model of each EVs charging flexibility. Results show that our approach is able to find a day-ahead consumption plan with comparable quality to the benchmark solution, without requiring an exact day-ahead model of each EVs charging flexibility.

Communication overlays and agents for dependable smart power grids

Smart grids rely on a dependable information infrastructure for the monitoring and control applications. Two elements can enhance the suitability of the communication and control infrastructure for such smart grid applications. Overlay networks allow to resiliently deal with nodes that appear and disappear, as well as with the dynamic nature of the power values these nodes represent in a smart grid. Agents-based modelling allows to simulate the smart grid applications in a scalable and flexible way before deployment. The paper discusses how both approaches can be combined for simulating a more dependable smart grid.

A comparison of two GIV mechanisms for providing ancillary services at the University of Delaware

At the University of Delaware, we are providing ancillary services by controlling the bidirectional power transfer between 15 EVs and the grid. To control this power transfer, a set of algorithms, models and interactions is used, called a “GIV (Grid Integrated Vehicle) mechanism”. In literature, many GIV mechanisms are proposed. However, because these mechanisms are evaluated independently in specific scenarios, their differences are not always clear. In this paper, we take a first step in tackling this challenge by comparing two different GIV mechanisms in the same scenario at the University of Delaware: a decentralized and a centralized mechanism. In the decentralized mechanism, which is currently operational at our test environment, EVs decide autonomously on the amount of power available for ancillary services. In the centralized mechanism, a central server gathers all EV information and makes a decision for all EVs. In evaluation, both GIV mechanisms are compared with each other. Simulation results show that the centralized mechanism outperforms its decentralized counterpart in terms of available power for ancillary services. On the other hand, the decentralized mechanism enables large-scale integration by distributing computations across all EVs.

Peak shaving of a heterogeneous cluster of residential flexibility carriers using reinforcement learning

Demand response is often defined as an optimal control problem. However, the practical application is challenged by computational complexity and lack of accurate models and data. In this work we extend upon previous work and combine batch reinforcement learning, using function approximators, with a market-based multi-agent system. The resulting adaptive control strategy is model-free and needs no prior knowledge of the cluster configuration. The strategy is evaluated for two distinct heterogeneous clusters of residential flexibility carriers. The evaluation shows that our self-learning strategy supports effective peak shaving and valley filling within a limited convergence time.