Adam Berry

Empirical evaluation methods for multiobjective reinforcement learning algorithms

While a number of algorithms for multiobjective reinforcement learning have been proposed, and a small number of applications developed, there has been very little rigorous empirical evaluation of the performance and limitations of these algorithms. This paper proposes standard methods for such empirical evaluation, to act as a foundation for future comparative studies. Two classes of multiobjective reinforcement learning algorithms are identified, and appropriate evaluation metrics and methodologies are proposed for each class. A suite of benchmark problems with known Pareto fronts is described, and future extensions and implementations of this benchmark suite are discussed. The utility of the proposed evaluation methods are demonstrated via an empirical comparison of two example learning algorithms.

Comparison of Mixed-Integer Programming and Genetic Algorithm Methods for Distributed Generation Planning

This paper applies recently developed mixed-integer programming (MIP) tools to the problem of optimal siting and sizing of distributed generators in a distribution network. We investigate the merits of three MIP approaches for finding good installation plans: a full AC power flow approach, a linear DC power flow approximation, and a nonlinear DC power flow approximation with quadratic loss terms, each augmented with integer generator placement variables. A genetic algorithm-based approach serves as a baseline for the comparison. A simple knapsack problem method involving generator selection is presented for determining lower bounds on the optimal design objective. Solution methods are outlined, and computational results show that the MIP methods, while lacking the speed of the genetic algorithm, can find improved solutions within conservative time requirements and provide useful information on optimality.

Building a microgrid laboratory

Microgrids are local area power systems, and are attracting increased attention due to their potential to provide a solution to integrate renewable energy into the wider grid. In order to facilitate experimental research, a microgrid laboratory has been built by CSIRO in Australia. Experiments have been carried out which investigate issues of integrating distributed generation, including renewable energy, into the electricity network. This paper describes some of the challenges involved in setting up such a facility and provides examples of experimental results. This facility is unique in its incorporation of three types of solar PV technologies, two types of wind power, three types of battery storage, and a programmable load bank. The availability of a flexible facility such as this is essential in advancing the science in this area, and is leading to valuable insights into microgrid operation.

What Role for Microgrids

The microgrid concept offers a stepping stone along the path to autonomous, intelligent low-emissions electricity systems, by creating a localized low-emissions smart grid that allows advanced control while being compatible with current electricity infrastructure. With such promise, microgrids are of growing interest to grid operators around the world, as ways of enhancing the performance of both developed and undeveloped electricity systems. The operation of microgrids, however, is not without its challenges. This chapter reviews microgrid technology, detailing worldwide research and developments of this technology, while also presenting examples of the technical challenges needing to be addressed before widespread rollout.

An introduction to multiobjective optimisation methods for decentralised power planning

Optimising the placement and sizing of distributed generators into decentralised power networks is receiving increasing research attention. The studies presented thus far, however, do not capitalise on the most contemporary techniques described in the multiobjective optimisation literature. This work illustrates, through both theoretical and empirical analyses, the consequences of this deficiency and provides a tutorial on how best to integrate core multiobjective concepts into the optimisation process. Initial results suggest that such integration may be of significant value to both researchers and design-engineers alike.

Residential precinct demand forecasting using optimised solar generation and battery storage

In the future there will be an increased uptake of solar and battery systems in the residential sector, driven by falling battery costs and increasing electricity tariffs. The increased uptake means we need new methods to forecast electricity demand when considering these technologies. This paper has achieved this goal using a two stage model. Stage 1: A machine learning demand model has been created applying adaptive boost to a regression tree algorithm, achieving an RMS error of 0.25. The model has been used to simulate the individual base-demand for 50 homes in a precinct. Stage 2: A linear programing model has been developed that determines the impact of solar and battery storage on that base demand, and optimizes the system capacities for each home in the precinct while limiting emissions. This model shows reducing emissions by 50% through solar and battery storage cost 2.6% more than the grid only scenario.

Minigrids: Analysing the state-of-play

Minigrids are often cited as an evolutionary path for electricity distribution systems- a way of integrating a high penetration of low emissions generation without needing to dramatically change system operation or design. Minigrids are not without their challenges though, and there is a growing body of work examining the technical issues related to the widespread uptake of minigrid technology. This survey offers a comprehensive collection and analysis of such research, painting a more complete minigrids picture, elucidating the progress that has been made thus far, and the challenges that still lay ahead.

A new approach to the design of multiple inverter systems using evolutionary optimization

The use of distributed generators (DGs), especially renewables like wind and solar photovoltaics, depends upon inverter technology to provide compatibility between multiple DGs on a local bus. It is becoming more common to group dissimilar DGs together to form microgrids, and desirable to allow DGs to be connected or disconnected without detrimental effects upon the stability of the microgrid. This has implications for the design of suitable control systems for operating inverters in a heterogeneous microgrid. Evolutionary computing provides a powerful optimization tool that has the potential to assist in the design of control systems for microgrids. As a first step, we introduce a feasibility study, where evolutionary design is used to optimize control parameters for an islanded microgrid with a large number of inverters. We show that the evolutionary optimization provides improved transient response when compared to a manual design, and propose that this approach is worthy of further investigation.

Designing multiple inverter systems with evolutionary multiobjective optimisation

Given the growth of microgrids and decentralised power, heterogeneous multi-inverter systems are becoming increasingly prevalent. Despite this, little is known about the interactive effects of such systems and how best to control them. In response, this work examines the use of traditional droop control and a contemporary multiobjective optimisation technique for automatically adapting parameters for a specified load profile in a heterogeneous ten-inverter system. Results indicate that the multiobjective approach offers a range of parameter sets that each outperform the manual droop settings with respect to both voltage sag and ripple objectives.

Steering approaches to Pareto-optimal multiobjective reinforcement learning

Abstract For reinforcement learning tasks with multiple objectives, it may be advantageous to learn stochastic or non-stationary policies. This paper investigates two novel algorithms for learning non-stationary policies which produce Pareto-optimal behaviour (w-steering and Q-steering), by extending prior work based on the concept of geometric steering. Empirical results demonstrate that both new algorithms offer substantial performance improvements over stationary deterministic policies, while Q-steering significantly outperforms w-steering when the agent has no information about recurrent states within the environment. It is further demonstrated that Q-steering can be used interactively by providing a human decision-maker with a visualisation of the Pareto front and allowing them to adjust the agent’s target point during learning. To demonstrate broader applicability, the use of Q-steering in combination with function approximation is also illustrated on a task involving control of local battery storage for a residential solar power system.