A multilayer, multi-timescale model approach for economic and frequency control in power grids
Lia Strenge, Paul Schultz, Jürgen Kurths, Jörg Raisch, Frank Hellmann
AA multiplex, multi-timescale model approach for economic and frequencycontrol in power grids.
Lia Strenge, Paul Schultz, Jürgen Kurths, Jörg Raisch, and Frank Hellmann a) Control Systems Group at Technische Universität Berlin, Germany Potsdam Institute for Climate Impact Research, Germany (Dated: 28 March 2020)
Power systems are subject to fundamental changes due to the increasing infeed of decentralised renewable energysources and storage. The decentralised nature of the new actors in the system requires new concepts for structuring thepower grid, and achieving a wide range of control tasks ranging from seconds to days. Here we introduce a multiplexdynamical network model covering all control timescales. Crucially, we combine a decentralised, self-organised low-level control and a smart grid layer of devices that can aggregate information from remote sources. The safety-criticaltask of frequency control is performed by the former, the economic objective of demand matching dispatch by the latter.Having both aspects present in the same model allows us to study the interaction between the layers. Remarkably, wefind that adding communication in the form of aggregation does not improve the performance in the cases considered.Instead, the self-organised state of the system already contains the information required to learn the demand structure inthe entire grid. The model introduced here is highly flexible, and can accommodate a wide range of scenarios relevantto future power grids. We expect that it is especially useful in the context of low-energy microgrids with distributedgeneration.
Highly decentralised power grids, possibly in the contextof prosumer systems, require new concepts for their stableoperation. We expect that both self-organised systems aswell as intelligent devices with communication capabilitythat can aggregate information from remote sources willplay a central role. Here we introduce a multiplex networkmodel that combines both aspects, and use it in a basicscenario and uncover surprising interactions between thelayers.
I. INTRODUCTION
Power systems are subject to fundamental changes causedby the increasing infeed of decentralised and fluctuating re-newable energy sources. One key change is that the conven-tional energy producers, i.e., big central power plants, are cur-rently also the locus of control resources for the power grid.The challenge facing future power grids lies in achieving astable and robust operation of the grid without such centrally-controlled actors.In general, the objective for a stable operation is to maintainthe frequency and voltage of the system and to keep the sys-tem in an economically desired state. To do so, it is necessaryto achieve an instantaneous balance between electricity gen-eration and consumption. Any imbalance is directly linked toa deviation of the grid frequency from its nominal value (50Hz/ 60 Hz). The control and stabilisation of frequency is tradi-tionally divided into primary, secondary and tertiary control.They respectively address instantaneous frequency stabilisa-tion (primary), i.e., keeping frequency within given bounds,restoring the nominal frequency (secondary), and achieving a) Electronic mail: [email protected] or restoring a desired economic state (tertiary). These threecontrol layers also typically come with a temporal hierarchy,with primary control acting at the scale of seconds, secondaryin minutes and tertiary in quarter-hours.In addition, the corresponding tasks require an increasinglevel of explicit communication and coordination of the ac-tors, who participate in various markets to ensure sufficientcontrol resources in an economically feasible way. While pri-mary control might be achieved through automated reactionsto frequency deviations, controls on slower timescales are typ-ically subject to active human decisions.The power required to operate the safety-critical primaryand secondary control as well as to balance out unforeseenload variations, is typically held in reserve, and has to beprovided at short notice. Both reserved capacity and energydrawn cause costs. On the other hand, the energy required toservice the expected load is bought days, weeks or even yearsin advance, based on past experience. This energy can bedispatched by the cheapest provider, and technical constraintsplay less of a role here. It can thus be expected to be cheaperoverall than primary or secondary control energy. Followinga fault, it is the role of tertiary control to restore this econom-ically favourable state.Control concepts for (prosumer-based) microgrids, whichcould form a key part of future grid designs, especially inemerging markets, follow the same hierarchy of tasks .Again, primary and secondary control are required for theproper functioning of the grid itself, whereas tertiary con-trol (also called energy management) chooses the economi-cally desirable source of energy. With this context in mind,we explore scenarios for achieving frequency stability as wellas economically optimal balancing. We focus on distributed,self-organised control actors, assuming that an analogous de-tailed market design for single microgrids is not feasible dueto their small sizes, decentralised power provision and low in-ertia.In future power grids, the control tasks will have to be per- a r X i v : . [ n li n . AO ] M a r formed by new distributed actors. A key question in their griddesign is how much coordination these actors will require andhow much can be achieved through self-organised means. Toobtain a system that can function in the face of communica-tion failures, it is natural to require that primary and secondarycontrol should be achieved in a fully decentralised and self-organised fashion. On the other hand, tertiary control is anoptimisation task that can make use of communication andcoordination infrastructure safely.To understand whether such a communication and coordi-nation layer is required, and how it performs with respect tocontrol and stabilisation, it is necessary to study the interac-tion of the different layers of the control hierarchy, which aretypically studied only separately. Most literature on hierarchi-cal control is reviewing existing approaches on their respec-tive timescales without explicitly studying their interaction .In this paper we introduce a model for the hierarchicaloperation of a power grid that aims to achieve two goals,robust control in the face of communication failures, andsome notion of an economically optimal dispatch under oper-ational constraints. We consider the power grid as a two layernetwork , where the layers are given by (i) the physicalelectricity network together with a fully decentralised real-time distributed primary/secondary control of the grid fre-quency, and (ii) an energy management layer. If we allowfor communication in the energy management layer, this canbe described as a multiplex network of a physical and acontrol layer, i.e., both layers have an identical set of nodes.In the latter, links correspond to a directed information trans-fer between controllers. This model allows us to study theinteraction of timescales ranging from seconds to days.We use this model to analyse a simple but illustrative sce-nario, where a subset of nodes in the system has the abilityto dispatch energy in hourly intervals, and optimise for an un-known periodic background demand that is inferred from thecontrol actions required by the primary/secondary layer. Us-ing this scenario we can compare the effect of various com-munication strategies.In order to study the performance of various approaches,we make use of probabilistic methods , that is, we definea scenario ensemble and evaluate the expected performanceof the system with respect to the ensemble average by sam-pling over the ensemble. This approach allows us to study theproperties of the system for the whole ensemble, rather thanin individual case studies. As our main aim here is to intro-duce the model and understand qualitatively the performanceof the control layers, our setup is rather conceptual and doesnot capture real power systems in detail. While the inspirationis drawn from the context of prosumer-based microgrids, weconsider this to provide a broader perspective as well. A. Multiplex aspects of power grids
The study of multilayer networks as (dynamical) systemswith an additional mesoscale structure experienced an activedevelopment in recent years (see Kivelä et al. for a review).Subsequently, a variety of statistical network characteristics (e.g., Donges et al. ) has been developed to quantify the mul-tilayer structure. A special multilayer structure is the multi-plex (or multilevel ) network which we employ in our model.Nodes are identical across the layers, hence, the topology be-tween the layers is fixed.We identify the network layers with their different func-tional roles within the system, i.e., electricity distribution andcontrol. Likewise, the coupling mechanism is different andgiven by the physical power flow and communication, respec-tively. Introducing an interdependence between the layers af-fects the overall system’s resilience to failures, a popular ex-ample is the 2003 Italian blackout with a multilayer cascad-ing failure. In particular, it has been shown that the intercon-nection of different networks can promote network breakdownin discontinuous first-order transitions .Note that there are a number of works that study consensus-based methods for achieving certain objectives (see, e.g., )by introducing an additional communication layer. Thesemultiplex networks differ in various ways from the setup stud-ied here. Most importantly in the fact that the layers cooper-ate on a single control objective and it is mostly secondarycontrol or quasi-stationary tertiary control that is consid-ered. B. Energy management
For a decentralised primary and secondary control we makethe most simple choice, and use a lag element, which can alsobe described as adapted distributed proportional-integral (PI)control. The main adaptation, discussed thoroughly in isthat the integral controller includes an exponential decay term.While this means that there remains a residual steady state er-ror, it makes the setup robust to unavoidable systematic errorsin the implementation of the integrator.The question of how to model energy management isfar more challenging, and less settled. A variety ofapproaches model energy markets directly. For tertiarycontrol (here referring to the technical implementation) di-rectly, we are not aware of any general models, though moreconcrete studies for the microgrid context exist . There arealso various works studying the relationship to congestionmanagement and frequency control . Our approach here isto side step the question of how exactly the dispatch is cho-sen, but instead focus on studying the steady state emergingonce the economic optimisation (given the available informa-tion) has been performed. We do so by introducing an iter-ative learning control (ILC) that considers the previousdays performance and attempts to iteratively improve it byscheduling a different dispatch for the next day. As the con-sumption/production fluctuates from day to day, this iterativeprocess should converge to an optimum dispatch. Note thatthe energy scheduled for the next day is known a day aheadbut not generated a day ahead.ILC is a control method which can be applied to track aperiodic output or reject periodic disturbances. The error isreduced over the iteration cycles and it can easily be com-bined with feedback controllers. ILC has previously beenused in power systems in other contexts, mainly for invertercontrol, e.g., . In addition, ILC is applied to an unin-terruptible power supply and for optimal residential loadscheduling . In building automation, data-driven methods fordemand response in the residential building sector are takeninto account ; ILC also addresses frequency control withhigh penetration of wind integration ; it is further applied toenergy management in electric vehicles . Hence, most of theliterature combining energy management and ILC focus onsingle nodes in a grid without emphasis on the overall grid per-spective. However, a review on ILC for energy managementin multi-agent systems states that the applicability of ILC tothe topic including physical constraints has a high researchpotential due to its (periodic) disturbance rejection capacityand distributed architecture for large-scale systems . ILC forphysically interconnected linear large-scale systems is studiedand applied to economic dispatch in power systems .In larger grids we expect that this learning would be re-placed, for example, by a market-based system. However, inautonomous microgrids, without the resources necessary toimplement a market-based solution, the ILC itself is a viableway of choosing dispatch. We leave a detailed discussion ofthe design, as well as a proof of linearised asymptotic stabil-ity in the iteration domain of the ILC in such a scenario to acompanion paper .This paper is structured as follows. In Section II, the over-all model with two control layers is presented. In Section III,we compare the performance of the system for different mul-tiplex topologies with sampling based numerical experiments.Finally, in Section IV, we discuss our main result that an ad-ditional communication layer is not needed in the proposedsetting and suggest further research directions. Notation
Let N be the set of nodes in the electricity network. Thenwe have the two graph layers. Firstly, the electricity network G = ( N , E ) is an undirected graph, i.e., E ⊆ N × N with ( i , j ) ∈ E ⇔ ( j , i ) ∈ E . Secondly, the communication layer isrepresented by a directed graph G C = ( N , E C ) , with a biparti-tion of N into N C with higher-layer control present and N n without higher-layer control. N C ˙ ∪ N n = N where ˙ ∪ is thedisjoint union. Note that E C = { ( i , j ) | i ∈ N , j ∈ N C } (edgesdirected from i to j ) is not necessarily a subset of E and datais available from all nodes in N . We call S ⊂ N a maximalindependent set in N , i.e., ∀ i ∈ N : i ∈ S ∨ N ( i ) ∩ S (cid:54) = /0where N ( i ) denotes the neighbors of i . We label the nodes j ∈ N = { , ..., N } . card( · ) denotes the cardinality of a setand × the Cartesian product of two sets. II. MODELLING
As noted above we use a straightforward and well studiedmodel for providing the basic frequency control of the systemwith bounded frequency deviation, cp. . time [h] po w e r de m and [ % pea k ] total demandbaseline FIG. 1. Exemplary demand curve (periodic and fluctuating compo-nent)
A. Lower layer
Our aim is to use a conceptual model for the dynamics of asingle node that can capture a variety of behaviours. In keep-ing with the inspiration of a fully distributed microgrid, we as-sume that all nodes have control capability. We further assumethat the distributed control mimics the relationship betweensynchronous frequency and the power balance of generationand demand found in traditional synchronous machines .Neglecting voltage dynamics this leads us to the formulationof the Kuramoto model with inertia , with the input power(i.e. basically the natural frequency of the oscillating units)controlled by the distributed control. Further we assume thatthere are some nodes that allow for slower, dispatchable en-ergy that is controlled by the ILC in the higher layer.The open-loop system equations for the nodes j are thengiven by M j ¨ φ j ( t ) = − P dj ( t ) + P LIj ( t ) + P ILCj ( t ) + F j ( t ) , (1a) F j ( t ) = − ∑ k ∈ N V j V k Y jk sin ( φ j ( t ) − φ k ( t )) , (1b)with the time t ∈ R ; φ j is the voltage phase angle of node j in the co-rotating frame and ω j : = ˙ φ j its instantaneous fre-quency deviation from the rated grid frequency. F j denotes theAC power flow from neighbouring nodes under the assump-tion of purely inductive lines. In general, our approach is notlimited to this assumption though. The parameters are the ef-fective inertia M j ; the steady-state voltages V j and the nodaladmittances Y jk , which encode the network topology.The system is driven by the balance between power demandand generation, here, P ILCj and P LIj are the power dispatchedby the higher layer (ILC) if available, and the distributed con-trol (LI), respectively. While we assume that P LIj can be setarbitrarily, P ILCj has restrictions. That is, P ILCj has to be cho-sen to lie within the achievable behaviour P ILCj ∈ B ILCj of thedispatchable energy at the node j . This can encode a variety ofconstraints such as minimal run times, maximum ramp rates,and finite storage. In order to mimic the behaviour of tradingmarkets, with their hourly or quarter-hourly dispatch, we heretake B ILCj to be the space of functions that are constant duringeach hour.The energy demand P dj is a priori unknown. In order toefficiently study the hierarchical control performance with re-gard to communication structure and the stochastic nature ofthe system, we use well-defined synthetic demand curves. Inparticular P dj = P pj + P fj is composed of P pj , which is a pe-riodic baseline demand with randomly selected amplitude ateach node (period T d of a single day), and P fj is composed ofadditive white noise with zero mean with a piece-wise linearinterpolation in intervals of 15 minutes. This is visualised inFig. 1. See Appendix B for an explicit formulation. Whilefor simplicity we here use a traditional separation of dispatchand demand, the model naturally accommodates fluctuatingproduction as well. Note that all power-related quantities hereare scaled with a rated power.The decentralised control in the lower layer is responsiblefor primary and secondary control tasks which is frequencystability and restoration. Hence, the addressed control objec-tive for nodes j ∈ N is to achieve a bounded frequency devi-ation: ∀ t ∀ j ∈ N ω j ( t ) ∈ [ ω min , ω max ] , (2)where we consider ω min = . = .
62 rad/s and ω max = . = .
42 rad/s. The lower-layer controller is cho-sen for this purpose. It has a small steady state error but itavoids instability caused by parallel integrators. The dynam-ics are as follows: P LIj ( t ) = − k p , j ω j ( t ) + χ j ( t ) , (3) T j ˙ χ j ( t ) = − ω j ( t ) − k I , j χ j ( t ) , (4)where χ j is the controller state, T j , k I , j and k P , j are parametersof the lower layer controller.The choice of the control parameters should be in compli-ance with known design criteria (Corollary 4). In order tospecifically choose the parameters within the remaining de-grees of freedom, we select them by probabilistic methods.The results are shown in Figs. 9 to 11 in Appendix C. Hence,we choose k P , j =
525 s/rad and k I , j = .
005 rad/s. The steadystate error (Corollary 4) for the maximum possible demandin this setting is 0.001655 rad/s.Concretely, we use two complementary measures of theperformance of the primary and secondary control, the maxi-mum observed frequency deviation ω top : = max j ∈ N max t ∈ [ t obs , start ; t end ] ω j ( t ) (5)and the exceedance, which is the total time that the frequencyis out of bounds within an observed interval, i.e., exc j : = t end − t obs , start (cid:90) t end t obs , start Θ ( | ω j ( t ) | − ∆ ω ) dt . (6)In this terminology, exc > ∆ ω = . (cid:2) t obs , start ; t end (cid:3) . B. The ILC control layer
The aim of the higher-layer controller is to achieve a statethat is in some sense economically optimal. As stated above,we study the equilibrium state rather than the convergence tothat state. We consider the former a sensible stand-in for othermethods that try to achieve an economically optimal situation.The design and performance of the ILC itself as a method formicrogrids is treated in a companion paper .Concretely, the higher-layer controller looks at the previousday and adjust the dispatch chosen from B ILCj in such a way asto minimise the overall system cost. Therefore, the economicobjective translates to minimising C total = N ∑ j = (cid:90) t end t obs , start λ | P LIj ( τ ) | + ( − λ ) | P ILCj ( τ ) | d τ , (7)with the (here constant) cost factor λ ∈ ( , ] . Hence, λ − λ isa notion of the price relation between P LI and P ILC . As notedabove, the modelling assumption here is that power planneda day in advance is technically less challenging, and thuscheaper than control power that needs to be provided as aninstantaneous reaction, and thus λ > .
5. This is aligned withcurrent market pricing where day-ahead markets trade energyat a much lower price than primary and secondary control en-ergy markets. A similar relation can be expected in micro-grids. Fig. 8 in Appendix A shows how the average overallsystem cost changes with the cost factor λ . For our analysis,we choose λ = . B ILCj , and the next day’s P ILCj are adjusted accordingly, possibly aggregating the updatesfrom communicating nodes as well. Concretely we adjust P ILCj proportionally to E hctrl , j = λ (cid:90) t h + t h sgn ( P LIj ( τ )) d τ + ¯ T ( λ − ) , (8)where t h = ( h − ) t with t ∈ [ t start , t end ] , t start , t end ∈ R ≥ , isthe beginning of hour h ∈ N and sgn( · ) the sign function and¯ T = j = , ..., N and each hour hP ILC , hj = P ILC , h − j + κ j E h − ctrl , j , (9)where P ILC , hj is the value of the hourly constant P ILCj ( t ) for t h ≤ t < t h + , that is, during the hour starting at t h , and κ j ∈ R is the learning gain.Let us now consider the genuine multiplex case, where weallow for communication, that is, the ILC nodes aggregateinformation about the expended control energy at differentnodes, and update using the total. Then, using the adjacencymatrix A C of the communication layer, we get P ILC , hj = P ILC , h − j + κ j d j + (cid:32) E h − ctrl , j + ∑ k (cid:54) = j A Cjk E h − ctrl , k (cid:33) , (10)for all j ∈ N C , where d j = ∑ k ∈ N A Cjk is the degree. In termsof control, this means we use a P-type (i.e., proportional) ILCcontroller. A Q-filter, implying a forgetting filter regardingprevious and upcoming hours of a day or a more sophisticatedinteraction between the nodes, is not considered here, hence Q = I where I is the identity matrix. A Q-filter may becomebeneficial if the demand amplitudes vary with the days. Fur-thermore, we simplify and use the same learning gain for allnodes, i.e., κ j = κ for all j ∈ N . III. PERFORMANCE COMPARISON FOR DIFFERENTMULTIPLEX TOPOLOGIES
Our main focus is to study whether the communication net-work is required in order to achieve sensible economic out-comes, or whether the decentralised robust operation of thegrid implicitly carries enough information between the nodesto achieve an acceptable. outcome without added communi-cation infrastructure.
A. Grid ensemble
As noted above, we apply a probabilistic approach. Thatis, we define a class of grids consisting of a random peak de-mand per node as well as random topology, and evaluate theexpected performance of the design for this grid class. Forsimplicity, and to focus on the effects of the topology in thehigher layer, we use a simple random regular graph with de-gree three for the topology in the lower layer. This is not in-tended to be a realistic choice for most power grids, but pro-vides a homogeneous backdrop on which the higher layer canoperate. The demand amplitudes are chosen uniformly at ran-dom. For details on the demand model, see Appendix B.We consider a sample size of S =
100 grids à card ( N ) =
24 nodes with random demand from the ensemble and thenintegrate the system for 50 days. Investigation of individ-ual trajectories reveals that this is highly sufficient to achieveequilibrium for the ILC in all cases considered (compare Ap-pendix D; Figs. 12 to 16 which show example trajectories forthe different higher-layer scenarios). We then use the perfor-mance on the last ten days to study the steady state properties.All initial conditions are set to zero, i.e. the ILC update setsin after the first day. Find all relevant simulation parametersin Tab. I.
TABLE I. Simulation parameters (node j = ,..., N ).W/W units refer to scaled quantities. Parameter Value Unit Description k I , j k P , j
525 Ws/(W rad) lower layer controlparameter κ j M j /(W rad) inertia N
24 - number of nodes1 / T j Y jk V j V k ∆ ω t end
50 days number of simulateddays t obs , start
40 days start of observationinterval S
100 - number of simulationsin one experiment
We evaluate the expected value of three quantities alreadyintroduced above. First, the maximum frequency deviation ω top to see how far the system deviates from the desired fre-quency, second, the frequency exceedance exc j which indi-cates the quality of the control achieved. The third quantity isthe total cost of higher-layer and lower-layer control energy inthe system C total defined in Eq. (7). B. Higher-layer topologies
We consider five different topologies chosen to illustratedifferent designs of communication and control infrastructure.The first baseline scenario (scenario ) is to study the per-formance of the decentralised control by itself, without anyILC, that is the higher layer is simply the empty graph, N C = /0. The second baseline scenario (scenario I ) is to assume thatevery node has the ability to dispatch energy, and optimises tosatisfy its demand locally, without taking the neighbours intoaccount, N C = N but E C = /0 and thus A C = II and III ) scenarios assumethat the location of the dispatchable power is chosen in somesense to be central in the underlying physical grid. To modelthis, we construct a maximal independent set N C , that is, a(non-unique) maximal set of vertices such that two nodes in N C are never adjacent. Hence, every node in the graph G C is either in N C or neighbour to one node in N C , cp. Fig. 2(top). The edges are directed, accounting for the directed in-formation transfer. By assigning the ILC to N C , every othernode in N n is adjacent to a node with dispatchable energy.With this set of nodes we can now define and compare the sce-narios II and III . Scenario II has communication from these controlgrid FIG. 2. An illustration of the multiplex network consisting of a phys-ical layer ("grid", bottom) and a control layer ("control", top). Di-rected edges are indicated with arrows. The vertical dashed linesidentify the nodes in the two layers. In the upper layer, filled circlesindicate the nodes v ∈ N C where control is present/active. neighbors and scenario III has no communication. In the for-mer case we have a directed communication graph with theadjacency matrix elements A Cjk = j ∈ N C and Y jk > A Cjk = IV ) that the ILC ispositioned at half the nodes at random, and communicate atrandom with three other nodes.The three scenarios II , III and IV thus represent sparse ILCwith no communication, structured communication and ran-dom communication. The scenarios are summarised in Tab.II. C. Results of the numerical experiments
We first consider the maximum frequency deviation to seehow far the system deviates from the desired frequency andthe exceedance of the frequency. These observables showthe quality of the control achieved by the lower layer in thepresence of the various higher-layer topologies. The first plotFig. 3 shows the distribution of ω top across the grid ensemble.In each scenario, ω min (cid:28) ω top (cid:28) ω max , i.e. the lower-layercontrol objective Eq. (2) is always achieved. We see that theperformance is very similar across the communication scenar-ios, it is slightly better for the no-ILC case ( ) and the localILC at all nodes without communication ( I ). Fig. 4 depictsthe exceedance exc j across the grid ensemble for all nodes j .We can observe that adding a higher-layer control reduces theexceedance drastically ( I - IV ). The scenarios with communi-cation ( II , IV ) perform slightly better than without ( I , III ). Itis apparent from the observation of both ω top and exc j thatthe addition of the higher layer the amplitude of transient de-viations but at the same time shortens their duration. Thisindicates that the ILC control suppresses demand fluctuationsthat drive the system out of the bounds given by ∆ ω .More interestingly, we can now consider the total system cost from Eq. (7) over the course of the last ten days of thesimulations, given the various choices of higher-layer control.We chose λ = .
8, i.e., instantaneous control energy beingfour times more expensive than energy at the day-ahead mar-ket. The baseline scenario , with no ILC at all, in the left col-umn of Fig. 5, gives us an idea of the total cost in the absenceof dispatch. Providing dispatchable energy at every node inscenario I reduces the total cost by a factor of almost threewith the parameters chosen, see the second column in Fig. 5.Turning now to the three main scenarios II - IV , we see thatthey all manage to reduce the cost even further compared toscenario I . For those scenarios, we obtain a cost reductionfactor of around four compared to the non-ILC scenario ,consistent with our choice of λ ( λ / − λ = . / . = I is due to con-flicting actions of the decantralised controllers that are elimi-nated by the communication infrastructure in the scenarios II - IV . If, however, the communication topology is not adaptedto the unerlying physical network but random, we also ob-serve a slight cost increase, showing that random aggregationis generally not beneficial over placing control at a maximalindependent set in scenario II .Another distinction between I and II - IV is the number ofnodes in the control set N C . To investigate the influence onthe overall performance, we systematically varied the num-ber of ILC nodes from 1 to card ( N ) =
24. The controllednodes were randomly drawn in each run and not connectedto any other nodes in the communication layer similar to sce-nario
III . The results are two-fold. The maximum frequencydeviation in Fig. 6 decreases monotonically, indicating thatthe best performance is achieved with the highest control ef-fort. Contrarily, Fig. 7 shows that the system is cost-optimalwhen about one third of nodes are equipped with ILC control.Whereas the significant improvement compared to is evi-dent for a small size of N C , a further distribution of controlaction across more nodes leads to slightly higher costs, still re-maining well below the baseline scenario. Interestingly, when N C = N , the expected cost is higher than in scenario I with-out communication. Concerning the cost difference between I and II - IV in Fig. 5, this experiment implies that the reductionis mainly achieved by a smaller-sized control set whereas theaddition of communication links actually increases the costs. IV. DISCUSSION, CONCLUSION AND WAY FORWARD
In this paper, we introduced a multiplex hierarchicalmodel of power grids that covers timescales from secondsto days, and allows studying the interaction of energy man-agement/tertiary control and self-organised primary and sec-ondary control.Remarkably, we find that a basic but natural communicationand aggregation scheme in the higher layer does not improvethe performance. The results indicate that the self-organiseddistributed control of the lower layer already carries sufficientinformation to learn the appropriate dispatch at those nodesthat are dispatchable. Adding explicit communication doesnot visibly reduce the cost of the system while the perfor-
TABLE II. Overview of the studied higher-layer topologies
Exp. Description Communication graph0 no ILC G C is the empty graph I local ILC at all nodes G C = ( N , /0 ) ,card( N C ) = card( N ) II ILC at nodes in a max. in-dependent set in the net-work graph and averagedupdate with all neigh-bouring nodes G C = ( N , E C ) , E C = E ∩ (cid:0) N C × N n (cid:1) , N C max. independent set III local ILC at nodes in amax. independent set inthe network graph G C = ( N , /0 ) , N C max. inde-pendent set IV ILC at 50 % of thenodes with averaged up-date with 3 random othernodes G C = ( N , E C ), E C = { ( i , j ) | i ∈ N C , j ∈ N \ { i }} ,card( N C ) = card( N n ) M a x f r equen cy de v i a t i on [ r ad / s ] FIG. 3. The maximum frequency deviations of the nodes for thevarious higher layers. The box plots show the quartiles and outliersof the system. The coloured box covers the second and third quartile,the middle line gives the median. The T bars give the extrema of thedistribution up to outliers. See Tab. II for the scenario definitions. E xc eedan c e [ % ] I II III IV0.050.100.150.200.250.30
FIG. 4. The exceedance of the nodes for the various higher layers,see Tab. II for the scenario definitions. O v e r a ll c o s t [ a . u .] I II III IV300350400450
FIG. 5. The overall energy cost, sum over all nodes, logarithmic axis,see Tab. II for the scenario definitions. M a x f r equen cy de v i a t i on [ r ad / s ] FIG. 6. Maximum frequency deviation for λ = . O v e r a ll c o s t ( a ll node s ) [ a . u .] FIG. 7. Overall cost for λ = . mance with respect to the lower-layer control objective is ex-pected to decrease slightly.Actually, there already exists implicit communicationthrough the power flows managed by the primary and sec-ondary control in each scenario. It seems that any additionalcommunication that tries to aggregate the local deviations,needs to take this into account. Also a carefully designedQ-filter and learning matrix may improve the performance ofadditional communication.Clearly for more complex and interesting control objec-tives, that a realistic energy management system has toachieve, communication is necessary. However, our resultsshow that even in these cases it might be worthwhile investi-gating the implicit communication already present in the sys-tem, and taking it into account since it leads to inputs takenfrom different parts of the system to be correlated, thus poten-tially causing an overcompensation.More broadly, we saw that for the parametrisation of thedecentralised control layer, probabilistic methods are a usefulcomplement to analytic bounds. The selection of k I , j is chosenaccording to the analytical bounds in (Corollary 4) and toavoid a frequency deviation independently of the choice of k P , j (cp. Fig. 9).Further, the presence of a higher layer with separate controlobjectives certainly has the potential to affect the performanceof the lower layer. For any added ILC there is a large effecton the exceedance (Fig. 4). This is not unexpected as thecontrol fundamentally changes the nature of all nodes. Still, itindicates that it is interesting to further study the interactionsof the layers in the future. In fact, the interaction betweenhigher-layer energy management and the frequency dynamicsof the power grid is an effect that is observed in real powergrids, where trading intervals are very visible in the statisticsof power grid frequency signals . We expect that the typeof model we have set up here is highly useful to study andreproduce some of these results without having to go to highlyspecific market models.The general setup we have chosen can serve as a wide rang-ing basis for the study of the interaction of dynamics and dis-patch constraints. The overall model can easily be extendedto include local limits on available storage and ramping times.Adding individual and time-varying prices for various formsof energy to give the ILC a more realistic target function tooptimise is also straightforward.Further, a more sophisticated higher layer can use the dis-tributed controllers to achieve more challenging goals thanmerely minimising price, or prioritising one type of energyover the other, i.e., in future work, we may consider, e.g., theset [ P ILCj ( t ) , T j ( t ) , k p , j ( t ) , k I , j ( t )] as an input to the ILC.Finally, we note that the type of system we introducedhere can be of independent interest in the context of theo-retical physics. For example, Nicosia et al. analyse thecoupling between different dynamics in a multiplex network,i.e., between a network of Kuramoto oscillators and a randomwalk. Under certain conditions, the coupling between the lay-ers then induces spontaneous explosive synchronisation tran-sitions. Since we study the synchronisation of Kuramoto os-cillators with inertia (Eq. (1)) and use proportional ILC that is also linear, this is mathematically similar to our model. Thusit would be interesting under which conditions models likeours can exhibit such properties as well. V. SOFTWARE
All code was written in Julia and is available on request oron the first authors github . The simulations were performedusing the DifferentialEquations.jl package and the Rodas4psolver . ACKNOWLEDGMENTS
This work was funded by the Deutsche Forschungsgemein-schaft (DFG, German Research Foundation) – KU 837/39-1 /RA 516/13-1.
REFERENCES J. M. Guerrero, J. C. Vasquez, J. Matas, L. G. De Vicuña, and M. Castilla,“Hierarchical control of droop-controlled AC and DC microgrids—a gen-eral approach toward standardization,” IEEE Transactions on IndustrialElectronics , 158–172 (2010). J. Schiffer, D. Zonetti, R. Ortega, A. Stankovicc, T. Sezid, and J. Raisch,“Modeling of microgrids—from fundamental physics to phasors and volt-age sources,” Automatica , 1–15 (2015). A. Bidram and A. Davoudi, “Hierarchical structure of microgrids controlsystem,” IEEE Transactions on Smart Grid , 1963–1976 (2012). F. Dörfler, J. W. Simpson-Porco, and F. Bullo, “Plug-and-play control andoptimization in microgrids,” in (IEEE, 2014) pp. 211–216. D. E. Olivares, A. Mehrizi-Sani, A. H. Etemadi, C. A. Cañizares, R. Ira-vani, M. Kazerani, A. H. Hajimiragha, O. Gomis-Bellmunt, M. Saeedifard,R. Palma-Behnke, et al. , “Trends in microgrid control,” IEEE Transactionson Smart Grid , 1905–1919 (2014). M. Aamir, K. A. Kalwar, and S. Mekhilef, “Uninterruptible power supply(ups) system,” Renewable and Sustainable Energy Reviews , 1395–1410(2016). H. Xin, R. Zhao, L. Zhang, Z. Wang, K. P. Wong, and W. Wei, “A decen-tralized hierarchical control structure and self-optimizing control strategyfor FP type DGs in islanded microgrids,” IEEE Transactions on Smart Grid , 3–5 (2015). Y. Han, H. Li, P. Shen, E. A. A. Coelho, and J. M. Guerrero, “Review ofactive and reactive power sharing strategies in hierarchical controlled mi-crogrids,” IEEE Transactions on Power Electronics , 2427–2451 (2016). Z. Li, C. Zang, P. Zeng, H. Yu, and S. Li, “Fully distributed hierarchicalcontrol of parallel grid-supporting inverters in islanded AC microgrids,”IEEE Transactions on Industrial Informatics , 679–690 (2017). J. Gao, S. V. Buldyrev, H. E. Stanley, and S. Havlin, “Networks formedfrom interdependent networks,” Nature Physics , 40–48 (2012). D. M. Gale, S. Kariv, and U. C. Systems,
American Economic Review ,edited by G. D’Agostino and A. Scala, Understanding Complex Systems,Vol. 97 (Springer International Publishing, Cham, 2014) pp. 99–103. M. Kivelä, A. Arenas, M. Barthelemy, J. P. Gleeson, Y. Moreno, and M. A.Porter, “Multilayer networks,” Journal of Complex Networks , 203–271(2014). M. De Domenico, A. Solé-Ribalta, E. Cozzo, M. Kivelä, Y. Moreno, M. A.Porter, S. Gómez, and A. Arenas, “Mathematical formulation of multilayernetworks,” Physical Review X , 1–15 (2014). V. Nicosia, G. Bianconi, V. Latora, and M. Barthelemy, “Growing multi-plex networks,” Physical Review Letters , 1–5 (2013). S. Boccaletti, G. Bianconi, R. Criado, C. I. del Genio, J. Gómez-Gardeñes,M. Romance, I. Sendiña-Nadal, Z. Wang, and M. Zanin, “The structure anddynamics of multilayer networks,” Physical Reports , 1–122 (2014). P. J. Menck, J. Heitzig, N. Marwan, and J. Kurths, “How basin stabilitycomplements the linear-stability paradigm,” Nature Physics , 89 (2013). P. J. Menck, J. Heitzig, J. Kurths, and H. J. Schellnhuber, “How dead endsundermine power grid stability,” Nature Communications , 3969 (2014). P. Schultz, J. Heitzig, and J. Kurths, “Detours around basin stability inpower networks,” New Journal of Physics , 125001 (2014). F. Hellmann, P. Schultz, C. Grabow, J. Heitzig, and J. Kurths, “Survivabil-ity of deterministic dynamical systems,” Scientific Reports , 29654 (2016). P. Schultz, F. Hellmann, K. N. Webster, and J. Kurths, “Bounding thefirst exit from the basin: Independence times and finite-time basin stabil-ity,” Chaos: An Interdisciplinary Journal of Nonlinear Science , 043102(2018). M. Lindner and F. Hellmann, “Stochastic basins of attraction and general-ized committor functions,” Physical Review E , 022124 (2019). H. Kim, S. H. Lee, and P. Holme, “Building blocks of the basin stability ofpower grids,” Physical Review E , 062318 (2016). H. Kim, S. H. Lee, J. Davidsen, and S.-W. Son, “Multistability and varia-tions in basin of attraction in power-grid systems,” New Journal of Physics , 113006 (2018). M. F. Wolff, P. G. Lind, and P. Maass, “Power grid stability under pertur-bation of single nodes: Effects of heterogeneity and internal nodes,” Chaos:An Interdisciplinary Journal of Nonlinear Science , 103120 (2018). J. F. Donges, H. C. Schultz, N. Marwan, Y. Zou, and J. Kurths, “Investigat-ing the topology of interacting networks: Theory and application to coupledclimate subnetworks,” European Physical Journal B , 635–651 (2011). R. Criado, J. Flores, A. García Del Amo, J. Gómez-Gardeñes, andM. Romance, “A mathematical model for networks with structures in themesoscale,” International Journal of Computer Mathematics , 291–309(2012). C. M. Schneider, N. Yazdani, N. A. Araújo, S. Havlin, and H. J. Herrmann,“Towards designing robust coupled networks,” Scientific Reports , 1969(2013). S. V. Buldyrev, R. Parshani, G. Paul, H. E. Stanley, and S. Havlin, “Catas-trophic cascade of failures in interdependent networks,” Nature , 1025–1028 (2010). A. Vespignani, “Complex networks: The fragility of interdependency,” Na-ture , 984–985 (2010). A. Krishna, J. Schiffer, N. Monshizadeh, and J. Raisch, “A consensus-based voltage control for reactive power sharing and PCc voltage regula-tion in microgrids with parallel-connected inverters,” in th Euro-pean Control Conference (ECC) (IEEE, 2019) pp. 542–547. M. S. Mahmoud, M. S. U. Rahman, and M.-S. Fouad, “Review of micro-grid architectures–a system of systems perspective,” IET Renewable PowerGeneration , 1064–1078 (2015). D. Wu, T. Dragicevic, J. C. Vasquez, J. M. Guerrero, and Y. Guan, “Sec-ondary coordinated control of islanded microgrids based on consensusalgorithms,” in (IEEE, 2014) pp. 4290–4297. S. Mao, Z. Dong, P. Schultz, Y. Tang, K. Meng, Z. Y. Dong, and F. Qian,“A finite-time distributed optimization algorithm for economic dispatch insmart grids,” IEEE Transactions on Systems, Man, and Cybernetics: Sys-tems (2019). E. Weitenberg, Y. Jiang, C. Zhao, E. Mallada, F. Dörfler, and C. De Persis,“Robust decentralized frequency control: A leaky integrator approach,” in (2018) pp. 764–769. A. Weidlich and D. Veit, “A critical survey of agent-based wholesale elec-tricity market models,” Energy Economics , 1728–1759 (2008). P. Ringler, D. Keles, and W. Fichtner, “Agent-based modelling and simula-tion of smart electricity grids and markets–a literature review,” Renewableand Sustainable Energy Reviews , 205–215 (2016). M. Mureddu and H. Meyer-Ortmanns, “Extreme prices in electricity bal-ancing markets from an approach of statistical physics,” Physica A: Statis-tical Mechanics and its Applications , 1324–1334 (2018). F. Claessen, B. Claessens, M. Hommelberg, A. Molderink, V. Bakker,H. Toersche, and M. van den Broek, “Comparative analysis of tertiary con-trol systems for smart grids using the flex street model,” Renewable Energy , 260–270 (2014). C. Zhao, E. Mallada, S. Low, and J. Bialek, “A unified framework forfrequency control and congestion management,” in (IEEE, 2016) pp. 1–7. N. Amann, D. H. Owens, and E. Rogers, “Predictive optimal iterative learn-ing control,” International Journal of Control , 203–226 (1998). D. A. Bristow, M. Tharayil, and A. G. Alleyne, “A survey of iterativelearning control,” IEEE Control Systems Magazine , 96–114 (2006). Z. Zeng, H. Yang, R. Zhao, and C. Cheng, “Topologies and control strate-gies of multi-functional grid-connected inverters for power quality enhance-ment: A comprehensive review,” Renewable and Sustainable Energy Re-views , 223–270 (2013). K.-T. Teng,
Repetitive and Iterative Learning Control for Power Converterand Precision Motion Control , Ph.D. thesis, UCLA (2014). B. Chai, Z. Yang, K. Gao, and T. Zhao, “Iterative learning for optimalresidential load scheduling in smart grid,” Ad Hoc Networks , 99–111(2016). A. Bampoulas, M. Saffari, F. Pallonetto, E. Mangina, and D. P. Finn, “Self-learning control algorithms for energy systems integration in the residentialbuilding sector,” in th World Forum on Internet of Things (WF-IoT) (IEEE, 2019) pp. 815–818. W. Guo, F. Liu, J. Si, D. He, R. Harley, and S. Mei, “Online supplementaryADP learning controller design and application to power system frequencycontrol with large-scale wind energy integration,” IEEE Transactions onNeural Networks and Learning Systems , 1748–1761 (2015). H.-Q. Guo, C.-Z. Liu, J.-W. Yong, X.-Q. Cheng, and F. Muhammad,“Model predictive iterative learning control for energy management ofplug-in hybrid electric vehicle,” IEEE Access (2019). D. H. Nguyen and D. Banjerdpongchai, “Iterative learning control of energymanagement system: Survey on multi-agent system framework,” Engineer-ing Journal , 1–4 (2016). J.-X. Xu and S. Yang, “Iterative learning based control and optimization forlarge scale systems,” IFAC Proceedings Volumes , 74–81 (2013). L. Strenge, X. Jing, T. Seel, F. Hellmann, and J. Raisch, “Iterative learningcontrol in prosumer-based microgrids with hierarchical control,” in submit-ted to IFAC World Congress 2020 (2020) p. tbd. D. J. Hill and G. Chen, “Power systems as dynamic networks,” in (IEEE, 2006) pp.4–pp. J. Machowski, J. Bialek, and J. Bumby,
Power system dynamics: stabilityand control (John Wiley \ & Sons, Ltd., 2011). J. Schiffer, R. Ortega, A. Astolfi, J. Raisch, and T. Sezi, “Conditionsfor stability of droop-controlled inverter-based microgrids,” Automatica ,2457–2469 (2014). B. Schäfer, C. Beck, K. Aihara, D. Witthaut, and M. Timme, “Non-Gaussian power grid frequency fluctuations characterized by lévy-stablelaws and superstatistics,” Nature Energy , 119 (2018). M. Anvari, L. R. Gorjão, M. Timme, D. Witthaut, B. Schäfer, and H. Kantz,“Stochastic properties of the frequency dynamics in real and syntheticpower grids,” arXiv:1909.09110 (2019). V. Nicosia, P. S. Skardal, A. Arenas, and V. Latora, “Collective PhenomenaEmerging from the Interactions between Dynamical Processes in MultiplexNetworks,” Physical Review Letters , 138302 (2017). Repository controlled-multi-timescale-powergrid at https://github.com/strangeli/controlled-multi-timescale-powergrid . C. Rackauckas and Q. Nie, “Differentialequations. jl–a performant andfeature-rich ecosystem for solving differential equations in julia,” Journalof Open Research Software (2017). G. Wanner and E. Hairer,
Solving ordinary differential equations II (Springer Berlin Heidelberg, 1996).
Appendix A: Motivating the ILC update law
We want to motivate the precise form of the update law usedin the ILC above. If we assume that the system is roughly heldin equilibrium, despite the low-amplitude fluctuations, Eq. (1)gives us00 ≈ − P dj ( t ) + P LIj ( t ) + P ILCj ( t ) + F j ( t ) , (A1)for each node j = , ... N . In the following, we omit thenode index for readability. If we neglect changes to the flows,we can approximately assume ∂ P LI ∂ P ILC ≈ − , (A2)i.e. a decrease in P LI is directly proportional to an increasein P ILC .The aim of the
ILC is adapting P ILC to optimize an observ-able O ( P LI ) . Take for instance O ( P LI ) = T (cid:90) | P LI | dt , (A3)the quadratic norm of the lower-layer control power. If wechange P ILC by a constant shift δ P ILC that does not depend on t we find that the variation of O ( P LI ) is approximated as δ O ( P LI ) = ∂ O ( P LI ) ∂ P ILC δ P ILC = T (cid:90) P LI ∂ P LI ∂ P ILC dt δ P ILC ≈ − T (cid:90) P LI dt δ P ILC . Thus, if we choose the
ILC update to be δ P ILC = (cid:82) P LI dt then the change in O is always negative and we gradient de-scend towards a local minimum. As our P ILC are constant onthe hour, this update law is sufficient to make sure that weminimize the square norm of P LI for each hour.A more economic objective function could be to integratethe total cost of energy used, taking into account that there aredifferent price points for energy bought (and scheduled) a dayahead or requested from a standing reserve of control energy.This suggests the objective function O λ ( P LI , P ILC ) = T (cid:90) (cid:2) λ | P LI | + ( − λ ) | P ILC | (cid:3) dt = T (cid:90) (cid:2) λ P LI sgn ( P LI ) + ( − λ ) P ILC sgn ( P ILC ) (cid:3) dt , (A4)where λ ∈ [
0; 1 ] is a real number. It should be noted thatin reality the composition of costs is considerably more com-plex. For example, capacity markets reward keeping a certain amount of generation available, whether it is used or not. Theobjective function Eq. (A4) only reflects the presence of dif-ferent price levels for different levels of flexibility.In order to calculate the variation of O λ with respect to asmall constant shift δ P ILC we make the further assumptionthat the contribution from the shift in the sgn functions is ofhigher order, as can be expected if P LI is sufficiently smooth.Then we obtain: δ O λ = T (cid:90) λ sgn ( P LI ) δ P LI dt + T ( − λ ) sgn ( P ILC ) δ P ILC = − λ T (cid:90) sgn ( P LI ) dt + T ( λ − ) sgn ( P ILC ) δ P ILC (A5)For λ =
1, that is day ahead energy is infinitely cheaperthan instantaneous energy, we can choose the
ILC update δ P ILC ∼ T (cid:90) sgn ( P LI ) dt (A6)(where ∼ means "proportional to"), which guarantees that δ O λ = is negative and we again descend to a sensible localminimum. Intuitively, this makes sense since we should in-crease the background power when there are more times whenpositive control energy is needed.For general λ , first note that as the ILC compensatesa positive background demand we can always assume thatsgn ( P ILC ) =
1. Then we have: δ O λ = − λ T (cid:90) sgn ( P LI ) dt + T ( λ − ) δ P ILC . (A7)Therefore, we want to chose the update law δ P ILC ∼ λ T (cid:90) sgn ( P LI ) dt + T ( λ − ) , (A8)to obtain an appropriate gradient descend.In summary we have the following economic update law: δ P ILC = k T (cid:90) (cid:2) λ sgn ( P LI ) + ( λ − ) (cid:3) dt (A9)Fig. 8 shows the average total cost for all numerical experi-ments performed in this paper. λ > . A v e r age t o t a l c o s t [ a . u .] I: local ILC all nodesII: local ILC at vc + neighbor. comIII: local ILC at vcIV: local ILC rand 50% + rand com
FIG. 8. Average mean cost over the cost factor λ (50 days simulated,last 10 days evaluated) Appendix B: Synthetic demand model
For every node j ∈ N in the network, we assume the de-mand is given by a periodic baseline P pj subject to fluctuations P fj , P pj ( t ) = A j sin (cid:18) π tT d (cid:19) , (B1) P fj ( t ) = ( t mod T q ) B j ( + (cid:98) t / T q (cid:99) ) + B j ( (cid:98) t / T q (cid:99) ) , (B2)where the period T d [s] is the duration of a day and the demandamplitudes A j ∼ U ([
0; 1 ]) are uniform i.i.d. random numbers.The fluctuation amplitudes B j vary over time in a Gaussianrandom walk with zero mean and a variance of 0.2. P fj islinearly interpolated between two consecutive updates, spacedapart by T q = A j and B j are normalized with a rated power. Appendix C: Selection of the lower-layer control parameters
The control parameters k p , j ∈ [ , ] s/rad and k I , j ∈ [ . , ] rad/s are varied in numerical experiments with 41and 40 values for each parameter, respectively. This results in41 · = k P , j and k I , j . We are interested ina low value for all given quantities. In combination with the analytic bounds given in the literature , we choose k P , j = k I , j = .
005 rad/s for all nodes j ∈ N . FIG. 9. Selection of control parameters: maximum frequency devia-tion; simulated and observed for 1 day, other parameters from Tab. Iexcept for k I , j and k P , j . The resulting maximum frequency deviationis 0 . FIG. 10. Selection of control parameters: exceedance; simulated andobserved for 1 day, other parameters from Tab. I except for k I , j and k P , j . The resulting exceedance is 4.53 · − %. Appendix D: Representative trajectories for all scenarios
Figs. 12 to 16 show exemplary trajectories of cumulativelower layer control energy used for 24 nodes for cases 0-IVover a time of 20 days including the initial learning phase.In case 0 the energy is cumulative over the whole time spanwhile in the cases I-IV it is cumulative over every hour onlyand then reset. Recall that for the equilibrium state analysisabove, the days 20-30 are chosen which are not shown here indetail. It can be observed that ILC at all nodes learns faster butthe equilibrium performance is worse than in the other cases.Other experiments performed show that during the learningprocess, faster learning leads to lower costs.2
FIG. 11. Selection of control parameters: frequency variance; simu-lated and observed for 1 day, other parameters from Tab. I except for k I , j and k P , j . The resulting frequency variance is 1.0586 rad/s.FIG. 12. Exemplary trajectories of lower layer control energy for24 nodes for case 0 over a time of 20 days incl. the initial learningphase. Note that in case 0 this is the total cumulative energy over thewhole time span since there are no resets.FIG. 13. Exemplary trajectories of lower layer control energy for 24nodes for case I over a time of 20 days incl. the initial learning phase.3