A Data-driven Nonlinear Recharge Controller for Energy Storage in Frequency Regulation
AA Data-driven Nonlinear Recharge Controller forEnergy Storage in Frequency Regulation
Wenting Ma
Dept. of Electrical EngineeringTsinghua UniversityBeijing, [email protected]
Bolun Xu
Dept. of Earth and Environmental EngineeringColumbia UniversityNew York, NY, [email protected]
Abstract —Battery energy storage boosts up the response speedof power system frequency regulation, but must be rechargedcarefully to minimize the distortion to the frequency regulationresponse. This paper proposes a nonlinear feedback controllerto optimize the recharge for storage resources in frequencyregulation. This controller is designed using a data-drivenbest-hindsight optimization framework, the resulting nonlinearrecharge controller’s gain depends on the storage state of chargeas well as its power and energy rating. The developed controlleris compared with two benchmark automatic generation controldesigns, one is a proportional-integral-based control from PJMInterconnection, the other one is based on linear-quadraticregulator. Simulation results using real area control error datafrom PJM Interconnection show the proposed controller achievessmaller deviations in both the area control error and the storagestate of charge compared to the two benchmark controllers undervarious storage configurations.
Index Terms —Frequency regulation, Energy storage, Nonlin-ear control, Data-driven control
I. I
NTRODUCTION
Frequency regulation was one of the first grid-scale appli-cations for battery energy storage (BES) and was consideredan ideal BES application due to its high requirement ofresponse speed and low requirement of energy capacity. Powersystem frequency stability depends on the balance betweengenerations and demands, and is critical to the reliable supplyof electric energy. Units participating in frequency regulationby adjusting their active power output according to automaticgeneration control (AGC) signals instructed by the systemoperator. With the increasing penetration of intermittent re-newable generations, conventional thermal units can hardlyfollow the disturbance dynamics resulting from the loss ofrotational inertia and increasing time-variant generation [1],and BES becomes an ideal choice to boost-up the responsespeed regulation services and reduce the capacity procurementrequirement [2].Yet, a new challenge arises in systems where BES partic-ipants are taking up a significant market share in frequencyregulation services: to manage the recharge need of these BESwhile minimizing the distortion to the regulation response .PJM Interconnection, the largest system operator in North
Wenting Ma was a visiting student at Columbia University in Spring 2020supported by the Columbia College Visiting Student Program and TsinghuaUniversity’s International Class of Energy Internet Program.
America, was a pioneer in integrating BES and its regulationmarket share of BES peaked 50% in 2018 [3]. These BESfollow a filtered AGC signal (RegD signal) to make sure thenet energy consumption is less than 15 minutes (normalized bythe rated power of the BES), so that BES with limited energycapacity can be utilized more effectively than through thetraditional regulation signal (RegA signal) [4]. However, PJMhas experienced cases that the recharge need for these storageout-weights the AGC, or these BES have no more energyleft when the system is still experiencing a contingency. As aresult, a higher BES ratio can worsen the regulation quality andweaken the system reliability, and PJM has revised its AGCdesign and enforced a 40% cap on storage participation [5].The key challenge in designing AGC for storage is to incor-porate its charge/discharge efficiency and tight energy limits,which are nonlinear dynamics that cannot be well addressed bycommon linear system controllers such as proportional-integral(PI) or linear-quadratic regulator (LQR) [6]. On the other hand,frequency fluctuations in power systems are non-Gaussian dueto human interactions and renewable intermittency [7], whichalso degrades the performance of traditional LQR controllers,while emerging researches on data-driven AGC have notconsidered the recharge need for storage resources [8]. Thenovelty of this paper is the design of a controller that learnsthe optimal control policies from historical regulation signalswithout assuming any underlying distributions and activelyincorporates storage constraints. The policy is implemented asa lookup table and can be seamlessly incorporated into existingPI-based AGC designs for managing storage recharge, whilethe rest of the AGC, especially the control loop for conven-tional generators, remains unchanged. The main contributionsof this paper can be summarized as follows: • We design a nonlinear feedback controller to manage stor-age state of charge (SoC) in frequency regulation usinga data-driven best-hindsight optimization [9] approach. • We conclude best-hindsight solutions into a feedback gainlookup table and incorporate it with AGC for conven-tional generators. • We tested the performance of the controller againstbenchmark AGCs under different BES configurationsusing historical PJM regulation signals. a r X i v : . [ ee ss . S Y ] J a n he rest of the paper is organized as follows: Section IIpresents the problem formulation and introduces benchmarkcontrollers, Section III describes the best-hindsight optimiza-tion problem and policy selection of proposed BES controller.Section IV evaluates the performance of the controller bysimulation, and performs a quantitative correlation analysisbetween initial battery states and controller gains, and finallysection V concludes with our findings.II. F ORMULATION AND B ENCHMARKS
We start by introducing the set-up for the frequency regu-lation simulation environment and unit response models, thenintroduce the two benchmark AGC designs. We also introducean anti-windup design into the AGC to adapt linear controllersfor handling tightly constrained BES units.
A. Simulation Framework
To compare the performance of different controllers, wefirst set up a frequency regulation simulation environmentaccording to the PJM AGC design manual [5]. Fig. 1 illustratesthe flowchart of this model. Signal input into this simulationis the uncorrected area control error (ACE) signal, which canbe reconstructed using the historical PJM regulation signals,unit capacity, and the unit response model (See Section II-B).The ACE signal is the sum of the corrected ACE signal plusthe response from all regulation units, and it is feed into theAGC and generates RegA and RegD signals to conventionalgenerators and BES participants. Responses from all regulationunits are added to the ACE. Note that this framework issimilar to signal tracking and does not consider the frequencydynamics, because in U.S. frequency regulation belongs to thesecondary frequency control tier and its propose is to mitigateACE and relieve the primary response, instead of stabilizingsystem dynamics [10].
Fig. 1. The AGC block diagram.
B. Modeling of the Conventional Generators and BES
Conventional generator response model [11] is described inFig. 2 (a). After the controller sends RegA signal to generators,a deadband is applied to prevent excessively frequent response. G g ( s ) = 1 / ( T g s + 1) is the governor dynamics. Power outputof generators P g is constrained by ramp rates and maximumgeneration capacity.The response model of BES is described in Fig. 2 (b),which considers charging and discharging efficiency as well Fig. 2. (a) Conventional generator model. (b) BES model as SoC constraints. e is the SoC of BES, and is fed back tothe controller. C. PJM Conditional Neutrality Controller
Beginning on Jan.9, 2017, the PJM’s new conditionalneutrality AGC [5] was put into production. It is a hybridproportional-integral (PI) controller including a RegD integralfeedback loop so that RegD is energy neutral. This AGC sendsslow RegA signal to conventional generators with limitedramping capability and unlimited energy, and send the remain-ing RegD signal to faster resources with limited energy[4].
Fig. 3. Structure of PJM Conditional Neutrality Controller.
The structure of this controller is shown in Fig. 3, where P ACE denotes filtered ACE, G f ( s ) = 1 / ( T a s + 1) is the lowpass filter for RegA signal, and G f ( s ) = 1 / ( T d s + 1) is thelow pass filter for RegD signal. The PI control implementedin this controller is given as the following: P AGC = − K P P ACE − K I I ACE (1a) I ACE = (cid:90) P ACE dt (1b) P AGC is the signal output of PI controller, K P and K I denotes controller gains of the proportional term and integralterm. Parameters of limiters are set according to output powerconstraints on RegA and RegD resources within the area. D. LQR-based AGC
The PJM controller determines the RegD signal as theresidual between the actual AGC and the low-passed RegAsignal. This prohibits adjustments of the control policy tofurther minimize corrected ACE ( P ACE in Fig. 1) for a lineartime-invariant system. Besides, a regulating action performedy BES suffers efficiency losses so that storage will slowlybecome empty. We introduce another benchmark AGC basedon LQR to determine the trade-off between AGC performanceand regulation costs. Although the use of LQR in designingAGC is questionable because the regulation system is non-linear and ACE disturbances are not Gaussian, our goal is toprovide another benchmark in the comparison, and this LQR-based AGC does show competitive performance compared tothe PJM AGC in simulation.The continues time-domain LQR AGC formulation is min x,u J := (cid:90) ∞ t x T Qx + u T Ru (2a) ˙ x = Ax + Bu + ω (2b)The LQR AGC uses a pre-set PI control for RegA andoptimizes the RegD control, the states include ACE, RegA,integral of ACE, and BES SoC. The control is the RegD signal. x =[ P ACE , RegA, I
ACE , e ] T (2c) u t = RegD (2d)Based on the modeling approach of system dynamics proposedin [12], the state and input matrices A and B can be given as A = − M M − K P T a − T a − T a K I (2e) B = (cid:2) M (cid:3) (2f)where M is the normalized inertia constant. The LQR gener-ates a state-feedback control for RegD as u = − Kx (2g)where K is the regulation control gain of system states. E. Anti-windup Control
We add anti-windup control [13] to all AGC designs to avoidintegral wind-up in the integrals caused by control saturation.Nonentities in regulation units, including governor dead-bands,ramp rates, and storage energy constraints, introduces nonlin-ear dynamics and sometimes even cause instability [14]. Whenthe actuator saturates as reached a constraint, the integral termin (1a) tends to accumulate the control error and dominate thecontroller, so called wind-up , and a return to normal controloperations becomes difficult due to the large integral value.Wind-up effects are more likely to occur in systems with highBES share as storage may frequently reach upper or lowerenergy limits [3], though they are less noticeable in traditionalAGC because conventional generators are less likely to beconstrained,We modify the integrator in (1b) by adding a comparingterm of the actual input and commanded input. As shownin Fig. 1, we suppose certain states of actuators can bemeasured and fed back to the controller. This includes the actual generators output P g , BES power output P e , and SoCof BES e . The integrator is now written as: I ACE = (cid:90) [ P ACE + (
RegA − P g ) + ( RegD − P e )] dt (3)The scheme is not in effect when system’s actual outputis synchronized with the control signal, but feeds back themismatch when non-linearities are encountered.III. O PTIMIZATION - AIDED C ONTROLLER D ESIGN
We introduce a data-driven best-hindsight optimizationproblem based on the existing PJM AGC to derive the optimalrecharge policy for BES given their nonlinear dynamics, andimplement this policy as a look-up table. Our storage control(RegD) is defined as a PI control combined with a nonlinearSoC feedback control as
RegD t = − K Dp P ACE,t − K DI I ACE,t − f SoC ( e t ) e t (4)where K Dp and K DI are proportional and integral control gainsfor the RegD signal in M W , while f SoC ( e t ) is the SoCfeedback control gain and it is a function dependent on thestorage state-of-charge value e t in M W · h .We choose the SoC feedback gain as the only nonlinearcontrol component based on the observation that during thedesign process the PI gain does not show meaningful depen-dency on the storage SoC. Also, system operators usually havepretty good knowledge about the choice of PI gains for theirAGC but are less familiar with how to manage storage SoC,therefore this study focused on the SoC feedback gain andassume it to be a scalar function to the storage SoC. A. Best-hindsight Optimization Formulation
We use a best-hindsight framework to design the feedbackgain for state-of-charge f soc ( e t ) in order to better incorporatethe BES non-linearity due to efficiency and energy limitssubjects to the non-Gaussian ACE signal with unknown dis-tribution ( p < . in both Jarque-Bera test and Lillieforstest). In the best-hindsight framework, we solve for the optimalSoC feedback gain K e over a relatively short period T witha starting time index t using historical ACE data as adeterministic optimization problem. We repeatedly solve thisoptimization for various t and initial SoC e , and recordall optimized SoC feedback gains ( K e ), which we eventuallyconcluded it into a function with respect to the storage state-of-charge. Such that the control policy is data-driven as it dependson the historical realizations of the ACE. The formulation forthis deterministic optimization problem is min K e J = Σ t + Tt = t [( P ACE,t ) + w e · e t ] (5a)s.t. RegA t = − K p P ACE,t − K I I ACE,t (5b)
RegD t = − K Dp P ACE,t − K DI I ACE,t − K e e t (5c)RegD control as in (4)system and unit response models as in Fig. 1 & 2his program optimizes a SoC feedback gain K e thatminimizes the objective using given PI control settings. (5a)is the cost function, where w e is the weight coefficient on itsSoC deviation term. By modifying the value of w e , the relativeimportance between reducing output ACE and keeping SoCclose to its desired state is adjusted. (5b) calculates the RegAsignal using a pre-set PI control setting, and (5c) calculatesthe RegD signal using a different PI control setting and aSoC feedback gain K e , which is the decision variable in thisproblem. The two regulation signals are then sent into unitresponse models as shown in Fig. 2 according to the simulationframework in Fig. 1. B. Solution Method
We obtain the feedback gain function f SoC by fitting resultsof K e in (5) to a uni-variate function depending on storageSoC e . (5) is a nonlinear optimization problem because of thequadratic objective and the use of saturation functions, and itis solved using the fmincon function from MATLAB.We train f SoC using historical ACE signal w t over a period t ∈ { , , . . . , H } with following steps.1) Discretize value range of SoC e ∈ [0 , E ] into N = 500 bins: { e n | n ∈ { , , . . . , N }} .2) For t = 1 : ( H − T ) Solve (5), record the solution K e and its correspondinginitial SoC level e : K e ( t ) , e ( t ) .3) f SoC ( e ) is obtained by computing the average value ofrecorded K e s: f SoC ( e ) = (cid:80) H − Tt =1 K e ( t ) { e n − ≤ e ( t ) < e n } (cid:80) H − Tt =1 { e n − ≤ e ( t ) < e n } (6) C. Training Results and Insights
We fix RegA capacity to C a = 400 M W according to PJMregulation requirements [5]. We train the lookup table f SoC ( e ) under various BES configurations. The obtained feedback gainfunction f SoC ( e ) is nonlinear, as visualized Fig. 4. Here,a larger BES energy capacity is represented by a longerdischarge time. Feedback gains from different BES settingsshare some common features, such as the gain is at minimalwhen the SoC is around the reference 50% level, and increasesas the SoC approaches storage limits. An interesting result isthat the feedback gain is reduced when BES is nearly empty(SoC less than 10%), which is likely due to that the systemis experiencing contingencies and has a persistent demandfor regulation generation that depletes the storage. A largerecharge power in this scenario may add too much stress tothe AGC and worsen the overall regulation quality, whichis a similar concept to PJM’s current practice of manuallydisabling the energy neutrality control during contingencies.IV. S IMULATION AND R ESULTS
We present simulation results of the proposed controller andbenchmark controllers using real uncorrected ACE data fromPJM, i.e., the ACE of the system assuming there is no reg-ulation response. The uncorrected ACE data is reconstructed
Fig. 4. Averaged recharge control policy function f SoC ( e ) under variousBES configurations. The feedback gain is applied to the SoC deviation to thereference target in MWh units. using historical regulation signals and unit response modelsaccording to the PJM AGC manual [5]. 30 days of 2-second-resolution data are used to construct the proposed rechargepolicy, and the controller performance is evaluated usinganother 15 days of data. All AGC have the same setting ofPI control gains for conventional regulation units (RegA) with K P = 0 , K I = 0 . . The rest of the benchmark PJM controllerparameters are based on the PJM manual [5]. The benchmarkLQR controller is tuned to achieve the best control perfor-mance using cost weights Q (1 ,
1) = 1 , Q (2 ,
2) = ( C d /C a ) , Q (3 ,
3) = 0 , Q (4 ,
4) = 1 /E , and R = 1 . The resulting state-feedback control gain K is K = [0 . , − . , . , .In our simulations, the LQR control performance is notsensitive to different storage settings so the same K is usedfor all storage cases. For the proposed controller, we set theRegD PI control setting to K DP = 1 and K DI = 0 . , thewindow length T for best-hindsight optimization in (5a) to15 minutes. The SoC deviation weight w e is tuned using anexhaustive search in each storage setting to achieve the bestoverall control performance.We simulate 400MW RegA capacities (i.e. conventionalgenerators) with varying RegD capacities as well as the BESenergy capacity. We assume all RegD is provided only by BES,and assume all BES units have identical physical parameters,initial SoC, and response characteristic, so all BES units canbe aggregated as a single BES in the design and simulations.The BES round-trip efficiency is 85%. Fig. 5 demonstratescomparisons of the uncorrected ACE, responses from differentAGC designs, and the resulting total SoC from all storage.We use two indexes to compare the control performanceof different AGC designs: the average values of output ACEsquare ( P ACE ) and the SoC deviation square e , where weset e = 50% as our desired state of BES. Table I showsthe results over the 15 day test period. Note that under theBES configuration of 200MW/60min, we adjust the parametersettings for the proposed controller to T =30min and w e = 10in order to obtain best performance. The proposed controlleris observed to reduce output ACE error by up to 10.74% ig. 5. Comparison of controller performance. Overall, the three controllersprovide similar shapes of ACE control results and SoC profiles, while theproposed control achieves the smallest ACE deviations and SoC deviations.TABLE IC ONTROLLER P ERFORMANCE U NDER D IFFERENT
BES C
ONFIGURATIONS
BES ( P ACE ) / × e Config.
Prop. LQR PJM Prop. LQR PJM200MW/15min 32.6 34.3 36.5 110.7 101.9 116.1200MW/20min 31.0 32.4 33.9 152.3 160.1 174.7200MW/30min 29.5 29.8 30.6 216.4 310.9 323.4200MW/60min 24.5 26.4 26.3 730.3 1015.7 1008.6300MW/15min 28.6 29.8 31.3 158.7 206.5 176.7400MW/15min 26.6 29.8 28.8 189.2 370.4 218.9 compared to LQR controller, and 10.68% compared to PJMcontroller. Moreover, it reduces the SoC deviation by up to48.92% compared to LQR controller, and 33.09% compared toPJM controller. This improvement in reducing extreme chargeand discharge is especially significant for BES with greaterenergy capacity such as 200MW/30min and 200WM/60min.It is worth noting that the performance of the proposedcontroller depends heavily upon the tuning of SoC deviationweight setting w e . The current setting of w e = 60 achievesthe overall best performance under a BES configuration of300MW/15min, where output ACE error is reduced by 4.16%compared to LQR controller, and SoC deviation is reduced by10.20%. Meanwhile, an improvement is expected by tuning w e under a rather under-performing configuration of BES.V. C ONCLUSION
We developed a data-driven nonlinear feedback control tooptimize the recharge need for battery energy storage infrequency regulation. This feedback controller uses a best-hindsight framework that models non-linear system dynamicsand non-stochastic disturbances, and can incorporate with anyexisting AGC design. Simulation results using real ACE data shows the proposed controller achieved significant improve-ments compared to two benchmark AGC in both control-ling ACE and preventing excessive use of ESS that mightexacerbate degrading. Moreover, it is capable of adaptingto different BES configuration by adjusting relative weightsaccordingly. The correlation between control gains and initialSoC is analyzed to help understand the effect of BES capacityconstraints on the controller’s choice of control policy.The key insight from our study is that energy storagerecharge can be better managed with nonlinear controllers,which were not considered by any existing AGC design.We hope results from this paper could incentivize new ideason how to better incorporate storage resources in ancillaryservices to achieve better grid reliability and sustainability.Future directions includes generalize this control method overdiverse storage resources and to large-scale interconnectedsystems composed of multiple balancing areas. Our resultsshow that new monitoring schemes and market designs arerequired to improve the utilization for storage resources.R
EFERENCES[1] A. Ulbig, T. S. Borsche, and G. Andersson, “Impact of low rotationalinertia on power system stability and operation,”
IFAC ProceedingsVolumes , vol. 47, no. 3, pp. 7290–7297, 2014.[2] Y. V. Makarov, S. Lu, J. Ma, and T. B. Nguyen, “Assessing the valueof regulation resources based on their time response characteristics,”Pacific Northwest National Lab.(PNNL), Richland, WA (United States),Tech. Rep., 2008.[3] B. Xu, Y. Shi, D. S. Kirschen, and B. Zhang, “Optimal battery partic-ipation in frequency regulation markets,”
IEEE Transactions on PowerSystems , vol. 33, no. 6, pp. 6715–6725, 2018.[4] T. Wang, “Battery assisted conventional generator in pjm frequencyregulation market,” in . IEEE, 2019, pp. 1–5.[5] “Implementation and rationale for pjm’s conditional neutrality regulationsignals,” [Available Online] http://pjm.com/ ∼ /media/committeesgroups/task-forces/rmistf/postings/regulation-market-whitepaper.ashx.[6] A. Isidori, Nonlinear control systems . Springer Science & BusinessMedia, 2013.[7] B. Sch¨afer, C. Beck, K. Aihara, D. Witthaut, and M. Timme, “Non-gaussian power grid frequency fluctuations characterized by l´evy-stablelaws and superstatistics,”
Nature Energy , vol. 3, no. 2, pp. 119–126,2018.[8] P. Hidalgo-Gonzalez, R. Henriquez-Auba, D. S. Callaway, and C. J.Tomlin, “Frequency regulation using data-driven controllers in powergrids with variable inertia due to renewable energy,” in . IEEE, 2019, pp. 1–5.[9] E. Hazan, S. Kakade, and K. Singh, “The nonstochastic control prob-lem,” in
Algorithmic Learning Theory . PMLR, 2020, pp. 408–421.[10] B. Xu, Y. Dvorkin, D. S. Kirschen, C. A. Silva-Monroy, and J.-P. Watson,“A comparison of policies on the participation of storage in us frequencyregulation markets,” in . IEEE, 2016, pp. 1–5.[11] K. Doenges, I. Egido, L. Sigrist, E. Lobato, and L. Rouco, “Improvingagc performance in power systems with regulation response accuracymargins using battery energy storage system (bess),”
IEEE Transactionson Power Systems , 2019.[12] Q. Liu and M. D. Ili´c, “Enhanced automatic generation control (e-agc)for future electric energy systems,” in . IEEE, 2012, pp. 1–8.[13] A. Zheng, M. V. Kothare, and M. Morari, “Anti-windup design forinternal model control,”
International Journal of Control , vol. 60, no. 5,pp. 1015–1024, 1994.[14] W. Tan, S. Chang, and R. Zhou, “Load frequency control of power sys-tems with non-linearities,”