Wentao Guo

2012 Special Issue: A boundedness result for the direct heuristic dynamic programming

Approximate/adaptive dynamic programming (ADP) has been studied extensively in recent years for its potential scalability to solve large state and control space problems, including those involving continuous states and continuous controls. The applicability of ADP algorithms, especially the adaptive critic designs has been demonstrated in several case studies. Direct heuristic dynamic programming (direct HDP) is one of the ADP algorithms inspired by the adaptive critic designs. It has been shown applicable to industrial scale, realistic and complex control problems. In this paper, we provide a uniformly ultimately boundedness (UUB) result for the direct HDP learning controller under mild and intuitive conditions. By using a Lyapunov approach we show that the estimation errors of the learning parameters or the weights in the action and critic networks remain UUB. This result provides a useful controller convergence guarantee for the first time for the direct HDP design.

Approximate dynamic programming based supplementary reactive power control for DFIG wind farm to enhance power system stability

Reactive power control of doubly fed induction generators (DFIGs) has been a heated topic in transient stability control of power systems in recent years. By using a new online supplementary learning control (OSLC) approach based on the theory of approximate dynamic programming (ADP), this paper develops an optimal and adaptive design method for the supplementary reactive power control of DFIGs to improve transient stability of power systems. To augment the reactive power command of the rotor-side converter (RSC), a supplementary controller is designed to reduce voltage sag at the common coupling point during a fault, and to mitigate active power oscillation of the wind farm after a fault. As a result, the transient stability of both DFIGs and the power system is enhanced. For the supplementary controller design, an action dependent cost function is introduced to make the OSLC model-free and completely data-driven. Furthermore, a least-squares based policy iteration algorithm is employed to train the supplementary controller with convergence and stability guarantee. By using such techniques, the supplementary reactive power controller can be trained directly from data measurements, and therefore, can adapt to system or external changes without an explicit offline system identification process. Simulations carried out in Power System Computer Aided Design/ Electro Magnetic Transient in DC System (PSCAD/EMTDC) show that the OSLC based supplementary reactive power controller can significantly improve the transient performance of the wind farm and enhance the transient stability of the power system after sever faults.

Online Supplementary ADP Learning Controller Design and Application to Power System Frequency Control With Large-Scale Wind Energy Integration

The emergence of smart grids has posed great challenges to traditional power system control given the multitude of new risk factors. This paper proposes an online supplementary learning controller (OSLC) design method to compensate the traditional power system controllers for coping with the dynamic power grid. The proposed OSLC is a supplementary controller based on approximate dynamic programming, which works alongside an existing power system controller. By introducing an action-dependent cost function as the optimization objective, the proposed OSLC is a nonidentifier-based method to provide an online optimal control adaptively as measurement data become available. The online learning of the OSLC enjoys the policy-search efficiency during policy iteration and the data efficiency of the least squares method. For the proposed OSLC, the stability of the controlled system during learning, the monotonic nature of the performance measure of the iterative supplementary controller, and the convergence of the iterative supplementary controller are proved. Furthermore, the efficacy of the proposed OSLC is demonstrated in a challenging power system frequency control problem in the presence of high penetration of wind generation.

Incorporating approximate dynamic programming-based parameter tuning into PD-type virtual inertia control of DFIGs

Doubly fed induction generators (DFIGs) are widely used in wind power generation. For controlling DFIGs to maintain network frequency within a safety range, the proportional-derivative (PD) type virtual inertia controllers (VIC) are used in the active power control of DFIGs. However, as is well known, wind power generation conditions change directly with wind conditions in nature. Such changes create great challenge for the VIC design and actually force the control designs to go beyond the traditional problem formulation of using explicit objective functions associated with specific optimality. Controller parameter tuning thus necessarily becomes a part of the controller design. In this paper, we propose an approximate dynamic programming (ADP) structure for online tuning of the PD type virtual inertia controller parameters. The proposed ADP structure naturally takes into account the PD control into design objective and provides the PD controller with online parameter tuning capability through learning. Design and implementation details of the proposed methodology, including neural network weight initialization, design of the reinforcement signal, data preprocessing, and a bound on the online tuned parameters are discussed in this paper. Simulation studies carried out on the Power System Computer Aided Design/ Electro Magnetic Transient in DC System (PSCAD/EMTDC) software are used to demonstrate the effectiveness and efficiency of the proposed ADP-based online VIC parameter tuning methodology.

Approximate dynamic programming based supplementary frequency control of thermal generators in power systems with large-scale renewable generation integration

Intermittent electricity generation from renewable sources is characterized by a wide range of fluctuations in frequency spectrum. The medium-frequency component of 0.01 Hz-1 Hz cannot be filtered out by system inertia and automatic generation control (AGC) and thus it results in deterioration of frequency quality. In this paper, an approximate dynamic programming (ADP) based supplementary frequency controller for thermal generators is developed to attenuate renewable generation fluctuation in medium-frequency range. A policy iteration based training algorithm is employed for online and model-free learning. Our simulation results demonstrate that the proposed supplementary frequency controller can effectively adapt to changes in the system and provide improved frequency control. Further sensitivity analysis validates that the supplementary frequency controller significantly attenuates the dependence of frequency deviation on the medium-frequency component of renewable generation fluctuation.

Reactive power control of DFIG wind farm using online supplementary learning controller based on approximate dynamic programming

Dynamic reactive power control of doubly fed induction generators (DFIGs) plays a crucially important role in maintaining transient stability of power systems with high penetration of DFIG based wind generation. Based on approximate dynamic programming (ADP), this paper proposes an optimal adaptive supplementary reactive power controller for DFIGs. By augmenting a corrective regulation signal to the reactive power command of rotor-side converter (RSC) of a DFIG, the supplementary controller is designed to reduce voltage sag at the point of common connection (PCC) during a fault, and to mitigate output active power oscillation of the wind farm after a fault. As a result, the transient stability of both DFIG and the power grid is enhanced. An action dependent cost function is introduced to provide real-time online ADP learning control. Furthermore, a policy iteration algorithm using high-efficiency least square method is employed to train the supplementary controller in an online model-free manner. By using such techniques, the supplementary reactive power controller is endowed with capability of online optimization and adaptation. Simulations carried out on a benchmark power system integrating a large DFIG wind farm show that the ADP based supplementary reactive power controller can significantly improve the transient system stability in changing operation conditions.

Policy iteration approximate dynamic programming using Volterra series based actor

There is an extensive literature on value function approximation for approximate dynamic programming (ADP). Multilayer perceptrons (MLPs) and radial basis functions (RBFs), among others, are typical approximators for value functions in ADP. Similar approaches have been taken for policy approximation. In this paper, we propose a new Volterra series based structure for actor approximation in ADP. The Volterra approx-imator is linear in parameters with global optima attainable. Given the proposed approximator structures, we further develop a policy iteration framework under which a gradient descent training algorithm for obtaining the optimal Volterra kernels can be obtained. Associated with this ADP design, we provide a sufficient condition based on actor approximation error to guarantee convergence of the value function iterations. A finite bound of the final convergent value function is also given. Finally, by using a simulation example we illustrate the effectiveness of the proposed Volterra actor for optimal control of a nonlinear system.

Policy Approximation in Policy Iteration Approximate Dynamic Programming for Discrete-Time Nonlinear Systems

Policy iteration approximate dynamic programming (DP) is an important algorithm for solving optimal decision and control problems. In this paper, we focus on the problem associated with policy approximation in policy iteration approximate DP for discrete-time nonlinear systems using infinite-horizon undiscounted value functions. Taking policy approximation error into account, we demonstrate asymptotic stability of the control policy under our problem setting, show boundedness of the value function during each policy iteration step, and introduce a new sufficient condition for the value function to converge to a bounded neighborhood of the optimal value function. Aiming for practical implementation of an approximate policy, we consider using Volterra series, which has been extensively covered in controls literature for its good theoretical properties and for its success in practical applications. We illustrate the effectiveness of the main ideas developed in this paper using several examples including a practical problem of excitation control of a hydrogenerator.

Online adaptation of controller parameters based on approximate dynamic programming

Controller parameter tuning is an integral part of control engineering practice. Existing tuning methods usually start with an accurate mathematical model of the controlled system, which may pose some challenges for practicing engineers dealing with real systems. As such, parameter optimization and adaptation are treated as two independent steps during tuning. To address these issues, we propose a new, online parameterized controller tuning method for a general nonlinear dynamic system. This tuning method is based on direct heuristic dynamic programming (direct HDP), a model-free algorithm in the approximated dynamic programming (ADP) family. By using a Lyapunov stability approach, we provide uniformly ultimately bounded (UUB) results under some mild conditions for controller parameters, the critic neural network weights, and the action neural network weights. Simulation studies based on the benchmark cart-pole system demonstrate adaptability and optimization capabilities of the proposed controller parameter tuning method.

Online and model-free supplementary learning control based on approximate dynamic programming

An approximate dynamic programming (ADP) based supplementary learning control method is developed to online improve the performance of existing controllers. The proposed supplementary learning structure can make full use of the prior knowledge of the pre-designed controller and endow the controller with learning ability. Moreover, by introducing the action dependent value function for policy evaluation, the supplementary learning control can work in a model-free manner. The policy iteration algorithm is employed to train the actor-critic structure of the ADP supplementary controller. Simulation studies are carried out on the cart-pole system to validate the optimization and the adaptation capability of the proposed methodology.