Paul J. Werbos

Stable adaptive control using critic designs

Classical adaptive control proves total-system stability for control of linear plants, but only for plants meeting very restrictive assumptions. Approximate Dynamic Programming (ADP) has the potential, in principle, to ensure stability without such tight restrictions. It also offers nonlinear and neural extensions for optimal control, with empirically supported links to what is seen in the brain. However, the relevant ADP methods in use today--TD, HDP, DHP, GDHP--and the Galerkin-based versions of these all have serious limitations when used here as parallel distributed real-time learning systems; either they do not possess quadratic unconditional stability (to be defined) or they lead to incorrect results in the stochastic case. (ADAC or Q- learning designs do not help.) After explaining these conclusions, this paper describes new ADP designs which overcome these limitations. It also addresses the Generalized Moving Target problem, a common family of static optimization problems, and describes a way to stabilize large-scale economic equilibrium models, such as the old long-term energy mode of DOE.

Consistency of HDP applied to a simple reinforcement learning problem

Abstract In “reinforcement learning over time,” a system of neural networks learns to control a system of motors or muscles so as to maximize some measure of performance or reinforcement in the future. Two architectures or designs are now widely used to address this problem in an engineering context: backpropagation through time and the adaptive critic family. This article begins with a brief review of these and other neurocontrol methods and their applications. Then it addresses the issue of consistency in using Heuristic Dynamic Programming (HDP), a procedure for adapting a “critic” neural network, closely related to Suttons method of temporal differences. In a multivariate linear environment, HDP converges to the correct system of weights. However, a variant of HDP—which appeals to common sense and which uses backpropagation with a complete gradient—leads to the wrong weights almost always. Similar consistency tests may be useful in evaluating architectures for neural nets to identify or emulate dynamic systems.

Biocomplexity: adaptive behavior in complex stochastic dynamical systems.

Existing methods of complexity research are capable of describing certain specifics of bio systems over a given narrow range of parameters but often they cannot account for the initial emergence of complex biological systems, their evolution, state changes and sometimes-abrupt state transitions. Chaos tools have the potential of reaching to the essential driving mechanisms that organize matter into living substances. Our basic thesis is that while established chaos tools are useful in describing complexity in physical systems, they lack the power of grasping the essence of the complexity of life. This thesis illustrates sensory perception of vertebrates and the operation of the vertebrate brain. The study of complexity, at the level of biological systems, cannot be completed by the analytical tools, which have been developed for non-living systems. We propose a new approach to chaos research that has the potential of characterizing biological complexity. Our study is biologically motivated and solidly based in the biodynamics of higher brain function. Our biocomplexity model has the following features, (1) it is high-dimensional, but the dimensionality is not rigid, rather it changes dynamically; (2) it is not autonomous and continuously interacts and communicates with individual environments that are selected by the model from the infinitely complex world; (3) as a result, it is adaptive and modifies its internal organization in response to environmental factors by changing them to meet its own goals; (4) it is a distributed object that evolves both in space and time towards goals that is continually re-shaping in the light of cumulative experience stored in memory; (5) it is driven and stabilized by noise of internal origin through self-organizing dynamics. The resulting theory of stochastic dynamical systems is a mathematical field at the interface of dynamical system theory and stochastic differential equations. This paper outlines several possible avenues to analyze these systems. Of special interest are input-induced and noise-generated, or spontaneous state-transitions and related stability issues.

Neurocontrol and fuzzy logic: connections and designs

Abstract Artificial neural networks (ANNs) and fuzzy logic are complementary technologies. ANNs extract information from systems to be learned or controlled, while fuzzy techniques most often use verbal information from experts. Ideally, the two sources of information should be combined. For example, one can learn rules in a hybrid fashion and then calibrate them for better whole-system performance. ANNs offer universal approximation theorems, pedagogical advantages, very high throughput hardware, and links to neurophysiology. Neurocontrol — the use of ANNs to directly control motors, actuators, etc. — uses five generalized designs, related to control theory, that can work on fuzzy logic systems as well as ANNs. These designs can copy what experts do instead of what they say, learn to track trajectories, generalize adaptive control, and maximize performance or minimize cost over time, even in noisy environments. Design trade-offs and future directions are discussed throughout. The final section mentions a few new ideas regarding reasoning, planning, and chunking, with biological parallels.

Beyond Feedforward Models Trained by Backpropagation: A Practical Training Tool for a More Efficient Universal Approximator

Cellular simultaneous recurrent neural network (SRN) has been shown to be a function approximator more powerful than the multilayer perceptron (MLP). This means that the complexity of MLP would be prohibitively large for some problems while SRN could realize the desired mapping with acceptable computational constraints. The speed of training of complex recurrent networks is crucial to their successful application. This work improves the previous results by training the network with extended Kalman filter (EKF). We implemented a generic cellular SRN (CSRN) and applied it for solving two challenging problems: 2-D maze navigation and a subset of the connectedness problem. The speed of convergence has been improved by several orders of magnitude in comparison with the earlier results in the case of maze navigation, and superior generalization has been demonstrated in the case of connectedness. The implications of this improvements are discussed.

Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities

Backwards calculation of derivatives – sometimes called the reverse mode, the full adjoint method, or backpropagation, has been developed and applied in many fields. This paper reviews several strands of history, advanced capabilities and types of application – particularly those which are crucial to the development of brain-like capabilities in intelligent control and artificial intelligence.

Putting more brain-like intelligence into the electric power grid: What we need and how to do it

Certain key features of brain-like intelligence are essential to fulfill the main goals of policy-makers and environmentalists for the “smart grid” - a key item in the new economic stimulus law, and a key item in a rational strategy for energy sustainability. This paper will explain why and how, and how the neural network community could play a crucial role in making this real.

Complete stability analysis of a heuristic approximate dynamic programming control design

This paper provides new stability results for Action-Dependent Heuristic Dynamic Programming (ADHDP), using a control algorithm that iteratively improves an internal model of the external world in the autonomous system based on its continuous interaction with the environment. We extend previous results by ADHDP control to the case of general multi-layer neural networks with deep learning across all layers. In particular, we show that the introduced control approach is uniformly ultimately bounded (UUB) under specific conditions on the learning rates, without explicit constraints on the temporal discount factor. We demonstrate the benefit of our results to the control of linear and nonlinear systems, including the cart-pole balancing problem. Our results show significantly improved learning and control performance as compared to the state-of-art.

Bell’s Theorem, Many Worlds and Backwards-Time Physics: Not Just a Matter of Interpretation

The classic “Bell’s Theorem” of Clauser, Holt, Shimony and Horne tells us that we must give up at least one of: (1) objective reality (aka “hidden variables”); (2) locality; or (3) time-forwards macroscopic statistics (aka “causality”). The orthodox Copenhagen version of physics gives up the first. The many-worlds theory of Everett and Wheeler gives up the second. The backwards-time theory of physics (BTP) gives up the third. Contrary to conventional wisdom, empirical evidence strongly favors Everett-Wheeler over orthodox Copenhagen. BTP allows two major variations—a many-worlds version and a neoclassical version based on Partial Differential Equations (PDE), in the spirit of Einstein. Section 2 of this paper discusses the origins of quantum measurement according to BTP, focusing on the issue of how we represent condensed matter objects like polarizers in a model “Bell’s Theorem” experiment. The backwards time telegraph (BTT) is not ruled out in BTP, but is highly speculative for now, as will be discussed.

Supervised Learning: Can it Escape its Local Minimum?

Tremendous progress and new research horizons have opened up in developing artificial neural network (ANN) systems which use supervised learning as an elementary building block in performing more brain-like, generic tasks such as control or planning or forecasting across time [1, 2]. However, many people believe that research into supervised learning itself is stuck in a kind of local minimum or plateau. Limited capabilities in supervised learning have created a serious dilemma, forcing us to choose between fast real-time learning versus good generalization with many inputs, without allowing us to have both together.