Hideki Satoh

Approximation and Analysis of Non-linear Equations in a Moment Vector Space

Moment vector equations (MVEs) are presented for use in approximating and analyzing multi-dimensional non-linear discrete- and continuous-time equations. A non-linear equation is expanded into simultaneous equations of generalized moments and then reduced to an MVE of a coefficient matrix and a moment vector. The MVE can be used to analyze the statistical properties, such as the mean, variance, covariance, and power spectrum, of the non-linear equation. Moreover, we can approximately express a combination of non-linear equations by using a combination of MVEs of the equations. Evaluation of the statistical properties of Lorenz equations and of a combination of logistic equations based on the MVE approach showed that MVEs can be used to approximate non-linear equations in statistical measurements.

A State Space Compression Method Based on Multivariate Analysis for Reinforcement Learning in High-Dimensional Continuous State Spaces

A state space compression method based on multivariate analysis was developed and applied to reinforcement learning for high-dimensional continuous state spaces. First, useful components in the state variables of an environment are extracted and meaningless ones are removed by using multiple regression analysis. Next, the state space of the environment is compressed by using principal component analysis so that only a few principal components can express the dynamics of the environment. Then, a basis of a feature space for function approximation is constructed based on orthonormal bases of the important principal components. A feature space is thus autonomously construct without preliminary knowledge of the environment, and the environment is effectively expressed in the feature space. An example synchronization problem for multiple logistic maps was solved using this method, demonstrating that it solves the curse of dimensionality and exhibits high performance without suffering from disturbance states.

A Nonlinear Approach to Robust Routing Based on Reinforcement Learning with State Space Compression and Adaptive Basis Construction

A robust routing algorithm was developed based on reinforcement learning that uses (1) reward-weighted principal component analysis, which compresses the state space of a network with a large number of nodes and eliminates the adverse effects of various types of attacks or disturbance noises, (2) activity-oriented index allocation, which adaptively constructs a basis that is used for approximating routing probabilities, and (3) newly developed control space compression based on a potential model that reduces the control space for routing probabilities. This algorithm takes all the network states into account and reduces the adverse effects of disturbance noises. The algorithm thus works well, and the frequencies of causing routing loops and falling to a local optimum are reduced even if the routing information is disturbed.

Reinforcement Learning for Continuous Stochastic Actions---An Approximation of Probability Density Function by Orthogonal Wave Function Expansion---

A function approximation based on an orthonormal wave function expansion in a complex space is derived. Although a probability density function (PDF) cannot always be expanded in an orthogonal series in a real space because a PDF is a positive real function, the function approximation can approximate an arbitrary PDF with high accuracy. It is applied to an actor-critic method of reinforcement learning to derive an optimal policy expressed by an arbitrary PDF in a continuous-action continuous-state environment. A chaos control problem and a PDF approximation problem are solved using the actor-critic method with the function approximation, and it is shown that the function approximation can approximate a PDF well and that the actor-critic method with the function approximation exhibits high performance.

Reinforcement Learning with Orthonormal Basis Adaptation Based on Activity-Oriented Index Allocation

An orthonormal basis adaptation method for function approximation was developed and applied to reinforcement learning with multi-dimensional continuous state space. First, a basis used for linear function approximation of a control function is set to an orthonormal basis. Next, basis elements with small activities are replaced with other candidate elements as learning progresses. As this replacement is repeated, the number of basis elements with large activities increases. Example chaos control problems for multiple logistic maps were solved, demonstrating that the method for adapting an orthonormal basis can modify a basis while holding the orthonormality in accordance with changes in the environment to improve the performance of reinforcement learning and to eliminate the adverse effects of redundant noisy states.