Roman Zajdel

Application of Reinforcement Learning Algorithms for the Adaptive Computation of the Smoothing Parameter for Probabilistic Neural Network

In this paper, we propose new methods for the choice and adaptation of the smoothing parameter of the probabilistic neural network (PNN). These methods are based on three reinforcement learning algorithms: Q(0)-learning, Q(λ)-learning, and stateless Q-learning. We regard three types of PNN classifiers: the model that uses single smoothing parameter for the whole network, the model that utilizes single smoothing parameter for each data attribute, and the model that possesses the matrix of smoothing parameters different for each data variable and data class. Reinforcement learning is applied as the method of finding such a value of the smoothing parameter, which ensures the maximization of the prediction ability. PNN models with smoothing parameters computed according to the proposed algorithms are tested on eight databases by calculating the test error with the use of the cross validation procedure. The results are compared with state-of-the-art methods for PNN training published in the literature up to date and, additionally, with PNN whose sigma is determined by means of the conjugate gradient approach. The results demonstrate that the proposed approaches can be used as alternative PNN training procedures.

Probabilistic neural network training procedure based on Q(0)-learning algorithm in medical data classification

In this article, an iterative procedure is proposed for the training process of the probabilistic neural network (PNN). In each stage of this procedure, the Q(0)-learning algorithm is utilized for the adaptation of PNN smoothing parameter (σ). Four classes of PNN models are regarded in this study. In the case of the first, simplest model, the smoothing parameter takes the form of a scalar; for the second model, σ is a vector whose elements are computed with respect to the class index; the third considered model has the smoothing parameter vector for which all components are determined depending on each input attribute; finally, the last and the most complex of the analyzed networks, uses the matrix of smoothing parameters where each element is dependent on both class and input feature index. The main idea of the presented approach is based on the appropriate update of the smoothing parameter values according to the Q(0)-learning algorithm. The proposed procedure is verified on six repository data sets. The prediction ability of the algorithm is assessed by computing the test accuracy on 10 %, 20 %, 30 %, and 40 % of examples drawn randomly from each input data set. The results are compared with the test accuracy obtained by PNN trained using the conjugate gradient procedure, support vector machine algorithm, gene expression programming classifier, k–Means method, multilayer perceptron, radial basis function neural network and learning vector quantization neural network. It is shown that the presented procedure can be applied to the automatic adaptation of the smoothing parameter of each of the considered PNN models and that this is an alternative training method. PNN trained by the Q(0)-learning based approach constitutes a classifier which can be treated as one of the top models in data classification problems.

Epoch-incremental reinforcement learning algorithms

Abstract In this article, a new class of the epoch-incremental reinforcement learning algorithm is proposed. In the incremental mode, the fundamental TD(0) or TD(λ) algorithm is performed and an environment model is created. In the epoch mode, on the basis of the environment model, the distances of past-active states to the terminal state are computed. These distances and the reinforcement terminal state signal are used to improve the agent policy.

Epoch-Incremental Queue-Dyna Algorithm

The basic reinforcement learning algorithm, as Q-learning, is characterized by short time-consuming single learning step, however, the number of epochs necessary to achieve the optimal policy is not satisfactory. There are many methods that reduce the number of necessary epochs, like TD(i¾?> 0), Dyna or prioritized sweeping, but their learning time is considerable. This paper proposes a combination of Q-learning algorithm performed in incremental mode with executed in epoch mode method of acceleration based on environment model and distance to terminal state. This approach ensures the maintenance of short time of a single learning step and high efficiency comparable with Dyna or prioritized sweeping. Proposed algorithm is compared with Q(i¾?)-learning, Dyna-Q and prioritized sweeping in the experiments on three maze tasks. The time-consuming learning process and number of epochs necessary to reach the terminal state is used to evaluate the efficiency of compared algorithms.

Fuzzy Q(λ)-learning algorithm

The adaptation of temporal differences method TD(λ>0) to reinforcement learning algorithms with fuzzy approximation of action-value function is proposed. Eligibility traces are updated using the normalized degree of activation of fuzzy rules. The two types of fuzzy reinforcement learning algorithm are formulated: with discrete and with continuous action values. These new algorithms are practically tested in the control of two typical models of continuous object, like ball-beam and cart-pole system. The achievement results are compared with two popular reinforcement learning algorithms with CMAC and table approximation of action-value function.

Stateless Q-Learning Algorithm for Training of Radial Basis Function Based Neural Networks in Medical Data Classification

In this article, the stateless Q-learning algorithm is used for the training process of two radial basis function based models: the radial basis function neural network (RBFNN) and the probabilistic neural network (PNN). The training process of considered models consists in the initialization and the adaptation of the smoothing parameter of the networks’ activation function in a hidden layer. The main idea of this approach is based on the appropriate computation of the smoothing parameter which relies on its update according to the stateless Q-learning algorithm. The proposed method is tested on six commonly available repository data sets. The prediction ability of the algorithm is assessed by computing the test set error on 10%, 20%, 30%, and 40% of examples drawn randomly from the entire input data. Obtained results are compared with the test errors achieved by PNN trained by means of the conjugate gradient procedure. It is shown that Q-learning method can be applied to the automatic adaptation of the smoothing parameter for both neural networks and provides better prediction ability results.

Fuzzy epoch-incremental reinforcement learning algorithm

The new epoch-incremental reinforcement learning algorithm with fuzzy approximation of action-value function is developed. This algorithm is practically tested in the control of the mobile robot which realizes goal seeking behavior. The obtained results are compared with results of fuzzy version of reinforcement learning algorithms, such as Q(0)-learning, Q(λ )-learning, Dyna-learning and prioritized sweeping. The adaptation of the fuzzy approximator to the model based reinforcement learning algorithms is also proposed.

Application of Reinforcement Learning to Stacked Autoencoder Deep Network Architecture Optimization

In this work, a new algorithm for the structure optimization of stacked autoencoder deep network (SADN) is introduced. It relies on the search for the numbers of the neurons in the first and the second layer of SADN through an approach based on reinforcement learning (RL). The Q(0)-learning based agent is constructed, which according to received reinforcement signal, picks appropriate values for the neurons. Considered network, with the architecture adjusted by the proposed algorithm, is applied to the task of MNIST digit database recognition. The classification quality is computed for SADN to determine its performance. It is shown that, using the proposed algorithm, the semi-optimal configuration of the number of hidden neurons can be achieved much faster than the successive exploration of the entire space of layers’ arrangement.

Epoch-incremental Dyna-learning and prioritized sweeping algorithms

Abstract Dyna-learning and prioritized sweeping (PS in short) are the most commonly used reinforcement learning algorithms which use the model of the environment. In this paper, the modified versions of these algorithms are presented. The modification exploits the breadth-first search (BFS) to conduct additional modifications of the policy in the epoch mode. The experiments, which are performed in the dynamic grid world and in the ball-beam system, showed that the proposed modifications improved the efficiency of the reinforcement learning algorithms.

Probabilistic Neural Network Training Procedure with the Use of SARSA Algorithm

In this paper, we present new probabilistic neural network (PNN) training procedure for classification problems. Proposed procedure utilizes the State-Action-Reward-State-Action algorithm (SARSA in short), which is the implementation of the reinforcement learning method. This algorithm is applied to the adaptive selection and computation of the smoothing parameter of the PNN model. PNNs with different forms of the smoothing parameter are regarded. The prediction ability for all the models is assessed by computing the test error with the use of a 10-fold cross validation (CV) procedure. The obtained results are compared with state-of-the-art methods for PNN training.