Orlando De Jesus

Backpropagation Algorithms for a Broad Class of Dynamic Networks

This paper introduces a general framework for describing dynamic neural networks-the layered digital dynamic network (LDDN). This framework allows the development of two general algorithms for computing the gradients and Jacobians for these dynamic networks: backpropagation-through-time (BPTT) and real-time recurrent learning (RTRL). The structure of the LDDN framework enables an efficient implementation of both algorithms for arbitrary dynamic networks. This paper demonstrates that the BPTT algorithm is more efficient for gradient calculations, but the RTRL algorithm is more efficient for Jacobian calculations

Training multi-loop networks

In this paper we investigate the training of time-lagged recurrent networks having multiple feedback paths and tapped-delay inputs. Network structures of this type are useful in approximating nonlinear dynamical systems. The introduction of additional feedback loops into a network structure may improve the modeling capability of the network, but a significant price can be paid in complexity and computational burden when calculating the dynamic derivatives needed for training. The focus of this paper is on the calculation of the dynamic derivatives which must be determined or approximated in order to use any of the popular methods employed in training neural networks. In this paper we illustrate the effect of multiple feedback loops on the formulation of the equations needed for calculating the dynamic derivatives. We also investigate the effect on network performance and computational complexity when various dynamic derivative approximations are used in training multiple feedback loop networks.