Andrew D. Back

Locally recurrent globally feedforward networks: a critical review of architectures

In this paper, we will consider a number of local-recurrent-global-feedforward (LRGF) networks that have been introduced by a number of research groups in the past few years. We first analyze the various architectures, with a view to highlighting their differences. Then we introduce a general LRGF network structure that includes most of the network architectures that have been proposed to date. Finally we will indicate some open issues concerning these types of networks.

Fir and iir synapses, a new neural network architecture for time series modeling

A new neural network architecture involving either local feedforward global feedforward, and/or local recurrent global feedforward structure is proposed. A learning rule minimizing a mean square error criterion is derived. The performance of this algorithm (local recurrent global feedforward architecture) is compared with a local-feedforward global-feedforward architecture. It is shown that the local-recurrent global-feedforward model performs better than the local-feedforward global-feedforward model.

A simplified gradient algorithm for iir synapse multilayer perceptrons

A network architecture with a global feedforward local recurrent construction was presented recently as a new means of modeling nonlinear dynamic time series (Back and Tsoi 1991a). The training rule used was based on minimizing the least mean square (LMS) error and performed well, although the amount of memory required for large networks may become significant if a large number of feedback connections are used. In this note, a modified training algorithm based on a technique for linear filters is presented, simplifying the gradient calculations significantly. The memory requirements are reduced from O[na(na + nb)Ns] to O[(2na + nb)Ns], where na is the number of feedback delays, and Ns is the total number of synapses. The new algorithm reduces the number of multiply-adds needed to train each synapse by na at each time step. Simulations indicate that the algorithm has almost identical performance to the previous one.

A unifying view of some training algorithms for multilayer perceptrons with FIR filter synapses

Concerns neural network architectures for modelling time-dependent signals. A number of algorithms have been published for multilayer perceptrons with synapses described by finite impulse response (FIR) and infinite impulse response (IIR) filters (the latter case is also known as locally recurrent globally feedforward networks). The derivations of these algorithms have used different approaches in calculating the gradients, and in this paper we present a short, but unifying account of how these different algorithms compare for the FIR case, both in derivation, and performance. A new algorithm is subsequently presented. In this paper, results are compared for the Mackey-Glass chaotic time series (1977) against a number of other methods including a standard multilayer perceptron, and a local approximation method.<<ETX>>

An adaptive lattice architecture for dynamic multilayer perceptrons

Time-series modeling is a topic of growing interest in neural network research. Various methods have been proposed for extending the nonlinear approximation capabilities to time-series modeling problems. A multilayer perceptron (MLP) with a global-feedforward local-recurrent structure was recently introduced as a new approach to modeling dynamic systems. The network uses adaptive infinite impulse response (IIR) synapses (it is thus termed an IIR MLP), and was shown to have good modeling performance. One problem with linear IIR filters is that the rate of convergence depends on the covariance matrix of the input data. This extends to the IIR MLP: it learns well for white input signals, but converges more slowly with nonwhite inputs. To solve this problem, the adaptive lattice multilayer perceptron (AL MLP), is introduced. The network structure performs Gram-Schmidt orthogonalization on the input data to each synapse. The method is based on the same principles as the Gram-Schmidt neural net proposed by Orfanidis (1990b), but instead of using a network layer for the orthogonalization, each synapse comprises an adaptive lattice filter. A learning algorithm is derived for the network that minimizes a mean square error criterion. Simulations are presented to show that the network architecture significantly improves the learning rate when correlated input signals are present.

Static and dynamic preprocessing methods in neural networks

Preprocessing is recognized as an important tool in modeling, particularly when the data or underlying physical process involves complex nonlinear dynamical interactions. This paper will give a review of preprocessing methods used in linear and nonlinear models. The problem of static preprocessing will be considered first, where no dependence on time between the input vectors is assumed. Then, dynamic preprocessing methods which involve the modification of time-dependent input values before they are used in the linear or nonlinear models will be considered. Furthermore, the problem of an insufficient number of input vectors is considered. It is shown that one way in which this problem can be overcome is by expanding the weight vector in terms of the available input vectors. Finally, a new problem which involves both cases of: (1) transformation of input vectors; and (2) insufficient number of input vectors is considered. It is shown how a combination of the techniques used to solve the individual problems can be combined to solve this composite problem. Some open issues in this type of preprocessing methods are discussed.

Static and Dynamic Preprocessing Methods in Neural Networks

Abstract Preprocessing is recognized as an important tool in modelling, particularly when the data or underlying physical process involves complex nonlinear dynamical interactions. In this paper, we will give a review of preprocessing methods used in linear and nonlinear models. We will consider first the problem of static preprocessing, where no dependence on time between the input vector elements is assumed. We then consider dynamic preprocessing methods which involve the modification of time dependent input values before they are used in the linear or nonlinear models. Furthermore the problem of insufficient number of input vectors is considered. It is shown that one way in which this problem can be overcome is by expanding the weight vector in terms of the available input vectors. Finally, a new problem which involves both cases of (1) transformation of input vectors, and (2) insufficient number of input vectors is considered. It is shown a combination of the techniques used to solve the individual problems can be combined to solve this composite problem. Some open issues in this type of preprocessing methods are discussed.

Internal representation of data in multilayer perceptrons with IIR synapses

Multilayer perceptrons having a dynamic synaptic structure, and the way in which data are represented in such networks are addressed. A decomposition principle that provides a clear interpretation of this type of network in terms of static and dynamic components is introduced. On the basis of this result a reduced complexity network is described.<<ETX>>

Constrained pole-zero filters as discrete-time operators for system approximation

Discrete-time models whether linear or nonlinear, often implicitly use the shift operator to obtain input regression vectors. It has been shown previously that the significantly better performance can be obtained in terms of coefficient sensitivity and output error by using alternative operators to the usual shift operator. These include the delta and gamma operators. In this paper the authors introduce second order pole-zero operators which have more general modelling properties than those previously considered. The authors provide some observations on the behaviour of the operators, considering representational issues and convergence characteristics in particular.

PauseCode: Computational Conversation Timing Analysis

Pauses play a critical role in adding, shifting or contradicting meaning in a conversation. To enable the study and incorporation of this important modality in computational discourse analytic and processing systems, we require extensible open source pause coding systems and associated software libraries. We designed and implemented a coding and visualisation system for pause and overlap detection and analysis, extending existing voicing and silence detection algorithms. Demonstrating the system using the TalkBank CallFriend and CallHome corpora we show how the approach can be used to code many different kinds of pauses and overlaps within and between interlocutors, and calculate the temporal distribution of these different types of pause and overlap. The coding schema is intended to be combined with other speech modalities to provide novel approaches to predicting social cues and markers, useful for designing more naturalistic conversational agents, and in new tools for measuring turn-taking structure of conversation in greater depth and accuracy.