Robert Hecht-Nielsen

On the geometry of feedforward neural network error surfaces

Many feedforward neural network architectures have the property that their overall input-output function is unchanged by certain weight permutations and sign flips. In this paper, the geometric structure of these equioutput weight space transformations is explored for the case of multilayer perceptron networks with tanh activation functions (similar results hold for many other types of neural networks). It is shown that these transformations form an algebraic group isomorphic to a direct product of Weyl groups. Results concerning the root spaces of the Lie algebras associated with these Weyl groups are then used to derive sets of simple equations for minimal sufficient search sets in weight space. These sets, which take the geometric forms of a wedge and a cone, occupy only a minute fraction of the volume of weight space. A separate analysis shows that large numbers of copies of a network performance function optimum weight vector are created by the action of the equioutput transformation group and that these copies all lie on the same sphere. Some implications of these results for learning are discussed.

A Biologically Motivated Solution to the Cocktail Party Problem

We present a new approach to the cocktail party problem that uses a cortronic artificial neural network architecture (Hecht-Nielsen, 1998) as the front end of a speech processing system. Our approach is novel in three important respects. First, our method assumes and exploits detailed knowledge of the signals we wish to attend to in the cocktail party environment. Second, our goal is to provide preprocessing in advance of a pattern recognition system rather than to separate one or more of the mixed sources explicitly. Third, the neural network model we employ is more biologically feasible than are most other approaches to the cocktail party problem. Although the focus here is on the cocktail party problem, the method presented in this study can be applied to other areas of information processing.

Conditional probability density function estimation with sigmoidal neural networks

Real-world problems can often be couched in terms of conditional probability density function estimation. In particular, pattern recognition, signal detection, and financial prediction are among the multitude of applications requiring conditional density estimation. Previous developments in this direction have used neural nets to estimate statistics of the distribution or the marginal or joint distributions of the input-output variables. We have modified the joint distribution estimating sigmoidal neural network to estimate the conditional distribution. Thus, the probability density of the output conditioned on the inputs is estimated using a neural network. We have derived and implemented the learning laws to train the network. We show that this network has computational advantages over a brute force ratio of joint and marginal distributions. We also compare its performance to a kernel conditional density estimator in a larger scale (higher dimensional) problem simulating more realistic conditions.

The Mechanism of Thought

A fast winners-take-all competition process, termed confabulation [1], [2], is proposed as the fundamental mechanism of all aspects of cognition (vision, hearing, planning, language, initiation of thought and movement, etc.). Multiple, contemporaneous, mutually interacting confabulations – in which millions of items of relevant knowledge are applied in parallel – are typically employed in thinking. At the beginning of such a multiconfabulation, billions of distinct, potentially viable, conclusion sets are considered. At the end, only one remains. This fast, massively parallel application of relevant knowledge (an alien kind of information processing with no analogue in todays computational intelligence, computational neurobiology, or computer science) is hypothesized to be the core explanation for the information processing effectiveness of thought. This paper presents a synopsis of this confabulation theory of human cortical and thalamic function.

A supervised learning neural network coprocessor for soft-decision maximum-likelihood decoding

A supervised learning neural network (SLNN) coprocessor which enhances the performance of a digital soft-decision Viterbi decoder used for forward error correction in a digital communication channel with either fading plus additive white Gaussian noise (AWGN) or pure AWGN has been investigated and designed. The SLNN is designed to cooperate with a phase shift keying (PSK) demodulator, an automatic gain control (AGC) circuit, and a 3-bit quantizer which is an analog to digital convertor. It is trained to learn the best uniform quantization step-size Delta (BEST) as a function of the mean and the standard deviation of various sets of Gaussian distributed random variables. The channel cutoff rate (R(0)) of the channel is employed to determine the best quantization threshold step-size (Delta(BEST)) that results in the minimization of the Viterbi decoder output bit error rate (BER). For a digital communication system with a SLNN coprocessor, consistent and substantial BER performance improvements are observed. The performance improvement ranges from a minimum of 9% to a maximum of 25% for a pure AWGN channel and from a minimum of 25% to a maximum of 70% for a fading channel. This neural network coprocessor approach can be generalized and applied to any digital signal processing system to decrease the performance losses associated with quantization and/or signal instability.

A simple control policy for achieving minimum jerk trajectories

Point-to-point fast hand movements, often referred to as ballistic movements, are a class of movements characterized by straight paths and bell-shaped velocity profiles. In this paper we propose a bang-bang optimal control policy that can achieve such movements. This optimal control policy is accomplished by minimizing the L∞ norm of the jerk profile of ballistic movements with known initial position, final position, and duration of movement. We compare the results of this control policy with human motion data recorded with a manipulandum. We propose that such bang-bang control policies are inherently simple for the central nervous system to implement and also minimize wear and tear on the bio-mechanical system. Physiological experiments support the possibility that some parts of the central nervous system use bang-bang control policies. Furthermore, while many computational neural models of movement control have used a bang-bang control policy without justification, our study shows that the use of such policies is not only convenient, but optimal.

Neural network tomography: Network replication from output surface geometry

Multilayer perceptron networks whose outputs consist of affine combinations of hidden units using the tanh activation function are universal function approximators and are used for regression, typically by reducing the MSE with backpropagation. We present a neural network weight learning algorithm that directly positions the hidden units within input space by numerically analyzing the curvature of the output surface. Our results show that under some sampling requirements, this method can reliably recover the parameters of a neural network used to generate a data set.

The Munificence of High Dimensionality

Many methods of mathematical analysis that are of practical value in low-dimensional Euclidean spaces have computational requirements that grow exponentially as the dimensionality of the space increases to tens, hundreds, or thousands. Mathematician Richard Bellman termed this explosion of difficulty “the curse of dimensionality.” The point of this talk is that this curse is not universal. In fact, there are some wonderfully beneficial properties of Euclidean spaces that improve with increasing dimension. These recently discovered and/or appreciated properties bear directly on studies of neural networks and related subjects.

III.3 – Theory of the Backpropagation Neural Network*

Publisher Summary This chapter presents a survey of the elementary theory of the basic backpropagation neural network architecture, covering the areas of architectural design, performance measurement, function approximation capability, and learning. The survey includes a formulation of the backpropagation neural network architecture to make it a valid neural network and a proof that the backpropagation mean squared error function exists and is differentiable. Also included in the survey is a theorem showing that any L2 function can be implemented to any desired degree of accuracy with a three-layer backpropagation neural network. An appendix presents a speculative neurophysiological model illustrating the way in which the backpropagation neural network architecture might plausibly be implemented in the mammalian brain for corticocortical learning between nearby regions of cerebral cortex. One of the crucial decisions in the design of the backpropagation architecture is the selection of a sigmoidal activation function.

A spiking network model of cerebellar Purkinje cells and molecular layer interneurons exhibiting irregular firing

While the anatomy of the cerebellar microcircuit is well-studied, how it implements cerebellar function is not understood. A number of models have been proposed to describe this mechanism but few emphasize the role of the vast network Purkinje cells (PKJs) form with the molecular layer interneurons (MLIs)—the stellate and basket cells. We propose a model of the MLI-PKJ network composed of simple spiking neurons incorporating the major anatomical and physiological features. In computer simulations, the model reproduces the irregular firing patterns observed in PKJs and MLIs in vitro and a shift toward faster, more regular firing patterns when inhibitory synaptic currents are blocked. In the model, the time between PKJ spikes is shown to be proportional to the amount of feedforward inhibition from an MLI on average. The two key elements of the model are: (1) spontaneously active PKJs and MLIs due to an endogenous depolarizing current, and (2) adherence to known anatomical connectivity along a parasagittal strip of cerebellar cortex. We propose this model to extend previous spiking network models of the cerebellum and for further computational investigation into the role of irregular firing and MLIs in cerebellar learning and function.