Mohamad H. Hassoun

A two-level Hamming network for high performance associative memory

This paper presents an analysis of a two-level decoupled Hamming network, which is a high performance discrete-time/discrete-state associative memory model. The two-level Hamming memory generalizes the Hamming memory by providing for local Hamming distance computations in the first level and a voting mechanism in the second level. In this paper, we study the effect of system dimension, window size, and noise on the capacity and error correction capability of the two-level Hamming memory. Simulation results are given for both random images and human face images.

Dynamic heteroassociative neural memories

Abstract A high-performance, high-capacity dynamic neural memory is proposed which is capable of simultaneous hetero- and autoassociative recall. The proposed memory utilizes two simple layers of neurons which implement a forward mapping and its inverse backward mapping, respectively, such that the range of one mapping is the domain of the other. This dynamic heteroassociative memory (DAM) employs the newly-developed Ho-Kashyap associative memory recording algorithm which optimally distributes the association process of each neural layer over individual neuron weighted-sum and activation function faculties. Various performance characteristics for the proposed DAM are tested and compared to those of correlation- and generalized inverse-recorded DAMs. Simulation results are presented which confirm the superiority of the proposed Ho-Kashyap-recorded DAM over correlation-recorded heteroassociative memories. The Ho-Kashyap recording algorithms performance is also known to exceed that of the optimal linear associative memory (OLAM) recording technique in the case of binary pattern associative storage and retrieval. The proposed DAMs high performance extends to a wide range of input/output association pattern dimensions and memory storage levels.

A coupled gradient network approach for static and temporal mixed-integer optimization

Utilizes the ideas of artificial neural networks to propose new solution methods for a class of constrained mixed-integer optimization problems. These new solution methods are more suitable to parallel implementation than the usual sequential methods of mathematical programming. Another attractive feature of the proposed approach is that some global search mechanisms may be easily incorporated into the computation, producing results which are more globally optimal. To formulate the solution method proposed in this paper, a penalty function approach is used to define a coupled gradient-type network with an appropriate architecture, energy function and dynamics such that high-quality solutions may be obtained upon convergence of the dynamics. Finally, it is shown how the coupled gradient net may be extended to handle temporal mixed-integer optimization problems, and simulations are presented which demonstrate the effectiveness of the approach.

High Performance Recording Algorithm For Hopfield Model Associative Memories

A new recording technique for Hopfield-type dynamic auto-associative memories is proposed. The new technique is based on the finite and exponentially convergent algorithm of Ho and Kashyap for the solution of a system of linear inequalities. Associative neural memories recorded with the proposed algorithm are shown to be superior to those recorded with the Hopfield outer-product and Kohonen generalized-inverse techniques. The new recording algorithm is characterized by high capacity, high convergence rates to stored memories and low convergence rates to false and oscillatory states. The issue of stable false and oscillatory states is raised, and it is shown that such states appear to have a direct Boolean logic relationship with the stored memories.

Adaptive Ho-Kashyap rules for perceptron training

Three adaptive versions of the Ho-Kashyap perceptron training algorithm are derived based on gradient descent strategies. These adaptive Ho-Kashyap (AHK) training rules are comparable in their complexity to the LMS and perceptron training rules and are capable of adaptively forming linear discriminant surfaces that guarantee linear separability and of positioning such surfaces for maximal classification robustness. In particular, a derived version called AHK II is capable of adaptively identifying critical input vectors lying close to class boundaries in linearly separable problems. The authors extend this algorithm as AHK III, which adds the capability of fast convergence to linear discriminant surfaces which are good approximations for nonlinearly separable problems. This is achieved by a simple built-in unsupervised strategy which allows for the adaptive grading and discarding of input vectors causing nonseparability. Performance comparisons with LMS and perceptron training are presented.

Nonlinear Hebbian rule: a statistical interpretation

Recently, the extension of Hebbian learning to nonlinear units has received increased attention. Some successful applications of this learning rule have been reported as well; however, a fundamental understanding of the capability of this learning rule is still lacking. In this paper, we pursue a better understanding of what the network is actually doing by exploring the statistical characteristics of the criterion function and interpreting the nonlinear unit as a probability integral transformation. To improve the capability of the nonlinear units, data preprocessing is suggested. A better data preprocessing leads to the development of a two-layer network which consists of linear units in the first layer and nonlinear units in the second layer. The linear units capture and filter the linear aspect of the data and the nonlinear units discover the nonlinear effects, such as clustering and other general nonlinear associations among the variables. Several potential applications are demonstrated through the simulation results given throughout this paper.<<ETX>>

Recurrent neural nets as dynamical Boolean systems with application to associative memory

Discrete-time/discrete-state recurrent neural networks are analyzed from a dynamical Boolean systems point of view in order to devise new analytic and design methods for the class of both single and multilayer recurrent artificial neural networks. With the proposed dynamical Boolean systems analysis, we are able to formulate necessary and sufficient conditions for network stability which are more general than the well-known but restrictive conditions for the class of single layer networks: (1) symmetric weight matrix with (2) positive diagonal and (3) asynchronous update. In terms of design, we use a dynamical Boolean systems analysis to construct a high performance associative memory. With this Boolean memory, we can guarantee that all fundamental memories are stored, and also guarantee the size of the basin of attraction for each fundamental memory.

The Hamming associative memory and its relation to the exponential capacity DAM

We analyze the capacity of the Hamming associative memory and explore its relation to the exponential DAM (Chiueh and Goodman, 1991). In particular, it is shown that for a sufficiently large but finite radix, the exponential DAM operates as a Hamming associative memory.

A Weighted Voting Model of Associative Memory

This paper presents an analysis of a random access memory (RAM)-based associative memory which uses a weighted voting scheme for information retrieval. This weighted voting memory can operate in heteroassociative or autoassociative mode, can store both real-valued and binary-valued patterns and, unlike memory models, is equipped with a rejection mechanism. A theoretical analysis of the performance of the weighted voting memory is given for the case of binary and random memory sets. Performance measures are derived as a function of the model parameters: pattern size, window size, and number of patterns in the memory set. It is shown that the weighted voting model has large capacity and error correction. The results show that the weighted voting model can successfully achieve high-detection and -identification rates and, simultaneously, low-false-acceptance rates

An RCE-based Associative Memory with Application to Human Face Recognition

Many models of neural network-based associative memory have been proposed and studied. However, most of these models do not have a rejection mechanism and hence are not practical for many real-world associative memory problems. For example, in human face recognition, we are given a database of face images and the identity of each image. Given an input image, the task is to associate when appropriate the image with the corresponding name of the person in the database. However, the input image may be that of a stranger. In this case, the system should reject the input. In this paper, we propose a practical associative memory model that has a rejection mechanism. The structure of the model is based on the restricted Coulomb energy (RCE) network. The capacity of the proposed memory is desibed by two measures: the ability of the system to correctly identify known individuals, and the ability of the system to reject individuals who are not in the database. Experimental results are given which show how the performance of the system varies as the size of the database increases up to 1000 individuals.