Lawrence D. Jackel

Backpropagation applied to handwritten zip code recognition

The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification.

Boosting and other ensemble methods

We compare the performance of three types of neural network-based ensemble techniques to that of a single neural network. The ensemble algorithms are two versions of boosting and committees of neural networks trained independently. For each of the four algorithms, we experimentally determine the test and training error curves in an optical character recognition (OCR) problem as both a function of training set size and computational cost using three architectures. We show that a single machine is best for small training set size while for large training set size some version of boosting is best. However, for a given computational cost, boosting is always best. Furthermore, we show a surprising result for the original boosting algorithm: namely, that as the training set size increases, the training error decreases until it asymptotes to the test error rate. This has potential implications in the search for better training algorithms.

Handwritten digit recognition: applications of neural network chips and automatic learning

Two novel methods for achieving handwritten digit recognition are described. The first method is based on a neural network chip that performs line thinning and feature extraction using local template matching. The second method is implemented on a digital signal processor and makes extensive use of constrained automatic learning. Experimental results obtained using isolated handwritten digits taken from postal zip codes, a rather difficult data set, are reported and discussed.<<ETX>>

Analog electronic neural network circuits

It is argued that the large interconnectivity and the precision required in neural network models present novel opportunities for analog computing. Analog circuits for a wide variety of problems such as pattern matching, optimization, and learning have been proposed and a few have been built. Most of the circuits built so far are relatively small, exploratory designs. Circuits implementing several different neural algorithms, namely, template matching, associative memory, learning, and two-dimensional resistor networks inspired by the architecture of the retina are discussed. The most mature circuits are those for template matching, and chips performing this function are now being applied to pattern-recognition problems. Examples of analog implementation are examined.<<ETX>>

An analog neural network processor with programmable topology

The architecture, implementation, and applications of a special-purpose neural network processor are described. The chip performs over 2000 multiplications and additions simultaneously. Its data path is particularly suitable for the convolutional topologies that are typical in classification networks, but can also be configured for fully connected or feedback topologies. Resources can be multiplexed to permit implementation of networks with several hundreds of thousands of connections on a single chip. Computations are performed with 6 b accuracy for the weights and 3 b for the neuron states. Analog processing is used internally for reduced power dissipation and higher density, but all input/output is digital to simplify system integration. The practicality of the chip is demonstrated with an implementation of a neural network for optical character recognition. This network contains over 130000 connections and was evaluated in 1 ms. >

Application of the ANNA neural network chip to high-speed character recognition

A neural network with 136000 connections for recognition of handwritten digits has been implemented using a mixed analog/digital neural network chip. The neural network chip is capable of processing 1000 characters/s. The recognition system has essentially the same rate (5%) as a simulation of the network with 32-b floating-point precision.

Electronic neural network chips

This paper reviews two custom electronic circuits that implement some simple models of neural function. The circuits include a thin-film array of read-only resistive synapses and an array of programmable synapses and amplifiers serving as electronic neurons. Circuit performance and architecture are discussed.

Microfabrication as a Scientific Tool

Research in microfabrication not only serves the microelectronics industry but also can provide research tools for studying the behavior of matter at submicrometer dimensions. A variety of techniques including optical, x-ray, and electron beam lithography and reactive ion etching can be used to make structures, devices, and arrays only hundreds of atoms across. Microfabrication techniques have been applied to experiments on surface-enhanced Raman scattering, transport in one-dimensional conductors, and macroscopic quantum tunneling. Recent progress is extending these techniques to scales of less than 100 angstroms.

50−nm silicon structures fabricated with trilevel electron beam resist and reactive‐ion etching

A trilevel electron beam resist has been used to make 25‐nm metal features on thick silicon substrates. Using this metal as a mask for reactive ion etching, silicon structures 0.33 μm deep have been fabricated. The resist consists of a thin upper layer of polymethylmethacrylate (PMMA), a middle layer of Ge, and a lower layer of co‐polymer of methylmethacrylate and methacrylic acid, P(MMA/MAA). High‐resolution patterns are written in the upper resist layer and are transferred to the lower layers by reactive‐ion etching. Completed resist stencils have 300‐nm high walls with near‐vertical profiles and are suitable for liftoff processing.

Shortest path segmentation: a method for training a neural network to recognize character strings

The authors describe a method which combines dynamic programming and a neural network recognizer for segmenting and recognizing character strings. The method selects the optimal consistent combination of cuts from a set of candidate cuts generated using heuristics. The optimal segmentation is found by representing the image, the candidate segments, and their scores as a graph in which the shortest path corresponds to the optimal interpretation. The scores are given by neural net outputs for each segment. A significant advantage of the method is that the labor required to segment images manually is eliminated. The system was trained on approximately 7000 unsegmented handwritten zip codes provided by the United States Postal Service. The system has achieved a per-zip-code raw recognition rate of 81% on a 2368 handwritten zip-code test set.<<ETX>>