Kunihiko Fukushima

Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position

A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by “learning without a teacher”, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname “neocognitron”. After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consits of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of “S-cells”, which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of “C-cells” similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any “teacher” during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cell of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the patterns position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern.

Recognition of partly occluded patterns: a neural network model

Abstract. Human beings are often able to read a letter or word partly occluded by contaminating ink stains. However, if the stains are completely erased and the occluded areas of the letter are changed to white, we usually have difficulty in reading the letter. In this article I propose a hypothesis explaining why a pattern is easier to recognize when it is occluded by visible objects than by invisible opaque objects. A neural network model is constructed based on this hypothesis.The visual system extracts various visual features from the input pattern and then attempts to recognize it. If the occluding objects are not visible, the visual system will have difficulty in distinguishing which features are relevant to the original pattern and which are newly generated by the occlusion. If the occluding objects are visible, however, the visual system can easily discriminate between relevant and irrelevant features and recognize the occluded pattern correctly.The proposed model is an extended version of the neocognitron model. The activity of the feature-extracting cells whose receptive fields cover the occluding objects is suppressed in an early stage of the hierarchical network. Since the irrelevant features generated by the occlusion are thus eliminated, the model can recognize occluded patterns correctly, provided the occlusion is not so large as to prevent recognition even by human beings.

Stereo algorithm that extracts a depth cue from interocularly unpaired points

In binocular vision, interocularly unpaired points are often generated as a result of occlusions. These points can only produce false matches, and have no information about binocular disparities. However, recent psychophysical experiments have suggested that interocularly unpaired points play an important role in human stereo perception. This article introduces a stereo algorithm that can extract depth cues from not only interocularly paired but also unpaired points. Computer simulation demonstrates that our algorithm can correctly process the da Vinci stereogram as well as natural stereo pairs.

Bistability of mixed states in a neural network storing hierarchical patterns

We discuss the properties of equilibrium states in an autoassociative memory model storing hierarchically correlated patterns (hereafter, hierarchical patterns). We will show that symmetric mixed states (hereafter, mixed states) are bistable on the associative memory model storing the hierarchical patterns in a region of the ferromagnetic phase. This means that the first-order transition occurs in this ferromagnetic phase. We treat these contents with a statistical mechanical method (SCSNA) and by computer simulation. Finally, we discuss a physiological implication of this model. Sugase et al (1999 Nature 400 869) analysed the time-course of the information carried by the firing of face-responsive neurons in the inferior temporal cortex. We also discuss the relation between the theoretical results and the physiological experiments of Sugase et al .

Self-organization of shift-invariant receptive fields

This paper proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. Namely, cells similar to simple and complex cells in the primary visual cortex are generated in a network trained by the new leaning rule. To demonstrate the new learning rule, we simulate a three-layered network that consists of an input layer (retina), a layer of S-cells (simple cells), and a layer of C-cells (complex cells). During the learning, straight lines of various orientations sweep across the input layer Both S- and C-cells are created through competition. Although S-cells compete depending on their instantaneous outputs, C-cells compete depending on the traces (or temporal averages) of their outputs. For the self-organization of S-cells, only winner S-cells have LTP (long term potentiation) in their input connections. For the self-organization of S-cells, however, loser S-cells have LTD (long term depression) in their input connections, while winners have LTP. Both S- and C-cells are accompanied by inhibitory cells. Modification of inhibitory connections together with excitatory connections is important for creation of C-cells as well as S-cells.

A neural network model for the mechanism of feature-extraction

We propose a new multilayered neural network model which has the ability of rapid self-organization. This model is a modified version of the cognitron (Fukushima, 1975). It has modifiable inhibitory feedback connections, as well as conventional modifiable excitatory feedforward connections, between the cells of adjoining layers. If a feature-extracting cell in the network is excited by a stimulus which is already familiar to the network, the cell immediately feeds back inhibitory signals to its presynaptic cells in the preceding layer, which suppresses their response. On the other hand, the feature-extracting cell does not respond to an unfamiliar feature, and the responses from its presynaptic cells are therefore not suppressed because they do not receive any feedback inhibition. Modifiable synapses in the new network are reinforced in a way similar to those in the cognitron, and synaptic connections from cells yielding a large sustained output are reinforced. Since familiar stimulus features do not elicit a sustained response from the cells of the network, only circuits which detect novel stimulus features develop. The network therefore quickly acquires favorable pattern-selectivity by the mere repetitive presentation of set of learning patterns.

Training neocognitron to recognize handwritten digits in the real world

Using a large-scale real-world database-the ETL-1 database of the Electrotechnical Laboratory in Japan-we show that a neocognitron trained by unsupervised learning with a winner-take-all process can recognize handwritten digits with a recognition rate higher than 97%. We use the technique of dual thresholds for feature-extracting S-cells, and higher threshold values are used in the learning than in the recognition phase. We discuss how the threshold values affect the recognition rate. The learning method for the cells of the highest stage of the network has been modified from the conventional one, in order to reconcile the unsupervised learning procedure with the use of information about the category names of the training patterns.

Neural network models for vision

Modeling neural networks is a powerful approach to uncover the mechanism of the brain, and the results of the research are ready to use for engineering applications. This paper introduces several models for vision from recent works by the author. (1) Increased recognition rate of the neocognitron: to increase the recognition rate of the neocognitron, several modifications have been applied to the network architecture and the learning method. For example, an inhibitory surround in the connections to C-cells is useful for this purpose. When trained with 3000 patterns, the neocognitron shows a recognition rate of 98.6% for a blind test set randomly sampled from a large database of handwritten digits (ETL-1), and 100% for the training set. (2) A neocognitron that can accept incremental learning, without giving severe damage to old memories or reducing learning speed: It uses competitive learning, and the learning of all stages of the hierarchical network progresses simultaneously. (3) A model that. has an ability to recognize and restore partly occluded patterns: even the identical image is perceived differently by human beings depending on the shape of occluding objects. The model responds in a similar way to human beings. It is a multi-layered hierarchical neural network, in which visual information is processed by interaction of bottom-up and top-down signals. Occluded parts of a pattern are restored mainly by feedback signals from the highest stage of the network, while the unoccluded parts are reproduced mainly by signals from lower stages. The model does not use a simple template matching method. It can recognize and restore even deformed versions of learned patterns.

Pattern recognition with eye movement: a neural network model

We offer a neural network model which has the ability of sift, scale, and deformation-invariant pattern recognition. The model extracts spatial relations between fixating feature and each of peripheral features. The mechanism of saccadic eye movement enables the model to analyze whole pattern-structure. The ability of the model has been tested on a computer.

Recognition of occluded patterns: a neural network model

This paper proposes a hypothesis explaining why a pattern is easier to be recognized when the occluding objects are visible. A neural network model is constructed based on the hypothesis and is demonstrated that the model responds to occluded patterns in a similar way as human beings. The visual system extracts various visual features from the input pattern and then recognizes it. If the occluding objects are invisible, the visual system will have difficulty in distinguishing which features are relevant to the original pattern and which are newly generate by the occlusion. If the occluding objects are visible, however, the visual system can easily discriminate relevant from irrelevant features and recognize the occluded pattern correctly. The proposed model is an extended version of the neocognitron model. The activity of the feature-extracting S-cells whose receptive fields cover the occluding objects is suppressed in the lowest stage of the hierarchical network.