Taeko Kamimura

Flexible feature discovery and structural information control

In this paper, we propose a new information theoretic method called structural information control for flexible feature discovery. The new method has three distinctive characteristics, which traditional competitive learning fails to offer. First, the new method can directly control competitive unit activation patterns, whereas traditional competitive learning does not have any means to control them. Thus, with the new method, it is possible to extract salient features not discovered by traditional methods. Second, competitive units compete witheach other by maximizing their information content about input patterns. Consequently, this information maximization makes it possible to control flexibly competition processes. Third, in structural information control, it is possible to define many different kinds of information content, and we can choose a specific type of information according to a given objective. When applied to competitive learning, structural information can be used to control the number of dead or spare units, and to extract macro as well as micro features of input patterns in explicit ways. We first applied this method to simple pattern classification to demonstrate that information can be controlled and that different neuron firing patterns can be generated. Second, a dipole problem was used to show that structural information could provide representations similar to those by the conventional competitive learning methods. Finally, we applied the method to a language acquisition problem in which networks must flexibly discover some linguistic rules by changing structural information. Especially, we attempted to examine the effect of the information parameter to control the number of dead neurons, and thus to examine how macro and micro features in input patterns can explicitly be discovered by structural information.

Greedy information acquisition algorithm: a new information theoretic approach to dynamic information acquisition in neural networks

In this paper, we proopose a new information theoretic approach to competitive learning. The new approach is called greedy information acquisition , because networks try to absorb as much information as possible in every stage of learning. In the first phase, with minimum network architecture for realizing competition, information is maximized. In the second phase, a new unit is added, and thereby information is again increased as much as possible. This proceess continues until no more increase in information is possible. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to three problems: a dipole problem; a language classification problem; and a phonological feature detection problem. Experimental results confirmed that information maximization can be repeatedly applied and that different features in input patterns are gradually discovered. We also compared our method with conventional competitive learning and multivariate analysis. The experimental results confirmed that our new method can detect salient features in input patterns more clearly than the other methods.

Argumentative Strategies in American and Japanese English.

This study examines the differences between argumentative strategies in English used by American and Japanese students. Two groups of English essays on capital punishment written by American and Japanese students were analyzed in terms of organization patterns, rhetorical appeals, diction, and cultural influences. It was found that the Japanese students used an organizational unit called ‘reservation’ more frequently and that this gave the impression of circularity to their essays. The American students employed more ‘rational’ appeals than the Japanese students, who by contrast used more affective appeals than the American counterparts. The types of diction preferred by the American students (such as ‘should,’ the + superlatives,’ and ‘I believe’) functioned as ‘emphatic devices’ while those preferred by the Japanese students (such as ‘I think’ and ‘maybe’) acted as ‘softening devices.’ Finally, the American students tended to exhibit cultural tokens such as references to ‘counseling,’‘Biblical references,’ and ‘the tax payer’s standpoint’; the Japanese students, on the other hand, tended to point out the suffering of the victim’s family and friends and concrete incidents, trying to evoke empathy in the reader’s mind.

Information theoretic rule discovery in neural networks

Proposes a new information-theoretic method called structural information, and argues that this new method should be substituted for the traditional competitive method. Structural information control is a more powerful and biologically sounder model, because it uses a soft winner-takes-all model instead of a hard winner-takes-all model. Experiments were conducted to apply the structural information to linguistic rule extraction in which the choice of different donatory verbs must be inferred in an unsupervised way. We found that the structural information control can detect linguistic rules more accurately than the traditional competitive learning method.

Competitive learning by mutual information maximization

We propose a new information maximization method for feature discovery and demonstrate that it can discover linguistic rules in unsupervised ways. The new method can directly control competitive unit activation patterns to which input-competitive connections are adjusted. This direct control of the activation patterns permits considerable flexibility for connections and shows the ability to discover salient features not captured by traditional methods. We applied the new method to a linguistic rule acquisition problem. In this problem, unsupervised methods are needed because children acquire rules even without any explicit instruction. Our results confirmed that only by maximizing information content in competitive units linguistic rules can be extracted. These results suggest that linguistic rule acquisition is induced by the processes of information maximization in living systems.

Embedded information enhancement for neuron selection in self-organizing maps.

In this paper, we improve the information enhancement method to extract important input variables. The information enhancement method has been developed to detect important components in neural systems. Our previous method focused on the detection of the important components; therefore, it has not fully incorporated the information contained in the components into learning processes. We here embed the information enhancement into the processes of self-organizing. The embedding is realized simply by adding the importance of input neurons to the self-organizing maps (SOM). We applied the method to an artificial data set and a real data set taken in the field of the second language learning (L2 learning). In the artificial data set, we could show that the symmetric properties of the data could be explicitly extracted. In the L2 learning data set, only one important variable could be detected. The detected input variable represented a grammatical item termed “the inanimate subject.” This variable was found to be the one which differentiated Japanese high school students from university students in the level of L2 grammatical competence. It has been claimed that students have difficulty in producing sentences with inanimate subjects in the field of L2 learning. Thus, the finding by the method well corresponds to the knowledge of L2 learning.

Hybrid information processing systems to generate self-organizing maps: combining SOM and information maximization for coherent activation patterns

We combine a self-organizing map (SOM) and information maximization to produce coherent competitive unit activation patterns in an artificial system. The new system is composed of a SOM component and an information maximization component. In the SOM component, the conventional SOM is used to cooperate neurons. In the information maximization component, information between input units and competitive units is increased as much as possible. The component plays a role to accentuate activation patterns obtained by the SOM component. We apply the new system to medical data analysis. Experimental results confirm that firing patterns obtained by the conventional SOM are reinforced and become clearer by the information maximization component.

Greedy information acquisition algorithm: a new information theoretic method for network growing

In this paper, we propose a new information theoretic network growing algorithm. The new approach is called greedy information acquisition, because networks try to absorb as much information as possible in every stage of learning. In the first stage, two competitive units compete with each other by maximizing mutual information. In the successive stages, new competitive units are gradually added and information is maximized. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to a language classification problem. Experimental results confirmed that different features in input patterns are gradually discovered.

Cooperative Information Control to Coordinate Competition and Cooperation

This paper proposes a novel information theoretic approach to self-organization called cooperative information control. The method aims to mediate between competition and cooperation among neurons by controlling the information content in the neurons. Competition is realized by maximizing information content in neurons. In the process of information maximization, only a small number of neurons win the competition, while all the others are inactive. Cooperation is implemented by having neighboring neurons behave similarly. These two processes are unified and controlled in the framework of cooperative information control. We applied the new method to political data analyses. Experimental results confirmed that competition and cooperation are flexibly controlled. In addition, controlled processes can yield a number of different neuron firing patterns, which can be used to detect macro as well as micro features in input patterns.

Information Maximization and Language Acquisition

In this paper, we propose a new information maximization method for feature discovery and demonstrate that it can discover linguistic rules in unsupervised ways. The new method can directly control competitive unit activation patterns to which input-competitive connections are adjusted. This direct control of the activation patterns permits considerable flexibility for connections and shows the ability to discover salient features not captured by traditional methods.We applied the new method to a linguistic rule acquisition problem. In this problem, unsupervised methods are needed because children acquire rules even without any explicit instruction. Our results confirmed that only by maximizing information content in competitive units linguistic rules can be extracted. These results suggest that linguistic rule acquisition is induced by the processes of information maximization in living systems.