Jeffrey L. Elman

Finding Structure in Time

Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic/semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly context-dependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type/token distinction.

The TRACE model of speech perception

Abstract We describe a model called the TRACE model of speech perception. The model is based on the principles of interactive activation. Information processing takes place through the excitatory and inhibitory interactions of a large number of simple processing units, each working continuously to update its own activation on the basis of the activations of other units to which it is connected. The model is called the TRACE model because the network of units forms a dynamic processing structure called “the Trace,” which serves at once as the perceptual processing mechanism and as the systems working memory. The model is instantiated in two simulation programs. TRACE I, described in detail elsewhere, deals with short segments of real speech, and suggests a mechanism for coping with the fact that the cues to the identity of phonemes vary as a function of context. TRACE II, the focus of this article, simulates a large number of empirical findings on the perception of phonemes and words and on the interactions of phoneme and word perception. At the phoneme level, TRACE II simulates the influence of lexical information on the identification of phonemes and accounts for the fact that lexical effects are found under certain conditions but not others. The model also shows how knowledge of phonological constraints can be embodied in particular lexical items but can still be used to influence processing of novel, nonword utterances. The model also exhibits categorical perception and the ability to trade cues off against each other in phoneme identification. At the word level, the model captures the major positive feature of Marslen-Wilsons COHORT model of speech perception, in that it shows immediate sensitivity to information favoring one word or set of words over others. At the same time, it overcomes a difficulty with the COHORT model: it can recover from underspecification or mispronunciation of a words beginning. TRACE II also uses lexical information to segment a stream of speech into a sequence of words and to find word beginnings and endings, and it simulates a number of recent findings related to these points. The TRACE model has some limitations, but we believe it is a step toward a psychologically and computationally adequate model of the process of speech perception.

Distributed Representations, Simple Recurrent Networks, And Grammatical Structure

In this paper three problems for a connectionist account of language are considered: 1. What is the nature of linguistic representations? 2. How can complex structural relationships such as constituent structure be represented? 3. How can the apparently open-ended nature of language be accommodated by a fixed-resource system? Using a prediction task, a simple recurrent network (SRN) is trained on multiclausal sentences which contain multiply-embedded relative clauses. Principal component analysis of the hidden unit activation patterns reveals that the network solves the task by developing complex distributed representations which encode the relevant grammatical relations and hierarchical constituent structure. Differences between the SRN state representations and the more traditional pushdown store are discussed in the final section.

Learning and evolution in neural networks

This article describes simulations on populations of neural networks that both evolve at the population level and learn at the individual level. Unlike other simulations, the evolutionary task (finding food in the environment) and the learning task (predicting the next position of food on the basis of present position and planned networks movement) are different tasks. In these conditions, learning influences evolution (without Lamarckian inheritance of learned weight changes) and evolution influences learning. Average but not peak fitness has a better evolutionary growth with learning than without learning. After the initial generations, individuals that learn to predict during life also improve their food-finding ability during life. Furthermore, individuals that inherit an innate capacity to find food also inherit an innate predisposition to learn to predict the sensory consequences of their movements. They do not predict better at birth, but they do learn to predict better than individuals of the initial generation given the same learning experience. The results are interpreted in terms of a notion of dynamic correlation between the fitness surface and the learning surface. Evolution succeeds in finding both individuals that have high fitness and individuals that, although they do not have high fitness at birth, end up with high fitness because they learn to predict.

Cognitive penetration of the mechanisms of perception: compensation for coarticulation of lexically restored phonemes

An important question in language processing is whether higher-level processes are able to interact directly with lower-level processes, as assumed by interactive models such as the TRACE model of speech perception. This issue is addressed in the present study by examining whether putative interlevel phenomena can trigger the operation of intralevel processes at lower levels. The intralevel process involved the perceptual compensation for the coarticulatory influences of one speech sound on another. TRACE predicts that this compensation can be triggered by illusory phonemes which are perceived as a result of topdown, lexical influences. In Experiment 1, we confirm this prediction. Experiments 2 to 4 replicate this finding and fail to support several potential alternative explanations of the results of Experiment 1. The basic finding that intralevel phenomena can be triggered by interlevel processes argues against the view that aspects of speech perception are encapsulated in a module impervious to influences from higher levels. Instead, it supports a central premise of interactive models, in which basic aspects of perceptual processing are subject to influences from higher levels.

An alternative view of the mental lexicon

An essential aspect of knowing language is knowing the words of that language. This knowledge is usually thought to reside in the mental lexicon, a kind of dictionary that contains information regarding a words meaning, pronunciation, syntactic characteristics, and so on. In this article, a very different view is presented. In this view, words are understood as stimuli that operate directly on mental states. The phonological, syntactic and semantic properties of a word are revealed by the effects it has on those states.

A Recurrent Neural Network that Learns to Count

Parallel distributed processing (PDP) architectures demonstrate a potentially radical alternative to the traditional theories of language processing that are based on serial computational models. However, learning complex structural relationships in temporal data presents a serious challenge to PDP systems. For example, automata theory dictates that processing strings from a context-free language (CFL) requires a stack or counter memory device. While some PDP models have been hand-crafted to emulate such a device, it is not clear how a neural network might develop such a device when learning a CFL. This research employs standard backpropagation training techniques for a recurrent neural network (RNN) in the task of learning to predict the next character in a simple deterministic CFL (DCFL). We show that an RNN can learn to recognize the structure of a simple DCFL. We use dynamical systems theory to identify how network states reflect that structure by building counters in phase space. The work is an empirical investigation which is complementary to theoretical analyses of network capabilities, yet original in its specific configuration of dynamics involved. The application of dynamical systems theory helps us relate the simulation results to theoretical results, and the learning task enables us to highlight some issues for understanding dynamical systems that process language with counters.

A basis for generating expectancies for verbs from nouns

We explore the implications of an event-based expectancy generation approach to language understanding, suggesting that one useful strategy employed by comprehenders is to generate expectations about upcoming words. We focus on two questions: (1) What role is played by elements other than verbs in generating expectancies? (2) What connection exists between expectancy generation and event-based knowledge? Because verbs follow their arguments in many constructions (particularly in verb-final languages), deferring expectations until the verb seems inefficient. Both human data and computational modeling suggest that other sentential elements may also play a role in predictive processing and that these constraints often reflect knowledge regarding typical events. We investigated these predictions, using both short and long stimulus onset asynchrony priming. Robust priming obtained when verbs were named aloud following typical agents, patients, instruments, and locations, suggesting that event memory is organized so that nouns denoting entities and objects activate the classes of events in which they typically play a role. These computations are assumed to be an important component of expectancy generation in sentence processing.

Sense and structure : Meaning as a determinant of verb subcategorization preferences

Readers are sensitive to the fact that verbs may allow multiple subcategorization frames that differ in their probability of occurrence. Although a verb’s overall subcategorization preferences can be described probabilistically, underlying non-random factors may determine those probabilities. One potential factor is verb semantics: Many verbs show sense differences, and a verb’s subcategorization profile can vary by sense. Thus, although find can occur with a direct object (DO) or a sentential complement (SC), when it is used to mean ‘locate’ it occurs only with a DO, whereas in its ‘realize’ sense it is SC-biased, but can take either frame. We used corpus analyses to identify verbs that occur with both frames, and found that their subcategorization probabilities differ by sense. Off-line sentence completions demonstrated that contexts can promote a specific sense of a verb, which subsequently influenced subcategorization probability. Finally, in a self-paced reading time experiment, verbs occurred in target sentences containing either a structurally unambiguous or ambiguous SC, following a context favoring the verb’s DO- or SC-biased sense. Sense-biasing context influenced reading times at that, and interacted with ambiguity in the disambiguating region. Thus, readers use sense-contingent subcategorization preferences during on-line language comprehension.

Connectionist models of cognitive development: where next?

Over the past two decades, connectionist models have generated a lively debate regarding the underlying mechanisms of cognitive development. This debate has in turn led to important empirical research that might not have occurred otherwise. More recently, advances in developmental neuroscience present a new set of challenges for modelers. In this article, I review some of the insights that have come from modeling work, focusing on (1) explanations for the shape of change; (2) new views on how knowledge may be represented; (3) the richness of experience. The article concludes by considering some of the new challenges and opportunities for modeling cognitive development.