Giosuè Baggio

The balance between memory and unification in semantics: A dynamic account of the N400

At least three cognitive brain components are necessary in order for us to be able to produce and comprehend language: a Memory repository for the lexicon, a Unification buffer where lexical information is combined into novel structures, and a Control apparatus presiding over executive function in language. Here we describe the brain networks that support Memory and Unification in semantics. A dynamic account of their interactions is presented, in which a balance between the two components is sought at each word-processing step. We use the theory to provide an explanation of the N400 effect.

Coercion and compositionality

Research in psycholinguistics and in the cognitive neuroscience of language has suggested that semantic and syntactic processing are associated with different neurophysiologic correlates, such as the N400 and the P600 in the ERPs. However, only a handful of studies have investigated the neural basis of the syntax–semantics interface, and even fewer experiments have dealt with the cases in which semantic composition can proceed independently of the syntax. Here we looked into one such case—complement coercion—using ERPs. We compared sentences such as, “The journalist wrote the article” with “The journalist began the article.” The second sentence seems to involve a silent semantic element, which is expressed in the first sentence by the head of the verb phrase (VP) “wrote the article.” The second type of construction may therefore require the reader to infer or recover from memory a richer event sense of the VP “began the article,” such as began writing the article, and to integrate that into a semantic representation of the sentence. This operation is referred to as “complement coercion.” Consistently with earlier reading time, eye tracking, and MEG studies, we found traces of such additional computations in the ERPs: Coercion gives rise to a long-lasting negative shift, which differs at least in duration from a standard N400 effect. Issues regarding the nature of the computation involved are discussed in the light of a neurocognitive model of language processing and a formal semantic analysis of coercion.

Language, linguistics and cognition

Experimental research during the last few decades has provided evidence that language is embedded in a mosaic of cognitive functions. An account of how language interfaces with memory, perception, action and control is no longer beyond the scope of linguistics, and can now be seen as part of an explanation of linguistic structure itself. However, although our view of language has changed, linguistic methodology is lagging behind. This chapter is a sustained argument for a diversification of the kinds of evidence applicable to linguistic questions at di↵erent levels of theory, and a defense of the role of linguistics in experimental cognitive science. 1.1 Linguistic methodology and cognitive science At least two conceptual issues are raised by current interactions between linguistics and cognitive science. One is whether the structures and rules described by linguists are cognitively real. There exist several opinions in this regard, that occupy di↵erent positions on the mentalism/anti-mentalism spectrum. At one extreme is cognitive linguistics [ Croft and Cruse, 2004 ] , endorsing both theoretical and methodological mentalism. The former is the idea that linguistic structures are related formally and causally to other mental entities. The latter calls for a revision of traditional linguistic methodology, and emphasizes the role of cognitive data in linguistics. At the opposite side of the spectrum lies formal semantics which, partly inspired by Frege’s anti-psychologistic stance on meaning and thought [ Frege, 1980; Lewis, 1970; Burge, 2005 ] , rejects both versions of mentalism. Somewhere between the two poles is Chomsky’s [ Chomsky, 1965 ] theoretical mentalism, which sees linguistic rules as ultimately residing in the brain of speakers. However, his commitment to the cognitive reality of grammar does not imply a revision of linguistic methodology, which is maintained in its traditional form based on native speakers’ intuitions and the competence/performance distinction. The second problem, in part dependent on the first, is whether experimental data on language acquisition, comprehension and production have any bearing on linguistic theory. On this point too, there is no consensus among linguists. The division between competence and performance has often been used to secure linguistics from experimental evidence of various sorts [ Bunge, 1984 ] , while

Linguistic Representations of Motion Do Not Depend on the Visual Motion System

Embodied semantics proposes that constructing the meaning of motion verb phrases relies on representations of motion in sensory cortex. However, the data reported by earlier studies as evidence for this claim are also explained by a symbolic-semantics view proposing interactions between dissociable systems. In the experiments reported here, participants were visually adapted to real and implied leftward or rightward motion, which produced a motion aftereffect opposite to the direction of the adapting stimulus. Participants then decided whether a directionally ambiguous or a leftward- or rightward-directional verb phrase implied leftward or rightward motion. Because the visual system is engaged in the motion aftereffect, embodied semantics predicts that responses in the motion-aftereffect direction (opposite to the direction of the adapting stimulus) are facilitated, whereas symbolic semantics predicts response facilitation in the direction of the adapting stimulus (opposite to the direction of the motion aftereffect). We found response facilitation in the direction of real- and implied-motion adapting stimuli in ambiguous and directional verb phrases. These results suggest that visual and linguistic representations of motion can be dissociated.

The Processing Consequences of the Imperfective Paradox

In this paper we present a semantic analysis of the imperfective paradox based on the Event Calculus (van Lambalgen & Hamm 2004), a planning formalism characterizing a class of models which can be computed by connectionist networks. We report the results of a questionnaire that support the semantic theory and suggest that different aspectual classes of VPs in the progressive give rise to different entailment patterns. Further, a processing model is outlined, combining the semantic analysis with the psycholinguistic principle of immediacy in the framework of recurrent networks. The model is used to derive predictions concerning the electrophysiological correlates of the computations described by the Event Calculus.

Logic as Marr's Computational Level: Four Case Studies

We sketch four applications of Marrs levels-of-analysis methodology to the relations between logic and experimental data in the cognitive neuroscience of language and reasoning. The first part of the paper illustrates the explanatory power of computational level theories based on logic. We show that a Bayesian treatment of the suppression task in reasoning with conditionals is ruled out by EEG data, supporting instead an analysis based on defeasible logic. Further, we describe how results from an EEG study on temporal prepositions can be reanalyzed using formal semantics, addressing a potential confound. The second part of the article demonstrates the predictive power of logical theories drawing on EEG data on processing progressive constructions and on behavioral data on conditional reasoning in people with autism. Logical theories can constrain processing hypotheses all the way down to neurophysiology, and conversely neuroscience data can guide the selection of alternative computational level models of cognition.

Role Asymmetry and Code Transmission in Signaling Games: An Experimental and Computational Investigation

In signaling games, a sender has private access to a state of affairs and uses a signal to inform a receiver about that state. If no common association of signals and states is initially available, sender and receiver must coordinate to develop one. How do players divide coordination labor? We show experimentally that, if players switch roles at each communication round, coordination labor is shared. However, in games with fixed roles, coordination labor is divided: Receivers adjust their mappings more frequently, whereas senders maintain the initial code, which is transmitted to receivers and becomes the common code. In a series of computer simulations, player and role asymmetry as observed experimentally were accounted for by a model in which the receiver in the first signaling round has a higher chance of adjusting its code than its partner. From this basic division of labor among players, certain properties of role asymmetry, in particular correlations with game complexity, are seen to follow.

Cultural Transmission and Evolution of Melodic Structures in Multi-generational Signaling Games

It has been proposed that languages evolve by adapting to the perceptual and cognitive constraints of the human brain, developing, in the course of cultural transmission, structural regularities that maximize or optimize learnability and ease of processing. To what extent would perceptual and cognitive constraints similarly affect the evolution of musical systems? We conducted an experiment on the cultural evolution of artificial melodic systems, using multi-generational signaling games as a laboratory model of cultural transmission. Signaling systems, using five-tone sequences as signals, and basic and compound emotions as meanings, were transmitted from senders to receivers along diffusion chains in which the receiver in each game became the sender in the next game. During transmission, structural regularities accumulated in the signaling systems, following principles of proximity, symmetry, and good continuation. Although the compositionality of signaling systems did not increase significantly across generations, we did observe a significant increase in similarity among signals from the same set. We suggest that our experiment tapped into the cognitive and perceptual constraints operative in the cultural evolution of musical systems, which may differ from the mechanisms at play in language evolution and change.

Motion words selectively modulate direction discrimination sensitivity for threshold motion

Can speech selectively modulate the sensitivity of a sensory system so that, in the presence of a suitable linguistic context, the discrimination of certain perceptual features becomes more or less likely? In this study, participants heard upward or downward motion words followed by a single visual field of random dots moving upwards or downwards. The time interval between the onsets of the auditory and the visual stimuli was varied parametrically. Motion direction could be either discriminable (suprathreshold motion) or non-discriminable (threshold motion). Participants had to judge whether the dots were moving upward or downward. Results show a double dissociation between discrimination sensitivity (d′) and reaction times depending on whether vertical motion was above or at threshold. With suprathreshold motion, responses were faster for congruent directions of words and dots, but sensitivity was equal across conditions. With threshold motion, sensitivity was higher for congruent directions of words and dots, but responses were equally fast across conditions. The observed differences in sensitivity and response times were largest when the dots appeared 450 ms after word onset, that is, consistently with electrophysiology, at the time the up/down semantics of the word had become available. These data suggest that word meanings can alter the balance between signal and noise within the visual system and affect the perception of low-level sensory features.

Selective alignment of brain responses by task demands during semantic processing

The way the brain binds together words to form sentences may depend on whether and how the arising cognitive representation is to be used in behavior. The amplitude of the N400 effect in event-related brain potentials is inversely correlated with the degree of fit of a words meaning into a semantic representation of the preceding discourse. This study reports a double dissociation in the latency characteristics of the N400 effect depending on task demands. When participants silently read words in a sentence context, without issuing a relevant overt response, greater temporal alignment over recording sites occurs for N400 onsets than peaks. If however a behavior is produced - here pressing a button in a binary probe selection task - exactly the opposite pattern is observed, with stronger alignment of N400 peaks than onsets. The peak amplitude of the N400 effect correlates best with the latency characteristic showing less temporal dispersion. These findings suggest that meaning construction in the brain is subtly affected by task demands, and that there is complex functional integration between semantic combinatorics and control systems handling behavioral goals.