Harm Brouwer

A time and place for language comprehension: mapping the N400 and the P600 to a minimal cortical network

We propose a new functional-anatomical mapping of the N400 and the P600 to a minimal cortical network for language comprehension. Our work is an example of a recent research strategy in cognitive neuroscience, where researchers attempt to align data regarding the nature and time-course of cognitive processing (from ERPs) with data on the cortical organization underlying it (from fMRI). The success of this “alignment” approach critically depends on the functional interpretation of relevant ERP components. Models of language processing that have been proposed thus far do not agree on these interpretations, and present a variety of complicated functional architectures. We put forward a very basic functional-anatomical mapping based on the recently developed Retrieval-Integration account of language comprehension (Brouwer et al., 2012). In this mapping, the left posterior part of the Middle Temporal Gyrus (BA 21) serves as an epicenter (or hub) in a neurocognitive network for the retrieval of word meaning, the ease of which is reflected in N400 amplitude. The left Inferior Frontal Gyrus (BA 44/45/47), in turn, serves a network epicenter for the integration of this retrieved meaning with the words preceding context, into a mental representation of what is being communicated; these semantic and pragmatic integrative processes are reflected in P600 amplitude. We propose that our mapping describes the core of the language comprehension network, a view that is parsimonious, has broad empirical coverage, and can serve as the starting point for a more focused investigation into the coupling of brain anatomy and electrophysiology.

A Neurocomputational Model of the N400 and the P600 in Language Processing

Abstract Ten years ago, researchers using event‐related brain potentials (ERPs) to study language comprehension were puzzled by what looked like a Semantic Illusion: Semantically anomalous, but structurally well‐formed sentences did not affect the N400 component—traditionally taken to reflect semantic integration—but instead produced a P600 effect, which is generally linked to syntactic processing. This finding led to a considerable amount of debate, and a number of complex processing models have been proposed as an explanation. What these models have in common is that they postulate two or more separate processing streams, in order to reconcile the Semantic Illusion and other semantically induced P600 effects with the traditional interpretations of the N400 and the P600. Recently, however, these multi‐stream models have been called into question, and a simpler single‐stream model has been proposed. According to this alternative model, the N400 component reflects the retrieval of word meaning from semantic memory, and the P600 component indexes the integration of this meaning into the unfolding utterance interpretation. In the present paper, we provide support for this “Retrieval–Integration (RI)” account by instantiating it as a neurocomputational model. This neurocomputational model is the first to successfully simulate the N400 and P600 amplitude in language comprehension, and simulations with this model provide a proof of concept of the single‐stream RI account of semantically induced patterns of N400 and P600 modulations.

Questions Left Unanswered: How the Brain Responds to Missing Information

It sometimes happens that when someone asks a question, the addressee does not give an adequate answer, for instance by leaving out part of the required information. The person who posed the question may wonder why the information was omitted, and engage in extensive processing to find out what the partial answer actually means. The present study looks at the neural correlates of the pragmatic processes invoked by partial answers to questions. Two experiments are presented in which participants read mini-dialogues while their Event-Related brain Potentials (ERPs) are being measured. In both experiments, violating the dependency between questions and answers was found to lead to an increase in the amplitude of the P600 component. We interpret these P600-effects as reflecting the increased effort in creating a coherent representation of what is communicated. This effortful processing might include the computation of what the dialogue participant meant to communicate by withholding information. Our study is one of few investigating language processing in conversation, be it that our participants were ‘eavesdroppers’ instead of real interactants. Our results contribute to the as of yet small range of pragmatic phenomena that modulate the processes underlying the P600 component, and suggest that people immediately attempt to regain cohesion if a question-answer dependency is violated in an ongoing conversation.

On the Proper Treatment of the N400 and P600 in Language Comprehension

Event-Related Potentials (ERPs)—stimulus-locked, scalp-recorded voltage fluctuations caused by post-synaptic neural activity—have proven invaluable to the study of language comprehension. Of interest in the ERP signal are systematic, reoccurring voltage fluctuations called components, which are taken to reflect the neural activity underlying specific computational operations carried out in given neuroanatomical networks (cf. Näätänen and Picton, 1987). For language processing, the N400 component and the P600 component are of particular salience (see Kutas et al., 2006, for a review). The typical approach to determining whether a target word in a sentence leads to differential modulation of these components, relative to a control word, is to look for effects on mean amplitude in predetermined time-windows on the respective ERP waveforms, e.g., 350–550 ms for the N400 component and 600–900 ms for the P600 component. The common mode of operation in psycholinguistics, then, is to tabulate the presence/absence of N400and/or P600-effects across studies, and to use this categorical data to inform neurocognitive models that attribute specific functional roles to the N400 and P600 component (see Kuperberg, 2007; Bornkessel-Schlesewsky and Schlesewsky, 2008; Brouwer et al., 2012, for reviews). Here, we assert that this Waveform-based Component Structure (WCS) approach to ERPs leads to inconsistent data patterns, and hence, misinforms neurocognitive models of the electrophysiology of language processing. The reason for this is that the WCS approach ignores the latent component structure underlying ERP waveforms (cf. Luck, 2005), thereby leading to conclusions about component structure that do not factor in spatiotemporal component overlap of the N400 and the P600. This becomes particularly problematic when spatiotemporal component overlap interacts with differential P600 modulations due to task demands (cf. Kolk et al., 2003). While the problem of spatiotemporal component overlap is generally acknowledged, and occasionally invoked to account for within-study inconsistencies in the data, its implications are often overlooked in psycholinguistic theorizing that aims to integrate findings across studies. We believeWCS-centric theorizing to be the single largest reason for the lack of convergence regarding the processes underlying the N400 and the P600, thereby seriously hindering the advancement of neurocognitive theories and models of language processing.

On the organization of the perisylvian cortex: Insights from the electrophysiology of language: Comment on "Towards a Computational Comparative Neuroprimatology: Framing the language-ready brain" by M.A. Arbib.

The Mirror System Hypothesis (MSH) on the evolution of the language-ready brain draws upon the parallel dorsal–ventral stream architecture for vision [1]. The dorsal “how” stream provides a mapping of parietally-mediated affordances onto the motor system (supporting preshape), whereas the ventral “what” stream engages in object recognition and visual scene analysis (supporting pantomime and verbal description). Arbib attempts to integrate this MSH perspective with a recent conceptual dorsal–ventral stream model of auditory language comprehension [5] (henceforth, the B&S model). In the B&S model, the dorsal stream engages in time-dependent combinatorial processing, which subserves syntactic structuring and linkage to action, whereas the ventral stream performs time-independent unification of conceptual schemata. These streams are integrated in the left Inferior Frontal Gyrus (lIFG), which is assumed to subserve cognitive control, and no linguistic processing functions. Arbib criticizes the B&S model on two grounds: (i) the time-independence of the semantic processing in the ventral stream (by arguing that semantic processing is just as time-dependent as syntactic processing), and (ii) the absence of linguistic processing in the lIFG (reconciling syntactic and semantic representations is very much linguistic processing proper). Here, we provide further support for these two points of criticism on the basis of insights from the electrophysiology of language. In the course of our argument, we also sketch the contours of an alternative model that may prove better suited for integration with the MSH. The B&S model is effectively a cortical instantiation of the extended Argument Dependency Model (eADM) [3,4]. The eADM posits a cascaded architecture, in which an algorithmic-driven processing stream (∼ the dorsal stream in the B&S model) works in parallel to a plausibility processing stream (∼ the ventral stream). The former serves to assign thematic roles to incoming noun phrases based on “prominence” information (e.g., animacy, case marking, and linear word order) and to link these to the argument structures of incoming verbs, whereas the latter determines the most plausible combination of the arguments and the verb in a sentence, while ignoring (linear, hence

Discourse semantics with information structure

The property of projection poses a challenge to formal semantic theories, due to its apparent non-compositional nature. Projected content is therefore typically analyzed as being different from and independent of asserted content. Recent evidence, however, suggests that these types of content in fact closely interact, thereby calling for a more integrated analysis that captures their similarities, while respecting their differences. Here, we propose such a unified, compositional semantic analysis of asserted and projected content. Our analysis captures the similarities and differences between presuppositions, anaphora, conventional implicatures and assertions on the basis of their information structure, that is, on basis of how their content is contributed to the unfolding discourse context. We formalize our analysis in an extension of the dynamic semantic framework of Discourse Representation Theory (DRT)—called Projective DRT (PDRT)—that employs projection variables to capture the information-structural aspects of semantic content; different constellations of such variables capture the differences between the different types of projected and asserted content within a single dimension of meaning. We formally derive the structural and compositional properties of PDRT, as well as its semantic interpretation. By instantiating PDRT as a mature semantic formalism, we argue that it paves way for a more focused investigation of the information-structural aspects of meaning.

A new insight into sentence comprehension: The impact of word associations in sentence processing as shown by invasive EEG recording

ABSTRACT The effect of word association on sentence processing is still a matter of debate. Some studies observe no effect while others found a dependency on sentence congruity or an independent effect. In an attempt to separate the effects of sentence congruity and word association in the spatio‐temporal domain, we jointly recorded scalp‐ and invasive‐EEG (iEEG). The latter provides highly localized spatial (unlike scalp‐EEG) and high temporal (unlike fMRI) resolutions. We recorded scalp‐ and iEEG in three patients with refractory epilepsy. The stimuli consisted of 280 sentences with crossed factors of sentence congruity and within sentence word‐association. We mapped semantic retrieval processes involved in sentence comprehension onto the left temporal cortex and both hippocampi, and showed for the first time that certain localized regions participate in the processing of word‐association in sentence context. Furthermore, simultaneous recording of scalp‐ and iEEG gave us a direct overview of signal change due to its propagation across the head tissues. HIGHLIGHTSThe brain mapping of sentence comprehension is still in progress.Invasive EEG provides the combination of high spatial and temporal resolutions.We mapped semantic processing of sentences on the left temporal cortex.Certain brain areas process word‐associations independent of sentence meaning.Scalp and invasive EEGs combined shows propagation of EEG signal through tissues.

Expectation-based Comprehension: Modeling the Interaction of World Knowledge and Linguistic Experience

ABSTRACT The processing difficulty of each word we encounter in a sentence is affected by both our prior linguistic experience and our general knowledge about the world. Computational models of incremental language processing have, however, been limited in accounting for the influence of world knowledge. We develop an incremental model of language comprehension that constructs—on a word-by-word basis—rich, probabilistic situation model representations. To quantify linguistic processing effort, we adopt Surprisal Theory, which asserts that the processing difficulty incurred by a word is inversely proportional to its expectancy (Hale, 2001; Levy, 2008). In contrast with typical language model implementations of surprisal, the proposed model instantiates a novel comprehension-centric metric of surprisal that reflects the likelihood of the unfolding utterance meaning as established after processing each word. Simulations are presented that demonstrate that linguistic experience and world knowledge are integrated in the model at the level of interpretation and combine in determining online expectations.