Neural dynamics of semantic composition

Abstract

Significance The rapid comprehension of speech is a remarkable but poorly understood human capacity. Central to this process is the integration of the meaning of each word, as it is heard, into the listener’s interpretation of the utterance. Here we focus on the real-time flow of neural activity that underpins this combinatorial process, using multivariate pattern analysis and computational semantic models to discover the contextual constraints that are constructed as each word is heard, and to determine how these constraints guide the interpretation of future words in the utterance. This combination of methods reveals a continuous information flow across the left-hemisphere language system, strongly constraining the immediate activation of word meanings and providing a neural substrate for seamless real-time speech comprehension. Human speech comprehension is remarkable for its immediacy and rapidity. The listener interprets an incrementally delivered auditory input, millisecond by millisecond as it is heard, in terms of complex multilevel representations of relevant linguistic and nonlinguistic knowledge. Central to this process are the neural computations involved in semantic combination, whereby the meanings of words are combined into more complex representations, as in the combination of a verb and its following direct object (DO) noun (e.g., “eat the apple”). These combinatorial processes form the backbone for incremental interpretation, enabling listeners to integrate the meaning of each word as it is heard into their dynamic interpretation of the current utterance. Focusing on the verb-DO noun relationship in simple spoken sentences, we applied multivariate pattern analysis and computational semantic modeling to source-localized electro/magnetoencephalographic data to map out the specific representational constraints that are constructed as each word is heard, and to determine how these constraints guide the interpretation of subsequent words in the utterance. Comparing context-independent semantic models of the DO noun with contextually constrained noun models reflecting the semantic properties of the preceding verb, we found that only the contextually constrained model showed a significant fit to the brain data. Pattern-based measures of directed connectivity across the left hemisphere language network revealed a continuous information flow among temporal, inferior frontal, and inferior parietal regions, underpinning the verb’s modification of the DO noun’s activated semantics. These results provide a plausible neural substrate for seamless real-time incremental interpretation on the observed millisecond time scales.