Mirko Grimaldi

Toward a neural theory of language: Old issues and new perspectives

Abstract The cognitive neuroscience of language is an exciting interdisciplinary perspective that suffers from unresolved epistemological and methodological issues. Despite the impressive amount of neural evidence accumulated until now, the field of research results fragmented and it is quite difficult to reach a unit of analysis and consensus on the object of study. This frustrating state of the art results in a detrimental reductionism consisting in the practice of associating linguistic computation hypothesized at theoretical level with neurobiological computation. However, these two entities are at the moment ontologically incommensurable. The problem lies in the fact that a theory of language consistent with a range of neurophysiological and neuroimaging techniques of investigation and verifiable through neural data is still lacking. In this article, I focus on the main issues, questions, and concerns that prevent the integrated study of language and brain and I explore a feasible way for linguistics to pursue a theory susceptible of neuroscientific testability in the light of recent neurocognitive models and of data on the functional-anatomic organization of language in the brain. Finally, I discuss a possible interdisciplinary program in order to achieve a theory capable of predictions on the real-time neural constrains characterizing the biological bases of language.

The use of phonetic motor invariants can improve automatic phoneme discrimination.

We investigate the use of phonetic motor invariants (MIs), that is, recurring kinematic patterns of the human phonetic articulators, to improve automatic phoneme discrimination. Using a multi-subject database of synchronized speech and lips/tongue trajectories, we first identify MIs commonly associated with bilabial and dental consonants, and use them to simultaneously segment speech and motor signals. We then build a simple neural network-based regression schema (called Audio-Motor Map, AMM) mapping audio features of these segments to the corresponding MIs. Extensive experimental results show that a small set of features extracted from the MIs, as originally gathered from articulatory sensors, are dramatically more effective than a large, state-of-the-art set of audio features, in automatically discriminating bilabials from dentals; the same features, extracted from AMM-reconstructed MIs, are as effective as or better than the audio features, when testing across speakers and coarticulating phonemes; and dramatically better as noise is added to the speech signal. These results seem to support some of the claims of the motor theory of speech perception and add experimental evidence of the actual usefulness of MIs in the more general framework of automated speech recognition.

A dataset of metaphors from the italian literature: exploring psycholinguistic variables and the role of context.

Defining the specific role of the factors that affect metaphor processing is a fundamental step for fully understanding figurative language comprehension, either in discourse and conversation or in reading poems and novels. This study extends the currently available materials on everyday metaphorical expressions by providing the first dataset of metaphors extracted from literary texts and scored for the major psycholinguistic variables, considering also the effect of context. A set of 115 Italian literary metaphors presented in isolation (Experiment 1) and a subset of 65 literary metaphors embedded in their original texts (Experiment 2) were rated on several dimensions (word and phrase frequency, readability, cloze probability, familiarity, concreteness, difficulty and meaningfulness). Overall, literary metaphors scored around medium-low values on all dimensions in both experiments. Collected data were subjected to correlation analysis, which showed the presence of a strong cluster of variables—mainly familiarity, difficulty, and meaningfulness—when literary metaphor were presented in isolation. A weaker cluster was observed when literary metaphors were presented in the original contexts, with familiarity no longer correlating with meaningfulness. Context manipulation influenced familiarity, concreteness and difficulty ratings, which were lower in context than out of context, while meaningfulness increased. Throughout the different dimensions, the literary context seems to promote a global interpretative activity that enhances the open-endedness of the metaphor as a semantic structure constantly open to all possible interpretations intended by the author and driven by the text. This dataset will be useful for the design of future experimental studies both on literary metaphor and on the role of context in figurative meaning, combining ecological validity and aesthetic aspects of language.

Numerical instabilities and three-dimensional electromagnetic articulography

The AG500 electromagnetic articulograph is widely used to reconstruct the movements of the articulatory organs. Nevertheless, some anomalies in its performance have been observed. It is well known that accuracy of the device is affected by electromagnetic interference and possible hardware failures or damage to the sensors. In this study, after eliminating any hardware or electromagnetic source of disturbance, a set of trials was carried out. The tests prove that anomalies in sensor position tracking are systematic in certain regions within the recording volume and, more importantly, show a specific pattern that can be clearly attributed to a wrong convergence of the calculation method.

Assimilation of L2 vowels to L1 phonemes governs L2 learning in adulthood: a behavioral and ERP study

According to the Perceptual Assimilation Model (PAM), articulatory similarity/dissimilarity between sounds of the second language (L2) and the native language (L1) governs L2 learnability in adulthood and predicts L2 sound perception by naïve listeners. We performed behavioral and neurophysiological experiments on two groups of university students at the first and fifth years of the English language curriculum and on a group of naïve listeners. Categorization and discrimination tests, as well as the mismatch negativity (MMN) brain response to L2 sound changes, showed that the discriminatory capabilities of the students did not significantly differ from those of the naïve subjects. In line with the PAM model, we extend the findings of previous behavioral studies showing that, at the neural level, classroom instruction in adulthood relies on assimilation of L2 vowels to L1 phoneme categories and does not trigger improvement in L2 phonetic discrimination. Implications for L2 classroom teaching practices are discussed.

Vowels and Consonants in the Brain: Evidence from Magnetoencephalographic Studies on the N1m in Normal-Hearing Listeners

Speech sound perception is one of the most fascinating tasks performed by the human brain. It involves a mapping from continuous acoustic waveforms onto the discrete phonological units computed to store words in the mental lexicon. In this article, we review the magnetoencephalographic studies that have explored the timing and morphology of the N1m component to investigate how vowels and consonants are computed and represented within the auditory cortex. The neurons that are involved in the N1m act to construct a sensory memory of the stimulus due to spatially and temporally distributed activation patterns within the auditory cortex. Indeed, localization of auditory fields maps in animals and humans suggested two levels of sound coding, a tonotopy dimension for spectral properties and a tonochrony dimension for temporal properties of sounds. When the stimulus is a complex speech sound, tonotopy and tonochrony data may give important information to assess whether the speech sound parsing and decoding are generated by pure bottom-up reflection of acoustic differences or whether they are additionally affected by top-down processes related to phonological categories. Hints supporting pure bottom-up processing coexist with hints supporting top-down abstract phoneme representation. Actually, N1m data (amplitude, latency, source generators, and hemispheric distribution) are limited and do not help to disentangle the issue. The nature of these limitations is discussed. Moreover, neurophysiological studies on animals and neuroimaging studies on humans have been taken into consideration. We compare also the N1m findings with the investigation of the magnetic mismatch negativity (MMNm) component and with the analogous electrical components, the N1 and the MMN. We conclude that N1 seems more sensitive to capture lateralization and hierarchical processes than N1m, although the data are very preliminary. Finally, we suggest that MEG data should be integrated with EEG data in the light of the neural oscillations framework and we propose some concerns that should be addressed by future investigations if we want to closely line up language research with issues at the core of the functional brain mechanisms.

Time Course and Neurophysiological Underpinnings of Metaphor in Literary Context

ABSTRACT Several theoretical proposals tried to account for the meaning open-endedness of metaphors in literature and for the effortful process they trigger in readers. However, very few experiments have tackled the neurophysiological underpinnings of literary metaphor. Here we used Event-Related brain Potentials (ERPs) to explore the temporal dynamics of comprehending metaphors from Italian poems and novels (e.g., grass of velvet), presented in their original context, as compared with literal expressions (e.g., throne of velvet). Results evidence a more negative ERP response for metaphors, unfolding in an N400 followed by a sustained negativity over frontal sites, suggesting a long-lasting effort in elaborating figurative meanings. Whereas the N400 might be indicative of lexical/semantic processes typical of metaphors and amplified by the literary context, the sustained negativity might reflect the manipulation of multiple meanings in working memory, possibly responsible for the poetic effect. Interestingly, the late negativity effect is driven by familiarity, with a more negative response for those metaphors that are less familiar. These findings offer material to discuss ideas put forward in pragmatics, literary studies, and cognitive neuroscience of literature, like the condensation of weak implicatures, the foregrounding, and the relation between a metaphor and its context.

Future perspectives in neurobiological investigation of language

Studying language as an object of the biological world requires the resolution of the mind-brain problem. While contemporary theoretical linguistics has addressed the problem adopting a dualistic approach (in which the representational and algorithmic nature of linguistic knowledge can be investigated independently by brain activity), cognitive neuroscience has privileged an anti-dualistic perspective (in which the direct observation of the brain can reveal the higher-level cognitive properties of the language faculty). These different epistemological views generated incommensurable ontologies that at the moment prevent the fertile integration of linguistics and cognitive neuroscience. The aim of this special issue is to redraw attention to unresolved shortcomings adopting an interdisciplinary perspective by comparing different research traditions, approaches and methods. The contributions come from linguistics, cognitive neuroscience, neurophysiology, computational neuroscience and computer science and discussed topics related with different aspects of the language faculty. We have tried to blend experimental works with theoretical contributions from linguistics addressing questions that can fruitfully join experimental evidence with abstract theorization. We conclude by outlining some future scenarios under the theme of integration that, although stimulated on the basis of the linguistic-cognitive neuroscience interface, represent general challenges for all interdisciplinary approaches in cognitive neuroscience.

Assessing the position tracking reliability of Carstens’ AG500 and AG501 electromagnetic articulographs during constrained movements and speech tasks

Abstract The goal of this study is to qualitatively and quantitatively assess the reliability of the AG501 Electromagnetic Articulograph (Carstens Medizinelektronik GmbH) and to compare it with the previous model, the AG500, which is still widely used in the 3D recording of articulatory movements. To explore and test the spatial accuracy of the articulographs in various areas of the recording volume, controlled sensor positions at fixed locations and along circular trajectories as well as movements of articulators of the speakers vocal tract during speech production tasks were tracked and analysed. It is well known that the AG500 trajectories are affected by perturbations which: (i) depend on the position of the sensors/subject within the recording volume and (ii) randomly occur along repetitions of the same sound in the same recording area. This study has shown that these issues do not affect the newer AG501, which not only performs according to the manufacturers claim of 0.3 mm dynamical accuracy within a 20-cm-wide spherical region inside the recording volume, but also performs well outside. Furthermore, while the AG500 shows perturbed trajectories in some instances, the AG501 consistently shows accurate results in reproducing the displacements of consonantal and vocalic gestures for the tested speech tasks. Our findings reveal that the AG501 is more stable and significantly more accurate than the previous model, the AG500, which, in turn, performs reasonably well only in specific limited conditions.

From brain noise to syntactic structures: A formal proposal within the oscillatory rhythms perspective

The neurobiology investigation of language seems limited by the impossibility to link directly linguistic computations with neural computations. To address this issue, we need to explore the hierarchical interconnections between the investigated fields trying to develop an inter-field theory. Considerable research has realized that event-related fluctuations in rhythmic, oscillatory EEG/MEG activity may provide a new window on the dynamics of functional neuronal networks involved in cognitive processing. Accordingly, this paper aims to outline a formal proposal on neuronal computation and representation of syntactic structures within the oscillatory neuronal dynamics. I briefly present the nature of event-related oscillations and how they work on the base of synchronization and de-synchronization processes. Then, I discuss some theoretical premises assuming that reentrant (hierarchical) properties of synchronized oscillatory rhythms constitute the biological endowment that allow the development of language in humans when exposed to appropriate inputs. The main rhythms involved in language and speech processing are examined: i.e. theta, alpha, beta, and gamma bands. A possible formal representation of the syntactic structures on the base of these oscillatory rhythms is discussed: in this model, the theta-gamma rhythms are cross-frequency coupled into the alpha-gamma-beta and into the gamma-beta-theta rhythms to generate the sentence along reentrant cortico-thalamic pathways through Merge, Label and Move operations. Finally, I present few conclusive remarks within an evolutionary perspective. “…The whole burden of philosophy seems to consist in this, from the phenomena of motions to investigate the forces of nature, and from these forces to demonstrate the other phenomena.” (Newton, 1687/1726)