Thomas J. Perrault

On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies

A growing number of brain imaging studies are being undertaken in order to better understand the contributions of multisensory processes to human behavior and perception. Many of these studies are designed on the basis of the physiological findings from single neurons in animal models, which have shown that multisensory neurons have the capacity for integrating their different sensory inputs and give rise to a product that differs significantly from either of the unisensory responses. At certain points these multisensory interactions can be superadditive, resulting in a neural response that exceeds the sum of the unisensory responses. Because of the difficulties inherent in interpreting the results of imaging large neuronal populations, superadditivity has been put forth as a stringent criterion for identifying potential sites of multisensory integration. In the present manuscript we discuss issues related to using the superadditive model in human brain imaging studies, focusing on population responses to multisensory stimuli and the relationship between single neuron measures and functional brain imaging measures. We suggest that the results of brain imaging studies be interpreted with caution in regards to multisensory integration. Future directions for imaging multisensory integration are discussed in light of the ideas presented.

Visual Experience Is Necessary for the Development of Multisensory Integration

Multisensory neurons and their ability to integrate multisensory cues develop gradually in the midbrain [i.e., superior colliculus (SC)]. To examine the possibility that early sensory experiences might play a critical role in these maturational processes, animals were raised in the absence of visual cues. As adults, the SC of these animals were found to contain many multisensory neurons, the large majority of which were visually responsive. Although these neurons responded robustly to each of their cross-modal inputs when presented individually, they were incapable of synthesizing this information. These observations suggest that visual experiences are critical for the SC to develop the ability to integrate multisensory information and lead to the prediction that, in the absence of such experience, animals will be compromised in their sensitivity to cross-modal events.

Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness

Single-neuron studies provide a foundation for understanding many facets of multisensory integration. These studies have used a variety of criteria for identifying and quantifying multisensory integration. While a number of techniques have been used, an explicit discussion of the assumptions, criteria, and analytical methods traditionally used to define the principles of multisensory integration is lacking. This was not problematic when the field was small, but with rapid growth a number of alternative techniques and models have been introduced, each with its own criteria and sets of implicit assumptions to define and characterize what is thought to be the same phenomenon. The potential for misconception prompted this reexamination of traditional approaches in order to clarify their underlying assumptions and analytic techniques. The objective here is to review and discuss traditional quantitative methods advanced in the study of single-neuron physiology in order to appreciate the process of multisensory integration and its impact.

The Development of Cortical Multisensory Integration

Although there are many perceptual theories that posit particular maturational profiles in higher-order (i.e., cortical) multisensory regions, our knowledge of multisensory development is primarily derived from studies of a midbrain structure, the superior colliculus. Therefore, the present study examined the maturation of multisensory processes in an area of cat association cortex [i.e., the anterior ectosylvian sulcus (AES)] and found that these processes are rudimentary during early postnatal life and develop only gradually thereafter. The AES comprises separate visual, auditory, and somatosensory regions, along with many multisensory neurons at the intervening borders between them. During early life, sensory responsiveness in AES appears in an orderly sequence. Somatosensory neurons are present at 4 weeks of age and are followed by auditory and multisensory (somatosensory–auditory) neurons. Visual neurons and visually responsive multisensory neurons are first seen at 12 weeks of age. The earliest multisensory neurons are strikingly immature, lacking the ability to synthesize the cross-modal information they receive. With postnatal development, multisensory integrative capacity matures. The delayed maturation of multisensory neurons and multisensory integration in AES suggests that the higher-order processes dependent on these circuits appear comparatively late in ontogeny.

Semantic confusion regarding the development of multisensory integration: a practical solution

There is now a good deal of data from neurophysiological studies in animals and behavioral studies in human infants regarding the development of multisensory processing capabilities. Although the conclusions drawn from these different datasets sometimes appear to conflict, many of the differences are due to the use of different terms to mean the same thing and, more problematic, the use of similar terms to mean different things. Semantic issues are pervasive in the field and complicate communication among groups using different methods to study similar issues. Achieving clarity of communication among different investigative groups is essential for each to make full use of the findings of others, and an important step in this direction is to identify areas of semantic confusion. In this way investigators can be encouraged to use terms whose meaning and underlying assumptions are unambiguous because they are commonly accepted. Although this issue is of obvious importance to the large and very rapidly growing number of researchers working on multisensory processes, it is perhaps even more important to the non‐cognoscenti. Those who wish to benefit from the scholarship in this field but are unfamiliar with the issues identified here are most likely to be confused by semantic inconsistencies. The current discussion attempts to document some of the more problematic of these, begin a discussion about the nature of the confusion and suggest some possible solutions.

Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration

The superior colliculus (SC) plays an important role in integrating visual, auditory and somatosensory information, and in guiding the orientation of the eyes, ears and head. Previously we have shown that cats with unilateral SC lesions showed a preferential loss of multisensory orientation behaviors for stimuli contralateral to the lesion. Surprisingly, this behavioral loss was seen even under circumstances where the SC lesion was far from complete. To assess the physiological changes induced by these lesions, we employed single unit electrophysiological methods to record from individual neurons in both the intact and damaged SC following behavioral testing in two animals. In the damaged SC of these animals, multisensory neurons were preferentially reduced in incidence, comprising less than 25% of the sensory-responsive population (as compared with 49% on the control side). In those multisensory neurons that remained following the lesion, receptive fields were nearly twofold larger, and less than 25% showed normal patterns of multisensory integration, with those that did being found in areas outside of the lesion. These results strongly suggest that the multisensory behavioral deficits seen following SC lesions are the combined result of a loss of multisensory neurons and a loss of multisensory integration in those neurons that remain.

Postnatal experiences influence how the brain integrates information from different senses.

Sensory processing disorder (SPD) is characterized by anomalous reactions to, and integration of, sensory cues. Although the underlying etiology of SPD is unknown, one brain region likely to reflect these sensory and behavioral anomalies is the superior colliculus (SC), a structure involved in the synthesis of information from multiple sensory modalities and the control of overt orientation responses. In the present review we describe normal functional properties of this structure, the manner in which its individual neurons integrate cues from different senses, and the overt SC-mediated behaviors that are believed to manifest this “multisensory integration.” Of particular interest here is how SC neurons develop their capacity to engage in multisensory integration during early postnatal life as a consequence of early sensory experience, and the intimate communication between cortex and the midbrain that makes this developmental process possible.

Non-stationarity in multisensory neurons in the superior colliculus.

The superior colliculus (SC) integrates information from multiple sensory modalities to facilitate the detection and localization of salient events. The efficacy of “multisensory integration” is traditionally measured by comparing the magnitude of the response elicited by a cross-modal stimulus to the responses elicited by its modality-specific component stimuli, and because there is an element of randomness in the system, these calculations are made using response values averaged over multiple stimulus presentations in an experiment. Recent evidence suggests that multisensory integration in the SC is highly plastic and these neurons adapt to specific anomalous stimulus configurations. This raises the question whether such adaptation occurs during an experiment with traditional stimulus configurations; that is, whether the state of the neuron and its integrative principles are the same at the beginning and end of the experiment, or whether they are altered as a consequence of exposure to the testing stimuli even when they are pseudo-randomly interleaved. We find that unisensory and multisensory responses do change during an experiment, and that these changes are predictable. Responses that are initially weak tend to potentiate, responses that are initially strong tend to habituate, and the efficacy of multisensory integration waxes or wanes accordingly during the experiment as predicted by the “principle of inverse effectiveness.” These changes are presumed to reflect two competing mechanisms in the SC: potentiation reflects increases in the expectation that a stimulus will occur at a given location relative to others, and habituation reflects decreases in stimulus novelty. These findings indicate plasticity in multisensory integration that allows animals to adapt to rapidly changing environmental events while suggesting important caveats in the interpretation of experimental data: the neuron studied at the beginning of an experiment is not the same at the end of it.