Ophelia Deroy

Why we are not all synesthetes (not even weakly so)

A little over a decade ago, Martino and Marks (Current Directions in Psychological Science 10:61–65, 2001) put forward the influential claim that cases of intuitive matchings between stimuli in different sensory modalities should be considered as a weak form of synesthesia. Over the intervening years, many other researchers have agreed—at the very least, implicitly—with this position (e.g., Bien, ten Oever, Goebel, & Sack NeuroImage 59:663–672, 2012; Eagleman Cortex 45:1266–1277, 2009; Esterman, Verstynen, Ivry, & Robertson Journal of Cognitive Neuroscience 18:1570–1576, 2006; Ludwig, Adachi, & Matzuzawa Proceedings of the National Academy of Sciences of the United States of America 108:20661–20665, 2011; Mulvenna & Walsh Trends in Cognitive Sciences 10:350–352, 2006; Sagiv & Ward 2006; Zellner, McGarry, Mattern-McClory, & Abreu Chemical Senses 33:211–222:2008). Here, though, we defend the separatist view, arguing that these cases are likely to form distinct kinds of phenomena despite their superficial similarities. We believe that crossmodal correspondences should be studied in their own right and not assimilated, either in terms of the name used or in terms of the explanation given, to synesthesia. To conflate these two phenomena is both inappropriate and potentially misleading. Below, we critically evaluate the evidence concerning the descriptive and constitutive features of crossmodal correspondences and synesthesia and highlight how they differ. Ultimately, we wish to provide a general definition of crossmodal correspondences as acquired, malleable, relative, and transitive pairings between sensory dimensions and to provide a framework in which to integrate the nonsystematic cataloguing of new cases of crossmodal correspondences, a tendency that has increased in recent years.

Crossmodal correspondences between odors and contingent features: odors, musical notes, and geometrical shapes

Olfactory experiences represent a domain that is particularly rich in crossmodal associations. Whereas associations between odors and tastes, or other properties of their typical sources such as color or temperature, can be straightforwardly explained by associative learning, other matchings are much harder to explain in these terms, yet surprisingly are shared across individuals: The majority of people, for instance, associate certain odors and auditory features, such as pitch (Belkin, Martin, Kemp, & Gilbert, Psychological Science 8:340–342, 1997; Crisinel & Spence, Chemical Senses 37:151–158, 2012b) or geometrical shapes (Hanson-Vaux, Crisinel, & Spence, Chemical Senses 38:161–166, 2013; Seo, Arshamian, et al., Neuroscience Letters 478:175–178, 2010). If certain odors might indeed have been encountered while listening to certain pieces of music or seeing certain geometrical shapes, these encounters are very unlikely to have been statistically more relevant than others; for this reason, associative learning from regular exposure is ruled out, and thus alternative explanations in terms of metaphorical mappings are usually defended. Here we argue that these associations are not primarily conceptual or linguistic, but are grounded in structural perceptual or neurological determinants. These cases of crossmodal correspondences established between contingent environmental features can be explained as amodal, indirect, and transitive mappings across modalities. Surprising associations between odors and contingent sensory features can be investigated as genuine cases of crossmodal correspondences, akin to other widespread cases of functional correspondences that hold, for instance, between auditory and visual features, and can help reveal the structural determinants weighing on the acquisition of these crossmodal associations.

As Light as your Footsteps: Altering Walking Sounds to Change Perceived Body Weight, Emotional State and Gait

An ever more sedentary lifestyle is a serious problem in our society. Enhancing peoples exercise adherence through technology remains an important research challenge. We propose a novel approach for a system supporting walking that draws from basic findings in neuroscience research. Our shoe-based prototype senses a persons footsteps and alters in real-time the frequency spectra of the sound they produce while walking. The resulting sounds are consistent with those produced by either a lighter or heavier body. Our user study showed that modified walking sounds change ones own perceived body weight and lead to a related gait pattern. In particular, augmenting the high frequencies of the sound leads to the perception of having a thinner body and enhances the motivation for physical activity inducing a more dynamic swing and a shorter heel strike. We here discuss the opportunities and the questions our findings open.

On tasty colours and colourful tastes? Assessing, explaining, and utilizing crossmodal correspondences between colours and basic tastes

Can basic tastes, such as sweet, sour, bitter, salty, and possibly also umami, be conveyed by means of colour? If so, how should we understand the relationship between colours and tastes: Is it universal or relative, innate or acquired, unidirectional or bidirectional? Here, we review the growing body of scientific research showing that people systematically associate specific colours with particular tastes. We highlight how these widely shared bidirectional crossmodal correspondences generalize across cultures and stress their difference from synaesthesia (with which they are often confused). The various explanations that have been put forward to account for such crossmodal mappings are then critically evaluated. Finally, we go on to look at some of the innovative ways in which chefs, culinary artists, designers, and marketers are taking—or could potentially push further—the latest insights from research in this area as inspiration for their own creative endeavours.

Crossmodal correspondences: Innate or learned?

“Are Chimpanzees synaesthetic?” An affirmative answer to this question appeared recently in a Nature commentary on a study by Ludwig, Adachi, and Matzuzawa (2011) that demonstrated crossmodal correspondences in both chimpanzees and humans. Here we question the claim that chimpanzees are synaesthetic. We also question the claim that certain crossmodal correspondences are innate. We suggest an alternative account for the crossmodal correspondence between auditory pitch and visual lightness in terms of the internalization of correlations present in the environment. We highlight the limitations of such natural correlation approaches to the study of crossmodal correspondences as well as how such claims could potentially be tested in future research.

Multisensory constraints on awareness

Given that multiple senses are often stimulated at the same time, perceptual awareness is most likely to take place in multisensory situations. However, theories of awareness are based on studies and models established for a single sense (mostly vision). Here, we consider the methodological and theoretical challenges raised by taking a multisensory perspective on perceptual awareness. First, we consider how well tasks designed to study unisensory awareness perform when used in multisensory settings, stressing that studies using binocular rivalry, bistable figure perception, continuous flash suppression, the attentional blink, repetition blindness and backward masking can demonstrate multisensory influences on unisensory awareness, but fall short of tackling multisensory awareness directly. Studies interested in the latter phenomenon rely on a method of subjective contrast and can, at best, delineate conditions under which individuals report experiencing a multisensory object or two unisensory objects. As there is not a perfect match between these conditions and those in which multisensory integration and binding occur, the link between awareness and binding advocated for visual information processing needs to be revised for multisensory cases. These challenges point at the need to question the very idea of multisensory awareness.

Composing with cross-modal correspondences: Music and odors in concert

We report two experiments designed to investigate cross-modal correspondences between a range of seven olfactory stimuli and both the pitch and instrument class of sounds as well as the angularity of visually presented shapes. The results revealed that odors were preferentially matched to musical features: For example, the odors of candied orange and iris flower were matched to significantly higher pitches than the odors of musk and roasted coffee. Meanwhile, the odor of crème brûlée was associated with a more rounded shape than the musk odor. Moreover, by simultaneously testing cross-modal correspondences between olfactory stimuli and matches in two other modalities, we were able to compare the ratings associated with each correspondence. Stimuli judged as happier, more pleasant, and sweeter tended to be associated to both higher pitch and a more rounded shape, whereas other ratings seemed to be more specifically correlated with the choice of either pitch or shape. Odors rated as more arousing tended to be associated with the angular shape, but not with a particular pitch; odors judged as brighter were associated with higher pitch and, to a lesser extent, rounder shapes. In a follow-up experiment, we investigated whether people could match specific pieces of music (composed to represent odors) to three of the odors (candied orange, crème brûlée, and ginger cookies). In one case (candied orange), a majority of the participants matched the odor to the intended piece of music. In another case (ginger cookies), another piece of music (than the one intended) was preferred. Finally, in the third case (crème brûlée), people showed no preference in matching the odor to the pieces of music. Both theoretical and practical implications of these results are discussed.

Reading the World through the Skin and Ears: A New Perspective on Sensory Substitution.

Sensory substitution devices aim at replacing or assisting one or several functions of a deficient sensory modality by means of another sensory modality. Despite the numerous studies and research programs devoted to their development and integration, sensory substitution devices have failed to live up to their goal of allowing one to “see with the skin” (White et al., 1970) or to “see with the brain” (Bach-y-Rita et al., 2003). These somewhat peremptory claims, as well as the research conducted so far, are based on an implicit perceptual paradigm. Such perceptual assumption accepts the equivalence between using a sensory substitution device and perceiving through a particular sensory modality. Our aim is to provide an alternative model, which defines sensory substitution as being closer to culturally implemented cognitive extensions of existing perceptual skills such as reading. In this article, we will show why the analogy with reading provides a better explanation of the actual findings, that is, both of the positive results achieved and of the limitations noticed across the field of research on sensory substitution. The parallel with the most recent two-route and interactive models of reading (e.g., Dehaene et al., 2005) generates a radically new way of approaching these results, by stressing the dependence of integration on the existing perceptual-semantic route. In addition, the present perspective enables us to generate innovative research questions and specific predictions which set the stage for future work.

Plating manifesto (II): the art and science of plating

It is well known that people serve themselves more, not to mention eat more, when dining from larger bowls and plates than from smaller ones. But what about the other visual qualities of the plateware? Does the colour, shape and finish also influence a diner’s behaviour? How important are these extrinsic visual properties, or even the visual arrangement of the elements on the plate itself, in terms of modulating a diner’s eating behaviours and experiences? At a time when so much is known about the influence of the colour of individual food products on taste and flavour perception, and when so many modernist restaurants are using an increasingly eclectic range of visual designs for their dishes, there has been surprisingly little scientific research on how the more complex visual properties and arrangement of food presentations may affect the diner. Below, we argue that the exploration of these effects constitutes the next natural step in an increasingly fruitful interdisciplinary collaboration between chefs, psychologists, sensory scientists and designers. The most important research questions, then, are to identify the kinds of effects that the presentation style has on the multisensory consumption experiences and behaviours of diners, and to study the interactions between the different visual cues that are provided. Taken together, the evidence reviewed here helps to emphasize the fact that getting both the plateware and the plating right constitute surprisingly important components to sublimate the flavours of the food, in the delivery and experience of a great meal.

The plating manifesto (I): from decoration to creation

At a time when a growing number of chefs and innovative food industries are starting to set up their own research kitchens and work with renowned scientists, it is surprising to see that issues related to the visual presentation of food on the plate are being left out of these successful exchanges. The variety of presentations created by chefs, and the number of varieties of tableware now available to achieve them, represent a formidable opportunity for cognitive scientists to study the more complex effects of vision on food experiences, which certainly should not be missed. Chefs can also benefit from the new insights that a scientific approach can bring to these areas, which previously have often been left to intuition. In this manifesto, we claim that this transfer of knowledge represents much more than merely another addition to the art and science of cuisine: it is its essential completion, as gastronomy moves more and more toward the ideal of a total multisensory art, as captivating for the eye as it is for the palate. Before turning to the scientific recommendations and review in the second part of our manifesto, we want to promote a different approach to plating, which breaks with the more functional and decorative purposes of plate ware, and puts experiments in visual presentation at the heart of modernist culinary expression.