Emmanuel Guzman-Martinez

Characteristic sounds facilitate visual search

In a natural environment, objects that we look for often make characteristic sounds. A hiding cat may meow, or the keys in the cluttered drawer may jingle when moved. Using a visual search paradigm, we demonstrated that characteristic sounds facilitated visual localization of objects, even when the sounds carried no location information. For example, finding a cat was faster when participants heard a meow sound. In contrast, sounds had no effect when participants searched for names rather than pictures of objects. For example, hearing “meow” did not facilitate localization of the word cat. These results suggest that characteristic sounds cross-modally enhance visual (rather than conceptual) processing of the corresponding objects. Our behavioral demonstration of object-based cross-modal enhancement complements the extensive literature on space-based cross-modal interactions. When looking for your keys next time, you might want to play jingling sounds.

Interactive Coding of Visual Spatial Frequency and Auditory Amplitude-Modulation Rate

Spatial frequency is a fundamental visual feature coded in primary visual cortex, relevant for perceiving textures, objects, hierarchical structures, and scenes, as well as for directing attention and eye movements. Temporal amplitude-modulation (AM) rate is a fundamental auditory feature coded in primary auditory cortex, relevant for perceiving auditory objects, scenes, and speech. Spatial frequency and temporal AM rate are thus fundamental building blocks of visual and auditory perception. Recent results suggest that crossmodal interactions are commonplace across the primary sensory cortices and that some of the underlying neural associations develop through consistent multisensory experience such as audio-visually perceiving speech, gender, and objects. We demonstrate that people consistently and absolutely (rather than relatively) match specific auditory AM rates to specific visual spatial frequencies. We further demonstrate that this crossmodal mapping allows amplitude-modulated sounds to guide attention to and modulate awareness of specific visual spatial frequencies. Additional results show that the crossmodal association is approximately linear, based on physical spatial frequency, and generalizes to tactile pulses, suggesting that the association develops through multisensory experience during manual exploration of surfaces.

Rapid eye-fixation training without eyetracking

Maintenance of stable central eye fixation is crucial for a variety of behavioral, electrophysiological, and neuroimaging experiments. Naive observers in these experiments are not typically accustomed to fixating, either requiring the use of cumbersome and costly eyetracking or producing confounds in results. We devised a flicker display that produced an easily detectable visual phenomenon whenever the eyes moved. A few minutes of training using this display dramatically improved the accuracy of eye fixation while observers performed a demanding spatial attention cuing task. The same amount of training using control displays did not produce significant fixation improvements, and some observers consistently made eye movements to the peripheral attention cue, contaminating the cuing effect. Our results indicate that (1) eye fixation can be rapidly improved in naive observers by providing real-time feedback about eye movements, and (2) our simple flicker technique provides an easy and effective method for providing this feedback. S. Suzuki, satoru@northwestern.edu

Sounds exaggerate visual shape

While perceiving speech, people see mouth shapes that are systematically associated with sounds. In particular, a vertically stretched mouth produces a /woo/ sound, whereas a horizontally stretched mouth produces a /wee/ sound. We demonstrate that hearing these speech sounds alters how we see aspect ratio, a basic visual feature that contributes to perception of 3D space, objects and faces. Hearing a /woo/ sound increases the apparent vertical elongation of a shape, whereas hearing a /wee/ sound increases the apparent horizontal elongation. We further demonstrate that these sounds influence aspect ratio coding. Viewing and adapting to a tall (or flat) shape makes a subsequently presented symmetric shape appear flat (or tall). These aspect ratio aftereffects are enhanced when associated speech sounds are presented during the adaptation period, suggesting that the sounds influence visual population coding of aspect ratio. Taken together, these results extend previous demonstrations that visual information constrains auditory perception by showing the converse - speech sounds influence visual perception of a basic geometric feature.

Audition dominates vision in duration perception irrespective of salience, attention, and temporal discriminability

Whereas the visual modality tends to dominate over the auditory modality in bimodal spatial perception, the auditory modality tends to dominate over the visual modality in bimodal temporal perception. Recent results suggest that the visual modality dominates bimodal spatial perception because spatial discriminability is typically greater for the visual than for the auditory modality; accordingly, visual dominance is eliminated or reversed when visual-spatial discriminability is reduced by degrading visual stimuli to be equivalent or inferior to auditory spatial discriminability. Thus, for spatial perception, the modality that provides greater discriminability dominates. Here, we ask whether auditory dominance in duration perception is similarly explained by factors that influence the relative quality of auditory and visual signals. In contrast to the spatial results, the auditory modality dominated over the visual modality in bimodal duration perception even when the auditory signal was clearly weaker, when the auditory signal was ignored (i.e., the visual signal was selectively attended), and when the temporal discriminability was equivalent for the auditory and visual signals. Thus, unlike spatial perception, where the modality carrying more discriminable signals dominates, duration perception seems to be mandatorily linked to auditory processing under most circumstances.

Lip Reading Without Awareness

Speech perception is vitally important, thus the visual system may possess exceptional sensitivity to track speech-related signals conveyed by lip movements even without awareness of the speaking face. We tested this possibility using a technique called continuous flash suppression (CFS) (e.g., Tsuchiya & Koch, 2005), in which a critical stimulus, here a speaking face, was presented to one eye and strong dynamic-masking noise was presented to the other eye (Figure 1A), rendering the speaking face invisible. We determined whether the visual system could still encode spatiotemporal patterns of lip movements when observers were aware of only the randomly flashing masking display. A. Trial sequence of a masked-face (face invisible) trial. Each trial began with a fixation cross (0.36° × 0.36°) lasting 3000 ms. While the face was presented to one eye, a strong dynamic mask, consisting of a random array of ... We measured the encoding of invisible lip movements as crossmodal facilitation of spoken word categorization (e.g., Sumby & Pollack, 1954). Participants determined whether each spoken word was a target word (a tool name) or a non-target word (a name of a non-tool object) while they concurrently viewed a face that either spoke the same word—the congruent condition—or a different word—the incongruent condition. Prior research suggests that spatial attention influences unaware as well as aware visual processing (e.g., Cohen et al., 2012), and that attention to the mouth region is necessary for lip movements to facilitate spoken word perception (e.g., Alsius et al., 2005; Driver & Spence, 1994). To help direct attention to the mouth region, on half of the trials, we presented the face without the dynamic mask (Figure 1B), and we instructed participants to localize a small probe briefly presented near the mouth (Supplementary Figure S1). These “attention-enforcement” trials were randomly intermixed with the critical masked-face (face invisible) trials. To further enforce attention to the mouth region on the masked-face trials, the probe also appeared on the masked face, and participants were instructed to report its location whenever the face became visible through the mask. Participants reported seeing the face on 5% of the masked-face trials, and the data from those trials were removed from the analyses. If the visual system automatically extracts lip movements even when they are invisible, spoken-word categorization should be facilitated by congruent lip movements even when the face is invisible on the masked-face trials. Indeed, responses to the spoken target words on the masked-face trails were significantly faster when the lip movements were congruent than incongruent, t45=2.66, p 3.50, ps<0.005). To control for the possibility that participants might have failed to report face visibility on some of the masked-face trials, we performed a second experiment incorporating a more stringent indicator of face visibility. A tinted translucent ellipse was placed over the mouth region of the face. On each trial, after responding to the spoken word, participants were asked to report the color of the ellipse (red, blue, green, or yellow); critically, they were required to guess if they thought that they had not seen the face. All masked-face trials on which participants correctly reported the color (30%) were removed from analysis. The same pattern of results was obtained. On the masked-face (face invisible) trials, responses to the spoken target words were significantly faster when the lip movements were congruent than incongruent, t23=2.12, p<0.05 (Figure 1D), with a mean accuracy of 92% with no evidence of a speed-accuracy trade-off. Also consistent with the original experiment, there was no congruency effect for target responses on the attention-enforcement trials, t23=0.80, n.s. (Figure 1F) or for non-target responses (1777ms [congruent] vs. 1743ms [incongruent], t23=1.06, n.s. on the masked-face trials, and 1708ms [congruent] vs. 1779ms [incongruent], t23=1.97, n.s. on the attention-enforcement trials.). These results demonstrate that even when a speaking face is rendered invisible by a dynamic mask with strong motion signals, the visual system accurately encodes invisible lip movements to facilitate auditory perception of the corresponding spoken words. This crossmodal effect is likely to occur at the level of encoding words; it has been shown that invisible lip movements do not generate a McGurk effect (Palmer & Ramsey, 2012), suggesting that invisible lip movements do not influence auditory perception at the level of encoding syllables. Dorsal motion processing mechanisms (e.g., V3a, V5) would have predominantly responded to the strong and visible flashing mask (e.g., Moutoussis et al., 2005). The invisible lip movements would thus likely have been processed through the ventral visual pathway including the superior temporal sulcus (STS), an area that selectively responds to biological motion and movements of facial features (e.g., Allison et al., 2000; Calvert & Campbell, 2003; Grossman et al., 2000), and facilitated spoken word perception via multimodal portions of the STS (e.g., Calvert et al., 2000). Sophisticated unconscious processing of static images (e.g., words, faces, sex of human body, and contextual congruence; Jiang et al., 2006; Jiang et al., 2007; Mudrik et al., 2011; Yang et al., 2007) has been demonstrated. Our results extend these prior findings to the processing of dynamic information. Static information can theoretically be extricated from a dynamic mask by temporal averaging. However, unconscious extrication of the subtle dynamics of lip movements from the overwhelming random dynamics of the mask requires sophisticated tuning of the ventral visual system to the behaviorally relevant dynamics.

Flicker adaptation of low-level cortical visual neurons contributes to temporal dilation

Several seconds of adaptation to a flickered stimulus causes a subsequent brief static stimulus to appear longer in duration. Nonsensory factors, such as increased arousal and attention, have been thought to mediate this flicker-based temporal-dilation aftereffect. In this study, we provide evidence that adaptation of low-level cortical visual neurons contributes to this aftereffect. The aftereffect was significantly reduced by a 45° change in Gabor orientation between adaptation and test. Because orientation-tuning bandwidths are smaller in lower-level cortical visual areas and are approximately 45° in human V1, the result suggests that flicker adaptation of orientation-tuned V1 neurons contributes to the temporal-dilation aftereffect. The aftereffect was abolished when the adaptor and test stimuli were presented to different eyes. Because eye preferences are strong in V1 but diminish in higher-level visual areas, the eye specificity of the aftereffect corroborates the involvement of low-level cortical visual neurons. Our results suggest that flicker adaptation of low-level cortical visual neurons contributes to expanding visual duration. Furthermore, this temporal-dilation aftereffect dissociates from the previously reported temporal-compression aftereffect on the basis of the differences in their orientation and flicker-frequency selectivity, suggesting that the visual system possesses at least two distinct and potentially complementary mechanisms for adaptively coding perceived duration.

A unique role of endogenous visual-spatial attention in rapid processing of multiple targets

Visual spatial attention can be exogenously captured by a salient stimulus or can be endogenously allocated by voluntary effort. Whether these two attention modes serve distinctive functions is debated, but for processing of single targets the literature suggests superiority of exogenous attention (it is faster acting and serves more functions). We report that endogenous attention uniquely contributes to processing of multiple targets. For speeded visual discrimination, response times are faster for multiple redundant targets than for single targets because of probability summation and/or signal integration. This redundancy gain was unaffected when attention was exogenously diverted from the targets but was completely eliminated when attention was endogenously diverted. This was not a result of weaker manipulation of exogenous attention because our exogenous and endogenous cues similarly affected overall response times. Thus, whereas exogenous attention is superior for processing single targets, endogenous attention plays a unique role in allocating resources crucial for rapid concurrent processing of multiple targets.

Direction of Auditory Pitch-Change Influences Visual Search for Slope From Graphs:

Linear trend (slope) is important information conveyed by graphs. We investigated how sounds influenced slope detection in a visual search paradigm. Four bar graphs or scatter plots were presented on each trial. Participants looked for a positive-slope or a negative-slope target (in blocked trials), and responded to targets in a go or no-go fashion. For example, in a positive-slope-target block, the target graph displayed a positive slope while other graphs displayed negative slopes (a go trial), or all graphs displayed negative slopes (a no-go trial). When an ascending or descending sound was presented concurrently, ascending sounds slowed detection of negative-slope targets whereas descending sounds slowed detection of positive-slope targets. The sounds had no effect when they immediately preceded the visual search displays, suggesting that the results were due to crossmodal interaction rather than priming. The sounds also had no effect when targets were words describing slopes, such as “positive,” “negative,” “increasing,” or “decreasing,” suggesting that the results were unlikely due to semantic-level interactions. Manipulations of spatiotemporal similarity between sounds and graphs had little effect. These results suggest that ascending and descending sounds influence visual search for slope based on a general association between the direction of auditory pitch-change and visual linear trend.

Spatial position influences perception of slope from graphs

We routinely examine linear trends from bar graphs and scatterplots while taking a science class, attending a business presentation, or reading a magazine article. Graphs are placed in different positions on a page or a presentation slide for aesthetic considerations. However, because left and right positions tend to be associated with lower and higher values in the conventional depiction of numerical values, we hypothesized that the perception of positive and negative slopes may be influenced by the placement of a graph. Using a visual search task, with each display containing four bar graphs or scatterplots (one per quadrant), we have demonstrated that the detection of a negative slope is selectively slowed in the upper-right quadrant (for both bar graphs and scatterplots), whereas the detection of a positive slope is selectively slowed in the upper-left quadrant (for bar graphs only). These results suggest that an upper-right position is incompatible with perceiving negative slopes and an upper-left position is incompatible with perceiving positive slopes. Although the origin of these specific associations is unclear, our results have implications for where to place a graph depending on the slope it displays.