Jeesun Kim

Task effects in masked cross-script translation and phonological priming

Abstract This study examined cross-language masked priming with Korean–English unbalanced bilinguals. The aim was to determine if the type of prime and target relationship and the type of response task would modulate across-language priming effects. The nature of the relationship between prime (Korean: L1) and target (English: L2) pairs was manipulated so that they shared semantics and phonology (cognate translations), semantics only (noncognate translations), phonology only (homophones), or neither phonology nor semantics (baseline). These prime types were tested in three different response tasks, i.e., lexical decision, naming, and semantic categorization. In the lexical decision task (Experiment 1), significant priming from cognate and noncognate translation primes was observed. Homophone primes did not produce a significant priming effect. In the naming task (Experiment 2), both cognate and homophone primes produced significant priming effects but the noncognate translation primes did not. Experiment 3 replicated the homophone priming effect and showed that it was unlikely to be due to the shared initial phonemes of primes and targets. Finally, the semantic categorization task (Experiment 4) showed cognate and noncognate translation priming but not homophone priming. The results indicated that priming was affected both by prime–target relationship and by task type; this outcome was discussed in terms of the regulation of lexical information by a task-decision system.

Investigating the audio-visual speech detection advantage

Abstract Seeing the moving face of the talker permits better detection of speech in noise compared to auditory only presentation, an Audio–Visual (AV) facilitation effect. Three experiments that used a masked speech detection task are reported. The experiments were designed to contrast two accounts of the AV facilitation effect (AV peak listening and AV grouping). In each experiment a different manipulation of the relationship between the auditory and visual signals was employed. The first experiment manipulated the sequencing of the visual and auditory information by presenting the displays time normal or time reversed. The results showed that AV facilitation only occurred for the time normal presentations where there was a high correlation between the AV signals. Experiment 2 examined the impact on AV facilitation of shifting the auditory signals earlier in time than its normal position (again with time normal and time reversed presentation). It was found that shifting the auditory component abolished the AV effect for both the time normal and reversed conditions. The final experiment examined the AV detection advantage using another situation in which the relationship between the AV signals differed. Two versions of AV speech produced by a virtual talker were investigated. In one version, based on text-to-speech synthesis, the video and auditory signals were more rapid than the human talker of Experiment 1. In the other version, the signals were lengthened to match the durations of the human talker. A small but reliable AV facilitation effect was only found for the second version. The results are consistent with a cross-modal peak listening account and are discussed in terms of constraints on the integration of auditory and visual speech.

Audio-visual interactions with intact clearly audible speech

The effects of viewing the face of the talker (visual speech) on the processing of clearly presented intact auditory stimuli were investigated using two measures likely to be sensitive to the articulatory motor actions produced in speaking. The aim of these experiments was to highlight the need for accounts of the effects of audio–visual (AV) speech that explicitly consider the properties of articulated action. The first experiment employed a syllable–monitoring task in which participants were required to monitor for target syllables within foreign carrier phrases. An AV effect was found in that seeing a talkers moving face (moving face condition) assisted in more accurate recognition (hits and correct rejections) of spoken syllables than of auditory–only still face (still face condition) presentations. The second experiment examined processing of spoken phrases by investigating whether an AV effect would be found for estimates of phrase duration. Two effects of seeing the moving face of the talker were found. First, the moving face condition had significantly longer duration estimates than the still face auditory–only condition. Second, estimates of auditory duration made in the moving face condition reliably correlated with the actual durations whereas those made in the still face auditory condition did not. The third experiment was carried out to determine whether the stronger correlation between estimated and actual duration in the moving face condition might have been due to generic properties of AV presentation. Experiment 3 employed the procedures of the second experiment but used stimuli that were not perceived as speech although they possessed the same timing cues as those of the speech stimuli of Experiment 2. It was found that simply presenting both auditory and visual timing information did not result in more reliable duration estimates. Further, when released from the speech context (used in Experiment 2), duration estimates for the auditory–only stimuli were significantly correlated with actual durations. In all, these results demonstrate that visual speech can assist in the analysis of clearly presented auditory stimuli in tasks concerned with information provided by viewing the production of an utterance. We suggest that these findings are consistent with there being a processing link between perception and action such that viewing a talker speaking will activate speech motor schemas in the perceiver.

Audio-visual speech perception off the top of the head

The study examined whether people can extract speech related information from the talkers upper face that was presented using either normally textured videos (Experiments 1 and 3) or videos showing only the outlined of the head (Experiments 2 and 4). Experiments 1 and 2 used within- and cross-modal matching tasks. In the within-modal task, observers were presented two pairs of short silent video clips that showed the top part of a talkers head. In the cross-modal task, pairs of audio and silent video clips were presented. The task was to determine the pair in which the talker said the same sentence. Performance on both tasks was better than chance for the outline as well as textured presentation suggesting that judgments were primarily based on head movements. Experiments 3 and 4 tested if observing the talkers upper face would help identify speech in noise. The results showed the viewing the talkers moving upper head produced a small but reliable improvement in speech intelligibility, however, this effect was only secure for the expressive sentences that involved greater head movements. The results suggest that people are sensitive to speech related head movements that extend beyond the mouth area and can use these to assist in language processing.

Emotional expressions evoke a differential response in the fusiform face area

It is widely assumed that the fusiform face area (FFA), a brain region specialized for face perception, is not involved in processing emotional expressions. This assumption is based on the proposition that the FFA is involved in face identification and only processes features that are invariant across changes due to head movements, speaking and expressing emotions. The present study tested this proposition by examining whether the response in the human FFA varies across emotional expressions with functional magnetic resonance imaging and brain decoding analysis techniques (n = 11). A one vs. all classification analysis showed that most emotional expressions that participants perceived could be reliably predicted from the neural pattern of activity in left and the right FFA, suggesting that the perception of different emotional expressions recruit partially non-overlapping neural mechanisms. In addition, emotional expressions could also be decoded from the pattern of activity in the early visual cortex (EVC), indicating that retinotopic cortex also shows a differential response to emotional expressions. These results cast doubt on the idea that the FFA is involved in expression invariant face processing, and instead indicate that emotional expressions evoke partially de-correlated signals throughout occipital and posterior temporal cortex.

Amodal processing of visual speech as revealed by priming

This study investigated the linguistic processing of visual speech (video of a talkers utterance without audio) by determining if such has the capacity to prime subsequently presented word and nonword targets. The priming procedure is well suited for the investigation of whether speech perception is amodal since visual speech primes can be used with targets presented in different modalities. To this end, a series of priming experiments were conducted using several tasks. It was found that visually spoken words (for which overt identification was poor) acted as reliable primes for repeated target words in the naming, written and auditory lexical decision tasks. These visual speech primes did not produce associative or reliable form priming. The lack of form priming suggests that the repetition priming effect was constrained by lexical level processes. That priming found in all tasks is consistent with the view that similar processes operate in both visual and auditory speech processing.

Cocaine increases endoplasmic reticulum stress protein expression in striatal neurons.

Cocaine administration upregulates the levels of extracellular glutamate and dopamine in the striatum. Activation of the receptors alters calcium homeostasis in striatal neurons leading to the expression of the endoplasmic reticulum (ER) stress proteins. It was therefore hypothesized that cocaine upregulates the expression of the ER stress proteins, immunoglobulin heavy chain binding protein (BiP), Ire1alpha and perk via glutamate and dopamine receptor activation. A novel glutamate microbiosensor and Western immunoblot analyses were mainly performed to test the hypothesis in the rat dorsal striatum. The results showed that i.p. injection of repeated cocaine (20 mg/kg) for nine consecutive days significantly increased extracellular glutamate levels while acute cocaine injection did not. However, the immunoreactivities (IR) of the ER stress proteins in the dorsal striatum were significantly increased by either acute or repeated cocaine injections as compared with saline controls. Intrastriatal injection (i.s.) of the selective group I metabotropic glutamate receptor (mGluR) antagonist N-phenyl-7-(hydroxyimino)cyclopropa[b]chromen-1a-carboxamide (PHCCC; 25 nmol) or the mGluR5 subtype antagonist 2-methyl-6-(phenylethynyl)pyridine hydrochloride (MPEP; 2 and 25 nmol) significantly decreased repeated cocaine-induced increases in the IR of the ER stress proteins in the injected dorsal striatum. Similarly, the selective D1 antagonist (R)-(+)-7-chloro-8-hydroxy-3-methyl-1-phenyl-2,3,4,5-tetrahydro-1H-3-benzazepine hydrochloride (SCH23390; 0.1 mg/kg, i.p.) or the N-methyl-d-aspartate antagonist dizocilpine/(5S,10R)-(+)-5-methyl-10,11-dihydro-5H-ibenzo[a,d]cyclohepten-5,10-imine maleate (MK801; 2 nmol, i.s.) decreased acute or repeated cocaine-induced the IR of the ER stress proteins in the dorsal striatum. These data suggest that cocaine upregulates expression of the ER stress proteins in striatal neurons via a mechanism involving activation of glutamate and dopamine receptors.

Prosody off the top of the head: Prosodic contrasts can be discriminated by head motion

The current study investigated peoples ability to discriminate prosody related head and face motion from videos showing only the upper face of the speaker saying the same sentence with different prosody. The first two experiments used a visual-visual matching task. These videos were either fully textured (Experiment 1) or showed only the outline of the speakers head (Experiment 2). Participants were presented with two stimulus pairs of silent videos, with their task to select the pair that had the same prosody. The overall results of the visual-visual matching experiments showed that people could discriminate same- from different-prosody sentences with a high degree of accuracy. Similar levels of discrimination performance were obtained for the fully textured (containing rigid and non-rigid motions) and the outline only (rigid motion only) videos. Good visual-visual matching performance shows that people are sensitive to the underlying factor that determined whether the movements were the same or not, i.e., the production of prosody. However, testing auditory-visual matching provides a more direct test concerning peoples sensitivity to how head motion/face motion relates to spoken prosody. Experiments 3 (with fully textured videos) and 4 (with outline only videos) employed a cross-modal matching task that required participants to match auditory with visual tokens that had the same prosody. As with the previous experiments, participants performed this discrimination very well. Similarly, no decline in performance was observed for the outline only videos. This result supports the proposal that rigid head motion provides an important visual cue to prosody.

Hearing Foreign Voices: Does Knowing What is Said Affect Visual-Masked-Speech Detection?

We investigated audio-visual (AV) perceptual integration by examining the effect of seeing the speakers synchronised moving face on masked-speech detection ability. Signal amplification and higher-level cognitive accounts of an AV advantage were contrasted, the latter by varying whether participants knew the language of the speaker. An AV advantage was shown for sentences whose mid-to-high-frequency acoustic envelope was highly correlated with articulator movement, regardless of knowledge of the language. For low-correlation sentences, knowledge of the language had a large impact; for participants with no knowledge of the language an AV inhibitory effect was found (providing support for reports of a compelling AV illusion). The results indicate a role for both sensory enhancement and higher-level cognitive factors in AV speech detection.

The effect of viewing speech on auditory speech processing is different in the left and right hemispheres

We used whole-head magnetoencephalograpy (MEG) to record changes in neuromagnetic N100m responses generated in the left and right auditory cortex as a function of the match between visual and auditory speech signals. Stimuli were auditory-only (AO) and auditory-visual (AV) presentations of /pi/, /ti/ and /vi/. Three types of intensity matched auditory stimuli were used: intact speech (Normal), frequency band filtered speech (Band) and speech-shaped white noise (Noise). The behavioural task was to detect the /vi/ syllables which comprised 12% of stimuli. N100m responses were measured to averaged /pi/ and /ti/ stimuli. Behavioural data showed that identification of the stimuli was faster and more accurate for Normal than for Band stimuli, and for Band than for Noise stimuli. Reaction times were faster for AV than AO stimuli. MEG data showed that in the left hemisphere, N100m to both AO and AV stimuli was largest for the Normal, smaller for Band and smallest for Noise stimuli. In the right hemisphere, Normal and Band AO stimuli elicited N100m responses of quite similar amplitudes, but N100m amplitude to Noise was about half of that. There was a reduction in N100m for the AV compared to the AO conditions. The size of this reduction for each stimulus type was same in the left hemisphere but graded in the right (being largest to the Normal, smaller to the Band and smallest to the Noise stimuli). The N100m decrease for the Normal stimuli was significantly larger in the right than in the left hemisphere. We suggest that the effect of processing visual speech seen in the right hemisphere likely reflects suppression of the auditory response based on AV cues for place of articulation.