Judith Schweppe

The Perceptual Basis of the Modality Effect in Multimedia Learning.

Various studies have demonstrated an advantage of auditory over visual text modality when learning with texts and pictures. To explain this modality effect, two complementary assumptions are proposed by cognitive theories of multimedia learning: first, the visuospatial load hypothesis, which explains the modality effect in terms of visuospatial working memory overload in the visual text condition; and second, the temporal contiguity assumption, according to which the modality effect occurs because solely auditory texts and pictures can be attended to simultaneously. The latter explanation applies only to simultaneous presentation, the former to both simultaneous and sequential presentation. This paper introduces a third explanation, according to which parts of the modality effect are due to early, sensory processes. This account predicts that-for texts longer than one sentence-the modality effect with sequential presentation is restricted to the information presented most recently. Two multimedia experiments tested the influence of text modality across three different conditions: simultaneous presentation of texts and pictures versus sequential presentation versus presentation of text only. Text comprehension and picture recognition served as dependent variables. An advantage for auditory texts was restricted to the most recent text information and occurred under all presentation conditions. With picture recognition, the modality effect was restricted to the simultaneous condition. These findings clearly support the idea that the modality effect can be attributed to early processes in perception and sensory memory rather than to a working memory bottleneck.

Shared representations in language processing and verbal short-term memory: The case of grammatical gender

Abstract The general idea of language-based accounts of short-term memory is that retention of linguistic materials is based on representations within the language processing system. In the present sentence recall study, we address the question whether the assumption of shared representations holds for morphosyntactic information (here: grammatical gender in German). Three experiments using a modified version of Potter and Lombardi’s (1990) intrusion paradigm tested whether the presentation of nouns similar in meaning to a noun included in a sentence led to more intrusions in sentence recall if the related nouns had the same grammatical gender as compared to a different gender. Indeed, morphosyntactic constraints resulted in weaker (or even no) intrusion effects for gender incongruent as compared to gender congruent intruders. This effect was observed independently of the degree to which phonological information was available during recall and independently of a determiner mismatch between the noun in the sentence and the incongruent intruder.

Evidence for a modality effect in sentence retention

It is well known that an acoustic-sensory code supports retention of linguistic materials whose storage is particularly based on phonological information (e.g., unrelated word lists). The present study investigates whether such a code also contributes to the retention of sentences. It has been shown that short-term sentence recall particularly depends on propositional and lexicosemantic information, which are assumed to be supplied independently of modality influences. We employed the intrusion paradigm of Potter and Lombardi (1990) and manipulated the availability of acoustic-sensory information. Participants were instructed to read sentences either silently or aloud. Since these two reading conditions also differ with respect to articulatory information, a further condition that provided articulatory but not acoustic-sensory information was introduced (i.e., silent mouthing). Our data suggest that acoustic-sensory information is used, if available, even in sentence recall.

Semantic and phonological information in sentence recall: Converging psycholinguistic and neuropsychological evidence

We present one experiment and a neuropsychological case study to investigate to what extent phonological and semantic representations contribute to short-term sentence recall. We modified Potter and Lombardis (1990) intrusion paradigm, in which retention of a list interferes with sentence recall such that on the list a semantically related lure is presented, which is expected to intrude into sentence recall. In our version, lure words are either semantically related to target words in the sentence or semantically plus phonologically related. With healthy participants, intrusions are more frequent when lure and target overlap phonologically in addition to semantically than when they solely overlap semantically. When this paradigm is applied to a patient with a phonological short-term memory impairment, both lure types induce the same amount of intrusions. These findings indicate that usually phonological information is retained in sentence recall in addition to semantic information.

Mood is linked to vowel type: the role of articulatory movements.

This study investigates the relation between vowel identity and emotional state. In Experiment 1, (pseudo)words were invented and articulated in a positive or negative mood condition. Subjects in a positive mood produced more words containing /i:/, a vowel involving the same muscle that is used in smiling--the zygomaticus major muscle (ZMM). Subjects in a negative mood produced more words containing /o:/, involving an antagonist of the ZMM--the orbicularis orbis muscle (OOM). We argue that the link between mood and vowel identity is related to orofacial muscle activity, which provides articulatory feedback to speakers on their emotional state. Experiment 2 tests this hypothesis more specifically. Participants rated the funniness of cartoons while repeatedly articulating either /i:/ (ZMM) or /o:/ (OOM). In line with our hypothesis, the cartoons were rated as funnier by subjects articulating /i:/ than by those articulating /o:/.

Integrating written text and graphics as a desirable difficulty in long-term multimedia learning

Many principles for the design of multimedia learning materials share the recommendation to facilitate processing. One prominent example is the modality principle, according to which pictures should be presented with auditory rather than visual texts. Research on desirable difficulties, however, indicates that - unlike short-term learning - long-term learning benefits when processing is more demanding and therefore more effortful. In a classroom experiment (Experiment 1) and in a laboratory study (Experiment 2), we tested whether the modality principle serves long-term learning. In a multimedia presentation on the formation of lightning, we varied the text modality (oral vs. written) and the delay between learning and test (retention and transfer performance tested immediately after instruction vs. one week later). In the immediate tests, there was either an auditory advantage (Experiment 1) or no difference (Experiment 2). However, when learning was tested after a delay, the combined processing of written text and animations led to better transfer performance than oral text and animations in both experiments. This suggests that written text presentation serves as a desirable difficulty that supports long-term learning. We investigate the modality effect in multimedia learning with immediate and delayed tests.Crucially, the written modality is superior in the delayed test.This suggests that the modality principle should be reversed for long-term learning.Studies on multimedia learning should not be restricted to immediate tests.

Two modality effects in verbal short-term memory: Evidence from sentence recall

This paper investigates the mechanisms underlying the standard modality effect (i.e., better recall performance for auditorily presented than for visually presented materials), and the modality congruency effect (i.e., better memory performance if the mode of recall and presentation are congruent rather than incongruent). We tested the assumption that the standard modality effect is restricted to the most recent word(s) of the sentences but occurs in both verbatim and gist recall (Experiments 1 and 2), whereas the modality congruency effect should be evident for the rest of the sentence when using verbatim recall (Experiment 3) but not when using gist recall (Experiment 4). All experiments used the Potter-Lombardi intrusion paradigm. When the target word was the most recent word of the sentence, a standard modality effect was found with both verbatim recall and gist recall. When the target word was included in the middle of the sentences, a modality congruency effect was found with verbatim recall but not with gist recall.

A modality congruency effect in verbal false memory

This experiment tests the modality congruency hypothesis, which suggests that in short-term recall of auditory or visual verbal information false memories can be suppressed more effectively if presentation modality and recall mode are in congruent relation (i.e., in oral recall of auditorily presented and written recall of visually presented materials). The experiment is based on the DRM paradigm (as used by Kellogg, 2001). The results demonstrate more intrusions if presentation and recall are in an incongruent relation than if they are in a congruent relation. This finding supports the assumption of a privileged pathway from the auditory input system to oral output on the one hand, and from the visual input system to written output on the other hand, which aids source monitoring.

Does visuo-spatial working memory generally contribute to immediate serial letter recall?

This work contributes to the understanding of the visual similarity effect in verbal working memory, a finding that suggests that the visuo-spatial sketch pad—the system in Baddeleys working memory model specialised in retaining nonverbal visual information—might be involved in the retention of visually presented verbal materials. Crucially this effect is implicitly interpreted by the most influential theory of multimedia learning as evidence for an obligatory involvement of the visuo-spatial sketch pad. We claim that it is only involved when the functioning of the working memory component normally used for processing verbal material is impaired. In this article we review the studies that give rise to the idea of obligatory involvement of the visuo-spatial sketch pad and suggest that some findings can be understood with reference to orthographic rather than visual similarity. We then test an alternative explanation of the finding that is most apt to serve as evidence for obligatory involvement of the visuo-spatial sketch pad. We conclude that, in healthy adults and under normal learning conditions, the visual similarity effect can be explained within the framework of verbal working memory proposed by Baddeley (e.g., 1986, 2000) without additional premises regarding the visuo-spatial sketch.

Beyond sentence boundaries : Grammatical gender information in short-term recall of texts

In the context of text recall it is often stated that surface representations are quickly forgotten. Jarvella (1971) and Sachs (1967) argued that what is retained beyond sentence boundaries is meaning information, whereas lexical and syntactic information is only available for the most recent constituent. We based a text recall experiment on Jarvella’s paradigm, in order to demonstrate that both meaning and grammatical gender information contribute to the recall of short text passages. Although it is known that grammatical gender information is used in anaphor resolution, even if noun and pronoun do not belong to adjacent sentences, there is no direct evidence for a gender contribution to text memory so far. The present experiment demonstrates that grammatical gender information, even when of no semantic importance, is retained beyond sentence boundaries and can contribute to the memory representation of subsequent text. All materials and additional statistics may be downloaded from mc.psychonomic-journals.org/content/supplemental.