Ralf Rummer

Recognizing Time Pressure and Cognitive Load on the Basis of Speech: An Experimental Study

In an experimental environment, we simulated the situation of a user who gives speech input to a system while walking through an airport. The time pressure on the subjects and the requirement to navigate while speaking were manipulated orthogonally. Each of the 32 subjects generated 80 utterances, which were coded semi-automatically with respect to a wide range of features, such as filled pauses. The experiment yielded new results concerning the effects of time pressure and cognitive load on speech. To see whether a system can automatically identify these conditions on the basis of speech input, we had this task performed for each subject by a Bayesian network that had been learned on the basis of the experimental data for the other subjects. The results shed light on the conditions that determine the accuracy of such recognition.

The Perceptual Basis of the Modality Effect in Multimedia Learning.

Various studies have demonstrated an advantage of auditory over visual text modality when learning with texts and pictures. To explain this modality effect, two complementary assumptions are proposed by cognitive theories of multimedia learning: first, the visuospatial load hypothesis, which explains the modality effect in terms of visuospatial working memory overload in the visual text condition; and second, the temporal contiguity assumption, according to which the modality effect occurs because solely auditory texts and pictures can be attended to simultaneously. The latter explanation applies only to simultaneous presentation, the former to both simultaneous and sequential presentation. This paper introduces a third explanation, according to which parts of the modality effect are due to early, sensory processes. This account predicts that-for texts longer than one sentence-the modality effect with sequential presentation is restricted to the information presented most recently. Two multimedia experiments tested the influence of text modality across three different conditions: simultaneous presentation of texts and pictures versus sequential presentation versus presentation of text only. Text comprehension and picture recognition served as dependent variables. An advantage for auditory texts was restricted to the most recent text information and occurred under all presentation conditions. With picture recognition, the modality effect was restricted to the simultaneous condition. These findings clearly support the idea that the modality effect can be attributed to early processes in perception and sensory memory rather than to a working memory bottleneck.

Assessment of a User’s Time Pressure and Cognitive Load on the Basis of Features of Speech

The project READY (1996-2004) approached the topic of resource-adaptive cognitive processes from a different angle than most of the other projects represented in this volume: The resources in question were the cognitive resources of computer users; the adaptation was done by the system that they were using.

Creating an empirical basis for adaptation decisions

How can an adaptive intelligent interface decide what particular action to perform in a given situation, as a function of perceived properties of the user and the situation? Ideally, such decisions should be made on the basis of an empirically derived causal model. In this paper we show how such a model can be constructed given an appropriately limited system and domain: On the basis of data from a controlled experiment, an influence diagram for making adaptation decisions is learned automatically. We then discuss why this method will often be infeasible in practice, and how parts of the method can nonetheless be used to create a more solid basis for adaptation decisions.

Phonological information in immediate and delayed sentence recall

Potter and Lombardi (1990) state in their conceptual regeneration hypothesis that immediate sentence recall is only based on conceptual and lexical information; phonological information does not contribute. As experimental evidence for this hypothesis, they reported that if a sentence is followed by a word list that included a lure word similar to one of the content words of the sentence (target word), the lure word frequently intrudes into sentence recall. We demonstrated that Potter and Lombardi did not observe any influence of phonological information because list presentation followed sentence presentation, and phonological information was discarded. We observed that phonological information influenced the intrusion rate if recall was not delayed by the subsequent presentation of a word list. With immediate recall, the lure intrusion effect disappeared in auditorily presented sentences. This shows that, if available, phonological information contributes to sentence recall.

Der Einfluss von Expertise auf den Prozess der schulischen Leistungsbeurteilung

Zusammenfassung. In zwei Experimenten wurde gepruft, ob unterschiedliche Ziele bei der Verarbeitung von Schulermerkmalen das Gedachtnis fur diese Merkmale und die padagogische Leistungsbeurteilung beeinflussen. In Experiment 1 erhielten Lehrer (Experten) Fallbeschreibungen von Schulern unter der Instruktion, sich entweder einen Eindruck uber die Personlichkeit des Schulers zu bilden oder die weitere Entwicklung des Schulers zu prognostizieren. Anschliesend waren die Beschreibungen so genau wie moglich zu reproduzieren und das Leistungsvermogen des beschriebenen Schulers zu beurteilen. Experiment 2 war eine Replikation mit Studierenden der Naturwissenschaften (Laien) als Probanden. Unter der Zielsetzung, zukunftige Schulerleistungen zu prognostizieren, wahlten Experten die aufmerksamkeitsintensive Strategie der Merkmalsintegration; unter dem Eindrucksbildungsziel aktivierten sie lediglich prototypische Kategorien. Bei Laien unterschieden sich die Verarbeitungsstrategien hingegen nicht in Abhangigkeit von d...

Shared representations in language processing and verbal short-term memory: The case of grammatical gender

Abstract The general idea of language-based accounts of short-term memory is that retention of linguistic materials is based on representations within the language processing system. In the present sentence recall study, we address the question whether the assumption of shared representations holds for morphosyntactic information (here: grammatical gender in German). Three experiments using a modified version of Potter and Lombardi’s (1990) intrusion paradigm tested whether the presentation of nouns similar in meaning to a noun included in a sentence led to more intrusions in sentence recall if the related nouns had the same grammatical gender as compared to a different gender. Indeed, morphosyntactic constraints resulted in weaker (or even no) intrusion effects for gender incongruent as compared to gender congruent intruders. This effect was observed independently of the degree to which phonological information was available during recall and independently of a determiner mismatch between the noun in the sentence and the incongruent intruder.

Evidence for a modality effect in sentence retention

It is well known that an acoustic-sensory code supports retention of linguistic materials whose storage is particularly based on phonological information (e.g., unrelated word lists). The present study investigates whether such a code also contributes to the retention of sentences. It has been shown that short-term sentence recall particularly depends on propositional and lexicosemantic information, which are assumed to be supplied independently of modality influences. We employed the intrusion paradigm of Potter and Lombardi (1990) and manipulated the availability of acoustic-sensory information. Participants were instructed to read sentences either silently or aloud. Since these two reading conditions also differ with respect to articulatory information, a further condition that provided articulatory but not acoustic-sensory information was introduced (i.e., silent mouthing). Our data suggest that acoustic-sensory information is used, if available, even in sentence recall.

Semantic and phonological information in sentence recall: Converging psycholinguistic and neuropsychological evidence

We present one experiment and a neuropsychological case study to investigate to what extent phonological and semantic representations contribute to short-term sentence recall. We modified Potter and Lombardis (1990) intrusion paradigm, in which retention of a list interferes with sentence recall such that on the list a semantically related lure is presented, which is expected to intrude into sentence recall. In our version, lure words are either semantically related to target words in the sentence or semantically plus phonologically related. With healthy participants, intrusions are more frequent when lure and target overlap phonologically in addition to semantically than when they solely overlap semantically. When this paradigm is applied to a patient with a phonological short-term memory impairment, both lure types induce the same amount of intrusions. These findings indicate that usually phonological information is retained in sentence recall in addition to semantic information.

The subordination effect: Evidence from self-paced reading and recall

Engelkamp and Rummer (2002) demonstrated that auditorily presented subordinate causal sentences are better retained than coordinate sentences. This subordination effect was explained by suggesting that subordinate sentences are merged more easily in memory than coordinate sentences. The present paper enlarges Engelkamp and Rummers findings with respect to three aspects. First, it demonstrates that the subordination effect can only be found if a verbatim but not a content related recall score is used. Second, using self-paced reading, it demonstrates a processing advantage for subordinate sentences. This reading time advantage was only observed if the subordinate clause preceded the matrix clause. Third, it excludes an alternative explanation of the subordination effect which attributes memory differences to the fact that the critical causal conjunction word was presented earlier in subordinate than in coordinate sentences. In sum, our findings suggest a direct contribution of syntactic information to sentence regeneration.