Mary Rudner

The Ease of Language Understanding (ELU) model: theoretical, empirical, and clinical advances

Working memory is important for online language processing during conversation. We use it to maintain relevant information, to inhibit or ignore irrelevant information, and to attend to conversation selectively. Working memory helps us to keep track of and actively participate in conversation, including taking turns and following the gist. This paper examines the Ease of Language Understanding model (i.e., the ELU model, Rönnberg, 2003; Rönnberg et al., 2008) in light of new behavioral and neural findings concerning the role of working memory capacity (WMC) in uni-modal and bimodal language processing. The new ELU model is a meaning prediction system that depends on phonological and semantic interactions in rapid implicit and slower explicit processing mechanisms that both depend on WMC albeit in different ways. It is based on findings that address the relationship between WMC and (a) early attention processes in listening to speech, (b) signal processing in hearing aids and its effects on short-term memory, (c) inhibition of speech maskers and its effect on episodic long-term memory, (d) the effects of hearing impairment on episodic and semantic long-term memory, and finally, (e) listening effort. New predictions and clinical implications are outlined. Comparisons with other WMC and speech perception models are made.

Cognition counts: a working memory system for ease of language understanding (ELU).

A general working memory system for ease of language understanding (ELU, Rönnberg, ) is presented. The purpose of the system is to describe and predict the dynamic interplay between explicit and implicit cognitive functions, especially in conditions of poorly perceived or poorly specified linguistic signals. In relation to speech understanding, the system based on (1) the quality and precision of phonological representations in long-term memory, (2) phonologically mediated lexical access speed, and (3) explicit, storage, and processing resources. If there is a mismatch between phonological information extracted from the speech signal and the phonological information represented in long-term memory, the system is assumed to produce a mismatch signal that invokes explicit processing resources. In the present paper, we focus on four aspects of the model which have led to the current, updated version: the language generality assumption; the mismatch assumption; chronological age; and the episodic buffer function of rapid, automatic multimodal binding of phonology (RAMBPHO). We evaluate the language generality assumption in relation to sign language and speech, and the mismatch assumption in relation to signal processing in hearing aids. Further, we discuss the effects of chronological age and the implications of RAMBPHO.

Cognition and hearing aids

The perceptual information transmitted from a damaged cochlea to the brain is more poorly specified than information from an intact cochlea and requires more processing in working memory before language content can be decoded. In addition to making sounds audible, current hearing aids include several technologies that are intended to facilitate language understanding for persons with hearing impairment in challenging listening situations. These include directional microphones, noise reduction, and fast-acting amplitude compression systems. However, the processed signal itself may challenge listening to the extent that with specific types of technology, and in certain listening situations, individual differences in cognitive processing resources may determine listening success. Here, current and developing digital hearing aid signal processing schemes are reviewed in the light of individual working memory (WM) differences. It is argued that signal processing designed to improve speech understanding may have both positive and negative consequences, and that these may depend on individual WM capacity.

When cognition kicks in: Working memory and speech understanding in noise

Perceptual load and cognitive load can be separately manipulated and dissociated in their effects on speech understanding in noise. The Ease of Language Understanding model assumes a theoretical position where perceptual task characteristics interact with the individuals implicit capacities to extract the phonological elements of speech. Phonological precision and speed of lexical access are important determinants for listening in adverse conditions. If there are mismatches between the phonological elements perceived and phonological representations in long-term memory, explicit working memory (WM)-related capacities will be continually invoked to reconstruct and infer the contents of the ongoing discourse. Whether this induces a high cognitive load or not will in turn depend on the individuals storage and processing capacities in WM. Data suggest that modulated noise maskers may serve as triggers for speech maskers and therefore induce a WM, explicit mode of processing. Individuals with high WM capacity benefit more than low WM-capacity individuals from fast amplitude compression at low or negative input speech-to-noise ratios. The general conclusion is that there is an overarching interaction between the focal purpose of processing in the primary listening task and the extent to which a secondary, distracting task taps into these processes.

Effects of noise and working memory capacity on memory processing of speech for hearing-aid users.

Abstract Objectives: It has been shown that noise reduction algorithms can reduce the negative effects of noise on memory processing in persons with normal hearing. The objective of the present study was to investigate whether a similar effect can be obtained for persons with hearing impairment and whether such an effect is dependent on individual differences in working memory capacity. Design: A sentence-final word identification and recall (SWIR) test was conducted in two noise backgrounds with and without noise reduction as well as in quiet. Working memory capacity was measured using a reading span (RS) test. Study sample: Twenty-six experienced hearing-aid users with moderate to moderately severe sensorineural hearing loss. Results: Noise impaired recall performance. Competing speech disrupted memory performance more than speech-shaped noise. For late list items the disruptive effect of the competing speech background was virtually cancelled out by noise reduction for persons with high working memory capacity. Conclusions: Noise reduction can reduce the adverse effect of noise on memory for speech for persons with good working memory capacity. We argue that the mechanism behind this is faster word identification that enhances encoding into working memory.

Hearing impairment and cognitive energy: the Framework for Understanding Effortful Listening (FUEL)

The Fifth Eriksholm Workshop on “Hearing Impairment and Cognitive Energy” was convened to develop a consensus among interdisciplinary experts about what is known on the topic, gaps in knowledge, the use of terminology, priorities for future research, and implications for practice. The general term cognitive energy was chosen to facilitate the broadest possible discussion of the topic. It goes back to Titchener (1908) who described the effects of attention on perception; he used the term psychic energy for the notion that limited mental resources can be flexibly allocated among perceptual and mental activities. The workshop focused on three main areas: (1) theories, models, concepts, definitions, and frameworks; (2) methods and measures; and (3) knowledge translation. We defined effort as the deliberate allocation of mental resources to overcome obstacles in goal pursuit when carrying out a task, with listening effort applying more specifically when tasks involve listening. We adapted Kahneman’s seminal (1973) Capacity Model of Attention to listening and proposed a heuristically useful Framework for Understanding Effortful Listening (FUEL). Our FUEL incorporates the well-known relationship between cognitive demand and the supply of cognitive capacity that is the foundation of cognitive theories of attention. Our FUEL also incorporates a motivation dimension based on complementary theories of motivational intensity, adaptive gain control, and optimal performance, fatigue, and pleasure. Using a three-dimensional illustration, we highlight how listening effort depends not only on hearing difficulties and task demands but also on the listener’s motivation to expend mental effort in the challenging situations of everyday life.

Working Memory Capacity May Influence Perceived Effort during Aided Speech Recognition in Noise

BACKGROUND Recently there has been interest in using subjective ratings as a measure of perceived effort during speech recognition in noise. Perceived effort may be an indicator of cognitive load. Thus, subjective effort ratings during speech recognition in noise may covary both with signal-to-noise ratio (SNR) and individual cognitive capacity. PURPOSE The present study investigated the relation between subjective ratings of the effort involved in listening to speech in noise, speech recognition performance, and individual working memory (WM) capacity in hearing impaired hearing aid users. RESEARCH DESIGN In two experiments, participants with hearing loss rated perceived effort during aided speech perception in noise. Noise type and SNR were manipulated in both experiments, and in the second experiment hearing aid compression release settings were also manipulated. Speech recognition performance was measured along with WM capacity. STUDY SAMPLE There were 46 participants in all with bilateral mild to moderate sloping hearing loss. In Experiment 1 there were 16 native Danish speakers (eight women and eight men) with a mean age of 63.5 yr (SD = 12.1) and average pure tone (PT) threshold of 47. 6 dB (SD = 9.8). In Experiment 2 there were 30 native Swedish speakers (19 women and 11 men) with a mean age of 70 yr (SD = 7.8) and average PT threshold of 45.8 dB (SD = 6.6). DATA COLLECTION AND ANALYSIS A visual analog scale (VAS) was used for effort rating in both experiments. In Experiment 1, effort was rated at individually adapted SNRs while in Experiment 2 it was rated at fixed SNRs. Speech recognition in noise performance was measured using adaptive procedures in both experiments with Dantale II sentences in Experiment 1 and Hagerman sentences in Experiment 2. WM capacity was measured using a letter-monitoring task in Experiment 1 and the reading span task in Experiment 2. RESULTS In both experiments, there was a strong and significant relation between rated effort and SNR that was independent of individual WM capacity, whereas the relation between rated effort and noise type seemed to be influenced by individual WM capacity. Experiment 2 showed that hearing aid compression setting influenced rated effort. CONCLUSIONS Subjective ratings of the effort involved in speech recognition in noise reflect SNRs, and individual cognitive capacity seems to influence relative rating of noise type.

The role of the episodic buffer in working memory for language processing

A body of work has accumulated to show that the cognitive process of binding information from different mnemonic and sensory sources as well as in different linguistic modalities can be fractionated from general executive functions in working memory both functionally and neurally. This process has been defined in terms of the episodic buffer (Baddeley in Trends Cogn Sci 4(11):417–423, 2000). This paper considers behavioural, neuropsychological and neuroimaging data that elucidate the role of the episodic buffer in language processing. We argue that the episodic buffer seems to be truly multimodal in function and that while formation of unitary multidimensional representations in the episodic buffer seems to engage posterior neural networks, maintenance of such representations is supported by frontal networks. Although, the episodic buffer is not necessarily supported by executive processes and seems to be supported by different neural networks, it may operate in tandem with the central executive during effortful language processing. There is also evidence to suggest engagement of the phonological loop during buffer processing. The hippocampus seems to play a role in formation but not maintenance of representations in the episodic buffer of working memory.

Dissociating cognitive and sensory neural plasticity in human superior temporal cortex

Disentangling the effects of sensory and cognitive factors on neural reorganization is fundamental for establishing the relationship between plasticity and functional specialization. Auditory deprivation in humans provides a unique insight into this problem, because the origin of the anatomical and functional changes observed in deaf individuals is not only sensory, but also cognitive, owing to the implementation of visual communication strategies such as sign language and speechreading. Here, we describe a functional magnetic resonance imaging study of individuals with different auditory deprivation and sign language experience. We find that sensory and cognitive experience cause plasticity in anatomically and functionally distinguishable substrates. This suggests that after plastic reorganization, cortical regions adapt to process a different type of input signal, but preserve the nature of the computation they perform, both at a sensory and cognitive level.

Neural representation of binding lexical signs and words in the episodic buffer of working memory

The episodic buffer accommodates formation and maintenance of unitary multidimensional representations based on information in different codes from different sources. Formation, based on submorphemic units, engages posterior brain regions, while maintenance engages frontal regions. Using a hybrid fMRI design, that allows separate analysis of transient and sustained components, an n-back task and an experimental group of 13 hearing native signers, with experience of Swedish Sign Language and Swedish since birth, we investigated binding of lexical signs and words in working memory. Results show that the transient component of these functions is supported by a buffer-specific network of posterior regions including the right middle temporal lobe, possibly relating to binding of phonological loop representations with semantic representations in long-term memory, as well as a loop-specific network, in line with predictions of a functional relationship between loop and buffer. The left hippocampus was engaged in transient and sustained components of buffer processing, possibly reflecting the meaningful nature of the stimuli. Only a minor role was found for executive functions in line with other recent work. A novel representation of the sustained component of working memory for audiovisual language in the right inferior temporal lobe may be related to perception of speech-related facial gestures. Previous findings of sign and speech loop representation in working memory were replicated and extended. Together, these findings support the notion of a module that mediates between codes and sources, such as the episodic buffer, and further our understanding of its nature.