Sharlene D. Newman

Learning to Decode Cognitive States from Brain Images

Over the past decade, functional Magnetic Resonance Imaging (fMRI) has emerged as a powerful new instrument to collect vast quantities of data about activity in the human brain. A typical fMRI experiment can produce a three-dimensional image related to the human subjects brain activity every half second, at a spatial resolution of a few millimeters. As in other modern empirical sciences, this new instrumentation has led to a flood of new data, and a corresponding need for new data analysis methods. We describe recent research applying machine learning methods to the problem of classifying the cognitive state of a human subject based on fRMI data observed over a single time interval. In particular, we present case studies in which we have successfully trained classifiers to distinguish cognitive states such as (1) whether the human subject is looking at a picture or a sentence, (2) whether the subject is reading an ambiguous or non-ambiguous sentence, and (3) whether the word the subject is viewing is a word describing food, people, buildings, etc. This learning problem provides an interesting case study of classifier learning from extremely high dimensional (105 features), extremely sparse (tens of training examples), noisy data. This paper summarizes the results obtained in these three case studies, as well as lessons learned about how to successfully apply machine learning methods to train classifiers in such settings.

Frontal and parietal participation in problem solving in the Tower of London: fMRI and computational modeling of planning and high-level perception

This study triangulates executive planning and visuo-spatial reasoning in the context of the Tower of London (TOL) task by using a variety of methodological approaches. These approaches include functional magnetic resonance imaging (fMRI), functional connectivity analysis, individual difference analysis, and computational modeling. A graded fMRI paradigm compared the brain activation during the solution of problems with varying path lengths: easy (1 and 2 moves), moderate (3 and 4 moves) and difficult (5 and 6 moves). There were three central findings regarding the prefrontal cortex: (1) while both the left and right prefrontal cortices were equally involved during the solution of moderate and difficult problems, the activation on the right was differentially attenuated during the solution of the easy problems; (2) the activation observed in the right prefrontal cortex was highly correlated with individual differences in working memory (measured independently by the reading span task); and (3) different patterns of functional connectivity were observed in the left and right prefrontal cortices. Results obtained from the superior parietal region also revealed left/right differences; only the left superior parietal region revealed an effect of difficulty. These fMRI results converged upon two hypotheses: (1) the right prefrontal area may be more involved in the generation of a plan, whereas the left prefrontal area may be more involved in plan execution; and (2) the right superior parietal region is more involved in attention processes while the left homologue is more of a visuo-spatial workspace. A 4CAPS computational model of the cognitive processes and brain activation in the TOL task integrated these hypothesized mechanisms, and provided a reasonably good fit to the observed behavioral and brain activation data. The multiple research approaches presented here converge on a deepening understanding of the combination of perceptual and conceptual processes in this type of visual problem solving.

Differential effects of syntactic and semantic processing on the subregions of Broca's area

This study attempts to specify the contribution of two subregions of Brocas area during syntactic and semantic processing of sentences by examining brain activation in a grammaticality judgment task. The processing of two types of ungrammatical sentences was examined. One type leaves the thematic interpretation generally unaffected, by violating the noun-verb agreement in number, while the other type introduces an extraneous verb, which cannot be incorporated into the developing thematic structure. Pars triangularis was more sensitive to the extra verb violation, whereas pars opercularis was more sensitive to the noun-verb agreement violation. The current study adds to the growing literature that suggests there are separable functional subregions of Brocas area, with pars triangularis more involved in thematic processing and pars opercularis more involved in syntactic processing. The posterior left temporal area was also involved in both types of processing.

Imagery in sentence comprehension: an fMRI study.

This study examined brain activation while participants read or listened to high-imagery sentences like The number eight when rotated 90 degrees looks like a pair of spectacles or low-imagery sentences, and judged them as true or false. The sentence imagery manipulation affected the activation in regions (particularly, the intraparietal sulcus) that activate in other mental imagery tasks, such as mental rotation. Both the auditory and visual presentation experiments indicated activation of the intraparietal sulcus area in the high-imagery condition, suggesting a common neural substrate for language-evoked imagery that is independent of the input modality. In addition to exhibiting greater activation levels during the processing of high-imagery sentences, the left intraparietal sulcus also showed greater functional connectivity in this condition with other cortical regions, particularly language processing regions, regardless of the input modality. The comprehension of abstract, nonimaginal information in low-imagery sentences was accompanied by additional activation in regions in the left superior and middle temporal areas associated with the retrieval and processing of semantic and world knowledge. In addition to exhibiting greater activation levels during the processing of low-imagery sentences, this left temporal region also revealed greater functional connectivity in this condition with other left hemisphere language processing regions and with prefrontal regions, regardless of the input modality. The findings indicate that sentence comprehension can activate additional cortical regions that process information that is not specifically linguistic but varies with the information content of the sentence (such as visual or abstract information). In particular, the left intraparietal sulcus area appears to be centrally involved in processing the visual imagery that a sentence can evoke, while activating in synchrony with some core language processing regions.

Differences in auditory processing of words and pseudowords: an fMRI study.

Although there has been great interest in the neuroanatomical basis of reading, little attention has been focused on auditory language processing. The purpose of this study was to examine the differential neuroanatomical response to the auditory processing of real words and pseudowords. Eight healthy right‐handed participants performed two phoneme monitoring tasks (one with real word stimuli and one with pseudowords) during a functional magnetic resonance imaging (fMRI) scan with a 4.1 T system. Both tasks activated the inferior frontal gyrus (IFG), the posterior superior temporal gyrus (pSTG) and the inferior parietal lobe (IPL). Pseudoword processing elicited significantly more activation within the posterior cortical regions compared with real word processing. Previous reading studies have suggested that this increase is due to an increased demand on the lexical access system. The left inferior frontal gyrus, on the other hand, did not reveal a significant difference in the amount of activation as a function of stimulus type. The lack of a differential response in IFG for auditory processing supports its hypothesized involvement in grapheme to phoneme conversion processes. These results are consistent with those from previous neuroimaging reading studies and emphasize the utility of examining both input modalities (e.g., visual or auditory) to compose a more complete picture of the language network. Hum. Brain Mapping 14:39–47, 2001.

Baseline conditions and subtractive logic in neuroimaging

Discrepancies in the patterns of cortical activation across studies may be attributable, in part, to differences in baseline tasks, and hence, reflect the limits of the subtractive logic underlying much of neuroimaging. To assess the extent of these effects, three of the most commonly used baseline conditions (rest, tone monitoring, and passive listening) were compared using phoneme discrimination as the experimental task. Eight participants were studied in a fMRI study with a 4.1 T system. The three baseline conditions systematically affected the amount of activation observed in the identical phoneme task with major affects in Brocas area, the left posterior superior temporal gyrus, and the left and right inferior parietal regions. Two central findings were: 1) a differential effect of baseline within each region, with the rest baseline condition producing the greatest amount of activation and the passive listening condition producing the least, and 2) systematic baseline task activation in the inferior parietal regions. These results emphasize the relativity of activation patterns observed in functional neuroimaging, and the necessity to specify the baseline processes in context to the experimental task processes. Hum. Brain Mapping 14:228–235, 2001.

The effect of semantic relatedness on syntactic analysis: An fMRI study

The sentences we process in normal conversation tend to refer to information that we are familiar with rather than abstract, unrelated information. This allows for the use of knowledge stores to help facilitate comprehension processes. In many sentence comprehension studies, the stimuli are designed such that the use of world knowledge is limited. Here, we investigated how the semantic relatedness of sentence constituents influences sentence processing. A three factor design was employed in which processing phase (sentence vs. probe), syntactic complexity (object-relative vs. conjoined active) and the semantic relatedness of the nouns within the sentence was examined. We found a differential effect in two sub-regions of the left inferior frontal gyrus (LIFG). BA 44 revealed an effect of syntactic complexity while inferior portions of the LIFG (BA 47) revealed an effect of relatedness as well as an interaction between complexity and relatedness during the probe phase. In addition, significant differences in activation were observed when comparing the sentence processing and probe phases with the sentence phase eliciting stronger semantic related activation while the probe phase elicited stronger working memory related activation.

The synchronization of the human cortical working memory network.

A verbal reasoning problem at the intersection of verbal working memory, problem-solving, and language comprehension was examined using event-related fMRI to distinguish differences in the differential timing of the response of the various cortical regions that compose the working memory network. Problems were developed such that the process demand as well as the timing of the manipulation of the contents of working memory (i.e., a demanding computation) was varied. Activation was observed in several regions including the dorsolateral prefrontal cortex, the inferior frontal gyrus, and the parietal lobe. Examination of the MR amplitude response revealed that the regions do not all activate simultaneously; instead, their activation time courses reveal differential responses that correspond to their theoretical processing role in the problem-solving task. The coordination of cortical area responses reveals how the various cortical regions synchronize and collaborate in order to accomplish a given cognitive function.

Volitional control of attention and brain activation in dual task performance.

This study used functional MRI (fMRI) to examine the neural effects of willfully allocating ones attention to one of two ongoing tasks. In a dual task paradigm, participants were instructed to focus either on auditory sentence comprehension, mental rotation, or both. One of the major findings is that the distribution of brain activation was amenable to strategic control, such that the amount of activation per task was systematically related to the attention‐dividing instructions. The activation in language processing regions was lower when attending to mental rotation than when attending to the sentences, and the activation in visuospatial processing regions was lower when attending to sentences than when attending to mental rotations. Additionally, the activation was found to be underadditive, with the dual‐task condition eliciting less activation than the sum of the attend sentence and attend rotation conditions. We also observed a laterality shift across conditions within language‐processing regions, with the attend sentence condition showing bilateral activation, while the dual task condition showed a left hemispheric dominance. This shift suggests multiple language‐processing modes and may explain the underadditivity in activation observed in the current and previous studies. Hum. Brain Mapp, 2007.

An fMRI study of the Tower of London: A look at problem structure differences

The aim of the current study was to explore the effects of problem structure, namely goal hierarchy and number of optimal solution paths, on the neural architecture that supports problem-solving and planning. Here, six-move problems with both an unambiguous and ambiguous goal hierarchy and single and multiple optimal solution paths were examined. In the task used, participants were encouraged to generate a solution plan before execution. The behavioral results revealed that problem-solving time and accuracy were both affected by both problem parameters. The fMRI activation results revealed three major findings. First, the right prefrontal cortex revealed a significantly different activation pattern than the other regions examined. This was the only region that revealed a larger response during the execution phase than the planning phase. Second, the effect of goal hierarchy was strongest during the execution phase. Finally, while there was no main effect of number of optimal solution paths, this parameter interacted with goal hierarchy in a number of regions across the brain. The present study also suggests that the minimum number of moves may not be the best measure of problem difficulty and that greater care be taken in the selection of TOL problems for both experimental studies as well as clinical assessment.