Kun-Han Lu

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing the bi-directional relationships between the CNN and the brain. Through the encoding models, the CNN-predicted areas covered not only the ventral stream, but also the dorsal stream, albeit to a lesser degree; single-voxel response was visualized as the specific pixel pattern that drove the response, revealing the distinct representation of individual cortical location; cortical activation was synthesized from natural images with high-throughput to map category representation, contrast, and selectivity. Through the decoding models, fMRI signals were directly decoded to estimate the feature representations in both visual and semantic spaces, for direct visual reconstruction and semantic categorization, respectively. These results corroborate, generalize, and extend previous findings, and highlight the value of using deep learning, as an all-in-one model of the visual cortex, to understand and decode natural vision.

Mapping white-matter functional organization at rest and during naturalistic visual perception

Abstract Despite the wide applications of functional magnetic resonance imaging (fMRI) to mapping brain activation and connectivity in cortical gray matter, it has rarely been utilized to study white‐matter functions. In this study, we investigated the spatiotemporal characteristics of fMRI data within the white matter acquired from humans both in the resting state and while watching a naturalistic movie. By using independent component analysis and hierarchical clustering, resting‐state fMRI data in the white matter were de‐noised and decomposed into spatially independent components, which were further assembled into hierarchically organized axonal fiber bundles. Interestingly, such components were partly reorganized during natural vision. Relative to resting state, the visual task specifically induced a stronger degree of temporal coherence within the optic radiations, as well as significant correlations between the optic radiations and multiple cortical visual networks. Therefore, fMRI contains rich functional information about the activity and connectivity within white matter at rest and during tasks, challenging the conventional practice of taking white‐matter signals as noise or artifacts. HighlightsICA applied to white‐matter fMRI signals reveals reproducible and hierarchical patterns.White‐matter ICA components are mostly preserved, but are in part distinct between the resting state and the task state.The distinction is specific to the axonal fibers involved in the task execution.White‐matter fMRI data are not noise or artifacts, but instead are signals of likely neuronal origin.

Influences of High-Level Features, Gaze, and Scene Transitions on the Reliability of BOLD Responses to Natural Movie Stimuli.

Complex, sustained, dynamic, and naturalistic visual stimulation can evoke distributed brain activities that are highly reproducible within and across individuals. However, the precise origins of such reproducible responses remain incompletely understood. Here, we employed concurrent functional magnetic resonance imaging (fMRI) and eye tracking to investigate the experimental and behavioral factors that influence fMRI activity and its intra- and inter-subject reproducibility during repeated movie stimuli. We found that widely distributed and highly reproducible fMRI responses were attributed primarily to the high-level natural content in the movie. In the absence of such natural content, low-level visual features alone in a spatiotemporally scrambled control stimulus evoked significantly reduced degree and extent of reproducible responses, which were mostly confined to the primary visual cortex (V1). We also found that the varying gaze behavior affected the cortical response at the peripheral part of V1 and in the oculomotor network, with minor effects on the response reproducibility over the extrastriate visual areas. Lastly, scene transitions in the movie stimulus due to film editing partly caused the reproducible fMRI responses at widespread cortical areas, especially along the ventral visual pathway. Therefore, the naturalistic nature of a movie stimulus is necessary for driving highly reliable visual activations. In a movie-stimulation paradigm, scene transitions and individuals’ gaze behavior should be taken as potential confounding factors in order to properly interpret cortical activity that supports natural vision.

Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex

Goal-driven and feedforward-only convolutional neural networks (CNN) have been shown to be able to predict and decode cortical responses to natural images or videos. Here, we explored an alternative deep neural network, variational auto-encoder (VAE), as a computational model of the visual cortex. We trained a VAE with a five-layer encoder and a five-layer decoder to learn visual representations from a diverse set of unlabeled images. Inspired by the “free-energy” principle in neuroscience, we modeled the brain’s bottom-up and top-down pathways using the VAE’s encoder and decoder, respectively. Following such conceptual relationships, we used VAE to predict or decode cortical activity observed with functional magnetic resonance imaging (fMRI) from three human subjects passively watching natural videos. Compared to CNN, VAE resulted in relatively lower accuracies for predicting the fMRI responses to the video stimuli, especially for higher-order ventral visual areas. However, VAE offered a more convenient strategy for decoding the fMRI activity to reconstruct the video input, by first converting the fMRI activity to the VAE’s latent variables, and then converting the latent variables to the reconstructed video frames through the VAE’s decoder. This strategy was more advantageous than alternative decoding methods, e.g. partial least square regression, by reconstructing both the spatial structure and color of the visual input. Findings from this study support the notion that the brain, at least in part, bears a generative model of the visual world.

Musical Imagery Involves Wernicke’s Area in Bilateral and Anti-Correlated Network Interactions in Musicians

Musical imagery is the human experience of imagining music without actually hearing it. The neural basis of this mental ability is unclear, especially for musicians capable of engaging in accurate and vivid musical imagery. Here, we created a visualization of an 8-minute symphony as a silent movie and used it as real-time cue for musicians to continuously imagine the music for repeated and synchronized sessions during functional magnetic resonance imaging (fMRI). The activations and networks evoked by musical imagery were compared with those elicited by the subjects directly listening to the same music. Musical imagery and musical perception resulted in overlapping activations at the anterolateral belt and Wernicke’s area, where the responses were correlated with the auditory features of the music. Whereas Wernicke’s area interacted within the intrinsic auditory network during musical perception, it was involved in much more complex networks during musical imagery, showing positive correlations with the dorsal attention network and the motor-control network and negative correlations with the default-mode network. Our results highlight the important role of Wernicke’s area in forming vivid musical imagery through bilateral and anti-correlated network interactions, challenging the conventional view of segregated and lateralized processing of music versus language.

Vagal nerve stimulation triggers widespread responses and alters large-scale functional connectivity in the rat brain

Vagus nerve stimulation (VNS) is a therapy for epilepsy and depression. However, its efficacy varies and its mechanism remains unclear. Prior studies have used functional magnetic resonance imaging (fMRI) to map brain activations with VNS in human brains, but have reported inconsistent findings. The source of inconsistency is likely attributable to the complex temporal characteristics of VNS-evoked fMRI responses that cannot be fully explained by simplified response models in the conventional model-based analysis for activation mapping. To address this issue, we acquired 7-Tesla blood oxygenation level dependent fMRI data from anesthetized Sprague–Dawley rats receiving electrical stimulation at the left cervical vagus nerve. Using spatially independent component analysis, we identified 20 functional brain networks and detected the network-wise activations with VNS in a data-driven manner. Our results showed that VNS activated 15 out of 20 brain networks, and the activated regions covered >76% of the brain volume. The time course of the evoked response was complex and distinct across regions and networks. In addition, VNS altered the strengths and patterns of correlations among brain networks relative to those in the resting state. The most notable changes in network-network interactions were related to the limbic system. Together, such profound and widespread effects of VNS may underlie its unique potential for a wide range of therapeutics to relieve central or peripheral conditions.

Contrast-Enhanced Magnetic Resonance Imaging of Gastric Emptying and Motility in Rats

The assessment of gastric emptying and motility in humans and animals typically requires radioactive imaging or invasive measurements. Here, we developed a robust strategy to image and characterize gastric emptying and motility in rats based on contrast-enhanced magnetic resonance imaging (MRI) and computer-assisted image processing. The animals were trained to naturally consume a gadolinium-labeled dietgel while bypassing any need for oral gavage. Following this test meal, the animals were scanned under low-dose anesthesia for high-resolution T1-weighted MRI in 7 Tesla, visualizing the time-varying distribution of the meal with greatly enhanced contrast against non-gastrointestinal (GI) tissues. Such contrast-enhanced images not only depicted the gastric anatomy, but also captured and quantified stomach emptying, intestinal filling, antral contraction, and intestinal absorption with fully automated image processing. Over four postingestion hours, the stomach emptied by 27%, largely attributed to the emptying of the forestomach rather than the corpus and the antrum, and most notable during the first 30 min. Stomach emptying was accompanied by intestinal filling for the first 2 h, whereas afterward intestinal absorption was observable as cumulative contrast enhancement in the renal medulla. The antral contraction was captured as a peristaltic wave propagating from the proximal to distal antrum. The frequency, velocity, and amplitude of the antral contraction were on average 6.34 ± 0.07 contractions per minute, 0.67 ± 0.01 mm/s, and 30.58 ± 1.03%, respectively. These results demonstrate an optimized MRI-based strategy to assess gastric emptying and motility in healthy rats, paving the way for using this technique to understand GI diseases, or test new therapeutics in rat models.

fMRI Data for Human Subjects During Musical Perception and Imagery

This fMRI dataset includes the original stimuli and the BOLD fMRI responses for a musical imagery study.

Vagus nerve stimulation promotes gastric emptying by increasing pyloric opening measured with magnetic resonance imaging

Vagus nerve stimulation (VNS) is an emerging electroceutical therapy for remedying gastric disorders that are poorly managed by pharmacological treatments and/or dietary changes. Such therapy seems promising as the vagovagal neurocircuitry modulates the enteric nervous system to influence gastric functions.

Proton density–weighted laryngeal magnetic resonance imaging in systemically dehydrated rats

Dehydrated vocal folds are inefficient sound generators. Although systemic dehydration of the body is believed to induce vocal fold dehydration, this causative relationship has not been demonstrated in vivo. Here we investigate the feasibility of using in vivo proton density (PD)–weighted magnetic resonance imaging (MRI) to demonstrate hydration changes in vocal fold tissue following systemic dehydration in rats.