Jukka-Pekka Kauppi

Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency

Cinema is a promising naturalistic stimulus that enables, for instance, elicitation of robust emotions during functional magnetic resonance imaging (fMRI). Inter-subject correlation (ISC) has been used as a model-free analysis method to map the highly complex hemodynamic responses that are evoked during watching a movie. Here, we extended the ISC analysis to frequency domain using wavelet analysis combined with non-parametric permutation methods for making voxel-wise statistical inferences about frequency-band specific ISC. We applied these novel analysis methods to a dataset collected in our previous study where 12 subjects watched an emotionally engaging movie “Crash” during fMRI scanning. Our results suggest that several regions within the frontal and temporal lobes show ISC predominantly at low frequency bands, whereas visual cortical areas exhibit ISC also at higher frequencies. It is possible that these findings relate to recent observations of a cortical hierarchy of temporal receptive windows, or that the types of events processed in temporal and prefrontal cortical areas (e.g., social interactions) occur over longer time periods than the stimulus features processed in the visual areas. Software tools to perform frequency-specific ISC analysis, together with a visualization application, are available as open source Matlab code.

A versatile software package for inter-subject correlation based analyses of fMRI

In the inter-subject correlation (ISC) based analysis of the functional magnetic resonance imaging (fMRI) data, the extent of shared processing across subjects during the experiment is determined by calculating correlation coefficients between the fMRI time series of the subjects in the corresponding brain locations. This implies that ISC can be used to analyze fMRI data without explicitly modeling the stimulus and thus ISC is a potential method to analyze fMRI data acquired under complex naturalistic stimuli. Despite of the suitability of ISC based approach to analyze complex fMRI data, no generic software tools have been made available for this purpose, limiting a widespread use of ISC based analysis techniques among neuroimaging community. In this paper, we present a graphical user interface (GUI) based software package, ISC Toolbox, implemented in Matlab for computing various ISC based analyses. Many advanced computations such as comparison of ISCs between different stimuli, time window ISC, and inter-subject phase synchronization are supported by the toolbox. The analyses are coupled with re-sampling based statistical inference. The ISC based analyses are data and computation intensive and the ISC toolbox is equipped with mechanisms to execute the parallel computations in a cluster environment automatically and with an automatic detection of the cluster environment in use. Currently, SGE-based (Oracle Grid Engine, Son of a Grid Engine, or Open Grid Scheduler) and Slurm environments are supported. In this paper, we present a detailed account on the methods behind the ISC Toolbox, the implementation of the toolbox and demonstrate the possible use of the toolbox by summarizing selected example applications. We also report the computation time experiments both using a single desktop computer and two grid environments demonstrating that parallelization effectively reduces the computing time. The ISC Toolbox is available in https://code.google.com/p/isc-toolbox/

Inter-Subject Correlation in fMRI: Method Validation against Stimulus-Model Based Analysis

Within functional magnetic resonance imaging (fMRI), the use of the traditional general linear model (GLM) based analysis methods is often restricted to strictly controlled research setups requiring a parametric activation model. Instead, Inter-Subject Correlation (ISC) method is based on voxel-wise correlation between the time series of the subjects, which makes it completely non-parametric and thus suitable for naturalistic stimulus paradigms such as movie watching. In this study, we compared an ISC based analysis results with those of a GLM based in five distinct controlled research setups. We used International Consortium for Brain Mapping functional reference battery (FRB) fMRI data available from the Laboratory of Neuro Imaging image data archive. The selected data included measurements from 37 right-handed subjects, who all had performed the same five tasks from FRB. The GLM was expected to locate activations accurately in FRB data and thus provide good grounds for investigating relationship between ISC and stimulus induced fMRI activation. The statistical maps of ISC and GLM were compared with two measures. The first measure was the Pearsons correlation between the non-thresholded ISC test-statistics and absolute values of the GLM Z-statistics. The average correlation value over five tasks was 0.74. The second was the Dice index between the activation regions of the methods. The average Dice value over the tasks and three threshold levels was 0.73. The results of this study indicated how the data driven ISC analysis found the same foci as the model-based GLM analysis. The agreement of the results is highly interesting, because ISC is applicable in situations where GLM is not suitable, for example, when analyzing data from a naturalistic stimuli experiment.

Towards brain-activity-controlled information retrieval: Decoding image relevance from MEG signals

We hypothesize that brain activity can be used to control future information retrieval systems. To this end, we conducted a feasibility study on predicting the relevance of visual objects from brain activity. We analyze both magnetoencephalographic (MEG) and gaze signals from nine subjects who were viewing image collages, a subset of which was relevant to a predetermined task. We report three findings: i) the relevance of an image a subject looks at can be decoded from MEG signals with performance significantly better than chance, ii) fusion of gaze-based and MEG-based classifiers significantly improves the prediction performance compared to using either signal alone, and iii) non-linear classification of the MEG signals using Gaussian process classifiers outperforms linear classification. These findings break new ground for building brain-activity-based interactive image retrieval systems, as well as for systems utilizing feedback both from brain activity and eye movements.

Differences in fMRI intersubject correlation while viewing unedited and edited videos of dance performance

Intersubject correlation (ISC) analysis of functional magnetic resonance imaging (fMRI) data provides insight into how continuous streams of sensory stimulation are processed by groups of observers. Although edited movies are frequently used as stimuli in ISC studies, there has been little direct examination of the effect of edits on the resulting ISC maps. In this study we showed 16 observers two audiovisual movie versions of the same dance. In one experimental condition there was a continuous view from a single camera (Unedited condition) and in the other condition there were views from different cameras (Edited condition) that provided close up views of the feet or face and upper body. We computed ISC maps for each condition, as well as created a map that showed the difference between the conditions. The results from the Unedited and Edited maps largely overlapped in the occipital and temporal cortices, although more voxels were found for the Edited map. The difference map revealed greater ISC for the Edited condition in the Postcentral Gyrus, Lingual Gyrus, Precentral Gyrus and Medial Frontal Gyrus, while the Unedited condition showed greater ISC in only the Superior Temporal Gyrus. These findings suggest that the visual changes associated with editing provide a source of correlation in maps obtained from edited film, and highlight the utility of using maps to evaluate the difference in ISC between conditions.

Hierarchical classification of dynamically varying radar pulse repetition interval modulation patterns

The central purpose of passive signal intercept receivers is to perform automatic categorization of unknown radar signals. Currently, there is an urgent need to develop intelligent classification algorithms for these devices due to emerging complexity of radar waveforms. Especially multifunction radars (MFRs) capable of performing several simultaneous tasks by utilizing complex, dynamically varying scheduled waveforms are a major challenge for automatic pattern classification systems. To assist recognition of complex radar emissions in modern intercept receivers, we have developed a novel method to recognize dynamically varying pulse repetition interval (PRI) modulation patterns emitted by MFRs. We use robust feature extraction and classifier design techniques to assist recognition in unpredictable real-world signal environments. We classify received pulse trains hierarchically which allows unambiguous detection of the subpatterns using a sliding window. Accuracy, robustness and reliability of the technique are demonstrated with extensive simulations using both static and dynamically varying PRI modulation patterns.

Functional Brain Segmentation Using Inter-Subject Correlation in fMRI

The human brain continuously processes massive amounts of rich sensory information. To better understand such highly complex brain processes, modern neuroimaging studies are increasingly utilizing experimental setups that better mimic daily‐life situations. A new exploratory data‐analysis approach, functional segmentation inter‐subject correlation analysis (FuSeISC), was proposed to facilitate the analysis of functional magnetic resonance (fMRI) data sets collected in these experiments. The method provides a new type of functional segmentation of brain areas, not only characterizing areas that display similar processing across subjects but also areas in which processing across subjects is highly variable. FuSeISC was tested using fMRI data sets collected during traditional block‐design stimuli (37 subjects) as well as naturalistic auditory narratives (19 subjects). The method identified spatially local and/or bilaterally symmetric clusters in several cortical areas, many of which are known to be processing the types of stimuli used in the experiments. The method is not only useful for spatial exploration of large fMRI data sets obtained using naturalistic stimuli, but also has other potential applications, such as generation of a functional brain atlases including both lower‐ and higher‐order processing areas. Finally, as a part of FuSeISC, a criterion‐based sparsification of the shared nearest‐neighbor graph was proposed for detecting clusters in noisy data. In the tests with synthetic data, this technique was superior to well‐known clustering methods, such as Wards method, affinity propagation, and K‐means ++ . Hum Brain Mapp 38:2643–2665, 2017.

Face prediction from fMRI data during movie stimulus: strategies for feature selection

We investigate the suitability of the multi-voxel pattern analysis approach to analyze diverse movie stimulus functional magnetic resonance imaging (fMRI) data. We focus on predicting the presence of faces in the drama movie based on the fMRI measurements of 12 subjects watching the movie. We pose the prediction as a regression problem where regression coefficients estimated from the training data are used to estimate the presence of faces in the stimulus for the test data. Because the number of features (voxels) exceeds the number of training samples, an emphasis is placed on the feature selection. We compare four automatic feature selection approaches. The best results were achieved by sparse regression models. The correlations between the face presence time-course predicted from fMRI data and manual face annotations were in the range from 0.43 to 0.62 depending on the subject and pre-processing options, i.e., the prediction was successful. This suggests that proposed methods are useful in testing novel research hypotheses with natural stimulus fMRI data.

Clustering inter-subject correlation matrices in functional magnetic resonance imaging

We present a novel clustering method to probe inter-subject variability in functional magnetic resonance imaging (fMRI) data acquired in complex audiovisual stimulus environments, such as during watching movies. We calculate voxel-wise inter-subject correlation matrices across individual subject fMRI time-series and cluster them over the cerebral cortex. We address correlation matrix clustering problem and modify a standard K-means algorithm to cope better with spurious observations. We investigate suitability of the modified K-means with hierarchical clustering based postprocessing to correlation matrix clustering with several artificially generated data sets. We also present clustering of fMRI movie data. Preliminary results suggest that our methodology can be a valuable tool to investigate inter-subject variability in brain activity in different brain regions, such as prefrontal cortex.

Comparing fMRI inter-subject correlations between groups

Inter-subject correlation (ISC) based analysis is a conceptually simple approach to analyze functional magnetic resonance imaging (fMRI) data acquired under naturalistic stimuli such as a movie. We describe and validate the statistical approaches for comparing ISCs between two groups of subjects implemented in the ISC toolbox, which is an open source software package for ISC-based analysis of fMRI data. The approaches are based on permutation tests. We validated the approaches using five different data sets from the ICBM functional reference battery tasks. In these experiments, we created two matched groups of subjects and assumed that no group difference exists. Based on the experiments, we recommend the usage of subject-wise permutations, instead of element-wise permutations following Chen et al. (2016). However, we observed that the null-distributions should be voxel-specific and not based on pooling all voxels across the brain as is typical in fMRI. This was the case even if studentized permutation tests were used. Additionally, we experimented with an fMRI dataset acquired using a dance movie stimulus for comparison of the group of adult males in autism spectrum to the matched control group. The experiment confirmed the differences between voxel-based permutation tests and global model based permutation tests.