Steven Scholte

Convolutional Neural Networks in the Brain: an fMRI study

Biologically inspired computational models replicate the hierarchical visual processing in the human ventral stream. One such recent model, Convolutional Neural Network (CNN) has achieved state of the art performance on automatic visual recognition tasks. The CNN architecture contains successive layers of convolution and pooling, and resembles the simple and complex cell hierarchy as proposed by Hubel and Wiesel. This makes it a candidate model to test against the human brain. In this study we look at 1) where in the brain different layers of the CNN account for brain responses, and 2) how the CNN network compares against existing and widely used hierarchical vision models such as Bag-of-Words (BoW) and HMAX. fMRI brain activity of 20 subjects obtained while viewing a short video clip was analyzed voxel-wise using a distance-based variation partitioning method. Variation partitioning was done on successive CNN layers to determine the unique contribution of each layers in explaining fMRI brain activity. We observe that each of the 7 different layers of CNN accounts for brain activity consistently across subjects in areas known to be involved in visual processing. In addition, we find a relation between the visual processing hierarchy in the brain and the 7 CNN layers: visual areas such as V1, V2 and V3 are sensitive to lower layers of the CNN while areas such as LO, TO and PPA are sensitive to higher layers. The comparison of CNN with HMAX and BoW furthermore shows that while all three models explain brain activity in early visual areas, the CNN additionally explains brain activity deeper in the brain. Overall, our results suggest that Convolutional Neural Networks provide a suitable computational basis for visual processing in the brain, allowing to decode feed-forward representations in the visual brain. Meeting abstract presented at VSS 2015.

The neural basis of different types of synesthesia

The condition synesthesia offers an extraordinary opportunity to study mechanisms involved in (hyper)-binding across different modalities. Currently, the majority of synesthesia research is aimed at a few types of synesthesia; colors evoked by written or spoken language and spatial arrangements evoked by sequential concepts. We do not yet know whether the processes involved in multisensory binding are similar or different across different types of synesthesia. In this project, brain structure measurements (connectivity and volume) were obtained from a large group of subjects (over 350 individuals). Furthermore, the subjects were presented with an extensive synesthesia questionnaire. In this subject group, different types of synesthetes were present; the inducers and concurrent ranged across perceptual modalities (e.g., visual, auditory, taste, tactile) but also involved non-sensory concepts (e.g., ‘personality’). This allows studying which of the previously obtained white-matter and grey-matter differences in synesthesia are specific to that (i.e., linguistic–color) type of synesthesia, and which are related more generally to having synesthesia. Furthermore, it allows examining whether synesthetes are, as a group, categorically different from all non-synesthetic subjects. Alternatively, each synesthete is only different from a non-synesthete in the binding of his or her specific inducer to his or her specific concurrent.