Richard Veale

How is visual salience computed in the brain? Insights from behaviour, neurobiology and modelling

Inherent in visual scene analysis is a bottleneck associated with the need to sequentially sample locations with foveating eye movements. The concept of a ‘saliency map’ topographically encoding stimulus conspicuity over the visual scene has proven to be an efficient predictor of eye movements. Our work reviews insights into the neurobiological implementation of visual salience computation. We start by summarizing the role that different visual brain areas play in salience computation, whether at the level of feature analysis for bottom-up salience or at the level of goal-directed priority maps for output behaviour. We then delve into how a subcortical structure, the superior colliculus (SC), participates in salience computation. The SC represents a visual saliency map via a centre-surround inhibition mechanism in the superficial layers, which feeds into priority selection mechanisms in the deeper layers, thereby affecting saccadic and microsaccadic eye movements. Lateral interactions in the local SC circuit are particularly important for controlling active populations of neurons. This, in turn, might help explain long-range effects, such as those of peripheral cues on tiny microsaccades. Finally, we show how a combination of in vitro neurophysiology and large-scale computational modelling is able to clarify how salience computation is implemented in the local circuit of the SC. This article is part of the themed issue ‘Auditory and visual scene analysis’.

Saliency-guided neural prosthesis for visual attention: Design and simulation

Recently the authors showed that a computational model of visual saliency could account for changes in gaze behavior of monkeys with damage in the primary visual cortex. Here we propose a neural prosthesis to restore eye gaze behavior by electrically stimulating the superior colliculus to drive visual attention. The saliency computational model is used to calculate the stimulation parameters from a real-time camera stream. Our simulations demonstrate that electrodes implanted in the superior colliculus at 1.0mm spacing are, in principle, able to recover specifically those visual attention behaviors which are lost when the primary visual cortex is damaged.

Applying differential evolution MCMC to parameterize large-scale spiking neural simulations

Parameterization of computationally expensive forward models of the brain is a novel research area that has only recently become tractable due to supercomputing resources. However, there is a lack of examples demonstrating how to achieve accurate estimates of neural, synaptic, and connectivity parameters of large-scale brain simulations within a reasonable period of time. We present the novel application of MT-DREAMZ, an existing parallel differential evolution Markov chain monte carlo (MCMC) method, to estimate the parameters of a complex spiking neuron circuit simulation in a short period of time. The parameters are estimated so that the neural simulations match empirical data collected from a midbrain visual/attention region called the superior colliculus. The results of the parameter sweeps reveal several regions of parameter space that fit the empirical data. The highest likelihood parameter regions show regularities consistent with anatomical properties of the original brain region, such as the wide horizontal inhibitory neurons. Our results demonstrate that evolutionary statistical techniques are highly effective tools for investigating complex models of the brain due to their efficiency and parallelizability. Not only do the sweeps find good fits for an incredibly complex non-linear problem, but the resulting posterior likelihood distributions show patterns that fit independently obtained data from anatomical and physiological studies of the same brain region that were not included in the fitness function. This demonstrates that hybrid evolutionary methods can be applied to increase our understanding of the underlying structural and dynamic properties of the brain using only reasonable amounts of behavioral data.

Temporal, Environmental, and Social Constraints of Word-Referent Learning in Young Infants: A Neurorobotic Model of Multimodal Habituation

Infants are able to adaptively associate auditory stimuli with visual stimuli even in their first year of life, as demonstrated by multimodal habituation studies. Different from language acquisition during later developmental stages, this adaptive learning in young infants is temporary and still very much stimulus-driven. Hence, temporal aspects of environmental and social factors figure crucially in the formation of prelexical multimodal associations. Study of these associations can offer important clues regarding how semantics are bootstrapped in real-world embodied infants. In this paper, we present a neuroanatomically based embodied computational model of multimodal habituation to explore the temporal and social constraints on the learning observed in very young infants. In particular, the model is able to explain empirical results showing that auditory word stimuli must be presented synchronously with visual stimulus movement for the two to be associated.

A parallelized dynamic programming approach to zero resource spoken term discovery

Zero resource spoken term discovery in continuous speech is the discovery of repeated patterns in acoustic signals without any higher level linguistic information. These patterns are then combined to define the compositional units of that speech. We describe and implement an algorithm that tags similar subsequences among sequences of acoustic features. We then discuss the use of this algorithm as part of a complete spoken term discovery system. Our implementation leverages parallelization via modern GPUs, allowing many independent comparisons to be executed concurrently. This parallelization enables the described system to analyze large data sets in tractable time frames. The accuracy and performance of our approach are compared to existing approaches as well as human transcriptions on two corpora of continuous natural speech. Our system improved on published results for multiple metrics.

Auditory habituation via spike-timing dependent plasticity in recurrent neural circuits

This paper presents evidence that spiking neuron models of parts of the human auditory system demonstrate habituation to real auditory word stimuli. This is accomplished via the simple addition of a model of spike-timing dependent plasticity to synapses. This result is interesting because the base neural circuit has also been used for pragmatically useful behaviors such as speech recognition. The model increases our understanding of the neural basis for behaviors in the developing human infant by showing that habituation learning can be implemented in the same neural substrate that underlies other types of learning (such as permanent word-learning).

Windshield Frame Shape and Awareness of the External World While Driving an Automobile

The vehicle windshield is supported and framed by the hood, roof, and pillars, which occlude the driver’s view of the outside. It has been previously shown that awareness of the external world changes according to differences in windshield shape. This directly affects the drivability of a vehicle. Thus, the windshield shape must be designed by considering driver’s visual performance so that it can be balanced with other performance measures such as weight and roominess to design the optimal cockpit. Visual performance during driving is affected by (1) bottom-up attention and (2) top-down attention, and (3) selection between them. This study focuses on Itti and Koch’s visual saliency that attracts bottom-up attention as a visual scene changes in shape and color around front windshield frame during driving. This paper aims to quantify the relationships between drivers’ gaze movements and visual saliency.

Technical Implementation of a Visual Attention Neuro-Prosthesis

Brain damage to visual cortex causes hemianopia, along with significant changes in looking behavior. Previously, we have proposed a visual attention neuro-prosthesis to correct the distribution of visual aattention in patients with damage to visual cortex. The prosthesis consists of an eye tracker, a forward-facing camera, and electric micro stimulation in superior colliculus (SC), a midbrain structure related to visual attention. We demonstrate in this paper that the prosthesis can be built using available hardware, and the only technical hurdle remaining is to implant stimulating electrodes of sufficient density in the midbrain.

Spatiotemporal dynamics in spiking simulations of superior colliculus fit via MCMC suggest disinhibition responsible for superlinear summation

The superior colliculus (SC) is a midbrain region with visually responsive neurons in the superficial layers and eye movement controlling neurons in the deeper layers. Recently, [1] performed in vitro experiments to elucidate lateral interactions within horizontal slices of the SC (Figure ​(Figure1A).1A). The experiments indicate that the superficial (visual) layers implement surround inhibition, and furthermore that strong stimulation at two adjacent locations (separation ~150 μm) produces an unexpected super-linear summation that is not seen in the deeper layers (Figure ​(Figure1B,1B, a+b). We used differential evolution Markov-chain Monte Carlo (MCMC) to estimate the parameters of a large-scale spiking neural circuit simulation to fit the slice data [2]. The model contains populations of inhibitory and excitatory neurons as well as input axons from retina/cortex. We included free anatomical (dendrite/axon spread) and dynamic (short-term synaptic plasticity) hyper-parameters in the model, and used MCMC estimate the posterior distribution of parameters that is most likely given the slice data. The resulting marginal distributions show promising agreement with verifiable anatomical parameters such as the lateral spread of dendrites and axons of the inhibitory and excitatory neuron populations in the superficial colliculus, even though no such constraints were coded into the model [2]. However, the posterior distributions for non-intuitive parameters (such as synaptic efficacies and facilitation/depression time constants) cannot be verified directly with existing data. Furthermore, it is not clear what the role of the dynamical parameters is in producing the behavior of the best-fit models, for example the local superlinearity described above. In this work, we take the additional step of analyzing the spatio-temporal dynamics of one of the best-fit regions of parameter space found via MCMC. The purpose is to provide a mechanistic explanation for the super-linear summation observed during two-point stimulation. Figure ​Figure1D1D shows the difference in spatio-temporal dynamics between actual two-point stimulation versus the linear sum of two independent stimulations. Thus, positive values indicate an increase in conductance sent to neurons at the horizontal axis position, arriving from neurons at the position indicated by the vertical axis (as explained in 1C). 16 ms from stimulation onset, there is a large increase in flow of inhibitory input from near the center of the circuit to inhibitory neurons all around the circuit, thus disinhibiting the circuit. 5 ms later, the excitatory neurons near the center receive input from excitatory neurons near the middle (i.e. recurrent activity), suggesting that cause for the super-linearity. Figure 1 (A) Experimental setup of [1], showing slice containing excitatory (red) and inhibitory (gray) neurons. Stimulation applied to electrodes at lateral distances while recording from an excitatory neuron. (B) Empirical results (dotted) and model fits. Purple: ...

Large-scale spiking circuit simulation of spatio-temporal dynamics in superior colliculus

We present computational simulations of the intrinsic circuitry of the superior colliculus using large-scale spiking neural circuit models. We reproduce recent results from slice experiments that showed different spatio-temporal patterns of interaction within the visual layers versus the eye-movement related layers of the superior colliculus. Specifically, the receptive fields of neurons in the visual layers implement a “center-surround” pattern of spatial competition, and furthermore additional input within the central region sums super-linearly. In contrast, the receptive fields of neurons in the motor-related regions implement spatially symmetric fields of overlaid excitation and inhibition, and additional inputs sum linearly. Our simulations investigate the circuit mechanisms and dynamics that differentiate the computational roles of these distinct but related regions. We constructed full-scale simulations of mouse brain slices using spiking neuron models. Connection parameters were then fit within physiological constraints to reproduce experimental data. Figure ​Figure1A1A shows results from a best fit of the superficial (visual) layers. Stimulation was applied at different horizontal distances from a central neuron in a tangential slice. The figure shows integrated post-synaptic potential (PSP) during stimulation at the indicated distance (purple). We also reproduced data from a “two-point” paradigm in which stimulation was simultaneously applied to the indicated electrode and the central electrode, simulating a “large” stimulus (red). The net response of the neuron sums super-linearly in comparison to the linear summation of the results from the two points independently (naive summation in black). Insets show corresponding slice data. Interestingly, anatomically distinct inhibitory and excitatory populations were not necessary to reproduce spatial asymmetry (“center-surround”) in the visual superficial layers. Rather, asymmetries in the temporal properties (synaptic dynamics) of connections were better able to account for observed data under all conditions. A further population of small disinhibitory neurons was necessary to maintain linear summation in the periphery while summing super-linearly in the center. Figure 1 This contrasts with the motor-related intermediate layers (Figure ​(Figure1B),1B), whose response to stimulation was best explained by both spatial and temporal symmetry between inhibitory and excitatory neural connections. Overall, the regions seem to have taken advantage of both spatial and temporal dynamics in their connections to specialize their computational function: The visual layers seem geared towards spatial competition and strengthening of salient stimuli, whereas the motor-related regions seem geared towards broad and long-term integration of input.