Rajagopal Ananthanarayanan

The cat is out of the bag: cortical simulations with 10 9 neurons, 10 13 synapses

In the quest for cognitive computing, we have built a massively parallel cortical simulator, C2, that incorporates a number of innovations in computation, memory, and communication. Using C2 on LLNLs Dawn Blue Gene/P supercomputer with 147, 456 CPUs and 144 TB of main memory, we report two cortical simulations -- at unprecedented scale -- that effectively saturate the entire memory capacity and refresh it at least every simulated second. The first simulation consists of 1.6 billion neurons and 8.87 trillion synapses with experimentally-measured gray matter thalamocortical connectivity. The second simulation has 900 million neurons and 9 trillion synapses with probabilistic connectivity. We demonstrate nearly perfect weak scaling and attractive strong scaling. The simulations, which incorporate phenomenological spiking neurons, individual learning synapses, axonal delays, and dynamic synaptic channels, exceed the scale of the cat cortex, marking the dawn of a new era in the scale of cortical simulations.

Anatomy of a cortical simulator

Insights into brains high-level computational principles will lead to novel cognitive systems, computing architectures, programming paradigms, and numerous practical applications. An important step towards this end is the study of large networks of cortical spiking neurons. We have built a cortical simulator, C2, incorporating several algorithmic enhancements to optimize the simulation scale and time, through: computationally efficient simulation of neurons in a clock-driven and synapses in an event-driven fashion; memory efficient representation of simulation state; and communication efficient message exchanges. Using phenomenological, single-compartment models of spiking neurons and synapses with spike-timing dependent plasticity, we represented a rat-scale cortical model (55 million neurons, 442 billion synapses) in STB memory of a 32, 768-processor BlueGene/L. With 1 millisecond resolution for neuronal dynamics and 1--20 milliseconds axonal delays, C2 can simulate 1 second of model time in 9 seconds per Hertz of average neuronal firing rate. In summary, by combining state-of-the-art hardware with innovative algorithms and software design, we simultaneously achieved unprecedented time-to-solution on an unprecedented problem size.

Think Global, Act Local; Projectome Estimation with BlueMatter

Estimating the complete set of white matter fascicles (the projectome) from diffusion data requires evaluating an enormous number of potential pathways; consequently, most algorithms use computationally efficient greedy methods to search for pathways. The limitation of this approach is that critical global parameters--such as data prediction error and white matter volume conservation--are not taken into account. We describe BlueMatter, a parallel algorithm for global projectome evaluation, which uniquely accounts for global prediction error and volume conservation. Leveraging the BlueGene/L supercomputing architecture, BlueMatter explores a massive database of 180 billion candidate fascicles. The candidates are derived from several sources, including atlases and multiple tractography algorithms. Using BlueMatter we created the highest resolution, volume-conserved projectome of the human brain.

Panache: a parallel WAN cache for clustered filesystems

Panache is a scalable, high-performance, remote file data caching solution integrated with the GPFS cluster file system. It leverages the inherent scalability of GPFS to provide a multi-node, consistent cache of data exported by a remote file system cluster. Panache exploits the soon-to-be standard pNFS protocol to move data in parallel from the remote file cluster. Furthermore, it provides a POSIX compliant file system interface making the cache completely transparent to applications. Panache can mask the fluctuating wide-area-network latencies and outages by supporting asynchronous and disconnected-mode operations. It allows concurrent updates to be made at the cache and at the remote cluster and synchronizes them by using conflict detection techniques to flag and handle conflicts. To maintain commercial viability, Panache relies on open standards for high-performance file serving and does not require any proprietary hardware or software to be deployed at the remote cluster. In this paper we present the overall architecture of Panache and its key features.