J. P. Grossman

Biomolecular Simulation: A Computational Microscope for Molecular Biology

Molecular dynamics simulations capture the behavior of biological macromolecules in full atomic detail, but their computational demands, combined with the challenge of appropriately modeling the relevant physics, have historically restricted their length and accuracy. Dramatic recent improvements in achievable simulation speed and the underlying physical models have enabled atomic-level simulations on timescales as long as milliseconds that capture key biochemical processes such as protein folding, drug binding, membrane transport, and the conformational changes critical to protein function. Such simulation may serve as a computational microscope, revealing biomolecular mechanisms at spatial and temporal scales that are difficult to observe experimentally. We describe the rapidly evolving state of the art for atomic-level biomolecular simulation, illustrate the types of biological discoveries that can now be made through simulation, and discuss challenges motivating continued innovation in this field.

Millisecond-scale molecular dynamics simulations on Anton

Anton is a recently completed special-purpose supercomputer designed for molecular dynamics (MD) simulations of biomolecular systems. The machines specialized hardware dramatically increases the speed of MD calculations, making possible for the first time the simulation of biological molecules at an atomic level of detail for periods on the order of a millisecond-about two orders of magnitude beyond the previous state of the art. Anton is now running simulations on a timescale at which many critically important, but poorly understood phenomena are known to occur, allowing the observation of aspects of protein dynamics that were previously inaccessible to both computational and experimental study. Here, we report Antons performance when executing actual MD simulations whose accuracy has been validated against both existing MD software and experimental observations. We also discuss the manner in which novel algorithms have been coordinated with Antons co-designed, application-specific hardware to achieve these results.

Anton, a special-purpose machine for molecular dynamics simulation

The ability to perform long, accurate molecular dynamics (MD) simulations involving proteins and other biological macro-molecules could in principle provide answers to some of the most important currently outstanding questions in the fields of biology, chemistry and medicine. A wide range of biologically interesting phenomena, however, occur over time scales on the order of a millisecond--about three orders of magnitude beyond the duration of the longest current MD simulations. In this paper, we describe a massively parallel machine called Anton, which should be capable of executing millisecond-scale classical MD simulations of such biomolecular systems. The machine, which is scheduled for completion by the end of 2008, is based on 512 identical MD-specific ASICs that interact in a tightly coupled manner using a specialized high-speed communication network. Anton has been designed to use both novel parallel algorithms and special-purpose logic to dramatically accelerate those calculations that dominate the time required for a typical MD simulation. The remainder of the simulation algorithm is executed by a programmable portion of each chip that achieves a substantial degree of parallelism while preserving the flexibility necessary to accommodate anticipated advances in physical models and simulation methods.

Anton 2: raising the bar for performance and programmability in a special-purpose molecular dynamics supercomputer

Anton 2 is a second-generation special-purpose supercomputer for molecular dynamics simulations that achieves significant gains in performance, programmability, and capacity compared to its predecessor, Anton 1. The architecture of Anton 2 is tailored for fine-grained event-driven operation, which improves performance by increasing the overlap of computation with communication, and also allows a wider range of algorithms to run efficiently, enabling many new software-based optimizations. A 512-node Anton 2 machine, currently in operation, is up to ten times faster than Anton 1 with the same number of nodes, greatly expanding the reach of all-atom bio molecular simulations. Anton 2 is the first platform to achieve simulation rates of multiple microseconds of physical time per day for systems with millions of atoms. Demonstrating strong scaling, the machine simulates a standard 23,558-atom benchmark system at a rate of 85 μs/day -- 180 times faster than any commodity hardware platform or general-purpose supercomputer.

A 32x32x32, spatially distributed 3D FFT in four microseconds on Anton

Anton, a massively parallel special-purpose machine for molecular dynamics simulations, performs a 32 × 32 × 32 FFT in 3.7 microseconds and a 64 × 64 × 64 FFT in 13.3 microseconds on a configuration with 512 nodes-an order of magnitude faster than all other FFT implementations of which we are aware. Achieving this FFT performance requires a coordinated combination of computation and communication techniques that leverage Antons underlying hardware mechanisms. Most significantly, Antons communication subsystem provides over 300 gigabits per second of bandwidth per node, message latency in the hundreds of nanoseconds, and support for word-level writes and single-ended communication. In addition, Antons general-purpose computation system incorporates primitives that support the efficient parallelization of small 1D FFTs. Although Anton was designed specifically for molecular dynamics simulations, a number of the hardware primitives and software implementation techniques described in this paper may also be applicable to the acceleration of FFTs on general-purpose high-performance machines.

High-throughput pairwise point interactions in Anton, a specialized machine for molecular dynamics simulation

Anton is a massively parallel special-purpose supercomputer designed to accelerate molecular dynamics (MD) simulations by several orders of magnitude, making possible for the first time the atomic-level simulation of many biologically important phenomena that take place over microsecond to millisecond time scales. The majority of the computation required for MD simulations involves the calculation of pairwise interactions between particles and/or gridpoints separated by no more than some specified cutoff radius. In Anton, such range-limited interactions are handled by a high-throughput interaction subsystem (HTIS). The HTIS on each of Antonpsilas 512 ASICs includes 32 computational pipelines running at 800 MHz, each producing a result on every cycle that would require approximately 50 arithmetic operations to compute on a general-purpose processor. In order to feed these pipelines and collect their results at a speed sufficient to take advantage of this computational power, Anton uses two novel techniques to limit inter- and intra-chip communication. The first is a recently developed parallelization algorithm for the range-limited N-body problem that offers major advantages in both asymptotic and absolute terms by comparison with traditional methods. The second is an architectural feature that processes pairs of points chosen from two point sets in time proportional to the product of the sizes of those sets, but with input and output volume proportional only to their sum. Together, these features allow Anton to perform pairwise interactions with very high throughput and unusually low latency, enabling MD simulations on time scales inaccessible to other general- and special-purpose parallel systems.

Incorporating flexibility in Anton, a specialized machine for molecular dynamics simulation

An effective special-purpose supercomputer for molecular dynamics (MD) requires much more than high-performance acceleration of computational kernels: such accelerators must be balanced with general-purpose computation and communication resources. Achieving this balance was a significant challenge in the design of Anton, a parallel machine that will accelerate MD simulations by several orders of magnitude. Anton executes its most computationally demanding calculations on a highly specialized, enormously parallel, but largely non-programmable high-throughput interaction subsystem (HTIS). Other elements of the simulation have a less uniform algorithmic structure, and may also change in response to future advances in physical models and simulation techniques. Such calculations are executed on Antonpsilas flexible subsystem, which combines programmability with the computational power required to avoid ldquoAmdahlpsilas Lawrdquo bottlenecks arising from the extremely high throughput of the HTIS. Antonpsilas flexible subsystem is a heterogeneous multiprocessor with 12 cores, each organized around a 128-bit data path. This subsystem includes hardware support for synchronization, data transfer and certain types of particle interactions, along with specialized instructions for geometric operations. All aspects of the flexible subsystem were designed specifically to accelerate MD simulations, and although it relies primarily on what may be regarded as ldquogeneral-purposerdquo processors, even this subsystem contains more application-specific features than many recently proposed ldquospecializedrdquo architectures.

Unifying on-chip and inter-node switching within the Anton 2 network

The design of network architectures has become increasingly complex as the chips connected by inter-node networks have emerged as distributed systems in their own right, complete with their own on-chip networks. In Anton 2, a massively parallel special-purpose supercomputer for molecular dynamics simulations, we managed this complexity by reusing the on-chip network as a switch for inter-node traffic. This unified network approach introduces several design challenges. Maintaining fairness within the inter-node network is difficult, as each hop becomes a sequence of many on-chip routing decisions. We addressed this problem with an inverse-weighted arbiter that ensures fairness with low implementation costs. Balancing the load of inter-node traffic across the on-chip network is also critical, and we adopted an optimization approach to design an appropriate routing algorithm. Finally, the on-chip routers carry inter-node traffic, so they must implement inter-node virtual channels to avoid deadlock. In order to keep the routers small and fast, we developed a deadlock-free routing algorithm that reduces the number of virtual channels by one-third relative to previous approaches. The resulting Anton 2 network implementation efficiently utilizes its inter-node channels and provides low messaging latency, while occupying a modest amount of silicon area.

Extending the Generality of Molecular Dynamics Simulations on a Special-Purpose Machine

Special-purpose computing hardware can provide significantly better performance and power efficiency for certain applications than general-purpose processors. Even within a single application area, however, a special-purpose machine can be far more valuable if it is capable of efficiently supporting a number of different computational methods that, taken together, expand the machines functionality and range of applicability. We have previously described a massively parallel special-purpose supercomputer, called Anton, and have shown that it executes traditional molecular dynamics simulations orders of magnitude faster than the previous state of the art. Here, we describe how we extended Antons software to support a more diverse set of methods, allowing scientists to simulate a broader class of biological phenomena at extremely high speeds. Key elements of our approach, which exploits Antons tightly integrated hardwired pipelines and programmable cores, are applicable to the hardware and software design of various other specialized or heterogeneous parallel computing platforms.

Hardware support for fine-grained event-driven computation in Anton 2

Exploiting parallelism to accelerate a computation typically involves dividing it into many small tasks that can be assigned to different processing elements. An efficient execution schedule for these tasks can be difficult or impossible to determine in advance, however, if there is uncertainty as to when each tasks input data will be available. Ideally, each task would run in direct response to the arrival of its input data, thus allowing the computation to proceed in a fine-grained event-driven manner. Realizing this ideal is difficult in practice, and typically requires sacrificing flexibility for performance. In Anton 2, a massively parallel special-purpose supercomputer for molecular dynamics simulations, we addressed this challenge by including a hardware block, called the dispatch unit, that provides flexible and efficient support for fine-grained event-driven computation. Its novel features include a many-to-many mapping from input data to a set of synchronization counters, and the ability to prioritize tasks based on their type. To solve the additional problem of using a fixed set of synchronization counters to track input data for a potentially large number of tasks, we created a software library that allows programmers to treat Anton 2 as an idealized machine with infinitely many synchronization counters. The dispatch unit, together with this library, made it possible to simplify our molecular dynamics software by expressing it as a collection of independent tasks, and the resulting fine-grained execution schedule improved overall performance by up to 16% relative to a coarse-grained schedule for precisely the same computation.