John K. Salmon

Atomic-Level Characterization of the Structural Dynamics of Proteins

Following Folding Fast Many protein functions involve conformational changes that occur on time-scales between tens of microseconds and milliseconds. This has limited the usefulness of all-atom molecular dynamics simulations, which are performed over shorter time-scales. Shaw et al. (p. 341) now report millisecond-scale, all-atom molecular dynamics simulations in an explicitly represented solvent environment. Simulation of the folding of a WW domain showed a well-defined folding pathway and simulation of the dynamics of bovine pancreatic trypsin inhibitor showed interconversion between distinct conformational states. Millisecond-scale simulations capture biologically relevant structural transitions during protein folding. Molecular dynamics (MD) simulations are widely used to study protein motions at an atomic level of detail, but they have been limited to time scales shorter than those of many biologically critical conformational changes. We examined two fundamental processes in protein dynamics—protein folding and conformational change within the folded state—by means of extremely long all-atom MD simulations conducted on a special-purpose machine. Equilibrium simulations of a WW protein domain captured multiple folding and unfolding events that consistently follow a well-defined folding pathway; separate simulations of the protein’s constituent substructures shed light on possible determinants of this pathway. A 1-millisecond simulation of the folded protein BPTI reveals a small number of structurally distinct conformational states whose reversible interconversion is slower than local relaxations within those states by a factor of more than 1000.

Scalable algorithms for molecular dynamics simulations on commodity clusters

Although molecular dynamics (MD) simulations of biomolecular systems often run for days to months, many events of great scientific interest and pharmaceutical relevance occur on long time scales that remain beyond reach. We present several new algorithms and implementation techniques that significantly accelerate parallel MD simulations compared with current state-of-the-art codes. These include a novel parallel decomposition method and message-passing techniques that reduce communication requirements, as well as novel communication primitives that further reduce communication time. We have also developed numerical techniques that maintain high accuracy while using single precision computation in order to exploit processor-level vector instructions. These methods are embodied in a newly developed MD code called Desmond that achieves unprecedented simulation throughput and parallel scalability on commodity clusters. Our results suggest that Desmonds parallel performance substantially surpasses that of any previously described code. For example, on a standard benchmark, Desmonds performance on a conventional Opteron cluster with 2K processors slightly exceeded the reported performance of IBMs Blue Gene/L machine with 32K processors running its Blue Matter MD code

Millisecond-scale molecular dynamics simulations on Anton

Anton is a recently completed special-purpose supercomputer designed for molecular dynamics (MD) simulations of biomolecular systems. The machines specialized hardware dramatically increases the speed of MD calculations, making possible for the first time the simulation of biological molecules at an atomic level of detail for periods on the order of a millisecond-about two orders of magnitude beyond the previous state of the art. Anton is now running simulations on a timescale at which many critically important, but poorly understood phenomena are known to occur, allowing the observation of aspects of protein dynamics that were previously inaccessible to both computational and experimental study. Here, we report Antons performance when executing actual MD simulations whose accuracy has been validated against both existing MD software and experimental observations. We also discuss the manner in which novel algorithms have been coordinated with Antons co-designed, application-specific hardware to achieve these results.

Anton, a special-purpose machine for molecular dynamics simulation

The ability to perform long, accurate molecular dynamics (MD) simulations involving proteins and other biological macro-molecules could in principle provide answers to some of the most important currently outstanding questions in the fields of biology, chemistry and medicine. A wide range of biologically interesting phenomena, however, occur over time scales on the order of a millisecond--about three orders of magnitude beyond the duration of the longest current MD simulations.n In this paper, we describe a massively parallel machine called Anton, which should be capable of executing millisecond-scale classical MD simulations of such biomolecular systems. The machine, which is scheduled for completion by the end of 2008, is based on 512 identical MD-specific ASICs that interact in a tightly coupled manner using a specialized high-speed communication network. Anton has been designed to use both novel parallel algorithms and special-purpose logic to dramatically accelerate those calculations that dominate the time required for a typical MD simulation. The remainder of the simulation algorithm is executed by a programmable portion of each chip that achieves a substantial degree of parallelism while preserving the flexibility necessary to accommodate anticipated advances in physical models and simulation methods.

Parallel random numbers: as easy as 1, 2, 3

Most pseudorandom number generators (PRNGs) scale poorly to massively parallel high-performance computation because they are designed as sequentially dependent state transformations. We demonstrate that independent, keyed transformations of counters produce a large alternative class of PRNGs with excellent statistical properties (long period, no discernable structure or correlation). These counter-based PRNGs are ideally suited to modern multi- core CPUs, GPUs, clusters, and special-purpose hardware because they vectorize and parallelize well, and require little or no memory for state. We introduce several counter-based PRNGs: some based on cryptographic standards (AES, Threefish) and some completely new (Philox). All our PRNGs pass rigorous statistical tests (including TestUOls BigCrush) and produce at least 264 unique parallel streams of random numbers, each with period 2128 or more. In addition to essentially unlimited parallel scalability, our PRNGs offer excellent single-chip performance: Philox is faster than the CURAND library on a single NVIDIA GPU.

Anton 2: raising the bar for performance and programmability in a special-purpose molecular dynamics supercomputer

Anton 2 is a second-generation special-purpose supercomputer for molecular dynamics simulations that achieves significant gains in performance, programmability, and capacity compared to its predecessor, Anton 1. The architecture of Anton 2 is tailored for fine-grained event-driven operation, which improves performance by increasing the overlap of computation with communication, and also allows a wider range of algorithms to run efficiently, enabling many new software-based optimizations. A 512-node Anton 2 machine, currently in operation, is up to ten times faster than Anton 1 with the same number of nodes, greatly expanding the reach of all-atom bio molecular simulations. Anton 2 is the first platform to achieve simulation rates of multiple microseconds of physical time per day for systems with millions of atoms. Demonstrating strong scaling, the machine simulates a standard 23,558-atom benchmark system at a rate of 85 μs/day -- 180 times faster than any commodity hardware platform or general-purpose supercomputer.

Evaluating the Effects of Cutoffs and Treatment of Long- range Electrostatics in Protein Folding Simulations

The use of molecular dynamics simulations to provide atomic-level descriptions of biological processes tends to be computationally demanding, and a number of approximations are thus commonly employed to improve computational efficiency. In the past, the effect of these approximations on macromolecular structure and stability has been evaluated mostly through quantitative studies of small-molecule systems or qualitative observations of short-timescale simulations of biological macromolecules. Here we present a quantitative evaluation of two commonly employed approximations, using a test system that has been the subject of a number of previous protein folding studies–the villin headpiece. In particular, we examined the effect of (i) the use of a cutoff-based force-shifting technique rather than an Ewald summation for the treatment of electrostatic interactions, and (ii) the length of the cutoff used to determine how many pairwise interactions are included in the calculation of both electrostatic and van der Waals forces. Our results show that the free energy of folding is relatively insensitive to the choice of cutoff beyond 9 Å, and to whether an Ewald method is used to account for long-range electrostatic interactions. In contrast, we find that the structural properties of the unfolded state depend more strongly on the two approximations examined here.

A 32x32x32, spatially distributed 3D FFT in four microseconds on Anton

Anton, a massively parallel special-purpose machine for molecular dynamics simulations, performs a 32 × 32 × 32 FFT in 3.7 microseconds and a 64 × 64 × 64 FFT in 13.3 microseconds on a configuration with 512 nodes-an order of magnitude faster than all other FFT implementations of which we are aware. Achieving this FFT performance requires a coordinated combination of computation and communication techniques that leverage Antons underlying hardware mechanisms. Most significantly, Antons communication subsystem provides over 300 gigabits per second of bandwidth per node, message latency in the hundreds of nanoseconds, and support for word-level writes and single-ended communication. In addition, Antons general-purpose computation system incorporates primitives that support the efficient parallelization of small 1D FFTs. Although Anton was designed specifically for molecular dynamics simulations, a number of the hardware primitives and software implementation techniques described in this paper may also be applicable to the acceleration of FFTs on general-purpose high-performance machines.

High-throughput pairwise point interactions in Anton, a specialized machine for molecular dynamics simulation

Anton is a massively parallel special-purpose supercomputer designed to accelerate molecular dynamics (MD) simulations by several orders of magnitude, making possible for the first time the atomic-level simulation of many biologically important phenomena that take place over microsecond to millisecond time scales. The majority of the computation required for MD simulations involves the calculation of pairwise interactions between particles and/or gridpoints separated by no more than some specified cutoff radius. In Anton, such range-limited interactions are handled by a high-throughput interaction subsystem (HTIS). The HTIS on each of Antonpsilas 512 ASICs includes 32 computational pipelines running at 800 MHz, each producing a result on every cycle that would require approximately 50 arithmetic operations to compute on a general-purpose processor. In order to feed these pipelines and collect their results at a speed sufficient to take advantage of this computational power, Anton uses two novel techniques to limit inter- and intra-chip communication. The first is a recently developed parallelization algorithm for the range-limited N-body problem that offers major advantages in both asymptotic and absolute terms by comparison with traditional methods. The second is an architectural feature that processes pairs of points chosen from two point sets in time proportional to the product of the sizes of those sets, but with input and output volume proportional only to their sum. Together, these features allow Anton to perform pairwise interactions with very high throughput and unusually low latency, enabling MD simulations on time scales inaccessible to other general- and special-purpose parallel systems.

Exploiting 162-Nanosecond End-to-End Communication Latency on Anton

Strong scaling of scientific applications on parallel architectures is increasingly limited by communication latency. This paper describes the techniques used to mitigate latency in Anton, a massively parallel special-purpose machine that accelerates molecular dynamics (MD) simulations by orders of magnitude compared with the previous state of the art. Achieving this speedup required a combination of hardware mechanisms and software constructs to reduce network latency, sender and receiver overhead, and synchronization costs. Key elements of Antons approach, in addition to tightly integrated communication hardware, include formulating data transfer in terms of counted remote writes, leveraging fine-grained communication, and establishing fixed, optimized communication patterns. Anton delivers software-to-software inter-node latency significantly lower than any other large-scale parallel machine, and the total critical-path communication time for an Anton MD simulation is less than 4% that of the next fastest MD platform.