Mahidhar Tatineni

The link between shocks, turbulence, and magnetic reconnection in collisionless plasmas

Global hybrid (electron fluid, kinetic ions) and fully kinetic simulations of the magnetosphere have been used to show surprising interconnection between shocks, turbulence, and magnetic reconnection. In particular, collisionless shocks with their reflected ions that can get upstream before retransmission can generate previously unforeseen phenomena in the post shocked flows: (i) formation of reconnecting current sheets and magnetic islands with sizes up to tens of ion inertial length. (ii) Generation of large scale low frequency electromagnetic waves that are compressed and amplified as they cross the shock. These “wavefronts” maintain their integrity for tens of ion cyclotron times but eventually disrupt and dissipate their energy. (iii) Rippling of the shock front, which can in turn lead to formation of fast collimated jets extending to hundreds of ion inertial lengths downstream of the shock. The jets, which have high dynamical pressure, “stir” the downstream region, creating large scale disturbances ...

Quantifying performance benefits of overlap using MPI-2 in a seismic modeling application

AWM-Olsen is a widely used ground motion simulation code based on a parallel finite difference solution of the 3-D velocity-stress wave equation. This application runs on tens of thousands of cores and consumes several million CPU hours on the TeraGrid Clusters every year. A significant portion of its run-time (37% in a 4,096 process run), is spent in MPI communication routines. Hence, it demands an optimized communication design coupled with a low-latency, high-bandwidth network and an efficient communication subsystem for good performance. In this paper, we analyze the performance bottlenecks of the application with regard to the time spent in MPI communication calls. We find that much of this time can be overlapped with computation using MPI non-blocking calls. We use both two-sided and MPI-2 one-sided communication semantics to re-design the communication in AWM-Olsen. We find that with our new design, using MPI-2 one-sided communication semantics, the entire application can be sped up by 12% at 4K processes and by 10% at 8K processes on a state-of-the-art InfiniBand cluster, Ranger at the Texas Advanced Computing Center (TACC).

Toward a Computational Steering Framework for Large-Scale Composite Structures Based on Continually and Dynamically Injected Sensor Data

Abstract Recent advances in simulation, optimization, structural health monitoring, and high-performance computing create a unique opportunity to combine the developments in these fields to formulate a Dynamics Data Driven Application System (DDDAS) framework. In this paper we propose such a framework, which consists of the following items and features: a structural health monitoring (SHM) system, an advanced fluid—structure interaction (FSI) simulation module, and a sensitivity analysis, optimization and control software module. High-performance computing (HPC) is employed for the various parts of the framework and is viewed as its essential element. The intended application of the developed framework is the analysis of medium-to-large-scale composite structures. These include aerospace structures, such as military aircraft fuselage and wings, helicopter blades, and unmanned aerial vehicles, and civil structures, such as wind turbine blades and towers. The proposed framework will continuously and dynamically integrate the SHM data into the FSI analysis of these structures. This capability allows one to: 1. Shelter the structures from excessive stress levels during operation; 2. Make informed decisions to perform structural maintenance and repair; and 3. Predict the remaining fatigue life of the structure.

In-situ visualization for global hybrid simulations

Petascale simulations have become mission critical in diverse areas of science and engineering. Knowledge discovery from such simulations remains a major challenge and is becoming more urgent as the march towards ultra-scale computing with millions of cores continues. One major issue with the current paradigm of running the simulations and saving the data to disk for post-processing is that it is only feasible to save the data at a small number of time slices. This low temporal resolution of the saved data is a serious handicap in many studies where the time evolution of the system is of principle interest. One way to address this I/O issue is through in-situ visualization strategies. The idea is to minimize data storage by extracting important features of the data and saving them, rather than raw data, at high temporal resolution. Parallel file systems of current petascale and future exascale systems are expensive shared resources and need to be utilized effectively, and similarly archival storage can be limited and both of these will benefit from in-situ visualization as it will lead to intelligent way of utilizing storage. In this paper, we present preliminary results from our in-situ visualization for global hybrid (electron fluid, kinetic ions) simulations which are used to study the interaction of the solar wind with planetary magnetospheres such as the Earth and Mercury. In particular, we examine the overhead and effect on code performance associated with the inline computations associated with in-situ visualization.

Data intensive analysis on the gordon high performance data and compute system

The Gordon data intensive computing system was designed to handle problems with large memory requirements that cannot easily be solved using standard workstations or distributed memory supercomputers. We describe the unique features of Gordon that make it ideally suited for data mining and knowledge discovery applications: memory aggregation using the vSMP software solution from ScaleMP, I/O nodes containing 4 TB of low-latency flash memory, and a high performance parallel file system with 4 PB capacity. We also demonstrate how a number of standard data mining tools (e.g. Matlab, WEKA, R) can be used effectively on Dash, an early prototype of the full Gordon system.

Group-based variant calling leveraging next-generation supercomputing for large-scale whole-genome sequencing studies

MotivationNext-generation sequencing (NGS) technologies have become much more efficient, allowing whole human genomes to be sequenced faster and cheaper than ever before. However, processing the raw sequence reads associated with NGS technologies requires care and sophistication in order to draw compelling inferences about phenotypic consequences of variation in human genomes. It has been shown that different approaches to variant calling from NGS data can lead to different conclusions. Ensuring appropriate accuracy and quality in variant calling can come at a computational cost.ResultsWe describe our experience implementing and evaluating a group-based approach to calling variants on large numbers of whole human genomes. We explore the influence of many factors that may impact the accuracy and efficiency of group-based variant calling, including group size, the biogeographical backgrounds of the individuals who have been sequenced, and the computing environment used. We make efficient use of the Gordon supercomputer cluster at the San Diego Supercomputer Center by incorporating job-packing and parallelization considerations into our workflow while calling variants on 437 whole human genomes generated as part of large association study.ConclusionsWe ultimately find that our workflow resulted in high-quality variant calls in a computationally efficient manner. We argue that studies like ours should motivate further investigations combining hardware-oriented advances in computing systems with algorithmic developments to tackle emerging ‘big data’ problems in biomedical research brought on by the expansion of NGS technologies.

SR-IOV: Performance Benefits for Virtualized Interconnects

The demand for virtualization within high-performance computing is rapidly growing as new communities, driven by both new application stacks and new computing modalities, continue to grow and expand. While virtualization has traditionally come with significant penalties in I/O performance that have precluded its use in mainstream large-scale computing environments, new standards such as Single Root I/O Virtualization (SR-IOV) are emerging that promise to diminish the performance gap and make high-performance virtualization possible. To this end, we have evaluated SR-IOV in the context of both virtualized InfiniBand and virtualized 10 gigabit Ethernet (GbE) using micro-benchmarks and real-world applications. We compare the performance of these interconnects on non-virtualized environments, Amazons SR-IOV-enabled C3 instances, and our own SR-IOV-enabled InfiniBand cluster and show that SR-IOV significantly reduces the performance losses caused by virtualization. InfiniBand demonstrates less than 2% loss of bandwidth and less than 10% increase in latency when virtualized with SR-IOV. Ethernet also benefits, although less dramatically, when SR-IOV is enabled on Amazons cloud.

Homology-based hydrogen bond information improves crystallographic structures in the PDB

The Protein Data Bank (PDB) is the global archive for structural information on macromolecules, and a popular resource for researchers, teachers, and students, amassing more than one million unique users each year. Crystallographic structure models in the PDB (more than 100,000 entries) are optimized against the crystal diffraction data and geometrical restraints. This process of crystallographic refinement typically ignored hydrogen bond (H‐bond) distances as a source of information. However, H‐bond restraints can improve structures at low resolution where diffraction data are limited. To improve low‐resolution structure refinement, we present methods for deriving H‐bond information either globally from well‐refined high‐resolution structures from the PDB‐REDO databank, or specifically from on‐the‐fly constructed sets of homologous high‐resolution structures. Refinement incorporating HOmology DErived Restraints (HODER), improves geometrical quality and the fit to the diffraction data for many low‐resolution structures. To make these improvements readily available to the general public, we applied our new algorithms to all crystallographic structures in the PDB: using massively parallel computing, we constructed a new instance of the PDB‐REDO databank (https://pdb-redo.eu). This resource is useful for researchers to gain insight on individual structures, on specific protein families (as we demonstrate with examples), and on general features of protein structure using data mining approaches on a uniformly treated dataset.

Using Gordon to accelerate LHC science

The discovery of the Higgs boson by the Large Hadron Collider (LHC) has garnered international attention. In addition to this singular result, the LHC may also uncover other fundamental particles, including dark matter. Much of this research is being done on data from one of the LHC experiments, the Compact Muon Solenoid (CMS). The CMS experiment was able to capture data at higher sampling frequencies than planned during the 2012 LHC operational period. The resulting data had been parked, waiting to be processed on CMS computers. While CMS has significant compute resources, by partnering with SDSC to incorporate Gordon into the CMS workflow, analysis of the parked data was completed months ahead of schedule. This allows scientists to review the results more quickly, and could guide future plans for the LHC.

Storage utilization in the long tail of science

The increasing expansion of computations in non-traditional domain sciences has resulted in an increasing demand for research cyberinfrastructure that is suitable for small- and mid-scale job sizes. The computational aspects of these emerging communities are coming into focus and being addressed through the deployment of several new XSEDE resources that feature easy on-ramps, customizable software environments through virtualization, and interconnects optimized for jobs that only use hundreds or thousands of cores; however, the data storage requirements for these emerging communities remains much less well characterized. To this end, we examined the distribution of file sizes on two of the Lustre file systems within the Data Oasis storage system at the San Diego Supercomputer Center (SDSC). We found that there is a very strong preference for small files among SDSCs users, with 90% of all files being less than 2 MB in size. Furthermore, 50% of all file system capacity is consumed by files under 2 GB in size, and these distributions are consistent on both scratch and projects storage file systems. Because parallel file systems like Lustre and GPFS are optimized for parallel IO to large, widestripe files, these findings suggest that parallel file systems may not be the most suitable storage solutions when designing cyberinfrastructure to meet the needs of emerging communities.