Tommy Minyard

Parallel load balancing for dynamic execution environments

The development and implementation of a new method for parallel dynamic load balancing is presented. The load balancer uses an octree-based method to calculate the amount of imbalance and determine a new partitioning for the hybrid mesh. The balancer is designed to redistribute the work when the loads on the processors change due to local mesh adaptation or to changes in the parallel execution system. For the case of a dynamic parallel system, the loads on the processors are determined by run time measurements and the balancer redistributes the work based on these timings. Since the load balancer uses the same octree-based technique employed by the partitioner, the overall change in the partitions is minimized and a reduced amount of data migration is realized. The qualities of the partitions are maintained even after multiple load balancings. The effectiveness of the dynamic load balancer is demonstrated via parallel execution times for adaptive turbulent flow simulations.

Adaptive hybrid grid methods

An adaptive refinement and redistribution method is presented for use on hybrid grids composed of prismatic, pyramidal and tetrahedral elements. The combined refinement and redistribution scheme minimizes the computational time and memory required for the solver and maximizes the numerical accuracy. Grid refinement is achieved via a priori source placement and also solution based splitting of the edges of the grid elements. The prismatic and pyramidal elements are divided directionally and the tetrahedral elements may be refined isotropically or directionally. The prismatic grid redistribution scheme attracts or repels points that are close to the wall surface so as to better capture boundary layers. A method is developed to place newly created boundary nodes on the actual splined surface instead of the previous discretized surface. This ensures that the grid refinement always represents the geometric model accurately. Several examples are presented to demonstrate the a priori source placement, the solution based grid refinement and the prismatic grid redistribution.

End-to-end framework for fault management for open source clusters: Ranger

The scale and complexity of both hardware and software on large open source software systems such as Ranger make occurrence of faults and failures inevitable. What is not inevitable is that they should be allowed to go undetected, nor that diagnosis and recovery from failures should continue to be largely manual and effort intensive. This paper presents a framework for end-to-end fault management for open source clusters which is being developed on Ranger, but which targets general open source software based clusters. The elements of the framework are: a rationalized system logging stack for Linux, low overhead log and status monitoring, and a multilevel suite of diagnostic analyses. This paper describes this framework, presents the accomplishments to date, the results which have been obtained with the elements of the framework which are in place, and the plans for future development including a solicitation for collaboration on the project.

Best practices for the deployment and management of production HPC clusters

Commodity-based Linux HPC clusters dominate the scientific computing landscape in both academia and industry ranging from small research clusters to petascale supercomputers supporting thousands of users. To support broad user communities and manage a user-friendly environment, end-user sites must combine a range of low-level system soft ware with multiple compiler chains, support libraries, and a suite of 3rd party applications. In addition, large sys tems require bare metal provisioning and a flexible software management strategy to maintain consistency and upgrade ability across thousands of compute nodes. This report documents a Linux operating system framework, (LosF), which has evolved over the last seven years to provide an integrated strategy for the deployment of multiple HPC systems at the Texas Advanced Computing Center. Documented within this effort is the high-level cluster configuration options and definitions, bare-metal provisioning, hierarchical HPC soft ware stack design, package-management, user environment management tools, user account synchronization, and local customization configurations.

Establishing Hypothesis for Recurrent System Failures from Cluster Log Files

A goal for the analysis of supercomputer logs is to establish causal relationships among events which reflect significant state changes in the system. Establishing these relationships is at the heart of failure diagnosis. In principle, a log analysis tool could automate many of the manual steps systems administrators must currently use to diagnose system failures. However, supercomputer logs are unstructured, incomplete and contain considerable ambiguity so that direct discovery of causal relationships is difficult. This paper describes the second generation FDiag log-based failure diagnostics framework that provides automation of the manual failure diagnosis process and determines with high confidence, the likely cause of the failure, the components involved and the event sequences which contain the times of the causal and terminal events. FDiag extracts relevant events from the system logs, performs correlation analysis on these events and from these correlations determines the components involved and the event sequences. The diagnostics capabilities of FDiag are validated by comparing its assessments on known instances of recurrent failures on the Ranger supercomputer at the University of Texas at Austin. We believe FDiag is the first log analyzer to demonstrate this level of diagnostics capability from the system logs of an open source software stack incorporating Linux and the Lustre file system. FDiag will be put into production use for support of failure diagnosis on Ranger in September, 2011.

Partitioning and dynamic load balancing of adaptive hybrid grids for large-scale turbulent flow simulations

A new partitioning method for complex 3-D hybrid prismatic/tetrahedral meshes is presented. The method uses an orthogonal recursive bisection approach on a special octree corresponding to the hybrid grid. The octree is generated automatically and handles any type of 3-D geometry and domain connectivity. It is employed for partitioning of static, as well as dynamic adaptive meshes. The method yields similar quality partitions for very divergent geometries, such as a sphere and an aircraft configuration. The strategy for load balancing is applied to octants which leads to an efficient parallel algorithm. Results include partitioning a hybrid grid around an aircraft configuration and octree-based load balancing for a dynamically adapted mesh. The method is applied to large-scale turbulent flow simulations around aircraft configurations.

Stampede 2: The Evolution of an XSEDE Supercomputer

The Stampede 1 supercomputer was a tremendous success as an XSEDE resource, providing more than eight million successful computational simulations and data analysis jobs to more than ten thousand users. In addition, Stampede 1 introduced new technology that began to move users towards many core processors. As Stampede 1 reaches the end of its production life, it is being replaced in phases by a new supercomputer, Stampede 2, that will not only take up much of the original systems workload, but continue the bridge to technologies on the path to exascale computing. This paper provides a brief summary of the experiences of Stampede 1, and details the design and architecture of Stampede 2. Early results are presented from a subset of Intel Knights Landing nodes that are bridging between the two systems.

Building Wrangler: A transformational data intensive resource for the open science community

With the growth of data in science and engineering fields and the I/O intense technologies used to carry out research with these massive datasets, it has become clear new solutions to support data research is required. In support of this, the Texas Advanced Computing Center presents Wrangler, the first open science research platform built from the ground up in support of data. Wrangler features a replicated 10 PB Lustre based parallel file system, compute capacity of 120 Intel Haswell nodes and 15 TB of RAM. In addition to the base system, Wrangler features a unique NAND flash-based storage system from DSSD, providing users with 0.5 PB of storage 1 TB/s bandwidth and 250 million IOP/s across the cluster. Supporting Hadoop, but not just Hadoop, Wrangler will provide current and future researchers with an environment supporting the most I/O intensive workflows in fields from astronomy to paleontology. With data at the forefront of Wranglers mission, support for ETL workflows, data curation, and data publication will enable users as they both discover new results and publish their own research. Support for both SQL and noSQL databases and GIS based extensions will also be provided, allowing users to leverage these tools for both data cataloging and cross-study integration. Wrangler will allow users to focus more on what is most important to them, the data and knowledge gained from its analysis, and less on the details of curation and I/O optimization.

Three-dimensional CFD analysis of two circular cylinders arranged perpendicular to each other

A three-dimensional numerical method combining solution of the incompressible Reynolds Averaged Navier-Stokes (RANS) equations with a rigid body structural dynamics response has been developed previously to aid in the prediction of the loads and motions of offshore structures. In this paper, we use the tool to compute the hydrodynamic flow around two tandem cylinders oriented perpendicularly to each other. The flow conditions and gap distances between the cylinders are chosen to match a set of water tunnel experiments carried out at the University of Queensland. Comparisons of Strouhal frequencies and example flowfield visualizations are presented between the experimental measurements and associated CFD results.Copyright