John N. Shadid

Multiphysics simulations: Challenges and opportunities

We consider multiphysics applications from algorithmic and architectural perspectives, where “algorithmic” includes both mathematical analysis and computational complexity, and “architectural” includes both software and hardware environments. Many diverse multiphysics applications can be reduced, en route to their computational simulation, to a common algebraic coupling paradigm. Mathematical analysis of multiphysics coupling in this form is not always practical for realistic applications, but model problems representative of applications discussed herein can provide insight. A variety of software frameworks for multiphysics applications have been constructed and refined within disciplinary communities and executed on leading-edge computer systems. We examine several of these, expose some commonalities among them, and attempt to extrapolate best practices to future systems. From our study, we summarize challenges and forecast opportunities.

Three‐dimensional wideband electromagnetic modeling on massively parallel computers

A method is presented for modeling the wideband, frequency domain electromagnetic (EM) response of a three-dimensional (3-D) earth to dipole sources operating at frequencies where EM diffusion dominates the response (less than 100 kHz) up into the range where propagation dominates (greater than 10 MHz). The scheme employs the modified form of the vector Helmholtz equation for the scattered electric fields to model variations in electrical conductivity, dielectric permitivity and magnetic permeability. The use of the modified form of the Helmholtz equation allows for perfectly matched layer ( PML) absorbing boundary conditions to be employed through the use of complex grid stretching. Applying the finite difference operator to the modified Helmholtz equation produces a linear system of equations for which the matrix is sparse and complex symmetrical. The solution is obtained using either the biconjugate gradient (BICG) or quasi-minimum residual (QMR) methods with preconditioning; in general we employ the QMR method with Jacobi scaling preconditioning due to stability. In order to simulate larger, more realistic models than has been previously possible, the scheme has been modified to run on massively parallel (MP) computer architectures. Execution on the 1840-processor Intel Paragon has indicated a maximum model size of 280 × 260 × 200 cells with a maximum flop rate of 14.7 Gflops. Three different geologic models are simulated to demonstrate the use of the code for frequencies ranging from 100 Hz to 30 MHz and for different source types and polarizations. The simulations show that the scheme is correctly able to model the air-earth interface and the jump in the electric and magnetic fields normal to discontinuities. For frequencies greater than 10 MHz, complex grid stretching must be employed to incorporate absorbing boundaries while below this normal (real) grid stretching can be employed.

A taxonomy and comparison of parallel block multi-level preconditioners for the incompressible Navier-Stokes equations

In recent years, considerable effort has been placed on developing efficient and robust solution algorithms for the incompressible Navier-Stokes equations based on preconditioned Krylov methods. These include physics-based methods, such as SIMPLE, and purely algebraic preconditioners based on the approximation of the Schur complement. All these techniques can be represented as approximate block factorization (ABF) type preconditioners. The goal is to decompose the application of the preconditioner into simplified sub-systems in which scalable multi-level type solvers can be applied. In this paper we develop a taxonomy of these ideas based on an adaptation of a generalized approximate factorization of the Navier-Stokes system first presented in A. Quarteroni, F. Saleri, A. Veneziani, Factorization methods for the numerical approximation of Navier-Stokes equations, Computational Methods in Applied Mechanical Engineering 188 (2000) 505-526]. This taxonomy illuminates the similarities and differences among these preconditioners and the central role played by efficient approximation of certain Schur complement operators. We then present a parallel computational study that examines the performance of these methods and compares them to an additive Schwarz domain decomposition (DD) algorithm. Results are presented for two and three-dimensional steady state problems for enclosed domains and inflow/outflow systems on both structured and unstructured meshes. The numerical experiments are performed using MPSalsa, a stabilized finite element code.

A Taxonomy of Consistently Stabilized Finite Element Methods for the Stokes Problem

Stabilized mixed methods can circumvent the restrictive inf-sup condition without introducing penalty errors. For properly chosen stabilization parameters these methods are well-posed for all conforming velocity-pressure pairs. However, their variational forms have widely varying properties. First, stabilization offers a choice between weakly or strongly coercive bilinear forms that give rise to linear systems with identical solutions but very different matrix properties. Second, coercivity may be conditional upon a proper choice of a stabilizing parameter. Here we focus on how these two aspects of stabilized methods affect their accuracy and efficient iterative solution. We present results that indicate a preference of Krylov subspace solvers for strongly coercive formulations. Stability criteria obtained by finite element and algebraic analyses are compared with numerical experiments. While for two popular classes of stabilized methods, sufficient stability bounds correlate well with numerical stability, our experiments indicate the intriguing possibility that the pressure-stabilized Galerkin method is unconditionally stable.

Globalization Techniques for Newton-Krylov Methods and Applications to the Fully Coupled Solution of the Navier-Stokes Equations

A Newton-Krylov method is an implementation of Newton’s method in which a Krylov subspace method is used to solve approximately the linear subproblems that determine Newton steps. To enhance robustness when good initial approximate solutions are not available, these methods are usually globalized, i.e., augmented with auxiliary procedures (globalizations) that improve the likelihood of convergence from a starting point that is not near a solution. In recent years, globalized Newton-Krylov methods have been used increasingly for the fully coupled solution of large-scale problems. In this paper, we review several representative globalizations, discuss their properties, and report on a numerical study aimed at evaluating their relative merits on large-scale two- and three-dimensional problems involving the steady-state Navier-Stokes equations.

Towards a scalable fully-implicit fully-coupled resistive MHD formulation with stabilized FE methods

This paper explores the development of a scalable, nonlinear, fully-implicit stabilized unstructured finite element (FE) capability for 2D incompressible (reduced) resistive MHD. The discussion considers the implementation of a stabilized FE formulation in context of a fully-implicit time integration and direct-to-steady-state solution capability. The nonlinear solver strategy employs Newton-Krylov methods, which are preconditioned using fully-coupled algebraic multilevel preconditioners. These preconditioners are shown to enable a robust, scalable and efficient solution approach for the large-scale sparse linear systems generated by the Newton linearization. Verification results demonstrate the expected order-of-accuracy for the stabilized FE discretization. The approach is tested on a variety of prototype problems, including both low-Lundquist number (e.g., an MHD Faraday conduction pump and a hydromagnetic Rayleigh-Bernard linear stability calculation) and moderately-high Lundquist number (magnetic island coalescence problem) examples. Initial results that explore the scaling of the solution methods are presented on up to 4096 processors for problems with up to 64M unknowns on a CrayXT3/4. Additionally, a large-scale proof-of-capability calculation for 1 billion unknowns for the MHD Faraday pump problem on 24,000 cores is presented.

Bifurcation and stability analysis of laminar isothermal counterflowing jets

We present a numerical study of the structure and stability of laminar isothermal flows formed by two counterflowing jets of an incompressible Newtonian fluid. We demonstrate that symmetric counterflowing jets with identical mass flow rates exhibit multiple steady states and, in certain cases, time-dependent (periodic) steady states. Two geometric configurations were studied based on the inlet jet shapes: planar and axisymmetric. Stagnation flows formed by planar counterflowing jets exhibit both steady-state multiplicity and time-dependent behaviour, while axisymmetric jets exhibit only a steady-state multiplicity. A linearized bifurcation and stability analysis based on the continuity and Navier–Stokes equations revealed transitions between a single (symmetric) steady state and multiple steady states or periodic steady states. The dimensionless quantities forming the parameter space of this system are the inlet Reynolds number (Re) and a geometric aspect ratio (α), equal to the jet inlet characteristic length (used for calculating Re) divided by the jet separation. The boundaries separating different flow regimes have been identified in the (Re, α) parameter space. The resulting flow maps are useful for the design and operation of counterflow jet reactors.

Performance of a parallel algebraic multilevel preconditioner for stabilized finite element semiconductor device modeling

In this study results are presented for the large-scale parallel performance of an algebraic multilevel preconditioner for solution of the drift-diffusion model for semiconductor devices. The preconditioner is the key numerical procedure determining the robustness, efficiency and scalability of the fully-coupled Newton-Krylov based, nonlinear solution method that is employed for this system of equations. The coupled system is comprised of a source term dominated Poisson equation for the electric potential, and two convection-diffusion-reaction type equations for the electron and hole concentration. The governing PDEs are discretized in space by a stabilized finite element method. Solution of the discrete system is obtained through a fully-implicit time integrator, a fully-coupled Newton-based nonlinear solver, and a restarted GMRES Krylov linear system solver. The algebraic multilevel preconditioner is based on an aggressive coarsening graph partitioning of the nonzero block structure of the Jacobian matrix. Representative performance results are presented for various choices of multigrid V-cycles and W-cycles and parameter variations for smoothers based on incomplete factorizations. Parallel scalability results are presented for solution of up to 10^8 unknowns on 4096 processors of a Cray XT3/4 and an IBM POWER eServer system.

Least Squares Preconditioners for Stabilized Discretizations of the Navier-Stokes Equations

This paper introduces two stabilization schemes for the least squares commutator (LSC) preconditioner developed by Elman, Howle, Shadid, Shuttleworth, and Tuminaro [SIAM J. Sci. Comput., 27 (2006), pp. 1651-1668] for the incompressible Navier-Stokes equations. This preconditioning methodology is one of several choices that are effective for Navier-Stokes equations, and it has the advantage of being defined from strictly algebraic considerations. It has previously been limited in its applicability to div-stable discretizations of the Navier-Stokes equations. This paper shows how to extend the same methodology to stabilized low-order mixed finite element approximation methods.

A parallel block multi-level preconditioner for the 3D incompressible Navier--Stokes equations

The development of robust and efficient algorithms for both steady-state simulations and fully implicit time integration of the Navier-Stokes equations is an active research topic. To be effective, the linear subproblems generated by these methods require solution techniques that exhibit robust and rapid convergence. In particular, they should be insensitive to parameters in the problem such as mesh size, time step, and Reynolds number. In this context, we explore a parallel preconditioner based on a block factorization of the coefficient matrix generated in an Oseen nonlinear iteration for the primitive variable formulation of the system. The key to this preconditioner is the approximation of a certain Schur complement operator by a technique first proposed by Kay, Loghin, and Wathen [SIAM J. Sci. Comput., 2002] and Silvester, Elman, Kay, and Wathen [J. Comput. Appl. Math. 128 (2001) 261]. The resulting operator entails subsidiary computations (solutions of pressure Poisson and convection-diffusion subproblems) that are similar to those required for decoupled solution methods; however, in this case these solutions are applied as preconditioners to the coupled Oseen system. One important aspect of this approach is that the convection-diffusion and Poisson subproblems are significantly easier to solve than the entire coupled system, and a solver can be built using tools developed for the subproblems. In this paper, we apply smoothed aggregation algebraic multigrid to both subproblems. Previous work has focused on demonstrating the optimality of these preconditioners with respect to mesh size on serial, two-dimensional, steady-state computations employing geometric multi-grid methods; we focus on extending these methods to large-scale, parallel, three-dimensional, transient and steady-state simulations employing algebraic multigrid (AMG) methods. Our results display nearly optimal convergence rates for steady-state solutions as well as for transient solutions over a wide range of CFL numbers on the two-dimensional and three-dimensional lid-driven cavity problem.