Fangzhou Wei

A hybrid message passing/shared memory parallelization of the adaptive integral method for multi-core clusters

Abstract A hybrid message passing and shared memory parallelization technique is presented for improving the scalability of the adaptive integral method (AIM), an FFT based algorithm, on clusters of identical multi-core processors. The proposed hybrid MPI/OpenMP parallelization scheme is based on a nested one-dimensional (1-D) slab decomposition of the 3-D auxiliary regular grid and the associated AIM calculations: If there are M processors and T cores per processor, the scheme (i) divides the regular grid into M slabs and MT sub-slabs, (ii) assigns each slab/sub-slab and the associated operations to one of the processors/cores, and (iii) uses MPI for inter-processor data communication and OpenMP for intra-processor data exchange. The MPI/OpenMP parallel AIM is used to accelerate the solution of the combined-field integral equation pertinent to the analysis of time-harmonic electromagnetic scattering from perfectly conducting surfaces. The scalability of the scheme is investigated theoretically and verified on a state-of-the-art multi-core cluster for benchmark scattering problems. Timing and speedup results on up to 1024 quad-core processors show that the hybrid MPI/OpenMP parallelization of AIM exhibits better strong scalability (fixed problem size speedup) than pure MPI parallelization of it when multiple cores are used on each processor.

Truncated multigrid versus pre-corrected FFT/AIM for bioelectromagnetics: When is O(N) better than O(NlogN)?

The effectiveness of multigrid and fast Fourier transform (FFT) based methods are investigated for accelerating the solution of volume integral equations encountered in bioelectromagnetics (BIOEM) analysis. The typical BIOEM simulation is in the mixed-frequency regime of analysis because the field variations in the simulation domain are dictated by a combination of the free space wavelength, geometrical features, and the wavelengths/skin depths in tissues. In this case, multigrid-based methods (when appropriately truncated at high-frequency levels) can achieve O(N) complexity that is asymptotically superior to the O(NlogN) complexity of FFT-based ones. Nevertheless, the constant in front of their asymptotic complexity estimate is larger and their accuracy-efficiency tradeoffs are different. Numerical experiments are performed to compare these methods and the results show that multigrid-based methods begin to outperform FFT-based ones for N∼103.

A More Scalable and Efficient Parallelization of the Adaptive Integral Method—Part II: BIOEM Application

The parallelization of the adaptive integral method proposed in Part I is used to solve 3-D scattering problems pertinent to bioelectromagnetic (BIOEM) analysis. Detailed numerical results are presented to quantify the computational complexity and parallel efficiency of the method on a petascale supercomputing cluster. Boundaries of acceptable parallelization regions of the method are identified in the N - P plane under realistic resource and efficiency constraints, where N and P denote the number of unknowns and processes, respectively. These regions are shown to be larger than those of an alternative parallelization method for benchmark BIOEM problems. The proposed method is also used to compute the power absorbed by an anatomical human body model to demonstrate its potential for solving complex BIOEM problems. The computations are performed for increasingly higher resolutions of the model and the power absorbed by the entire model and specific tissues in it are compared to safety standards. The highest resolution body model leads to an extreme problem with about 1.2 billion unknowns; the absolute parallel efficiency of the power-absorption simulation with this model is shown to be approximately 78% for matrix fill operations, 66% for memory requirement, and 30% for iterative solution on 8192 processes.

A 2-D decomposition based parallelization of AIM for 3-D BIOEM problems

A novel parallelization of the adaptive integral method (AIM) is presented for accelerating the large-scale solution of volume integral equations pertinent to bioelectromagnetics (BIOEM) problems. The proposed method improves the parallel scalability of AIM by (i) using different workload distribution strategies to load balance the different steps of the algorithm (rather than a single global decomposition strategy) and (ii) using a 2-D rather than a 1-D stencil decomposition of the auxiliary grid to parallelize the 3-D FFTs and the related anterpolation/interpolation steps. With the proposed modifications, the maximum number of processes that can be used effectively scales as Pmax ∼ N2/3, where N denotes the number of volumetric unknowns. Numerical results demonstrate the scalability of the method and its application to BIOEM problems.

Error measures for comparing bioelectromagnetic simulators

Various error norms that can be used to quantify the accuracy of large-scale, high-fidelity bioelectromagnetic simulations are presented. The merits of the norms are highlighted and they are used to compare large-scale benchmark simulations for voxel-based and CAD models of multi-layered spherical head and leg phantoms.

FDTD vs. AIM for bioelectromagnetic analysis

The effectiveness of a time-domain differential-equation and a frequency-domain integral-equation solver are contrasted for bioelectromagnetic analysis. The two fundamentally different methods are compared empirically in terms of their accuracy and efficiency for benchmark problems.

A multiregion integral-equation method for antennas implanted in anatomical human models

To aid the design of power- and spectrum-efficient implanted antennas, efficient computational methods that can account for the presence of nearby inhomogeneous and dispersive human tissues are needed. While layered planar or spherical tissue models are often used to represent the antenna environment, the increasing fidelity and availability of anatomical human models can enable site-specific modeling, more accurate analysis, and better designs. Simulating radiation from antennas near/on/in anatomical human models, however, gives rise to large-scale problems as the latest high-fidelity models are composed of over 100 million voxels (J. W. Massey et al., 34th Annu. Conf. Bioelectromagn. Soc., June 2012). Such large problems can be solved by coupling the surface and volume electric-field integral equations and using a preconditioned, parallel FFT-accelerated iterative solver (F. Wei and A. E. Yilmaz, USNC/URSI Rad. Sci. Meet., July 2013). Unlike traditional finite-difference time-domain based methods, this approach (i) does not require the antenna model to conform to a regular grid to avoid staircasing errors and (ii) accurately models complex antennas by using irregular meshes. Moreover, as is the case for integral-equation methods in general, it requires meshing neither free space (to propagate fields) nor an extended computational domain (to truncate the problem with local boundary conditions that approximate the radiation condition); therefore, for antennas outside the body, this approach does not require the region between the antenna and the body to be meshed. For antennas implanted in voxel-based anatomical human models (by removing tissue voxels at the antenna site from the human model and inserting the antenna mesh), however, the method becomes impractical because it requires the transition region between the antenna and human tissues to be meshed such that the mesh conforms to both the irregular (triangular/tetrahedral) antenna mesh and the voxel tissue mesh.

A More Scalable and Efficient Parallelization of the Adaptive Integral Method—Part I: Algorithm

A more effective parallelization of the adaptive integral method (AIM) is proposed for solving the 3-D volume electric field integral equation. The AIM computations are distributed among processes by employing two workload decomposition strategies: 1) a novel 2-D pencil decomposition of the auxiliary regular grid is used to parallelize the anterpolation, interpolation, and 3-D FFT-accelerated propagation steps; and 2) a balanced 3-D block decomposition of the scattering volume is used to parallelize the correction step. The scalability of the proposed parallelization method is investigated theoretically. To compare it with competing methods, the concepts of resource constraints, parallel efficiency constraints, scalability limits, and acceptable parallelization regions are introduced by using N-P plots, where N and P denote the number of unknowns and processes, respectively. Using these concepts, it is shown that the proposed parallelization of AIM has better weak and strong scalability compared to the traditional ones, i.e., it enables not only larger problems to be solved but also faster solution of a given problem by increasing P.

Surface-preconditioned AIM-accelerated surface-volume integral equation solution for bioelectromagnetics

The adaptive integral method (AIM) is used to accelerate the iterative method-of-moments (MOM) solution of the frequency-domain surface-volume electric-field integral equation pertinent to the analysis of antennas near realistic human models. The iterative solution convergence is improved by using a block diagonal preconditioner: One block of the preconditioner contains the full inverse of the surface-tested portion of the MOM impedance matrix; the other block contains the diagonal inverse of the volume-tested portion. Various antenna-body configurations are simulated.

Mixed basis functions for fast analysis of antennas near voxel-based human models

To support the continuing proliferation of wireless devices that operate near/on/in the human body in the UHF band (0.3-3 GHz), antenna properties (input impedance, radiation patterns, etc.) must be characterized in the presence of human models. In recent years, a significant number of different anatomically accurate high-fidelity human models have been developed for these problems (Massey et al., 34th Annu. Conf. Bioelectromagnetics Society, 2012). With a few exceptions, almost all human models developed to date are based on voxels because the underlying data sets are 2-D medical images, e.g., CT, MRI, and cross-sectional images, and because it is extremely challenging to smooth these models while maintaining their accuracy. Given the difficulty of the problem, it should be expected that voxel-based models will continue to dominate the available human models in the future. As a result, computational methods that are based on regular meshes, such as the finite-difference time-domain (FDTD) and conjugate gradient FFT (CG-FFT) methods, have clear advantages for analyzing scattering from human models; in fact, these methods remain the two most popular approaches in bioelectromagnetics. When antennas must be modeled, however, the classical FDTD and CG-FFT methods constrain the antenna to conform to a regular mesh to avoid significant staircasing errors. Recently, a massively parallel and preconditioned version of the adaptive integral method (AIM) has been used to analyze antennas near human models (F. Wei and A. E. Yılmaz, Int. Conf. on Electromag. in Advanced Applicat., 869-872, 2012). The AIM approach allows irregular meshes to be used when discretizing integral equations because it introduces an auxiliary regular grid to accelerate the method of moments solution. Thus, arbitrarily shaped, located, and oriented antennas can be modeled accurately by using triangular surface and tetrahedral volume meshes when AIM is used. Discretizing voxel-based human models with tetrahedral volume meshes, however, is inefficient. It requires splitting each voxel into five or more tetrahedra and assigning the voxels material properties to these tetrahedra; this increases the number of elements/unknowns without adding any information on material properties and boundary locations. In this article, the AIM procedure is modified to use mixed basis functions; specifically, voxel-based volume basis functions (rooftops) are used in the human model and triangle-based surface and tetrahedron-based volume basis functions are used in the antenna region. While a single auxiliary grid is used to enclose both the human and antennas models, the remaining AIM parameters (number of auxiliary grid points and the near-zone correction size assigned to the basis functions) are optimized separately for the different types of basis functions. The mixed basis functions are observed to reduce the iterative solution time by a factor of ~5-10.