Fearghal O'Donncha

Parallelisation study of a three-dimensional environmental flow model

There are many simulation codes in the geosciences that are serial and cannot take advantage of the parallel computational resources commonly available today. One model important for our work in coastal ocean current modelling is EFDC, a Fortran 77 code configured for optimal deployment on vector computers. In order to take advantage of our cache-based, blade computing system we restructured EFDC from serial to parallel, thereby allowing us to run existing models more quickly, and to simulate larger and more detailed models that were previously impractical. Since the source code for EFDC is extensive and involves detailed computation, it is important to do such a port in a manner that limits changes to the files, while achieving the desired speedup. We describe a parallelisation strategy involving surgical changes to the source files to minimise error-prone alteration of the underlying computations, while allowing load-balanced domain decomposition for efficient execution on a commodity cluster. The use of conjugate gradient posed particular challenges due to implicit non-local communication posing a hindrance to standard domain partitioning schemes; a number of techniques are discussed to address this in a feasible, computationally efficient manner. The parallel implementation demonstrates good scalability in combination with a novel domain partitioning scheme that specifically handles mixed water/land regions commonly found in coastal simulations. The approach presented here represents a practical methodology to rejuvenate legacy code on a commodity blade cluster with reasonable effort; our solution has direct application to other similar codes in the geosciences.

Parameterizing suspended canopy effects in a three-dimensional hydrodynamic model

ABSTRACT This paper presents an amendment of an existing three-dimensional hydro-environmental model (Environmental Fluid Dynamics Code) to incorporate effects of suspended canopies on the vertical flow structure. Five different modelling approaches are investigated, encompassing hydrodynamic form drag imparted by the suspended canopy, an amended two-equation turbulence scheme representing turbulence generated locally by elements within the canopy, and three separate formulations for vertical profiles of drag coefficients. Data from laboratory experiments with rigid cylinders are used to validate the calculations of velocity and shear stresses. The results show that the most accurate reproduction of the canopy flow was obtained using a vertically varying drag coefficient along with a two-equation turbulence closure scheme that includes additional turbulence production and dissipation terms. The numerical model reproduced velocity profiles accurately, but the shear stresses are slightly overestimated.

Parallelisation of hydro-environmental model for simulating marine current devices

There are many serial simulation codes in the geosciences that cannot take advantage of the computational resources commonly available today. Further, the initial design of many parallel application codes targeted relatively small-scale commodity clusters and does not take full advantage of modern cache-based, blade computing system. In this paper, we discuss the porting of a widely used hydro-environmental code, EFDC, to parallel using both the MPI and OpenMP paradigms. The objective of the research is to better elucidate the strengths and weaknesses of both approaches to further a hybrid parallelisation strategy that leverages the capabilities of both.

Deploying and optimizing performance of a 3D hydrodynamic model on cloud

Container-based cloud computing, as standardised and popularised by the open-source docker project has many potential opportunities for scientific application in highperformance computing. It promises highly flexible and available compute capabilities via cloud, without the resource overheads of traditional virtual machines. Further, productivity gains can be made by easy repackaging of images with additional developments, automated deployments, and version-control integrations. Nevertheless, the impact of container overhead and overlay network implementation and performance are areas that requires detailed study to allow for well-defined quality of service for typical HPC applications. This papers presents details on deploying the Environmental Fluid Dynamics Code (EFDC) on a container-based cloud environment. Results are compared to a bare metal deployment. Application-specific benchmarking tests are complemented by detailed network tests that evaluate isolated MPI communication protocols both at intra-node and inter-node level with varying degrees of self-contention. Cloud-based simulations report significant performance loss in mean run-times. A containerised environment increases simulation time by up to 50%. More detailed analysis demonstrates that much of this performance penalty is a result of large variance in MPI communciation times. This manifests as simulation runtime variance on container cloud that hinders both simulation run-time and collection of well-defined quality-of-service metrics.

Calibration of a 3D hydrodynamic aquaculture model

Aquaculture structures comprising suspended canopies extending from the free surface to a point above the sediment are represented as porous blockages in a hydrodynamic surface-water model. These canopies alter flow conditions and material transport within and around the structure, especially between the bottom of the canopy and the sediment bed. These altered flow conditions can have significant implications for sediment, nutrient, and waste transport in the environments in which aquaculture structures are placed. This study uses an augmented version of the Environmental Fluid Dynamic Code to model flow conditions in and around a suspended canopy calibrated against data obtained from a series of flume tests. The parameter estimation code, PEST, was used to calibrate various model parameters including horizontal momentum diffusivity, vertical eddy viscosity, turbulence closure constants, and depth-dependent drag coefficients to ensure that model simulations match the experimental (calibration) data comprising vertical velocity and turbulent kinetic energy profiles. The model was then validated against vertical profiles of turbulent kinetic energy production rate, which were not used during the calibration process. Ultimately, trends of increasing average drag coefficient with decreasing canopy blockage ratio and increasing average flume flow speed were observed. The calibrated canopy-turbulence parameters may yield improved predictions of the hydrodynamic and material transport conditions resulting from the various aquaculture structures and flow systems. In turn, these predictions will help develop methods to minimize environmental impacts and to increase production from aquaculture farms.

Assessment and quantification of HF radar uncertainty

A large body of work exists concerning uncertainty in ocean current measuring high-frequency radar (HFR) systems. This study investigates the magnitude of uncertainty present in a HFR system in the lower Chesapeake Bay region of Virginia. A method of assessing the fundamental performance of the HFR is comparing the radial velocities measured by two facing HF radars at the centre point of their baseline. In an error-free network, radial vectors from the two sites would be equal and opposite at a point on the baseline, so the magnitude of their sum represents a measure of imperfection in the data. Often essential information lies not in any individual process variable but in how the variables change with respect to one another, i.e. how they co-vary. PCA is a data-driven modelling technique that transforms a set of correlated variables into a smaller set of uncorrelated variables while retaining most of the original information. This paper adopts PCA to detect anomalies in data coming from the individual HF stations. A PCA model is developed based on a calibration set of historical data. The model is used with new process data to detect changes in the system by application of PCA in combination with multivariate statistical techniques. Based on a comprehensive analysis the study presents an objective preconditioning methodology for preprocessing of HFR data prior to assimilation into coastal ocean models or other uses sensitive to the divergence of the flow.

Deployment and parametrisation of couplec hydrodynamic and wave models

This study investigates near-shore circulation and wave characteristics applied to a case-study site in Santa Cruz, California. Hourly regional wave-condition data were gathered from the National Oceanic and Atmospheric Administrations National Data Buoy Center. Circulation patterns are resolved by the Environmental Fluid Dynamics Code, a three-dimensional hydrodynamic model, while wave information is computed using the Simulating Waves Nearshore model, a third-generation wave model that computes wind-generated waves in coastal and inland waters. Input data include bathymetry, hydrodynamic boundary conditions to the circulation model, current data for the wave model, and wind speeds and directions. The study assesses the performance of numerical models to resolve both wave and currents. Considerations to provide accurate forecasts are provided and the advantages of coupled model simulations evaluated.