Emanuele Ragnoli

Parallelisation study of a three-dimensional environmental flow model

There are many simulation codes in the geosciences that are serial and cannot take advantage of the parallel computational resources commonly available today. One model important for our work in coastal ocean current modelling is EFDC, a Fortran 77 code configured for optimal deployment on vector computers. In order to take advantage of our cache-based, blade computing system we restructured EFDC from serial to parallel, thereby allowing us to run existing models more quickly, and to simulate larger and more detailed models that were previously impractical. Since the source code for EFDC is extensive and involves detailed computation, it is important to do such a port in a manner that limits changes to the files, while achieving the desired speedup. We describe a parallelisation strategy involving surgical changes to the source files to minimise error-prone alteration of the underlying computations, while allowing load-balanced domain decomposition for efficient execution on a commodity cluster. The use of conjugate gradient posed particular challenges due to implicit non-local communication posing a hindrance to standard domain partitioning schemes; a number of techniques are discussed to address this in a feasible, computationally efficient manner. The parallel implementation demonstrates good scalability in combination with a novel domain partitioning scheme that specifically handles mixed water/land regions commonly found in coastal simulations. The approach presented here represents a practical methodology to rejuvenate legacy code on a commodity blade cluster with reasonable effort; our solution has direct application to other similar codes in the geosciences.

Parallelisation of hydro-environmental model for simulating marine current devices

There are many serial simulation codes in the geosciences that cannot take advantage of the computational resources commonly available today. Further, the initial design of many parallel application codes targeted relatively small-scale commodity clusters and does not take full advantage of modern cache-based, blade computing system. In this paper, we discuss the porting of a widely used hydro-environmental code, EFDC, to parallel using both the MPI and OpenMP paradigms. The objective of the research is to better elucidate the strengths and weaknesses of both approaches to further a hybrid parallelisation strategy that leverages the capabilities of both.

An optimal interpolation scheme for assimilation of HF radar current data into a numerical ocean model

In this work we present a technique for the assimilation of ocean surface current measurements into a numerical ocean model using data from High Frequency Radar (HFR) systems. Optimal interpolation of the model-predicted velocity and HFR observations is computed in order to derive a supplementary forcing applied at the surface boundary. This work is the first stage in the development of an HFR data assimilation system for hydrodynamic modelling in Galway Bay; it demonstrates the viability of adopting data assimilation techniques to improve the performance of numerical models in regions characterized by significant wind-driven flows.

Surface flow dynamics within an exposed wind-driven bay: Combined HF radar observations and model simulations

In this study, observations from a high-frequency radar (HFR) deployment are combined with numerical model simulations to investigate the relative contributions of surface current forcing. A tidal decomposition of the relevant datasets was applied by fitting the flow profile to a finite set of sinusoids at specific frequencies related to astronomical parameters. The resultant time-series pair comprised tidal harmonic constituents and residuals composed of primarily wind-driven surface flows. Complex correlation coefficients between these data and both numerical simulations and winds were used to further investigate both bay dynamics and numerical model performance. Results of analysis demonstrate good agreement between HFR and model data, particularly for tidal harmonics. Analysis of wind driven surface currents illustrate the complex nature of flow dynamics and the many factors requiring consideration.

Turbulence modelling using dynamic parameterization with data assimilation

ABSTRACT This research assesses the application of a novel approach to parameterization of turbulence models. Dynamic parameterization is used to improve performance of two turbulence schemes incorporated in a coastal hydrodynamic model code: the Prandtl mixing length (PML) model and the k- model. The 3D variational data assimilation scheme is used to assess model skill and facilitate optimization of the turbulence schemes. Neither the PML nor the k- models are particularly suitable for recirculating flows of complex turbulence structure when default empirical constants are used. Static parameterization improves model predictions but the degree of improvement varies across the flow. Dynamic parameterization is superior to static parameterization due to its general solution for a range of flows and the self-updating process does not require costly pre-processed determination of turbulence constants. When using dynamic parameterization, the PML model exhibits comparable levels of accuracy to the k- model while retaining its computational efficiency and ease of application.

Deploying and optimizing performance of a 3D hydrodynamic model on cloud

Container-based cloud computing, as standardised and popularised by the open-source docker project has many potential opportunities for scientific application in highperformance computing. It promises highly flexible and available compute capabilities via cloud, without the resource overheads of traditional virtual machines. Further, productivity gains can be made by easy repackaging of images with additional developments, automated deployments, and version-control integrations. Nevertheless, the impact of container overhead and overlay network implementation and performance are areas that requires detailed study to allow for well-defined quality of service for typical HPC applications. This papers presents details on deploying the Environmental Fluid Dynamics Code (EFDC) on a container-based cloud environment. Results are compared to a bare metal deployment. Application-specific benchmarking tests are complemented by detailed network tests that evaluate isolated MPI communication protocols both at intra-node and inter-node level with varying degrees of self-contention. Cloud-based simulations report significant performance loss in mean run-times. A containerised environment increases simulation time by up to 50%. More detailed analysis demonstrates that much of this performance penalty is a result of large variance in MPI communciation times. This manifests as simulation runtime variance on container cloud that hinders both simulation run-time and collection of well-defined quality-of-service metrics.

Assessment and quantification of HF radar uncertainty

A large body of work exists concerning uncertainty in ocean current measuring high-frequency radar (HFR) systems. This study investigates the magnitude of uncertainty present in a HFR system in the lower Chesapeake Bay region of Virginia. A method of assessing the fundamental performance of the HFR is comparing the radial velocities measured by two facing HF radars at the centre point of their baseline. In an error-free network, radial vectors from the two sites would be equal and opposite at a point on the baseline, so the magnitude of their sum represents a measure of imperfection in the data. Often essential information lies not in any individual process variable but in how the variables change with respect to one another, i.e. how they co-vary. PCA is a data-driven modelling technique that transforms a set of correlated variables into a smaller set of uncorrelated variables while retaining most of the original information. This paper adopts PCA to detect anomalies in data coming from the individual HF stations. A PCA model is developed based on a calibration set of historical data. The model is used with new process data to detect changes in the system by application of PCA in combination with multivariate statistical techniques. Based on a comprehensive analysis the study presents an objective preconditioning methodology for preprocessing of HFR data prior to assimilation into coastal ocean models or other uses sensitive to the divergence of the flow.

Domain Decomposition for a linear advection-diffusion equation by means of minimax filtering

In this work a novel strategy that combines Domain Decomposition, Differential Algebraic Equations (DAE) and Minimax Estimation is created to contruct a numerical solution for a linear non stationary advection-diffusion equation. The proposed approach helps to control the error introduced by Domain Decomposition and FEM discretisation and allows to combine observations with solutions of the advection-diffusion equation.

Surrogate modeling and risk-based analysis for solute transport simulations

Abstract This study is driven by the question of how quickly a solute will be flushed from an aquatic system after input of the solute into the system ceases. Simulating the fate and transport of a solute in an aquatic system can be performed at high spatial and temporal resolution using a computationally demanding state-of-the-art hydrodynamics simulator. However, uncertainties in the system often require stochastic treatment and risk-based analysis requires a large number of simulations rendering the use of a physical model impractical. A surrogate model that represents a second-level physical abstraction of the system is developed and coupled with a Monte Carlo based method to generate volumetric inflow scenarios. The surrogate model provides an approximate 8 orders of magnitude speed-up over the full physical model enabling uncertainty quantification through Monte Carlo simulation. The approach developed here consists of an stochastic inflow generator, a solute concentration prediction mechanism based on the surrogate model, and a system response risk assessment method. The probabilistic outcome provided relates the uncertain quantities to the relevant response in terms of the system’s ability to remove the solute. We develop a general approach that can be applied in a generality of system configurations and types of solute. As a test case, we present a study specific to salinization of a lake.

Localised filtering for transport models

Interface control is an important area in applications of Domain Decomposition (DD) for linear advection-diffusion equations, since it attempts to minimize the errors committed by DD methods. In this work a localised control and estimation strategy, confined to selected sub-domains, that combines DD and filtering is proposed for linear non-stationary advection-diffusion equations. This approach mitigates the error introduced by DD and Finite Element Method (FEM) discretization and, in addition, it allows to combine observations (for instance, sensors readings) with numerical solutions of advection-diffusion equations. The latter is of utmost importance for data assimilation methods widely applied in geophysics.