David Richards

Parallelization of the WASH123D Code—Phase III: 1-Dimensional Channel, 2-Dimensional Overland, and 3-Dimensional Subsurface Flows

Watershed models are used to simulate and predict major hydrological processes, such as surface or subsurface flows, which may occur on different spatial domains and temporal scales. A key feature of watershed models is the ability to model interactions among different processes and domains. Such interactions can be strongly or weakly coupled depending on the relevant time scales for each process. WASH123D is a first-principles, physics-based model for simulating a coupled system of channel flow, overland flow, and subsurface flow. A parallel version, pWASH, has been used for the calibration, validation, and evaluation of proposed alternatives of the Biscayne Bay Coastal Wetlands project. The goal is to rehydrate wetlands using the best alternative based on the simulation results. In the pWASH code, channel flow is modeled as a one-dimensional (1-D) channel network, overland flow as a 2-D process, and subsurface flow as a 3-D process. Different algorithms are implemented to account for the interactions between these different domains. The pWASH code is designed to tackle large watershed problems on parallel high performance computers. A software-engineering approach was used to efficiently parallelize the complex coupling algorithms. The resulting software toolkit encapsulates the parallel data structures and message passing required for multi- domain/process interactions. In this paper, the authors briefly describe the numerical methods in WASH123D, the parallelization of pWASH, and the scalability of coupled flow problems running on parallel high performance computers.

A parallel software development for watershed simulations

A watershed software application is designed to model a coupled system of multiple physics on multiple domains. Tremendous computational resources are required to integrate the system equations on large spatial domains with multiple temporal scales among them. Supported by the Department of Defense Common High Performance Computing Software Initiative, the parallel WASH123D software development aims to efficiently simulate one aspect (i.e., soil and land) of the battlespace environment. Currently, the coupled two-dimensional overland and three-dimensional subsurface flows have been completed. Different numerical approaches are implemented to solve different components of the coupled system. The parallelization of such a complex system is developed on an IT-based approach—modular, hierarchical model construction, portable, scalable, and embedded parallel computational tools development and integration. Experimental results are presented to demonstrate the successful implementation of the parallel algorithms. Detailed profiling is also provided to show the imposed light-weight communication overhead.

Parallelization of the WASH123D Code—Phase II: Coupled Two-Dimensional Overland and Three-Dimensional Subsurface Flows

The parallel WASH123D is designed to simulate watershed systems using a coupled system of one-dimensional (1-D) channels, 2-D overland areas, and 3- D subsurface media on parallel scalable computers. The U.S. Department of Defense High Performance Computing Modernization Program funds the parallelization of the watershed model through the Common High Performance Computing Software Initiative in order to solve one aspect of the battlespace environment, which in- cludes space, weather, ocean, and soil, to develop a complete coupled battlespace environment. Tasks in this project include parallel algorithm development, software toolkit development, and performance studies. Currently, the parallel version, which includes coupled 2-D overland and 3-D subsurface flows, is completed and packaged for application projects, such as calibration and validation of the coupled 2-D/3-D Biscayne Bay Coastal Wetland inland flow model presented in this paper, on various computer architectures.

Parallelization of the WASH123D code—Phase I: 2-dimensional overland and 3-dimensional subsurface flows

The parallel WASH123D, which is supported by the DoD CHSSI (Department of Defense Common High Performance Computing Software Support Initiative), is designed to solve watershed problems on scalable computing systems. WASH123D is a first-principle, physics-based model to compute water flow and/or contaminant and sediment transport within a watershed system. In the WASH123D model, a watershed is conceptualized as a coupled system of one-dimensional (1-D) channel network, two-dimensional (2-D) overland regime, and three-dimensional (3-D) subsurface media. It aims to address the environmental issues concerning both water quantity and quality. To reach numerical solutions with reasonable and tolerable computer time for simulations that embrace large meshes, numerical algorithm improvement and code parallelization are two essential tasks. Mathematically, 1-D channel flow and 2-D overland flow are described with the St. Venant equations, which are solved with either the Semi-Lagrangian or the Eulerian finite element method. The 3-D subsurface flow is governed by the modified Richards equation, which is solved with the Eulerian finite element method. The contaminant transport and sediment transport equations, which are solved with the Lagrangian-Eulerian finite element method, are derived based on the mass conservation principle. A parallel in-element particle-tracking algorithm for unsteady flow is applied to backtrack fictitious particles from global nodes to determine the so-called Lagrangian values when the Semi-Lagrangian or the Lagrangian-Eulerian method is used. This paper addresses the parallelization of such a complex numerical model. In phase I, tasks including data structure and software design, software tool development, as well as tool integration are accomplished. The 2- and 3-D flow modules are expected to be employed in the production stage.

Numerical Strategies to Model Surface and Groundwater Interactions for the Biscayne Bay Coastal Wetlands Project Alternatives

WASH123D is a first-principle, physics-based numerical model that computes flow and transport in a watershed system that is conceptualized as a combination of 1-D channel network, 2-D overland regimes, and 3-D subsurface media. It has been selected as the tool to help evaluate the proposed alternatives of the Biscayne Bay Coastal Wetlands project that is one of the 40 projects included in the Florida Comprehensive Everglade Restoration Plan. In order to best rehydrate wetlands and reduce point source discharge to Biscayne Bay, rule-controlled coastal canal structures, rule-controlled pump stations, spreader swales, stormwater treatment areas, flowways, levees, culverts, roads, and backfilling canals are included in these project alternatives. Specified target freshwater flows for Biscayne Bay and the wetlands within the redistribution system are to be computed in each alternative, which will be used in performance measurement to determine the most adequate alternative for further investigation. In this paper, the numerical strategies to incorporate all the aforementioned hydrological features and processes included in the alternatives are presented. An example alternative will be used for demonstration.

Using the Parallel WASH123D Code to Simulate Overland-Subsurface Interactions

WASH123D model is a first-principle, physics-based model, where water flow and/or contaminant and sediment transport within a watershed system are computed. In the WASH123D model, a watershed is conceptualized as a coupled system of 1-D canal /stream network, 2-D overland regime, and 3-D subsurface media. It is designed to answer the environmental issues concerning both water quantity and quality. To reach numerical solutions with reasonable and tolerable computer time for a regional scale watershed simulation, numerical algorithm improvement and code parallelization are two essential tasks when a distributed numerical model, WASH123D, is used. This paper presents the code parallelization approach, followe d by demonstrating its scalability performance. The test problem of a large mesh uses the topographic data in the C -111 Spreader Canal Project. Background The C-111 Spreader Canal Project is one component of the more than 60 restoration plans under the Comprehensive Environmental Restoration Plan (CERP) and has a goal to provide water deliveries that will enhance the connection between the natural areas in the Southern Glades and Model Lands area of South Dade County. The spreader canal

Modeling Surface and Subsurface Hydrologic Interactions in the Biscayne Bay Coastal Wetlands

Restoration of the South Florida ecosystem is a major undertaking for the U.S. Army Corps of Engineers and the South Florida Water Management District in the federally approved Comprehensive Environmental Restoration Plan (CERP). Many competing entities have an interest in the restoration process which will probably include physical changes to the land surface and adjustments to water deliveries. Restoration plans will be developed using numerical hydrologic models that optimize the benefits among the various stakeholders. Since the health of the South Florida ecosystem is impacted by subtle exchanges of surface and subsurface water, there is a need for great accuracy in the models that will identify the best restoration plans. The accuracy needed for design level simulations of the various projects approaches +/0.1 ft in the water elevations. Models with insufficient physical processes and/or highly schematized models with large grid or mesh induced error will not be sufficient to make design level simulation of South Florida hydrologic processes. Additionally, since a large number of models will be set up and analyzed, effective and efficient graphical user interfaces are needed for the models. This paper presents the results of testing WASH123D code to simulate surface water and subsurface interactions in the groundwater system.

Investigating The Application of Channel Boundary Conditions for Model Calibration and Validation

This study investigated how the stage-type and the flow-type boundary conditions may impact the channel flow solutions in order to address issues concerning adequate channel boundary conditions for model calibration and validation. It is revealed that using the stage-type boundary condition and using the flow-type boundary condition yield the same results as long as the conditions are consistent. Because flow may be more sensitive to the change of system input (e.g., boundary condition, source/sink, model parameter, etc.) than stage (or depth), it is suggested that the more accurate measured stage data be used for calibration with the less accurate measured flow data employed as a secondary check point to ensure correct calibration-validation outcomes. A sensitivity analysis to determine an adequate time-step size for the desired computational mesh is essential for valid model calibration and validation.

Model Calibration and Validation of South Florida Biscayne Bay Coastal Wetlands Watershed

Among the 40 projects in the Florida Comprehensive Everglade Restoration Plan, the Biscayne Bay Coastal Wetlands (BBCW) project has a purpose of re-hydrating wetlands and reducing point source discharge to Biscayne Bay. Seven project alternatives that include rule-controlled coastal canal structures, rule-controlled pump stations, spreader swales, stormwater treatment areas, flowways, levees, culverts, roads, and backfilling canals have been proposed to achieve the purpose. These alternatives are evaluated with the WASH123D model-a first-principle, physics-based numerical model that computes flow and transport in a watershed system that is conceptualized as a combination of 1-D channel network, 2-D overland regimes, and 3- D subsurface media. The WASH123D-BBCW model needs to be calibrated and validated before being used to evaluate the alternatives. This paper presents the calibration and validation of the model. A three-step approach that is used for model calibration and validation is presented, discussed, and evaluated in this paper. The three steps are: calibration of the overland Manning’s roughness coefficients (n2) and subsurface hydraulic conductivities (K) with the coupled 2- D/3-D WASH123D-BBCW model where the historical data of canal stage is used as the boundary condition; calibration of the canal Manning’s roughness coefficients (n1) where the calibrated n2 and K from the previous step are used; and validation of the calibrated n1, n2, and K with a second set of historical data. The calibration- validation results of the WASH123D-BBCW model are also presented in the paper.