Stephen M. Westenbroek

Approaches in highly parameterized inversion: TSPROC, a general time-series processor to assist in model calibration and result summarization

The TSPROC (Time Series PROCessor) computer software uses a simple scripting language to process and analyze time series. It was developed primarily to assist in the calibration of environmental models. The software is designed to perform calculations on time-series data commonly associated with surface-water models, including calculation of flow volumes, transformation by means of basic arithmetic operations, and generation of seasonal and annual statistics and hydrologic indices. TSPROC can also be used to generate some of the key input files required to perform parameter optimization by means of the PEST (Parameter ESTimation) computer software. Through the use of TSPROC, the objective function for use in the model-calibration process can be focused on specific components of a hydrograph. Background This report documents and describes the use of TSPROC (Time Series PROCessor), a software package designed to assist in the calibration of models by editing and distilling time series datasets into more meaningful observations to be used in the optimization objective function. This report is designed to supplement existing literature and documents regarding PEST (Parameter ESTimation) and TSPROC software. A short summary of a selection of existing documents is provided in this section. TSPROC has existed in one form or another for several years. The Doherty (2008) version of TSPROC included several supplemental tools that are not included or discussed in this report. These additional tools include: • adjobs—— adjusts observation weights for different observation groups in a PEST control file by use of a smple user-adjustable expression, • plt2smp—build a site sample file on the basis of a Hydrological Simulation Program—Fortran (HSPF)generated plot file; used as part of a composite model run by PEST, • smpchek—checks a site sample file (*.ssf) for correctness, • smp2hyd—rewrites the contents of a site sample file for a user-specified list of sites in a form suitable for plotting against time, and • smp2vol— calculates volumes between arbitrary dates and times for flow samples listed in a site sample file. This report does not document the support software referenced above, because many of the functions are either model-specific or can largely be accomplished by TSPROC software. The application of TSPROC in a model optimization exercise implies the use of PEST software (Doherty 2010a, b). TSPROC is designed to work with two new additions to the PEST family: • PEST++ (Welter and others, 2012) is a new code that aims to implement the most popular features of the venerable PEST program while shielding the user from some of the trickier implementation details, • GENIE (Muffles and others, 2012) is a new parallel run manager designed to make it easier to run models on multiple machines enabled by communications over TCP/IP protocol. The reader should also be aware of publications that give examples of PEST and TSPROC application to surface-water models, including Doherty and Skahill (2006), Skahill and Doherty (2006), Cocca and others (2004), and Doherty and Johnston (2003). 2 Approaches in Highly Parameterized Inversion: TSPROC, a General Time-Series Processor Introduction TSPROC is a program designed to assist in model parameter estimation, especially surface-water models, by means of the widely used PEST software (Doherty, 2010a, b). TSPROC may also be used to create some of the basic input files required to make use of PEST. In addition to being a utility for working within the PEST environment, TSPROC is a general purpose tool for working with time series and developing statistics and reports from observations and model results. Surface-water models are capable of simulating daily (or more frequent) discharge values at many locations and for many years. The thousands of data points contained in each hydrograph are serially related, and in the case of surfacewater models, more of this type of data is not necessarily better; therefore, comparing statistical indices or summaries between the simulated and observed data is useful. Although parameter estimation may be successfully applied through the use of daily hydrograph data, the resulting model calibration will not necessarily be able to replicate the aspects of the hydrograph pertinent to the project at hand. In this situation, it is necessary to process the data into a form that focuses the calibration process on the aspects of the hydrograph that are most important to the model’s intended use. Because thousands or tens of thousands of data records are involved, automation of model post-processing tasks and PEST input file preparation is desirable. This publication documents the TSPROC software package and provides details about each of the program commands and options in the scripting language. Appendix 1 describes the format of a “site sample file,” a generic American Standard Code for Information Exchange (ASCII) file understood by TSPROC that may be used for storing, importing, and exporting time-series data. Appendix 2 describes the development and basic use of TSPROC when packaged as a Python module. Parameter Estimation and TSPROC This section discusses general TSPROC capabilities and describes how TSPROC may be used as part of a composite model subject to parameter optimization. TSPROC Capabilities TSPROC fills two roles. First, it is a time-series processor, having the ability to perform many different types of operations on observed and model-generated time series. Second, it automates the generation of PEST input files for calibration tasks of arbitrary complexity based on these time series. Many of the operations performed by TSPROC are designed specifically for use in a model parameter-estimation context. To make valid comparisons between the modeled and observed time series, the model-generated time series must be interpolated to the times when observations were made. Because observations of a particular environmental quantity are often intermittent rather than regular, TSPROC does not assume that any time series that it manipulates has a constant sample interval. TSPROC makes possible the incorporation of some or all of the following data types into the model-calibration process. 1. Raw data. All or part of complete observation time series can be included; TSPROC can facilitate this by interpolating the model-generated series to field observation times. 2. Processed or filtered data. Time-series data can be decomposed or filtered using the digital filtering or hydrograph separation techniques. TSPROC includes digital filtering capabilities that allow the separation of high, medium, and low frequency components of any time series. This can be useful in baseflow separation, as implemented in the Doherty (2008) version of TSPROC (Nathan and McMahon, 1990). The version of TSPROC documented in this report also incorporates U.S. Geological Survey (USGS) HYSEP (HYdrograph SEParation) modules that include three different techniques for computing baseflow separation (Sloto and Crouse, 1996). Modeled and observed processed or filtered counterparts can be individually matched through the calibration process. Parameter Estimation and TSPROC 3 3. Accumulated volumes and masses. Flow volumes and constituent masses can be accumulated between any number of arbitrary dates and times occurring within the model simulation period. Inclusion of volumetric and mass data, calculated on the basis of field observations and on model-generated flows and constituent concentrations (interpolated to field observation times), respectively, can bring numerical stability to the parameter-estimation process and result in more robust estimates of parameter values. 4. Exceedance-time characteristics. As with volumetric and mass data, the inclusion of exceedance-time characteristics in the inversion process can decrease the likelihood of numerical instability while it promotes estimation of a realistic set of parameter values. Furthermore, in many modeling applications it is crucial that a model predict exceedance-time characteristics as accurately as possible under future climatic or management conditions. 5. Summary statistics and period statistics. Basic statistics (mean, sum, median, maximum, minimum, range, and standard deviation) can be calculated from the terms of a time series or functions of these terms over varying time intervals; the period statistics module allows for these statistics to be calculated on a monthly or annual basis over the length of the time series (for example, mean monthly), or as a series composed of monthly values (for example, monthly mean). 6. Functions of arbitrary complexity and data patterns. New time series may be calculated on the basis of one or more measured or modeled time series. In many instances of model calibration, it may be better to include a comparison of derived time series, rather than raw time series, in the parameter-estimation process. TSPROC allows the user to calculate any number of new time series based on relationships of arbitrary complexity between existing time series. For example, in some calibration contexts, it may be beneficial to compare the log (or some other function) of an observation type with its model-generated counterpart over all or part of the model simulation time. In other contexts, it may be useful to compare a combination of today’s and yesterday’s flow with the model-generated equivalent of this same quantity. Minimization of the discrepancies between two such composite time series may result in better parameter estimates, as well as better estimates of the uncertainties associated with these parameters, because it incorporates the correlation structure of flow and constituent observations into the parameter-estimation process (Kuczera, 1983). Relationships such as those used by the USGS LOADEST program (Runkel and others, 2004) may be suitable in some cases. 7. Hydrologic indices. When TSPROC is applie