Daniel Williamson

The Art and Science of Climate Model Tuning

AbstractThe process of parameter estimation targeting a chosen set of observations is an essential aspect of numerical modeling. This process is usually named tuning in the climate modeling community. In climate models, the variety and complexity of physical processes involved, and their interplay through a wide range of spatial and temporal scales, must be summarized in a series of approximate submodels. Most submodels depend on uncertain parameters. Tuning consists of adjusting the values of these parameters to bring the solution as a whole into line with aspects of the observed climate. Tuning is an essential aspect of climate modeling with its own scientific issues, which is probably not advertised enough outside the community of model developers. Optimization of climate models raises important questions about whether tuning methods a priori constrain the model results in unintended ways that would affect our confidence in climate projections. Here, we present the definition and rationale behind model...

Identifying and removing structural biases in climate models with history matching

We describe the method of history matching, a method currently used to help quantify parametric uncertainty in climate models, and argue for its use in identifying and removing structural biases in climate models at the model development stage. We illustrate the method using an investigation of the potential to improve upon known ocean circulation biases in a coupled non-flux-adjusted climate model (the third Hadley Centre Climate Model; HadCM3). In particular, we use history matching to investigate whether or not the behaviour of the Antarctic Circumpolar Current (ACC), which is known to be too strong in HadCM3, represents a structural bias that could be corrected using the model parameters. We find that it is possible to improve the ACC strength using the parameters and observe that doing this leads to more realistic representations of the sub-polar and sub-tropical gyres, sea surface salinities (both globally and in the North Atlantic), sea surface temperatures in the sinking regions in the North Atlantic and in the Southern Ocean, North Atlantic Deep Water flows, global precipitation, wind fields and sea level pressure. We then use history matching to locate a region of parameter space predicted not to contain structural biases for ACC and SSTs that is around 1 % of the original parameter space. We explore qualitative features of this space and show that certain key ocean and atmosphere parameters must be tuned carefully together in order to locate climates that satisfy our chosen metrics. Our study shows that attempts to tune climate model parameters that vary only a handful of parameters relevant to a given process at a time will not be as successful or as efficient as history matching.

Exploratory ensemble designs for environmental models using k-extended Latin Hypercubes

In this paper we present a novel, flexible, and multi-purpose class of designs for initial exploration of the parameter spaces of computer models, such as those used to study many features of the environment. The idea applies existing technology aimed at expanding a Latin Hypercube (LHC) in order to generate initial LHC designs that are composed of many smaller LHCs. The resulting design and its component parts are designed so that each is approximately orthogonal and maximises a measure of coverage of the parameter space. Designs of the type advocated for in this paper are particularly useful when we want to simultaneously quantify parametric uncertainty and any uncertainty due to the initial conditions, boundary conditions, or forcing functions required to run the model. This makes the class of designs particularly suited to environmental models, such as climate models that contain all of these features. The proposed designs are particularly suited to initial exploratory ensembles whose goal is to guide the design of further ensembles aimed at, for example, calibrating the model. We introduce a new emulator diagnostic that exploits the structure of the advocated ensemble designs and allows for the assessment of structural weaknesses in the statistical modelling. We provide illustrations of the method through a simple example and describe a 400 member ensemble of the Nucleus for European Modelling of the Ocean (NEMO) ocean model designed using the method. We build an emulator for NEMO using the created design to illustrate the use of our emulator diagnostic test.

Bayesian Policy Support for Adaptive Strategies Using Computer Models for Complex Physical Systems

In this paper, we discuss combining expert knowledge and computer simulators in order to provide decision support for policy makers managing complex physical systems. We allow future states of the complex system to be viewed after initial policy is made, and for those states to influence revision of policy. The potential for future observations and intervention impacts heavily on optimal policy for today and this is handled within our approach. We show how deriving policy dependent system uncertainty using computer models leads to an intractable backwards induction problem for the resulting decision tree. We introduce an algorithm for emulating an upper bound on our expected loss surface for all possible policies and discuss how this might be used in policy support. To illustrate our methodology, we look at choosing an optimal CO2 abatement strategy, combining an intermediate complexity climate model and an economic utility model with climate data.

Evolving Bayesian Emulators for Structured Chaotic Time Series, with Application to Large Climate Models

We develop Bayesian dynamic linear model Gaussian processes for emulation of time series output for computer models that may exhibit chaotic behavior, but where this behavior retains some underlying structure. The statistical technology is particularly suited to emulating the time series output of large climate models that exhibit this feature and where we want samples from the posterior of the emulator to evolve in the same way as dynamic processes in the computer model do. The methodology combines key features of good uncertainty quantification (UQ) methods such as using complex mean functions to capture large-scale signals within parameter space, with dynamic linear models in a way that allows UQ to borrow strength from the Bayesian time series literature. We present an MCMC algorithm for sampling from the posterior of the emulator parameters when the roughness lengths of the Gaussian process are unknown. We discuss an interpretation of the results of this algorithm that allows us to use MCMC to fix th...

A comparison of statistical emulation methodologies for multi-wave calibration of environmental models

Expensive computer codes, particularly those used for simulating environmental or geological processes, such as climate models, require calibration (sometimes called tuning). When calibrating expensive simulators using uncertainty quantification methods, it is usually necessary to use a statistical model called an emulator in place of the computer code when running the calibration algorithm. Though emulators based on Gaussian processes are typically many orders of magnitude faster to evaluate than the simulator they mimic, many applications have sought to speed up the computations by using regression‐only emulators within the calculations instead, arguing that the extra sophistication brought using the Gaussian process is not worth the extra computational power. This was the case for the analysis that produced the UK climate projections in 2009. In this paper, we compare the effectiveness of both emulation approaches upon a multi‐wave calibration framework that is becoming popular in the climate modeling community called “history matching.” We find that Gaussian processes offer significant benefits to the reduction of parametric uncertainty over regression‐only approaches. We find that in a multi‐wave experiment, a combination of regression‐only emulators initially, followed by Gaussian process emulators for refocussing experiments can be nearly as effective as using Gaussian processes throughout for a fraction of the computational cost. We also discover a number of design and emulator‐dependent features of the multi‐wave history matching approach that can cause apparent, yet premature, convergence of our estimates of parametric uncertainty. We compare these approaches to calibration in idealized examples and apply it to a well‐known geological reservoir model.

Uncertainty quantification for computer models with spatial output using calibration-optimal bases

ABSTRACT The calibration of complex computer codes using uncertainty quantification (UQ) methods is a rich area of statistical methodological development. When applying these techniques to simulators with spatial output, it is now standard to use principal component decomposition to reduce the dimensions of the outputs in order to allow Gaussian process emulators to predict the output for calibration. We introduce the “terminal case,” in which the model cannot reproduce observations to within model discrepancy, and for which standard calibration methods in UQ fail to give sensible results. We show that even when there is no such issue with the model, the standard decomposition on the outputs can and usually does lead to a terminal case analysis. We present a simple test to allow a practitioner to establish whether their experiment will result in a terminal case analysis, and a methodology for defining calibration-optimal bases that avoid this whenever it is not inevitable. We present the optimal rotation algorithm for doing this, and demonstrate its efficacy for an idealized example for which the usual principal component methods fail. We apply these ideas to the CanAM4 model to demonstrate the terminal case issue arising for climate models. We discuss climate model tuning and the estimation of model discrepancy within this context, and show how the optimal rotation algorithm can be used in developing practical climate model tuning tools. Supplementary materials for this article are available online.

Posterior belief assessment: Extracting meaningful subjective judgements from Bayesian analyses with complex statistical models

Danny Williamson is funded by an EPSRC fellowship grant number EP/K019112/1. We wish to thank an associate editor and two referees for their thoughtful and detailed comments regarding the paper. Their suggestions have helped to improve it a great deal. We also thank Adam Blaker and Bablu Sinha for running the NEMO ocean model to generate the ensemble that appears in our application.