Paul D. Arendt

Quantification of Model Uncertainty: Calibration, Model Discrepancy, and Identifiability

To use predictive models in engineering design of physical systems, one should first quantify the model uncertainty via model updating techniques employing both simulation and experimental data. While calibration is often used to tune unknown calibration parameters of a computer model, the addition of a discrepancy function has been used to capture model discrepancy due to underlying missing physics, numerical approximations, and other inaccuracies of the computer model that would exist even if all calibration parameters are known. One of the main challenges in model updating is the difficulty in distinguishing between the effects of calibration parameters versus model discrepancy. We illustrate this identifiability problem with several examples, explain the mechanisms behind it, and attempt to shed light on when a system may or may not be identifiable. In some instances, identifiability is achievable under mild assumptions, whereas in other instances, it is virtually impossible. In a companion paper, we demonstrate that using multiple responses, each of which depends on a common set of calibration parameters, can substantially enhance identifiability.

Toward a better understanding of model validation metrics

Model validation metrics have been developed to provide a quantitative measure that characterizes the agreement between predictions and observations. In engineering design, the metrics become useful for model selection when alternative models are being considered. Additionally, the predictive capability of a computational model needs to be assessed before it is used in engineering analysis and design. Due to the various sources of uncertainties in both computer simulations and physical experiments, model validation must be conducted based on stochastic characteristics. Currently there is no unified validation metric that is widely accepted. In this paper, we present a classification of validation metrics based on their key characteristics along with a discussion of the desired features. Focusing on stochastic validation with the consideration of uncertainty in both predictions and physical experiments, four main types of metrics, namely classical hypothesis testing, Bayes factor, frequentist’s metric, and area metric, are examined to provide a better understanding of the pros and cons of each. Using mathematical examples, a set of numerical studies are designed to answer various research questions and study how sensitive these metrics are with respect to the experimental data size, the uncertainty from measurement error, and the uncertainty in unknown model parameters. The insight gained from this work provides useful guidelines for choosing the appropriate validation metric in engineering applications. [DOI: 10.1115/1.4004223]

A Hierarchical Statistical Sensitivity Analysis Method for Multilevel Systems With Shared Variables

Statistical sensitivity analysis (SSA) is an effective methodology to examine the impact of variations in model inputs on the variations in model outputs at either a prior or posterior design stage. A hierarchical statistical sensitivity analysis (HSSA) method has been proposed in literature to incorporate SSA in designing complex engineering systems with a hierarchical structure. However, the original HSSA method only deals with hierarchical systems with independent subsystems. For engineering systems with dependent subsystem responses and shared variables, an extended HSSA method with shared variables (named HSSA-SV) is developed in this work. A top-down strategy, the same as in the original HSSA method, is employed to direct SSA from the top level to lower levels. To overcome the limitation of the original HSSA method, the concept of a subset SSA is utilized to group a set of dependent responses from the lower level submodels in the upper level SSA and the covariance of dependent responses is decomposed into the contributions from individual shared variables. An extended aggregation formulation is developed to integrate local submodel SSA results to estimate the global impact of lower level inputs on the top level response. The effectiveness of the proposed HSSA-SV method is illustrated via a mathematical example and a multiscale design problem.

A preposterior analysis to predict identifiability in the experimental calibration of computer models

ABSTRACT When using physical experimental data to adjust, or calibrate, computer simulation models, two general sources of uncertainty that must be accounted for are calibration parameter uncertainty and model discrepancy. This is complicated by the well-known fact that systems to be calibrated are often subject to identifiability problems, in the sense that it is difficult to precisely estimate the parameters and to distinguish between the effects of parameter uncertainty and model discrepancy. We develop a form of preposterior analysis that can be used, prior to conducting physical experiments but after conducting the computer simulations, to predict the degree of identifiability that will result after conducting the physical experiments for a given experimental design. Specifically, we calculate the preposterior covariance matrix of the calibration parameters and demonstrate that, in the examples that we consider, it provides a reasonable prediction of the actual posterior covariance that is calculated after the experimental data are collected. Consequently, the preposterior covariance can be used as a criterion for designing physical experiments to help achieve better identifiability in calibration problems.

Updating Predictive Models: Calibration, Bias Correction and Identifiability

Model updating, which utilizes mathematical means to combine model simulations with physical observations for improving model predictions, has been viewed as an integral part of a model validation process. While calibration is often used to “tune” uncertain model parameters, bias-correction has been used to capture model inadequacy due to a lack of knowledge of the physics of a problem. While both sources of uncertainty co-exist, these two techniques are often implemented separately in model updating. This paper examines existing approaches to model updating and presents a modular Bayesian approach as a comprehensive framework that accounts for many sources of uncertainty in a typical model updating process and provides stochastic predictions for the purpose of design. In addition to the uncertainty in the computer model parameters and the computer model itself, this framework accounts for the experimental uncertainty and the uncertainty due to the lack of data in both computer simulations and physical experiments using the Gaussian process model. Several challenges are apparent in the implementation of the modular Bayesian approach. We argue that distinguishing between uncertain model parameters (calibration) and systematic inadequacies (bias correction) is often quite challenging due to an identifiability issue. We present several explanations and examples of this issue and bring up the needs of future research in distinguishing between the two sources of uncertainty.Copyright

Extended Objective-Oriented Sequential Sampling Method for Robust Design of Complex Systems Against Design Uncertainty

In robust design of complex systems, metamodeling techniques are commonly used to replace expensive computer simulations. To improve the sampling efficiency, efforts have been made towards developing objective-oriented sequential sampling methods for deterministic problems. In this paper, an extended objective-oriented sequential sampling method is proposed for robust design, with an emphasis on those problems with uncertainty in design variables. The method involves quantitative assessment of the effects of metamodeling uncertainty on the robust responses, as well as a sequential strategy of choosing samples to adaptively improve the predicted robust response. To validate the benefits of the sequential strategy, two mathematical examples are illustrated first. This is followed by an automotive crashworthiness design example, a highly expensive and non-linear problem. Results show that the proposed method can mitigate the effect of both metamodeling uncertainty and design uncertainty, and more efficiently identify the robust solution compared to the one-stage sampling approach that is commonly used in practice.Copyright

Improving identifiability in model calibration using multiple responses

In physics-based engineering modeling, the two primary sources of model uncertainty, which account for the differences between computer models and physical experiments, are parameter uncertainty and model discrepancy. Distinguishing the effects of the two sources of uncertainty can be challenging. For situations in which identifiability cannot be achieved using only a single response, we propose to improve identifiability by using multiple responses that share a mutual dependence on a common set of calibration parameters. To that end, we extend the single response modular Bayesian approach for calculating posterior distributions of the calibration parameters and the discrepancy function to multiple responses. Using an engineering example, we demonstrate that including multiple responses can improve identifiability (as measured by posterior standard deviations) by an amount that ranges from minimal to substantial, depending on the characteristics of the specific responses that are combined.

Objective–Oriented Sequential Sampling for Simulation Based Robust Design Considering Multiple Sources of Uncertainty

Sequential sampling strategies have been developed for managing complexity when using computationally expensive computer simulations in engineering design. However, much of the literature has focused on objective-oriented sequential sampling methods for deterministic optimization. These methods cannot be directly applied to robust design which must account for uncontrollable variations in certain input variables (i.e., noise variables). Obtaining a robust design that is insensitive to variations in the noise variables is more challenging. Even though methods exist for sequential sampling in design under uncertainty, the majority of the existing literature does not systematically take into account the interpolation uncertainty that results from limitations on the number of simulation runs, the effect of which is inherently more severe than in deterministic design. In this paper, we develop a systematic objective-oriented sequential sampling approach to robust design with consideration of both noise variable uncertainty and interpolation uncertainty. The method uses Gaussian processes to model the costly simulator and quantify the interpolation uncertainty within a robust design objective. We examine several criteria, including our own proposed criteria, for sampling the design and noise variables and provide insight into their performance behaviors. We show that for both of the examples considered in this paper the proposed sequential algorithm is more efficient in finding the robust design solution than a one-shot space filling design.Copyright

Simulation of the diffusion of doxorubicin (DOX) in polymer pores for a multiscale nanodiamond based drug delivery system

A drug delivery system utilizing nanodiamonds has been proposed for the controlled release of the chemotherapeutic doxorubicin (DOX) [1,2]. The drug delivery system consists of a patch which is envisioned to be implanted immediately after a tumor removal surgery to target residual cancerous cells such that tumor reoccurrence can be effectively prevented. Therefore, long release times are desired for the patch to prevent any new cancerous tissue from forming. The patch is to be assembled in three different layers beginning with a bottom layer comprised of nonporous parylene. A layer of DOX loaded nanodimonds is deposited onto the base layer. Then a porous layer of parylene is accumulated on top of the nanodimonds. Nanodimonds are utilized because their release rates are based on pH, which changes near cancerous cells. Due to the complexity of the drug delivery system, models are created at several different scales.© 2010 ASME

An extended hierarchical statistical sensitivity analysis method for multilevel systems with shared variables

Statistical sensitivity analysis (SSA) is an effective methodology to examine the impact of variations in model inputs on the variations in model outputs at either a prior or posterior design stage. A hierarchical statistical sensitivity analysis (HSSA) method has been proposed in literature to incorporate SSA in designing complex engineering systems with a hierarchical structure. However, the original HSSA method only deals with hierarchical systems with independent subsystems. Due to the existence of shared variables at lower levels, responses from lower level submodels that act as inputs to a higher level subsystem are both functionally and statistically dependent. For designing engineering systems with dependent subsystem responses, an extended hierarchical statistical sensitivity analysis (EHSSA) method is developed in this work to provide a ranking order based on the impact of lower level model inputs on the top level system performance. A top-down strategy, same as in the original HSSA method, is employed to direct SSA from the top level to lower levels. To overcome the limitation of the original HSSA method, the concept of a subset SSA is utilized to group a set of dependent responses from lower level submodels in the upper level SSA. For variance decomposition at a lower level, the covariance of dependent responses is decomposed into the contributions from individual shared variables. To estimate the global impact of lower level inputs on the top level output, an extended aggregation formulation is developed to integrate local submodel SSA results. The importance sampling technique is also introduced to re-use the existing data from submodels SSA during the aggregation process. The effectiveness of the proposed EHSSA method is illustrated via a mathematical example and a multiscale design problem.Copyright