You Ling

Quantitative model validation techniques: New insights

This paper develops new insights into quantitative methods for the validation of computational model prediction. Four types of methods are investigated, namely classical and Bayesian hypothesis testing, a reliability-based method, and an area metric-based method. Traditional Bayesian hypothesis testing is extended based on interval hypotheses on distribution parameters and equality hypotheses on probability distributions, in order to validate models with deterministic/stochastic output for given inputs. Two types of validation experiments are considered - fully characterized (all the model/experimental inputs are measured and reported as point values) and partially characterized (some of the model/experimental inputs are not measured or are reported as intervals). Bayesian hypothesis testing can minimize the risk in model selection by properly choosing the model acceptance threshold, and its results can be used in model averaging to avoid Type I/II errors. It is shown that Bayesian interval hypothesis testing, the reliability-based method, and the area metric-based method can account for the existence of directional bias, where the mean predictions of a numerical model may be consistently below or above the corresponding experimental observations. It is also found that under some specific conditions, the Bayes factor metric in Bayesian equality hypothesis testing and the reliability-based metric can both be mathematically related to the p-value metric in classical hypothesis testing. Numerical studies are conducted to apply the above validation methods to gas damping prediction for radio frequency (RF) microelectromechanical system (MEMS) switches. The model of interest is a general polynomial chaos (gPC) surrogate model constructed based on expensive runs of a physics-based simulation model, and validation data are collected from fully characterized experiments.

Separation of aleatory and epistemic uncertainty in probabilistic model validation

This paper investigates model validation under a variety of different data scenarios and clarifies how different validation metrics may be appropriate for different scenarios. In the presence of multiple uncertainty sources, model validation metrics that compare the distributions of model prediction and observation are considered. Both ensemble validation and point-by-point approaches are discussed, and it is shown how applying the model reliability metric point-by-point enables the separation of contributions from aleatory and epistemic uncertainty sources. After individual validation assessments are made at different input conditions, it may be desirable to obtain an overall measure of model validity across the entire domain. This paper proposes an integration approach that assigns weights to the validation results according to the relevance of each validation test condition to the overall intended use of the model in prediction. Since uncertainty propagation for probabilistic validation is often unaffordable for complex computational models, surrogate models are often used; this paper proposes an approach to account for the additional uncertainty introduced in validation by the uncertain fit of the surrogate model. The proposed methods are demonstrated with a microelectromechanical system (MEMS) example.

Dynamic Bayesian Network for Aircraft Wing Health Monitoring Digital Twin

Current airframe health monitoring generally relies on deterministic physics models and ground inspections. This paper uses the concept of a dynamic Bayesian network to build a versatile probabilis...

Challenging issues in Bayesian calibration of multi-physics models

This paper investigates several challenging aspects of Bayesian calibration for multiphysics models, including: (1) calibration with different forms of experimental data (e.g., interval data and time series data), (2) determination of the identifiability of model parameters when the analytical expression of model is known or unknown, (3) calibration of multiple physics models sharing common parameters, and (4) efficient use of available data in a multi-model calibration problem especially when the experimental resources are limited. A Bayesian network-based method is developed for the calibration of multi-physics models, integrating various sources of uncertainty with information from computational models and experimental data. We adopt the well-known Kennedy and O’Hagan (KOH) framework for model calibration under uncertainty, and develop extensions to multi-physics models and various scenarios of available data. Both aleatoric uncertainty (due to natural variability) and epistemic uncertainty (due to lack of information, including data uncertainty and model uncertainty) are accounted for in the calibration process.A first-order Taylor series expansion-based method is proposed to determine which model parameters are identifiable, i.e., to find the parameters that can be calibrated with the available data. Following the KOH framework, a probabilistic discrepancy term is estimated and added to the prediction of the calibrated model, attempting to account for model uncertainty. This discrepancy term is modeled as a Gaussian process when sufficient data are available for multiple model input combinations, and is modeled as a random variable when the available data set is small and limited. The overall approach is illustrated using two application examples related to microelectromechanical system (MEMS) devices: (1) calibration of a dielectric charging model with time-series data, and (2) calibration of two physics models (pull-in voltage and creep) using measurements of different physical quantities in different devices.

Options for the inclusion of model discrepancy in Bayesian calibration

One of the challenging issues in the well-known Kennedy and O’Hagan framework for Bayesian calibration is to formulate the prior of model discrepancy function, which can significantly affect the results of calibration. In the absence of physical knowledge on model inadequacy, it is often not clear how to construct a suitable prior, whereas an inappropriate selection of prior may lead to biased or useless parameter estimation. Aiming to address the uncertainty arising from the selection of a particular prior, this paper conducts an extensive study on possible formulations of model discrepancy function, and proposes a three-step (calibration, validation, and combination) approach in order to inform the decision on the construction of model discrepancy priors. In the validation step, a reliability-based metric is used to evaluate the predictions based on calibrated model parameters and discrepancy in the validation domain. The validation metric serves as a quantitative measure of how well the discrepancy formulation captures the physics missing in the model. In the combination step, the posterior distributions of model parameter and discrepancy corresponding to different priors are combined into a single distribution based on the probabilistic weights derived from the validation step. The combined distribution acknowledges the uncertainty in the prior formulation of model discrepancy function.

Confidence assessment in model-based structural health monitoring

This paper presents a methodology for confidence assessment in model-based structural health monitoring, using the domain of fatigue crack growth analysis. Several models - finite element model, crack growth model, surrogate model, etc. - are connected through a Bayes network that aids in model calibration, uncertainty quantification, and model validation. Three types of uncertainty are included in both uncertainty quantification and model validation: (1) natural variability in loading and material properties; (2) data uncertainty due to measurement errors, sparse data, and different inspection scenarios (crack not detected, crack detected but size not measured, and crack detected with size measurement); and (3) modeling uncertainty and errors during crack growth analysis, numerical approximations, and finite element discretization. Global sensitivity analysis is used to quantify the contribution of each source of uncertainty to the overall prediction uncertainty and identify the important parameters that need to be calibrated. Bayesian hypothesis testing is used for model validation and the Bayes factor metric is used to quantify the confidence in the model prediction. 1 2

Model Calibration for Fatigue Crack Growth Analysis under Uncertainty

This paper presents a Bayesian methodology for model calibration applied to fatigue crack growth analysis of structures with complicated geometry and subjected to multi-axial variable amplitude loading conditions. The crack growth analysis uses the concept of equivalent initial flaw size to replace small crack growth calculations and makes direct use of a long crack growth model. The equivalent initial flaw size is calculated from material and geometrical properties of the specimen. A surrogate model, trained by a few finite element runs, is used to calculate the stress intensity factor used in crack growth calculations. This eliminates repeated use of an expensive finite element model in each cycle and leads to rapid computation, thereby making the methodology efficient and inexpensive. Three different kinds of models – finite element models, surrogate models and crack growth models - are connected in this framework. Various sources of uncertainty – natural variability, data uncertainty and modeling errors - are considered in this procedure. The various component models, their model parameters and the modeling errors are integrated using a Bayesian approach. Using inspection data, the parameters of the crack growth model and the modeling error are updated using Bayes theorem. The proposed method is illustrated using an application problem, surface cracking in a cylindrical structure.

Fatigue Crack Growth Analysis under Uncertainty

This paper presents a methodology to quantify the uncertainty in fatigue crack growth analysis, applied to structures with complicated geometry and subjected to variable amplitude multi-axial loading. Finite element analysis is used to address the complicated geometry and calculate the stress intensity factors. Crack growth under variable ampli tude loading is modeled using a modified Paris law that includes retardation effects. During cycle by-cycle integration of the crack growth law, a Gaussian process surrogate model is used to replace the expensive finite element analysis. The effect of diff erent kinds of uncertainty – physical variability, data uncertainty and modeling errors – on crack growth prediction is investigated. The various sources of uncertainty include, but not limited to, variability in loading conditions, material parameters, experimental data, model uncertainty, etc. Three different kinds of modeling errors – crack growth model error, discretization error and surrogate model error – are included in analysis. The different kinds of uncertainty are incorporated into the crack growth prediction methodology to predict the probability distribution of crack size as a function of number of load cycles. The proposed method is illustrated using an application problem, surface cracking in a cylindrical structure.