Dirk Gorissen

A Novel Hybrid Sequential Design Strategy for Global Surrogate Modeling of Computer Experiments

Many complex real-world systems can be accurately modeled by simulations. However, high-fidelity simulations may take hours or even days to compute. Because this can be impractical, a surrogate model is often used to approximate the dynamic behavior of the original simulator. This model can then be used as a cheap, drop-in replacement for the simulator. Because simulations can be very expensive, the data points, which are required to build the model, must be chosen as optimally as possible. Sequential design strategies offer a huge advantage over one-shot experimental designs because they can use information gathered from previous data points in order to determine the location of new data points. Each sequential design strategy must perform a trade-off between exploration and exploitation, where the former involves selecting data points in unexplored regions of the design space, while the latter suggests adding data points in regions which were previously identified to be interesting (for example, highly nonlinear regions). In this paper, a novel hybrid sequential design strategy is proposed which uses a Monte Carlo-based approximation of a Voronoi tessellation for exploration and local linear approximations of the simulator for exploitation. The advantage of this method over other sequential design methods is that it is independent of the model type, and can therefore be used in heterogeneous modeling environments, where multiple model types are used at the same time. The new method is demonstrated on a number of test problems, showing that it is a robust, competitive, and efficient sequential design strategy.

A novel sequential design strategy for global surrogate modeling

In mathematical/statistical modeling of complex systems, the locations of the data points are essential to the success of the algorithm. Sequential design methods are iterative algorithms that use data acquired from previous iterations to guide future sample selection. They are often used to improve an initial design such as a Latin hypercube or a simple grid, in order to focus on highly dynamic parts of the design space. In this paper, a comparison is made between different sequential design methods for global surrogate modeling on a real-world electronics problem. Existing exploitation and exploration-based methods are compared against a novel hybrid technique which incorporates both an exploitation criterion, using local linear approximations of the objective function, and an exploration criterion, using a Monte Carlo Voronoi tessellation. The test results indicate that a considerable improvement of the average model accuracy can be achieved by using this new approach.

Blind Kriging: Implementation and performance analysis

When analysing data from computationally expensive simulation codes or process measurements, surrogate modelling methods are firmly established as facilitators for design space exploration, sensitivity analysis, visualisation and optimisation. Kriging is a popular surrogate modelling technique for data based on deterministic computer experiments. There exist several types of Kriging, mostly differing in the type of regression function used. Recently a promising new variable selection technique was proposed to identify a regression function in the Kriging framework. In this paper this type of Kriging, i.e., blind Kriging, has been efficiently implemented in Matlab(R) and has been extended. The implementation is validated and tested on several examples to illustrate the strength and weaknesses of this new, promising modelling technique. It is shown that the performance of blind Kriging is as good as, or better than ordinary Kriging. Though, blind Kriging comes at double the computational cost with respect to ordinary Kriging.

Sequential modeling of a low noise amplifier with neural networks and active learning

The use of global surrogate models has become commonplace as a cost effective alternative for performing complex high fidelity computer simulations. Due to their compact formulation and negligible evaluation time, global surrogate models are very useful tools for exploring the design space, what-if analysis, optimization, prototyping, visualization, and sensitivity analysis. Neural networks have been proven particularly useful in this respect due to their ability to model high dimensional, non-linear responses accurately. In this article, we present the results of an extensive study on the performance of neural networks as compared to other modeling techniques in the context of active learning. We investigate the scalability and accuracy in function of the number design variables and number of datapoints. The case study under consideration is a high dimensional, parametrized low noise amplifier RF circuit block.

Multiobjective global surrogate modeling, dealing with the 5-percent problem

When dealing with computationally expensive simulation codes or process measurement data, surrogate modeling methods are firmly established as facilitators for design space exploration, sensitivity analysis, visualization, prototyping and optimization. Typically the model parameter (=hyperparameter) optimization problem as part of global surrogate modeling is formulated in a single objective way. Models are generated according to a single objective (accuracy). However, this requires an engineer to determine a single accuracy target and measure upfront, which is hard to do if the behavior of the response is unknown. Likewise, the different outputs of a multi-output system are typically modeled separately by independent models. Again, a multiobjective approach would benefit the domain expert by giving information about output correlation and enabling automatic model type selection for each output dynamically. With this paper the authors attempt to increase awareness of the subtleties involved and discuss a number of solutions and applications. In particular, we present a multiobjective framework for global surrogate model generation to help tackle both problems and that is applicable in both the static and sequential design (adaptive sampling) case.

Evolutionary Neuro-Space Mapping Technique for Modeling of Nonlinear Microwave Devices

This paper presents a new advance in Neuro-space mapping (Neuro-SM) techniques for modeling nonlinear microwave devices. Suppose that existing device models (namely, coarse models) cannot match the behavior of a new device (referred to as the fine model). By neural network mapping of the voltage and current signals from the coarse to the fine models, Neuro-SM can modify the behavior of the coarse model to match that of the fine model. However, the efficiency of mapping depends on both the mapping structure and the coarse model. In this paper, a structural optimization technique is presented to achieve optimal combinations of mapping structure and coarse model. An aggressive optimization formulation exploring detailed structural variations in both the mapping and the coarse model is proposed, where the internal branches of coarse models and separate mappings for the voltage and current at gate and drain are used as basic topology variables. The formulation of such a structural optimization by an evolutionary optimization algorithm is proposed. Numerical examples of metal-semiconductor field-effect transistor and high electron-mobility transistor modeling demonstrate that, by using the proposed algorithm, optimal combinations of space mapping and coarse model structures can be achieved leading to the best modeling accuracy with the simplest mapping function.

Automatic model type selection with heterogeneous evolution: An application to RF circuit block modeling

Many complex, real world phenomena are difficult to study directly using controlled experiments. Instead, the use of computer simulations has become commonplace as a cost effective alternative. However, regardless of Moorepsilas law, performing high fidelity simulations still requires a great investment of time and money. Surrogate modeling (metamodeling) has become indispensable as an alternative solution for relieving this burden. Many surrogate model types exist (support vector machines, Kriging, RBF models, neural networks, ...) but no type is optimal in all circumstances. Nor is there any hard theory available that can help make this choice. The same is true for setting the surrogate model parameters (bias- variance trade-off). Traditionally, the solution to both problems has been a pragmatic one, guided by intuition, prior experience or simply available software packages. In this paper we present a more founded approach to these problems. We describe an adaptive surrogate modeling environment, driven by speciated evolution, to automatically determine the optimal model type and complexity. Its utility and performance is presented on a case study from electronics.

Adaptive distributed metamodeling

Simulating and optimizing complex physical systems is known to be a task of considerable time and computational complexity. As a result, metamodeling techniques for the efficient exploration of the design space have become standard practice since they reduce the number of simulations needed. However, conventionally such metamodels are constructed sequentially in a one-shot manner, without exploiting inherent parallelism. To tackle this inefficient use of resources we present an adaptive framework where modeler and simulator interact through a distributed environment, thus decreasing model generation and simulation turnaround time. This paper provides evidence that such a distributed approach for adaptive sampling and modeling is worthwhile investigating. Research in this new field can lead to even more innovative automated modeling tools for complex simulation systems.

An alternative approach to avoid overfitting for surrogate models

Surrogate models are data-driven models used to accurately mimic the complex behavior of a system. They are often used to approximate computationally expensive simulation code in order to speed up the exploration of design spaces. A crucial step in the building of surrogate models is finding a good set of hyperparameters, which determine the behavior of the model. This is especially important when dealing with sparse data, as the models are in that case more prone to overfitting. Cross-validation is often used to optimize the hyperparameters of surrogate models, however it is computationally expensive and can still lead to overfitting or other erratic model behavior. This paper introduces a new auxiliary measure for the optimization of the hyperparameters of surrogate models which, when used in conjunction with a cheap accuracy measure, is fast and effective at avoiding unexplained model behavior.

Evolutionary regression modeling with active learning: an application to rainfall runoff modeling

Many complex, real world phenomena are difficult to study directly using controlled experiments. Instead, the use of computer simulations has become commonplace as a feasible alternative. However, due to the computational cost of these high fidelity simulations, the use of neural networks, kernel methods, and other surrogate modeling techniques has become indispensable. Surrogate models are compact and cheap to evaluate, and have proven very useful for tasks such as optimization, design space exploration, visualization, prototyping, and sensitivity analysis. Consequently, there is great interest in techniques that facilitate the construction of such regression models, while minimizing the computational cost and maximizing model accuracy. The model calibration problem in rainfall runoff modeling is an important problem from hydrology that can benefit from advances in surrogate modeling and machine learning in general. This paper presents a novel, fully automated approach to tackling this problem. Drawing upon advances in machine learning, hyperparameter optimization, model type selection, and sample selection (active learning) are all handled automatically. Increasing the utility of such methods for the domain expert.