Mihai Alexe

Second-order adjoints for solving PDE-constrained optimization problems

Inverse problems are of the utmost importance in many fields of science and engineering. In the variational approach, inverse problems are formulated as partial differential equation-constrained optimization problems, where the optimal estimate of the uncertain parameters is the minimizer of a certain cost functional subject to the constraints posed by the model equations. The numerical solution of such optimization problems requires the computation of derivatives of the model output with respect to model parameters. The first-order derivatives of a cost functional (defined on the model output) with respect to a large number of model parameters can be calculated efficiently through first-order adjoint (FOA) sensitivity analysis. Second-order adjoint (SOA) models give second derivative information in the form of matrix–vector products between the Hessian of the cost functional and user-defined vectors. Traditionally, the construction of second-order derivatives for large-scale models has been considered too costly. Consequently, data assimilation applications employ optimization algorithms that use only first-order derivative information, such as nonlinear conjugate gradients and quasi-Newton methods. In this paper, we discuss the mathematical foundations of SOA sensitivity analysis and show that it provides an efficient approach to obtain Hessian-vector products. We study the benefits of using second-order information in the numerical optimization process for data assimilation applications. The numerical studies are performed in a twin experiment setting with a two-dimensional shallow water model. Different scenarios are considered with different discretization approaches, observation sets, and noise levels. Optimization algorithms that employ second-order derivatives are tested against widely used methods that require only first-order derivatives. Conclusions are drawn regarding the potential benefits and the limitations of using high-order information in large-scale data assimilation problems.

Forward and adjoint sensitivity analysis with continuous explicit Runge–Kutta schemes

Abstract We study the numerical solution of tangent linear, first and second order adjoint models with high-order explicit, continuous Runge–Kutta pairs. The approaches currently implemented in popular packages such as SUNDIALS or DASPKADJOINT are based on linear multistep methods. For adaptive time integration of nonlinear models, interpolation of the forward model solution is required during the adjoint model simulation. We propose to use the dense output mechanism built in the continuous Runge–Kutta schemes as a highly accurate and cost-efficient interpolation method in the inverse problem run. We implement our approach in a Fortran library called DENSERKS , which is found to compare well to other similar software on a number of test problems.

On the discrete adjoints of adaptive time stepping algorithms

We investigate the behavior of adaptive time stepping numerical algorithms under the reverse mode of automatic differentiation (AD). By differentiating the time step controller and the error estimator of the original algorithm, reverse mode AD generates spurious adjoint derivatives of the time steps. The resulting discrete adjoint models become inconsistent with the adjoint ODE, and yield incorrect derivatives. To regain consistency, one has to cancel out the contributions of the non-physical derivatives in the discrete adjoint model. We demonstrate that the discrete adjoint models of one-step, explicit adaptive algorithms, such as the Runge-Kutta schemes, can be made consistent with their continuous counterparts using simple code modifications. Furthermore, we extend the analysis to cover second order adjoint models derived through an extra forward mode differentiation of the discrete adjoint code. Several numerical examples support the mathematical derivations.

Space-time adaptive solution of inverse problems with the discrete adjoint method

This paper develops a framework for the construction and analysis of discrete adjoint sensitivities in the context of time dependent, adaptive grid, adaptive step models. Discrete adjoints are attractive in practice since they can be generated with low effort using automatic differentiation. However, this approach brings several important challenges. The space–time adjoint of the forward numerical scheme may be inconsistent with the continuous adjoint equations. A reduction in accuracy of the discrete adjoint sensitivities may appear due to the inter-grid transfer operators. Moreover, the optimization algorithm may need to accommodate state and gradient vectors whose dimensions change between iterations. This work shows that several of these potential issues can be avoided through a multi-level optimization strategy using discontinuous Galerkin (DG) hp-adaptive discretizations paired with Runge–Kutta (RK) time integration. We extend the concept of dual (adjoint) consistency to space–time RK-DG discretizations, which are then shown to be well suited for the adaptive solution of time-dependent inverse problems. Furthermore, we prove that DG mesh transfer operators on general meshes are also dual consistent. This allows the simultaneous derivation of the discrete adjoint for both the numerical solver and the mesh transfer logic with an automatic code generation mechanism such as algorithmic differentiation (AD), potentially speeding up development of large-scale simulation codes. The theoretical analysis is supported by numerical results reported for a two-dimensional non-stationary inverse problem.

Obtaining and using second order derivative information in the solution of large scale inverse problems

Inverse problems are of utmost importance in many fields of science and engineering. In the variational approach inverse problems are formulated as constrained optimization problems, where the optimal estimate of the uncertain parameters is the minimizer of a certain cost function subject to the model constraints. The numerical solution of such optimization problems requires the derivatives of a chosen cost function I dependent on the model parameters. Given that the parameter space is large in real-life problems, the derivatives of I can be calculated efficiently through first order adjoint sensitivity analysis. Second order adjoint models give second derivative information in the form of products between the Hessian of the cost functional and a user defined vector. In this paper we review the mathematical foundations of the second order adjoint sensitivity method. We then evaluate their performance in several data assimilation, sensitivity analysis, and uncertainty quantification scenarios, for a two dimensional shallow water flow simulation. In the data assimilation problem, we compare the performance of several well-known optimization methods that make use of first and second order information.

An investigation of discrete adjoints for flux-limited numerical schemes

In this paper we investigate the construction of discrete adjoints of flux-limited numerical schemes for the advection equation. Discrete adjoints are attractive since they can be generated easily via automatic differentiation. However, a careful analysis is needed in order to understand their properties. We discuss several issues posed by the differentiation of the limiter functions and propose an alternative implementation of the forward model that leads to stable discrete adjoint solutions.

On the adaptive solution of space-time inverse problems with the adjoint method

Abstract Adaptivity in space and time is ubiquitous in modern numerical simulations. The large number of unknowns associated with todays typical inverse problem may run in the millions, or more. To capture small scale phenomena in regions of interest, adaptive mesh and temporal step refinements are required, since uniform refinements quickly make the problem computationally intractable. To date, there is still a considerable gap between the state–of–the–art techniques used in direct (forward) simulations, and those employed in the solution of inverse problems, which have traditionally relied on fixed meshes and time steps. This paper describes a framework for building a space-time consistent adjoint discretization for a general discrete forward problem, in the context of adaptive mesh, adaptive time step models. The discretize–then–di_erentiate approach to optimization is a very attractive approach in practice, because the adjoint model code may be generated using automatic di_erentiation (AD). However, several challenges are introduced when using an adaptive forward solver. First, one may have consistency problems with the adjoint of the forward numerical scheme. Similarly, intergrid transfer operators may reduce the accuracy of the discrete adjoint sensitivities. The optimization algorithm may need to be specifically tailored to handle variations in the state and gradient vector sizes. This work shows that several of these potential issues can be avoided when using the Runge–Kutta discontinuous Galerkin (DG) method, an excellent candidate method for h=p-adaptive parallel simulations. Selective application of automatic di_erentiation on individual numerical algorithms may simplify considerably the adjoint code development. A numerical data assimilation example illustrates the e_ectiveness of the primal/dual RK–DG methods when used in inverse simulations.