John A. Trangenstein

Initial Value Problems

This chapter is devoted to initial values problems for ordinary differential equations. It discusses theory for existence, uniqueness and continuous dependence on the data of the problem. Special techniques for linear ordinary differential equations with constant coefficients are discussed in terms of matrix exponentials and their approximations. Next, linear multistep methods are introduced and analyzed, leading to a presentation of important families of linear multistep methods and their stability. These methods are implemented through predictor-corrector methods, and techniques for automatically selecting stepsize and order are discussed. Afterwards, deferred correction and Runge-Kutta methods are examined. The chapter ends with the selection of numerical methods for stiff problems, and a discussion of nonlinear stability.

Boundary Value Problems

This chapter is devoted to boundary value problems for ordinary differential equations. It begins with analysis of the existence and uniqueness of solutions to these problems, and the effect of perturbations to the problem. The first numerical approach is the shooting method. This is followed by finite differences and collocation. Finite elements allow for the development of very high order methods for many boundary value problems, but their analysis typically requires sophisticated ideas from real analysis. The chapter ends with the application of deferred correction to both collocation and finite elements.

Introduction to Scientific Computing

This chapter introduces five basic steps in scientific computing applied to an initial value problem. The first step constructs a mathematical model, consisting of an ordinary differential equation and an initial value. The second step examines the mathematical well-posedness of the model problem, to see if the a solution exists, is unique, and depends continuously on the data. In the third step, we construct a simple numerical method for this problem, and in the fourth step we develop computer programs to execute this method. In the final step, we perform a mathematical analysis of the numerical method to determine its stability and convergence properties, even in the presence of computer roundoff errors. This analysis helps us to choose appropriate parameters for the numerical method, such as the time step size, and to compare the relative efficiency of competing methods.

Least Squares Problems

This chapter examines the linear optimization problem of finding the best solution to over-determined, underdetermined or rank-deficient systems of linear equations. The existence theory for these least squares problems is necessarily more complicated than the theory for nonsingular linear systems. Pseudo-inverses, perturbation analysis and a posteriori error estimates are developed. Successive reflection factorization is developed and analyzed for over-determined, under-determined and rank-deficient problems. The effect of rounding errors and the use iterative improvement are also discussed. Alternative factorizations by successive orthogonal projection and successive orthogonal rotation are presented next. The singular value decomposition is introduced, but its full development depends on a detailed knowledge of eigenvalues, which are discussed in a later chapter. The chapter ends with a discussion of quadratic programming problems.

Differentiation and Integration

This chapter develops numerical methods for computing derivatives and integrals. Numerical differentiation of polynomials can be performed by synthetic division, or through special properties of trigonometric polynomials or orthogonal polynomials. For derivatives of more general functions, finite differences lead to difficulties with rounding errors that can be largely overcome by clever post-processing, such as Richardson extrapolation. Integration is a more complicated topic. The Lebesgue integral is related to Monte Carlo methods, and Riemann sums are improved by trapezoidal and midpoint rules. Analysis of the errors leads to the Euler-MacLaurin formula. Various polynomial interpolation techniques lead to specialized numerical integration methods. The chapter ends with discussions of tricks for difficult integrals, adaptive quadrature, and integration in multiple dimensions.

Iterative Linear Algebra

This chapter describes iterative methods for approximately solving systems of linear equations. This discussion begins by presenting the concept of a sparse matrix, where it arises and how it might be represented in a computer. Next, simple methods based on iterative improvement are presented, along with termination criteria for the iteration. Afterwards, more elaborate gradient methods are examined, such as conjugate gradients and biconjugate gradients. The chapter proceeds tominimum resiual methods, and ends with a fairly detailed discussion of multigrid methods.

Eigenvalues and Eigenvectors

This chapter begins with the basic theory of eigenvalues and eigenvectors of matrices. Essential concepts such as characteristic polynomials, the Fundamental Theorem of Algebra, the Gerschgorin circle theorem, invariant subspaces, change of basis, spectral radius and the distance between subspaces are developed. Hermitian matrices are analyzed through the spectral theorem, and a perturbation analysis of their eigenvalue problem is performed. This chapter presents and examines algorithms for finding eigenvalues of Hermitian tridiagonal matrices, such as bisection, the power method, QL, QR, implicit QR, divide and conquer and dqds. Reduction of general Hermitian matrices to tridigonal form, and the Lanczos process are also discussed. Next, the eigenvalue problem for general matrices is examined. Theory for the Schur decomposition and the Jordan form are presented. Perturbation theory and conditions numbers lead to a posteriori estimates for general eigenvalue problems. Numerical methods for upper Hessenberg matricces are discussed, followed by general techniques for orthogonal similarity transformation to upper Hessenberg form. Then the chapter turns to the singular value decomposition, with theory discussing its existence, pseudo-inverses and the minimax theorem. Methods for reducing general matrices to bidiagonal form, and techniques for finding singular value decompositions of bidiagonal matrices follow next. The chapter ends with discussions of linear recurrences and functions of matrices.

Interpolation and Approximation

This chapter begins with a discussion of interpolation. Polynomial interpolation for a function of a single variable is analyzed, and implemented through Newton, Lagrange and Hermite forms. Intelligent selection of interpolation points is discussed, and extensions to multi-dimensional polynomial interpolation are presented. Rational polynomial interpolation is studied next, and connected to quadric surfaces. Then the discussion turns to piecewise polynomial interpolation and splines. The study of interpolation concludes with a presentation of parametric curves. Afterwards, the chapter moves on to least squares approximation, orthogonal polynomials and trigonometric polynomials. Trigonometric polynomial interpolation or approximation is implemented by the fast Fourier transform. The chapter concludes with wavelets, as well as their application to discrete data and continuous functions.