John Skilling

Algorithms and Applications

Maximum entropy, using the Shannon/Jaynes form -Σ p log p, is an enormously powerful tool for reconstructing positive, additive images from a wide variety of types of data. The alternative form Σ log f, due to Burg, is shown to be inappropriate for image reconstruction in general, including radio astronomy, and also for the reconstruction of the profiles of power spectra.

Classic Maximum Entropy

This paper presents a fully Bayesian derivation of maximum entropy image reconstruction. The argument repeatedly goes from the particular to the general, in that if there are general theories then they must apply to special cases. Two such special cases, formalised as the “Cox axioms “, lead to the well-known fact that Bayesian probability theory is the only consistent language of inference. Further cases, formalised as the axioms of maximum entropy, show that the prior probability distribution for any positive, additive distribution must be monotonic in the entropy. Finally, a quantified special case shows that this monotonic function must be the exponential, leaving only a single dimensional scaling factor to be determined a posteriori. Many types of distribution, including probability distributions themselves, are positive and additive, so the entropy exponential is very general.

Exponential sampling, an alternative method for sampling in two-dimensional NMR experiments

Abstract A new method for sampling in two-dimensional nuclear magnetic resonance experiments is proposed and tested using one-dimensional spectra as models. The free induction decays are sampled exponentially, using many points where the signal-to-noise ratio (S/N) is high and a few where it is low. Using the maximum entropy method to reconstruct spectra, much higher resolution can be obtained than by using conventional sampling (for a given number of data points). The method is shown to work for FIDs having even very poor S/N. It should prove valuable in the future for 2D NMR experiments where at present valuable high-resolution information is lost as a result of the necessity for truncation of data sets in t1 in order to optimize sensitivity.

The Axioms of Maximum Entropy

Maximum entropy is presented as a universal method of finding a “best” positive distribution constrained by incomplete data. The generalised entropy ∑(f - m - f log(f/m))) is the only form which selects acceptable distributions f in particular cases. It holds even if f is not normalised, so that maximum entropy applies directly to physical distributions other than probabilities. Furthermore, maximum entropy should also be used to select “best” parameters if the underlying model m has such freedom.

Reconstruction of phase-sensitive two-dimensional NMR spectra by maximum entropy

Abstract The problems of reconstructing phase-sensitive 2D NMR spectra using conventional methods and the advantages of using our previously proposed implementation of the maximum entropy method (MEM) are analyzed. It is shown that when a phase-sensitive 2D spectrum is reconstructed using MEM, a higher resolution can be obtained for a given measuring time. The method should prove to be most useful, however, when applied to the reconstruction of spectra that cannot after conventional data processing be phased such that all peaks have pure absorption phase. MEM is shown to be capable of producing such spectra. For example, when applied to the reconstruction of a phase-sensitive COSY spectrum the cross peaks and diagonal peaks can be separated into subspectra where all peaks have pure absorption phase. In effect this also removes the diagonal from the COSY spectrum. Finally the method is shown to be capable of reducing t1 noise.

QUANTIFIED MAXIMUM ENTROPY

This tutorial paper discusses the theoretical basis of quantified maximum entropy, as a technique for obtaining probabilistic estimates of images and other positive additive distributions from noisy and incomplete data. The analysis is fully Bayesian, with estimates always being obtained as probability distributions from which appropriate error bars can be found. This supersedes earlier techniques, even those using maximum entropy, which aimed to produce a single optimal distribution.

Maximum entropy reconstruction of spectra containing antiphase peaks

Two-dimensional nuclear magnetic resonance methods have in recent years been used to great effect to study the three-dimensional structure and conformation of small proteins, DNA fragments, and other biological macromolecules. To maximize resolution and sensitivity (and consequently information content), spectra have been recorded in the phase-sensitive mode giving particularly impressive results (I, 2). Nevertheless, currently available NMR methods are limited in their applicability to larger macromolecules by problems caused by (1) increased linewidths and spectral crowding and (2) decreased sensitivity. In an attempt to alleviate these problems in two-dimensional spectra we are investigating the use of the maximum entropy method (MEM) for reconstruction of NMR spectra from free induction decay data. In previous papers we have demonstrated the potential of MEM reconstructions for simultaneous noise suppression and resolution enhancement in both ‘H and i3C one-dimensional NMR spectra (3, 4). This feature coupled with the ability of MEM to suppress truncation and other artifacts suggested that the method could be particularly useful for processing 2D NMR spectra. However, an apparent difficulty arises because signals in many phase-sensitive two-dimensional spectra are antiphase. Maximum entropy necessarily reconstructs positive oscillator number densities. Fortunately, the solution is straightforward. We merely allow two types of oscillators, one giving positive signals and the other negative, and seek the maximum entropy distribution over both frequency and type. Here, we report our first reconstructions of spectra containing antiphase peaks. For simplicity, we used one-dimensional data containing both positive and negative spectral lines. The entropy is simply

Prior Distributions on Measure Space

SUMMARY A measure is the formal representation of the non-negative additive functions that abound in science. We review and develop the art of assigning Bayesian priors to measures. Where necessary, spatial correlation is delegated to correlating kernels imposed on otherwise uncorrelated priors. The latter must be infinitely divisible (ID) and hence described by the Levy-Khinchin representation. Thus the fundamental object is the Levy measure, the choice of which corresponds to different ID process priors. The general case of a Levy measure comprising a mixture of assigned base measures leads to a prior process comprising a convolution of corresponding processes. Examples involving a single base measure are the gamma process, the Dirichlet process (for the normalized case) and the Poisson process. We also discuss processes that we call the supergamma and superDirichlet processes, which are double base measure generalizations of the gamma and Dirichlet processes. Examples of multiple and continuum base measures are also discussed. We conclude with numerical examples of density estimation.

The Eigenvalues of Mega-dimensional Matrices

Often, we need to know some integral property of the eigenvalues {x} of a large N × N symmetric matrix A. For example, determinants det (A) = exp(∑ log (x)) play a role in the classic maximum entropy algorithm [Gull, 1988] . Likewise in physics, the specific heat of a system is a temperature- -dependent sum over the eigenvalues of the Hamiltonian matrix. However, the matrix may be so large that direct O (N 3 calculation of all N eigenvalues is prohibited. Indeed, if A is coded as a “fast” procedure, then O (N 2 operations may also be prohibited.

On Parameter Estimation and Quantified Maxent

We give a Bayesian comparison between parameter estimation and free-form reconstruction by quantified MaxEnt. The evidence favours the latter prior for the example analysed, and we suggest that this may hold more generally.