Peter Sollich

Bayesian Methods for Support Vector Machines: Evidence and Predictive Class Probabilities

I describe a framework for interpreting Support Vector Machines (SVMs) as maximum a posteriori (MAP) solutions to inference problems with Gaussian Process priors. This probabilistic interpretation can provide intuitive guidelines for choosing a ‘good’ SVM kernel. Beyond this, it allows Bayesian methods to be used for tackling two of the outstanding challenges in SVM classification: how to tune hyperparameters—the misclassification penalty C, and any parameters specifying the ernel—and how to obtain predictive class probabilities rather than the conventional deterministic class label predictions. Hyperparameters can be set by maximizing the evidence; I explain how the latter can be defined and properly normalized. Both analytical approximations and numerical methods (Monte Carlo chaining) for estimating the evidence are discussed. I also compare different methods of estimating class probabilities, ranging from simple evaluation at the MAP or at the posterior average to full averaging over the posterior. A simple toy application illustrates the various concepts and techniques.

Model Selection for Support Vector Machine Classification

We address the problem of model selection for Support Vector Machine (SVM) classification. For fixed functional form of the kernel, model selection amounts to tuning kernel parameters and the slack penalty coefficient C. We begin by reviewing a recently developed probabilistic framework for SVM classification. An extension to the case of SVMs with quadratic slack penalties is given and a simple approximation for the evidence is derived, which can be used as a criterion for model selection. We also derive the exact gradients of the evidence in terms of posterior averages and describe how they can be estimated numerically using Hybrid Monte-Carlo techniques. Though computationally demanding, the resulting gradient ascent algorithm is a useful baseline tool for probabilistic SVM model selection, since it can locate maxima of the exact (unapproximated) evidence. We then perform extensive experiments on several benchmark data sets. The aim of these experiments is to compare the performance of probabilistic model selection criteria with alternatives based on estimates of the test error, namely the so-called “span estimate” and Wahbas Generalized Approximate Cross-Validation (GACV) error. We find that all the “simple” model criteria (Laplace evidence approximations, and the span and GACV error estimates) exhibit multiple local optima with respect to the hyperparameters. While some of these give performance that is competitive with results from other approaches in the literature, a significant fraction lead to rather higher test errors. The results for the evidence gradient ascent method show that also the exact evidence exhibits local optima, but these give test errors which are much less variable and also consistently lower than for the simpler model selection criteria.

Predicting phase equilibria in polydisperse systems

Many materials containing colloids or polymers are polydisperse: they comprise particles with properties (such as particle diameter, charge, or polymer chain length) that depend continuously on one or several parameters. This review focuses on the theoretical prediction of phase equilibria in polydisperse systems; the presence of an effectively infinite number of distinguishable particle species makes this a highly non-trivial task. I first describe qualitatively some of the novel features of polydisperse phase behaviour, and outline a theoretical framework within which they can be explored. Current techniques for predicting polydisperse phase equilibria are then reviewed. I also discuss applications to some simple model systems including homopolymers and random copolymers, spherical colloids and colloid-polymer mixtures, and liquid crystals formed from rod- and plate-like colloidal particles; the results surveyed give an idea of the rich phenomenology of polydisperse phase behaviour. Extensions to the study of polydispersity effects on interfacial behaviour and phase separation kinetics are outlined briefly.

Equilibrium Phase Behavior of Polydisperse Hard Spheres

We calculate the phase behavior of hard spheres with size polydispersity, using accurate free energies for the fluid and solid phases. Cloud and shadow curves are found exactly by the moment free energy method, but we also compute the complete phase diagram, taking full account of fractionation. In contrast to earlier, simplified treatments we find no point of equal concentration between fluid and solid or reentrant melting at higher densities. Rather, the fluid cloud curve continues to the largest polydispersity that we study (14%); from the equilibrium phase behavior a terminal polydispersity can thus be defined only for the solid, where we find it to be around 7%. At sufficiently large polydispersity, fractionation into several solid phases can occur, consistent with previous approximate calculations; we find, in addition, that coexistence of several solids with a fluid phase is also possible.

Observable Dependence of Fluctuation-Dissipation Relations and Effective Temperatures

We study the nonequilibrium fluctuation-dissipation theorem (FDT) in the glass phase of Bouchauds trap model. We incorporate an arbitrary observable m and obtain its correlation and response functions in closed form. A limiting nonequilibrium FDT plot is approached at long times for most choices of m. In contrast to standard mean field models, however, the shape of the plot depends nontrivially on the observable, and its slope varies continuously even though there is a single scaling of relaxation times with age. Nonequilibrium FDT plots can therefore not be used to define a meaningful effective temperature T(eff) in this model. Consequences for the wider applicability of an FDT-derived T(eff) are discussed.

Simplified Onsager theory for isotropic–nematic phase equilibria of length polydisperse hard rods

Polydispersity is believed to have important effects on the formation of liquid crystal phases in suspensions of rodlike particles. To understand such effects, we analyze the phase behavior of thin hard rods with length polydispersity. Our treatment is based on a simplified Onsager theory, obtained by truncating the series expansion of the angular dependence of the excluded volume. We describe the model and give the full phase equilibrium equations; these are then solved numerically using the moment free energy method which reduces the problem from one with an infinite number of conserved densities to one with a finite number of effective densities that are moments of the full density distribution. The method yields exactly the onset of nematic ordering. Beyond this, results are approximate but we show that they can be made essentially arbitrarily precise by adding adaptively chosen extra moments, while still avoiding the numerical complications of a direct solution of the full phase equilibrium conditions. We investigate in detail the phase behavior of systems with three different length distributions: a (unimodal) Schulz distribution, a bidisperse distribution, and a bimodal mixture of two Schulz distributions which interpolates between these two cases. A three-phase isotropic-nematic-nematic coexistence region is shown to exist for the bimodal and bidisperse length distributions if the ratio of long and short rod lengths is sufficiently large, but not for the unimodal one. We systematically explore the topology of the phase diagram as a function of the width of the length distribution and of the rod length ratio in the bidisperse and bimodal cases.

Large deviations and ensembles of trajectories in stochastic models

We consider ensembles of trajectories associated with large deviations of time-integrated quantities in stochastic models. Motivated by proposals that these ensembles are relevant for physical processes such as shearing and glassy relaxation, we show how they can be generated directly using auxiliary stochastic processes. We illustrate our results using the Glauber-Ising chain, for which biased ensembles of trajectories can exhibit ferromagnetic ordering. We discuss the relation between such biased ensembles and quantum phase transitions.

2005 Special Issue: Bayesian approach to feature selection and parameter tuning for support vector machine classifiers

A Bayesian point of view of SVM classifiers allows the definition of a quantity analogous to the evidence in probabilistic models. By maximizing this one can systematically tune hyperparameters and, via automatic relevance determination (ARD), select relevant input features. Evidence gradients are expressed as averages over the associated posterior and can be approximated using Hybrid Monte Carlo (HMC) sampling. We describe how a Nyström approximation of the Gram matrix can be used to speed up sampling times significantly while maintaining almost unchanged classification accuracy. In experiments on classification problems with a significant number of irrelevant features this approach to ARD can give a significant improvement in classification performance over more traditional, non-ARD, SVM systems. The final tuned hyperparameter values provide a useful criterion for pruning irrelevant features, and we define a measure of relevance with which to determine systematically how many features should be removed. This use of ARD for hard feature selection can improve classification accuracy in non-ARD SVMs. In the majority of cases, however, we find that in data sets constructed by human domain experts the performance of non-ARD SVMs is largely insensitive to the presence of some less relevant features. Eliminating such features via ARD then does not improve classification accuracy, but leads to impressive reductions in the number of features required, by up to 75%.

Fractionation effects in phase equilibria of polydisperse hard-sphere colloids

The equilibrium phase behavior of hard spheres with size polydispersity is studied theoretically. We solve numerically the exact phase equilibrium equations that result from accurate free energy expressions for the fluid and solid phases, while accounting fully for size fractionation between coexisting phases. Fluids up to the largest polydispersities that we can study (around 14%) can phase separate by splitting off a solid with a much narrower size distribution. This shows that experimentally observed terminal polydispersities above which phase separation no longer occurs must be due to nonequilibrium effects. We find no evidence of reentrant melting; instead, sufficiently compressed solids phase separate into two or more solid phases. Under appropriate conditions, coexistence of multiple solids with a fluid phase is also predicted. The solids have smaller polydispersities than the parent phase as expected, while the reverse is true for the fluid phase, which contains predominantly smaller particles but also residual amounts of the larger ones. The properties of the coexisting phases are studied in detail; mean diameter, polydispersity, and volume fraction of the phases all reveal marked fractionation. We also propose a method for constructing quantities that optimally distinguish between the coexisting phases, using principal component analysis in the space of density distributions. We conclude by comparing our predictions to Monte Carlo simulations at imposed chemical potential distribution, and find excellent agreement.

Crystalline phases of polydisperse spheres

We use specialized Monte Carlo simulation methods and moment free energy calculations to provide conclusive evidence that dense polydisperse spheres at equilibrium demix into coexisting fcc phases, with more phases appearing as the spread of diameters increases. We manage to track up to four coexisting phases. Each of these is fractionated: it contains a narrower distribution of particle sizes than is present in the system overall. We also demonstrate that, surprisingly, demixing transitions can be nearly continuous, accompanied by fluctuations in local particle size correlated over many lattice spacings.