Persi Diaconis

On the histogram as a density estimator:L2 theory

Let f be a probability density on an interval I, finite or infinite: I includes its finite endpoints, if any; and f vanishes outside of I. Let X1, . . . ,X k be independent random variables, with common density f The empirical histogram for the Xs is often used to estimate f To define this object, choose a reference point xosI and a cell width h. Let Nj be the number of Xs falling in the j th class interval:

Longest increasing subsequences: from patience sorting to the Baik-Deift-Johansson theorem

We describe a simple one-person card game, patience sorting. Its analysis leads to a broad circle of ideas linking Young tableaux with the longest increasing subsequence of a random permutation via the Schensted correspondence. A recent highlight of this area is the work of Baik-Deift-Johansson which yields limiting probability laws via hard analysis of Toeplitz determinants.

SHUFFLING CARDS AND STOPPING-TIMES

EXAMPLE1. Top in at random shuffle. Consider the following method of mixing a deck of cards: the top card is removed and inserted into the deck at a random position. This procedure is repeated a number of times. The following argument should convince the reader that about n log n shuffles suffice to mix up n cards. The argument depends on following the bottom card of the deck. This card stays at the bottom until the first time (TI) a card is inserted below it. Standard calculations, reviewed below, imply this takes about n shuffles. As the shuffles continue, eventually a second card is inserted below the original bottom card (this takes about n/2 further shuffles). Consider the instant (T,) that a second card is inserted below the original bottom card. The two cards under the original bottom card are equally likely to be in relative order low-high or high-low. Similarly, the first time a h r d card is inserted below the original bottom card, each of the 6 possible orders of the 3 bottom cards is equally likely. Now consider the first time T,-, that the original bottom card comes up to the top. By an inductive argument, all (n l ) ! arrangements of the lower cards are equally likely. When the original bottom card is inserted at random, at time T = q,-, + 1, then all n! possible arrangements of the deck are equally likely.

Updating Subjective Probability

Abstract Jeffreys rule for revising a probability P to a new probability P* based on new probabilities P* (Ei ) on a partition {Ei } i = 1 n is P*(A) = Σ P(A| Ei ) P* (Ei ). Jeffreys rule is applicable if it is judged that P* (A | Ei ) = P(A | Ei ) for all A and i. This article discusses some of the mathematical properties of this rule, connecting it with sufficient partitions, and maximum entropy updating of contingency tables. The main results concern simultaneous revision on two partitions.

Sequential Monte Carlo Methods for Statistical Analysis of Tables

We describe a sequential importance sampling (SIS) procedure for analyzing two-way zero–one or contingency tables with fixed marginal sums. An essential feature of the new method is that it samples the columns of the table progressively according to certain special distributions. Our method produces Monte Carlo samples that are remarkably close to the uniform distribution, enabling one to approximate closely the null distributions of various test statistics about these tables. Our method compares favorably with other existing Monte Carlo-based algorithms, and sometimes is a few orders of magnitude more efficient. In particular, compared with Markov chain Monte Carlo (MCMC)-based approaches, our importance sampling method not only is more efficient in terms of absolute running time and frees one from pondering over the mixing issue, but also provides an easy and accurate estimate of the total number of tables with fixed marginal sums, which is far more difficult for an MCMC method to achieve.

Strong uniform times and finite random walks

There are several techniques for obtaining bounds on the rate of convergence to the stationary distribution for Markov chains with strong symmetry properties, in particular random walks on finite groups. An elementary method, strong uniform times, is often effective. We prove such times always exist, and relate this method to coupling and Fourier analysis.

A Sequential Importance Sampling Algorithm for Generating Random Graphs with Prescribed Degrees

Abstract Random graphs with given degrees are a natural next step in complexity beyond the Erdős–Rényi model, yet the degree constraint greatly complicates simulation and estimation. We use an extension of a combinatorial characterization due to Erdős and Gallai to develop a sequential algorithm for generating a random labeled graph with a given degree sequence. The algorithm is easy to implement and allows for surprisingly efficient sequential importance sampling. The resulting probabilities are easily computed on the fly, allowing the user to reweight estimators appropriately, in contrast to some ad hoc approaches that generate graphs with the desired degrees but with completely unknown probabilities. Applications are given, including simulating an ecological network and estimating the number of graphs with a given degree sequence.

The Markov chain Monte Carlo revolution

The use of simulation for high dimensional intractable computations has revolutionized applied mathematics. Designing, improving and understanding the new tools leads to (and leans on) fascinating mathematics, from representation theory through micro-local analysis.

Hammersley's interacting particle process and longest increasing subsequences

SummaryIn a famous paper [8] Hammersley investigated the lengthLn of the longest increasing subsequence of a randomn-permutation. Implicit in that paper is a certain one-dimensional continuous-space interacting particle process. By studying a hydrodynamical limit for Hammersleys process we show by fairly “soft” arguments that limn′1/2ELn=2. This is a known result, but previous proofs [14, 11] relied on hard analysis of combinatorial asymptotics.

Methods for Studying Coincidences

This article illustrates basic statistical techniques for studying coincidences. These include data-gathering methods (informal anecdotes, case studies, observational studies, and experiments) and methods of analysis (exploratory and confirmatory data analysis, special analytic techniques, and probabilistic modeling, both general and special purpose). We develop a version of the birthday problem general enough to include dependence, inhomogeneity, and almost multiple matches. We review Fisher’s techniques for giving partial credit for close matches. We develop a model for studying coincidences involving newly learned words. Once we set aside coincidences having apparent causes, four principles account for large numbers of remaining coincidences: hidden cause; psychology, including memory and perception; multiplicity of endpoints, including the counting of “close” or nearly alike events as if they were identical; and the law of truly large numbers which says that when enormous numbers of events and people and their interactions cumulate over time, almost any outrageous event is bound to occur. These sources account for much of the force of synchronicity.