Hock Peng Chan

Scan Statistics With Weighted Observations

We examine scan statistics for one-dimensional marked Poisson processes. Such statistics tabulate the maximum weighted count of event occurrences within a window of predetermined width over all windows within an observed interval. We derive analytical formulas and also give an importance sampling method for approximating the tail probabilities of scan statistics. Because high-throughput genomic sequencing has led to the availability of massive amounts of biomolecular sequence data, it is often of interest to search long DNA or protein sequences for local regions that are enriched for a certain characteristic. Thus scan statistics have become a useful tool in modern computational biology. We illustrate the application of our p value approximations with such examples.

Multiscale Adaptive Inference on Conditional Moment Inequalities

This paper considers inference for conditional moment inequality models using a multiscale statistic. We derive the asymptotic distribution of this test statistic and use the result to propose feasible critical values that have a simple analytic formula. We also propose critical values based on a modified bootstrap procedure and prove their asymptotic validity. The asymptotic distribution is extreme value, and the proof uses new techniques to overcome several technical obstacles. We provide power results that show that our test detects local alternatives that approach the identified set at the best possible rate under a set of conditions that hold generically in the set identified case in a broad class of models, and that our test is adaptive to the smoothness properties of the data generating process. Our results also have implications for the use of moment selection procedures in this setting. We provide a monte carlo study and an empirical illustration to inference in a regression model with endogenously censored and missing data.

A general theory of particle filters in hidden Markov models and some applications

By making use of martingale representations, we derive the asymptotic normality of particle filters in hidden Markov models and a relatively simple formula for their asymptotic variances. Although repeated resamplings result in complicated dependence among the sample paths, the asymptotic variance formula and martingale representations lead to consistent estimates of the standard errors of the particle filter estimates of the hidden states.

Detection of spatial clustering with average likelihood ratio test statistics

Generalized likelihood ratio (GLR) test statistics are often used in the detection of spatial clustering in case-control and case-population datasets to check for a significantly large proportion of cases within some scanning window. The traditional spatial scan test statistic takes the supremum GLR value over all windows, whereas the average likelihood ratio (ALR) test statistic that we consider here takes an average of the GLR values. Numerical experiments in the literature and in this paper show that the ALR test statistic has more power compared to the spatial scan statistic. We develop in this paper accurate tail probability approximations of the ALR test statistic that allow us to by-pass computer intensive Monte Carlo procedures to estimate p-values. In models that adjust for covariates, these Monte Carlo evaluations require an initial fitting of parameters that can result in very biased p-value estimates.

A SEQUENTIAL MONTE CARLO APPROACH TO COMPUTING TAIL PROBABILITIES IN STOCHASTIC MODELS

Sequential Monte Carlo methods which involve sequential importance sampling and resampling are shown to provide a versatile approach to computing probabilities of rare events. By making use of martingale representations of the sequential Monte Carlo estimators, we show how resampling weights can be chosen to yield logarithmically ecient Monte Carlo estimates of large deviation probabilities for multidimensional Markov random walks.

Rare-event simulation of heavy-tailed random walks by sequential importance sampling and resampling

We introduce a new approach to simulating rare events for Markov random walks with heavy-tailed increments. This approach involves sequential importance sampling and resampling, and uses a martingale representation of the corresponding estimate of the rare-event probability to show that it is unbiased and to bound its variance. By choosing the importance measures and resampling weights suitably, it is shown how this approach can yield asymptotically efficient Monte Carlo estimates.

Maxima of moving sums in a Poisson random field

In this paper we examine the extremal tail probabilities of moving sums in a marked Poisson random field. These sums are computed by adding up the weighted occurrences of events lying within a scanning set of fixed shape and size. We also provide an alternative representation of the constants of the asymptotic formulae in terms of the occupation measure of the conditional local random field at zero, and extend these representations to the constants of asymptotic tail probabilities of Gaussian random fields.

Importance Sampling of Word Patterns in DNA and Protein Sequences

Monte Carlo methods can provide accurate p-value estimates of word counting test statistics and are easy to implement. They are especially attractive when an asymptotic theory is absent or when either the search sequence or the word pattern is too short for the application of asymptotic formulae. Naive direct Monte Carlo is undesirable for the estimation of small probabilities because the associated rare events of interest are seldom generated. We propose instead efficient importance sampling algorithms that use controlled insertion of the desired word patterns on randomly generated sequences. The implementation is illustrated on word patterns of biological interest: palindromes and inverted repeats, patterns arising from position-specific weight matrices (PSWMs), and co-occurrences of pairs of motifs.

Optimal detection of multi-sample aligned sparse signals

We describe, in the detection of multi-sample aligned sparse signals, the critical boundary separating detectable from nondetectable signals, and construct tests that achieve optimal detectability: penalized versions of the Berk-Jones and the higher-criticism test statistics evaluated over pooled scans, and an average likelihood ratio over the critical boundary. We show in our results an inter-play between the scale of the sequence length to signal length ratio, and the sparseness of the signals. In particular the difficulty of the detection problem is not noticeably affected unless this ratio grows exponentially with the number of sequences. We also recover the multiscale and sparse mixture testing problems as illustrative special cases.

Discussion on “Is Average Run Length to False Alarm Always an Informative Criterion?” by Yajun Mei

Abstract We comment on Meis critique of the standard minimax formulation with the ARL to false alarm of a sequential detection rule as its operating characteristic, and provide an alternative Bayesian approach for his use of a mixing distribution to handle pre-change parameters. We also consider computational issues in implementing the ARL and better alternatives for evaluating sequential detection rules in complex stochastic systems.