Longhai Li

Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC

Looking at predictive accuracy is a traditional method for comparing models. A natural method for approximating out-of-sample predictive accuracy is leave-one-out cross-validation (LOOCV)—we alternately hold out each case from a full dataset and then train a Bayesian model using Markov chain Monte Carlo without the held-out case; at last we evaluate the posterior predictive distribution of all cases with their actual observations. However, actual LOOCV is time-consuming. This paper introduces two methods, namely iIS and iWAIC, for approximating LOOCV with only Markov chain samples simulated from a posterior based on a full dataset. iIS and iWAIC aim at improving the approximations given by importance sampling (IS) and WAIC in Bayesian models with possibly correlated latent variables. In iIS and iWAIC, we first integrate the predictive density over the distribution of the latent variables associated with the held-out without reference to its observation, then apply IS and WAIC approximations to the integrated predictive density. We compare iIS and iWAIC with other approximation methods in three kinds of models: finite mixture models, models with correlated spatial effects, and a random effect logistic regression model. Our empirical results show that iIS and iWAIC give substantially better approximates than non-integrated IS and WAIC and other methods.

Compressing Parameters in Bayesian High-order Models with Application to Logistic Sequence Models ∗

Abstract. Bayesian classification and regression with high order interactions is largely infeasible because Markov chain Monte Carlo (MCMC) would need to be applied with a great many parameters, whose number increases rapidly with the order. In this paper we show how to make it feasible by effectively reducing the number of parameters, exploiting the fact that many interactions have the same values for all training cases. Our method uses a single “compressed” parameter to represent the sum of all parameters associated with a set of patterns that have the same value for all training cases. Using symmetric stable distributions as the priors of the original parameters, we can easily find the priors of these compressed parameters. We therefore need to deal only with a much smaller number of compressed parameters when training the model with MCMC. The number of compressed parameters may have converged before considering the highest possible order. After training the model, we can split these compressed parameters into the original ones as needed to make predictions for test cases. We show in detail how to compress parameters for logistic sequence prediction models. Experiments on both simulated and real data demonstrate that a huge number of parameters can indeed be reduced by our compression method.

Robust descriptive discriminant analysis for repeated measures data

Discriminant analysis (DA) procedures based on parsimonious mean and/or covariance structures have recently been proposed for repeated measures data. However, these procedures rest on the assumption of a multivariate normal distribution. This study examines repeated measures DA (RMDA) procedures based on maximum likelihood (ML) and coordinatewise trimming (CT) estimation methods and investigates bias and root mean square error (RMSE) in discriminant function coefficients (DFCs) using Monte Carlo techniques. Study parameters include population distribution, covariance structure, sample size, mean configuration, and number of repeated measurements. The results show that for ML estimation, bias in DFC estimates was usually largest when the data were normally distributed, but there was no consistent trend in RMSE. For non-normal distributions, the average bias of CT estimates for procedures that assume unstructured group means and structured covariances was at least 40% smaller than the values for corresponding procedures based on ML estimators. The average RMSE for the former procedures was at least 10% smaller than the average RMSE for the latter procedures, but only when the data were sampled from extremely skewed or heavy-tailed distributions. This finding was observed even when the covariance and mean structures of the RMDA procedure were mis-specified. The proposed robust procedures can be used to identify measurement occasions that make the largest contribution to group separation when the data are sampled from multivariate skewed or heavy-tailed distributions.

An online Bayesian mixture labelling method by minimizing deviance of classification probabilities to reference labels

Solving label switching is crucial for interpreting the results of fitting Bayesian mixture models. The label switching originates from the invariance of posterior distribution to permutation of component labels. As a result, the component labels in Markov chain simulation may switch to another equivalent permutation, and the marginal posterior distribution associated with all labels may be similar and useless for inferring quantities relating to each individual component. In this article, we propose a new simple labelling method by minimizing the deviance of the class probabilities to a fixed reference labels. The reference labels can be chosen before running Markov chain Monte Carlo (MCMC) using optimization methods, such as expectation-maximization algorithms, and therefore the new labelling method can be implemented by an online algorithm, which can reduce the storage requirements and save much computation time. Using the Acid data set and Galaxy data set, we demonstrate the success of the proposed labelling method for removing the labelling switching in the raw MCMC samples.

Bias-Corrected Hierarchical Bayesian Classification With a Selected Subset of High-Dimensional Features

Class prediction based on high-dimensional features has received a great deal of attention in many areas of application. For example, biologists are interested in using microarray gene expression profiles for diagnosis or prognosis of a certain disease (e.g., cancer). For computational and other reasons, it is necessary to select a subset of features before fitting a statistical model, by evaluating how strongly the features are related to the response. However, such a feature selection procedure will result in overconfident predictive probabilities for future cases, because the signal-to-noise ratio in the retained features is exacerbated by the feature selection. In this article we develop a hierarchical Bayesian classification method that can correct for this feature selection bias. Our method, which we term bias-corrected Bayesian classification with selected features (BCBCSF), uses the partial information from the feature selection procedure, in addition to the retained features, to form a correct (unbiased) posterior distribution of certain hyperparameters in the hierarchical Bayesian model that control the signal-to-noise ratio of the dataset. We take a Markov chain Monte Carlo (MCMC) approach to inferring the model parameters. We then use MCMC samples to make predictions for future cases. Because of the simplicity of the models, the inferred parameters from MCMC are easy to interpret, and the computation is very fast. Simulation studies and tests with two real microarray datasets related to complex human diseases show that our BCBCSF method provides better predictions than two widely used high-dimensional classification methods, prediction analysis for microarrays and diagonal linear discriminant analysis. The R package BCBCSF for the method described here is available from http://math.usask.ca/longhai/software/BCBCSF and CRAN.

Fully Bayesian Logistic Regression with Hyper-Lasso Priors for High-dimensional Feature Selection

ABSTRACT Feature selection arises in many areas of modern science. For example, in genomic research, we want to find the genes that can be used to separate tissues of different classes (e.g. cancer and normal). One approach is to fit regression/classification models with certain penalization. In the past decade, hyper-LASSO penalization (priors) have received increasing attention in the literature. However, fully Bayesian methods that use Markov chain Monte Carlo (MCMC) for regression/classification with hyper-LASSO priors are still in lack of development. In this paper, we introduce an MCMC method for learning multinomial logistic regression with hyper-LASSO priors. Our MCMC algorithm uses Hamiltonian Monte Carlo in a restricted Gibbs sampling framework. We have used simulation studies and real data to demonstrate the superior performance of hyper-LASSO priors compared to LASSO, and to investigate the issues of choosing heaviness and scale of hyper-LASSO priors.

Estimating cross-validatory predictive p-values with integrated importance sampling for disease mapping models

An important statistical task in disease mapping problems is to identify divergent regions with unusually high or low risk of disease. Leave-one-out cross-validatory (LOOCV) model assessment is the gold standard for estimating predictive p-values that can flag such divergent regions. However, actual LOOCV is time-consuming because one needs to rerun a Markov chain Monte Carlo analysis for each posterior distribution in which an observation is held out as a test case. This paper introduces a new method, called integrated importance sampling (iIS), for estimating LOOCV predictive p-values with only Markov chain samples drawn from the posterior based on a full data set. The key step in iIS is that we integrate away the latent variables associated the test observation with respect to their conditional distribution without reference to the actual observation. By following the general theory for importance sampling, the formula used by iIS can be proved to be equivalent to the LOOCV predictive p-value. We compare iIS and other three existing methods in the literature with two disease mapping datasets. Our empirical results show that the predictive p-values estimated with iIS are almost identical to the predictive p-values estimated with actual LOOCV and outperform those given by the existing three methods, namely, the posterior predictive checking, the ordinary importance sampling, and the ghosting method by Marshall and Spiegelhalter (2003). Copyright

Approximating Cross-validatory Predictive P-values with Integrated IS for Disease Mapping Models

An important statistical task in disease mapping problems is to identify divergent regions with unusually high or low risk of disease. Leave-one-out cross-validatory (LOOCV) model assessment is the gold standard for estimating predictive p-values that can flag such divergent regions. However, actual LOOCV is time-consuming because one needs to rerun a Markov chain Monte Carlo analysis for each posterior distribution in which an observation is held out as a test case. This paper introduces a new method, called integrated importance sampling (iIS), for estimating LOOCV predictive p-values with only Markov chain samples drawn from the posterior based on a full data set. The key step in iIS is that we integrate away the latent variables associated the test observation with respect to their conditional distribution without reference to the actual observation. By following the general theory for importance sampling, the formula used by iIS can be proved to be equivalent to the LOOCV predictive p-value. We compare iIS and other three existing methods in the literature with two disease mapping datasets. Our empirical results show that the predictive p-values estimated with iIS are almost identical to the predictive p-values estimated with actual LOOCV and outperform those given by the existing three methods, namely, the posterior predictive checking, the ordinary importance sampling, and the ghosting method by Marshall and Spiegelhalter (2003). Copyright

Are Bayesian Inferences Weak for Wasserman's Example?

An example was given in the textbook All of Statistics (Wasserman, 2004, pp. 186–188) for arguing that, in the problems with a great many parameters Bayesian inferences are weak, because they rely heavily on the likelihood function that captures information of only a tiny fraction of the total parameters. Alternatively, he suggested non Bayesian Horwitz–Thompson estimator, which cannot be obtained from a likelihood-based approaches, including Bayesian approaches. He argued that Horwitz–Thompson estimator is good since it is unbiased and consistent. In this article, the mean square errors of Horwitz–Thompson estimator is compared with a Bayes estimator at a wide range of parameter configurations. These two estimators are also simulated to visualize them directly. From these comparisons, the conclusion is that the simple Bayes estimator works better than Horwitz–Thompson estimator for most parameter configurations. Hence, Bayesian inferences are not weak for this example.

Estimating cross-validatory predictivep-values with integrated importance sampling for disease mapping models: L. LI, C. X. FENG AND S. QIU

An important statistical task in disease mapping problems is to identify divergent regions with unusually high or low risk of disease. Leave-one-out cross-validatory (LOOCV) model assessment is the gold standard for estimating predictive p-values that can flag such divergent regions. However, actual LOOCV is time-consuming because one needs to rerun a Markov chain Monte Carlo analysis for each posterior distribution in which an observation is held out as a test case. This paper introduces a new method, called integrated importance sampling (iIS), for estimating LOOCV predictive p-values with only Markov chain samples drawn from the posterior based on a full data set. The key step in iIS is that we integrate away the latent variables associated the test observation with respect to their conditional distribution without reference to the actual observation. By following the general theory for importance sampling, the formula used by iIS can be proved to be equivalent to the LOOCV predictive p-value. We compare iIS and other three existing methods in the literature with two disease mapping datasets. Our empirical results show that the predictive p-values estimated with iIS are almost identical to the predictive p-values estimated with actual LOOCV and outperform those given by the existing three methods, namely, the posterior predictive checking, the ordinary importance sampling, and the ghosting method by Marshall and Spiegelhalter (2003). Copyright