Vivekananda Roy

On Monte Carlo methods for Bayesian multivariate regression models with heavy-tailed errors

We consider Bayesian analysis of data from multivariate linear regression models whose errors have a distribution that is a scale mixture of normals. Such models are used to analyze data on financial returns, which are notoriously heavy-tailed. Let @p denote the intractable posterior density that results when this regression model is combined with the standard non-informative prior on the unknown regression coefficients and scale matrix of the errors. Roughly speaking, the posterior is proper if and only if n>=d+k, where n is the sample size, d is the dimension of the response, and k is number of covariates. We provide a method of making exact draws from @p in the special case where n=d+k, and we study Markov chain Monte Carlo (MCMC) algorithms that can be used to explore @p when n>d+k. In particular, we show how the Haar PX-DA technology studied in Hobert and Marchev (2008) [11] can be used to improve upon Lius (1996) [7] data augmentation (DA) algorithm. Indeed, the new algorithm that we introduce is theoretically superior to the DA algorithm, yet equivalent to DA in terms of computational complexity. Moreover, we analyze the convergence rates of these MCMC algorithms in the important special case where the regression errors have a Students t distribution. We prove that, under conditions on n, d, k, and the degrees of freedom of the t distribution, both algorithms converge at a geometric rate. These convergence rate results are important from a practical standpoint because geometric ergodicity guarantees the existence of central limit theorems which are essential for the calculation of valid asymptotic standard errors for MCMC based estimates.

Improving the Convergence Properties of the Data Augmentation Algorithm with an Application to Bayesian Mixture Modelling

Every reversible Markov chain defines an operator whose spectrum encodes the convergenceproperties of the chain. When the state space is finite, the spectrum is just the set ofeigenvalues of the corresponding Markov transition matrix. However, when the state space isinfinite, the spectrum may be uncountable, and is nearly always impossible to calculate. In mostapplications of the data augmentation (DA) algorithm, the state space of the DA Markov chainis infinite. However, we show that, under regularity conditions that include the finiteness of theaugmented space, the operators defined by the DA chain and Hobert and Marchev’s (2008) alternativechain are both compact, and the corresponding spectra are both finite subsets of [0; 1).Moreover, we prove that the spectrum of Hobert and Marchev’s (2008) chain dominates thespectrum of the DA chain in the sense that the ordered elements of the former are all less thanor equal to the corresponding elements of the latter. As a concrete example, we study a widelyused DA algorithm for the exploration of posterior densities associated with Bayesian mixturemodels (Diebolt and Robert, 1994). In particular, we compare this mixture DA algorithm withan alternative algorithm proposed by Fr¨uhwirth-Schnatter (2001) that is based on random labelswitching.

Convergence rates for MCMC algorithms for a robust Bayesian binary regression model

Abstract: Most common regression models for analyzing binary random variables are logistic and probit regression models. However it is well known that the estimates of regression coefficients for these models are not robust to outliers [26]. The robit regression model [1, 16] is a robust alternative to the probit and logistic models. The robit model is obtained by replacing the normal (logistic) distribution underlying the probit (logistic) regression model with the Student’s t−distribution. We consider a Bayesian analysis of binary data with the robit link function. We construct a data augmentation (DA) algorithm that can be used to explore the corresponding posterior distribution. Following [10] we further improve the DA algorithm by adding a simple extra step to each iteration. Though the two algorithms are basically equivalent in terms of computational complexity, the second algorithm is theoretically more efficient than the DA algorithm. Moreover, we analyze the convergence rates of these Markov chain Monte Carlo (MCMC) algorithms. We prove that, under certain conditions, both algorithms converge at a geometric rate. The geometric convergence rate has important theoretical and practical ramifications. Indeed, the geometric ergodicity guarantees that the ergodic averages used to approximate posterior expectations satisfy central limit theorems, which in turn allows for the construction of asymptotically valid standard errors. These standard errors can be used to choose an appropriate (Markov chain) Monte Carlo sample size and allow one to use the MCMC algorithms developed in this paper with the same level of confidence that one would have using classical (iid) Monte Carlo. The results are illustrated using a simple numerical example.

Efficient estimation and prediction for the Bayesian binary spatial model with flexible link functions

Spatial generalized linear mixed models (SGLMMs) are popular models for spatial data with a non-Gaussian response. Binomial SGLMMs with logit or probit link functions are often used to model spatially dependent binomial random variables. It is known that for independent binomial data, the robit regression model provides a more robust (against extreme observations) alternative to the more popular logistic and probit models. In this article, we introduce a Bayesian spatial robit model for spatially dependent binomial data. Since constructing a meaningful prior on the link function parameter as well as the spatial correlation parameters in SGLMMs is difficult, we propose an empirical Bayes (EB) approach for the estimation of these parameters as well as for the prediction of the random effects. The EB methodology is implemented by efficient importance sampling methods based on Markov chain Monte Carlo (MCMC) algorithms. Our simulation study shows that the robit model is robust against model misspecification, and our EB method results in estimates with less bias than full Bayesian (FB) analysis. The methodology is applied to a Celastrus Orbiculatus data, and a Rhizoctonia root data. For the former, which is known to contain outlying observations, the robit model is shown to do better for predicting the spatial distribution of an invasive species. For the latter, our approach is doing as well as the classical models for predicting the disease severity for a root disease, as the probit link is shown to be appropriate. Though this article is written for Binomial SGLMMs for brevity, the EB methodology is more general and can be applied to other types of SGLMMs. In the accompanying R package geoBayes, implementations for other SGLMMs such as Poisson and Gamma SGLMMs are provided.

Monte Carlo methods for improper target distributions

Abstract: Monte Carlo methods (based on iid sampling or Markov chains) for estimating integrals with respect to a proper target distribution (that is, a probability distribution) are well known in the statistics literature. If the target distribution π happens to be improper then it is shown here that the standard time average estimator based on Markov chains with π as its stationary distribution will converge to zero with probability 1, and hence is not appropriate. In this paper, we present some limit theorems for regenerative sequences and use these to develop some algorithms to produce strongly consistent estimators (called regeneration and ratio estimators) that work whether π is proper or improper. These methods may be referred to as regenerative sequence Monte Carlo (RSMC) methods. The regenerative sequences include Markov chains as a special case. We also present an algorithm that uses the domination of the given target π by a probability distribution π0. Examples are given to illustrate the use and limitations of our algorithms.

Selection of Tuning Parameters, Solution Paths and Standard Errors for Bayesian Lassos

Penalized regression methods such as the lasso and elastic net (EN) have become popular for simultaneous variable selection and coefficient estimation. Implementation of these methods require selection of the penalty parameters. We propose an empirical Bayes (EB) methodology for selecting these tuning parameters as well as computation of the regularization path plots. The EB method does not suffer from the “double shrinkage problem” of frequentist EN. Also it avoids the difficulty of constructing an appropriate prior on the penalty parameters. The EB methodology is implemented by efficient importance sampling method based on multiple Gibbs sampler chains. Since the Markov chains underlying the Gibbs sampler are proved to be geometrically ergodic, Markov chain central limit theorem can be used to provide asymptotically valid confidence band for profiles of EN coefficients. The practical effectiveness of our method is illustrated by several simulation examples and two real life case studies. Although this article considers lasso and EN for brevity, the proposed EB method is general and can be used to select shrinkage parameters in other regularization methods. MSC 2010 subject classifications: primary 62F15, 62J07; secondary 60J05.

Interweaving Markov Chain Monte Carlo Strategies for Efficient Estimation of Dynamic Linear Models

ABSTRACT In dynamic linear models (DLMs) with unknown fixed parameters, a standard Markov chain Monte Carlo (MCMC) sampling strategy is to alternate sampling of latent states conditional on fixed parameters and sampling of fixed parameters conditional on latent states. In some regions of the parameter space, this standard data augmentation (DA) algorithm can be inefficient. To improve efficiency, we apply the interweaving strategies of Yu and Meng to DLMs. For this, we introduce three novel alternative DAs for DLMs: the scaled errors, wrongly scaled errors, and wrongly scaled disturbances. With the latent states and the less well known scaled disturbances, this yields five unique DAs to employ in MCMC algorithms. Each DA implies a unique MCMC sampling strategy and they can be combined into interweaving and alternating strategies that improve MCMC efficiency. We assess these strategies using the local level model and demonstrate that several strategies improve efficiency relative to the standard approach and the most efficient strategy interweaves the scaled errors and scaled disturbances. Supplementary materials are available online for this article.

A novel sandwich algorithm for empirical Bayes analysis of rank data

Rank data arises frequently in marketing, finance, organizational behavior, and psychology. Most analysis of rank data reported in the literature assumes the presence of one or more variables (sometimes latent) based on whose values the items are ranked. In this paper we analyze rank data using a purely probabilistic model where the observed ranks are assumed to be perturbed versions of the true rank and each perturbation has a specific probability of occurring. We consider the general case when covariate information is present and has an impact on the rankings. An empirical Bayes approach is taken for estimating the model parameters. The Gibbs sampler is shown to converge very slowly to the target posterior distribution and we show that some of the widely used empirical convergence diagnostic tools may fail to detect this lack of convergence. We propose a novel, fast mixing sandwich algorithm for exploring the posterior distribution. An EM algorithm based on Markov chain Monte Carlo (MCMC) sampling is developed for estimating prior hyperparameters. A real life rank data set is analyzed using the methods developed in the paper. The results obtained indicate the usefulness of these methods in analyzing rank data with covariate information.

A new algorithm to estimate monotone nonparametric link functions and a comparison with parametric approach

The generalized linear model (GLM) is a class of regression models where the means of the response variables and the linear predictors are joined through a link function. Standard GLM assumes the link function is fixed, and one can form more flexible GLM by either estimating the flexible link function from a parametric family of link functions or estimating it nonparametically. In this paper, we propose a new algorithm that uses P-spline for nonparametrically estimating the link function which is guaranteed to be monotone. It is equivalent to fit the generalized single index model with monotonicity constraint. We also conduct extensive simulation studies to compare our nonparametric approach for estimating link function with various parametric approaches, including traditional logit, probit and robit link functions, and two recently developed link functions, the generalized extreme value link and the symmetric power logit link. The simulation study shows that the link function estimated nonparametrically by our proposed algorithm performs well under a wide range of different true link functions and outperforms parametric approaches when they are misspecified. A real data example is used to illustrate the results.

MCMC diagnostics for higher dimensions using Kullback Leibler divergence

ABSTRACT In the existing literature of MCMC diagnostics, we have identified two areas for improvement. Firstly, the density-based diagnostic tools currently available in the literature are not equipped to assess the joint convergence of multiple variables. Secondly, in case of multi-modal target distribution if the MCMC sampler gets stuck in one of the modes, then the current diagnostic tools may falsely detect convergence. The Tool 1 proposed in this article makes use of adaptive kernel density estimation, symmetric Kullback Leibler divergence and a testing of hypothesis framework to assess the joint convergence of multiple variables. In cases where Tool 1 detects divergence of multiple chains, started at distinct initial values, we propose a visualization tool that can help to investigate reasons behind their divergence. The Tool 2 proposed in this article makes a novel use of the target distribution (known up till the unknown normalizing constant), to detect divergence when an MCMC sampler gets stuck in one of the modes of a multi-modal target distribution. The usefulness of the tools proposed in this article is illustrated using a multi-modal distribution, a mixture of bivariate normal distribution and a Bayesian logit model example.