Jonah Gabry

Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparison of predictive errors between two models. We implement the computations in an R package called loo and demonstrate using models fit with the Bayesian inference package Stan.

Erratum to: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparison of predictive errors between two models. We implement the computations in an R package called loo and demonstrate using models fit with the Bayesian inference package Stan.

Antiemetic prophylaxis as a marker of health care disparities in the national anesthesia clinical outcomes registry

BACKGROUND: US health care disparities persist despite repeated countermeasures. Research identified race, ethnicity, gender, and socioeconomic status as factors, mediated through individual provider and/or systemic biases; little research exists in anesthesiology. We investigated antiemetic prophylaxis as a surrogate marker for anesthesia quality by individual providers because antiemetics are universally available, indicated contingent on patient characteristics (gender, age, etc), but independent of comorbidities and not yet impacted by regulatory or financial constraints. We hypothesized that socioeconomic indicators (measured as insurance status or median income in the patients’ home zip code area) are associated with the utilization of antiemetic prophylaxis (as a marker of anesthesia quality). METHODS: We tested our hypothesis in several subsets of electronic anesthesia records from the National Anesthesia Clinical Outcomes Registry (NACOR), fitting frequentist and novel Bayesian multilevel logistic regression models. RESULTS: NACOR contained 12 million cases in 2013. Six institutions reported on antiemetic prophylaxis for 441,645 anesthesia cases. Only 173,133 cases included details on insurance information. Even fewer (n = 92,683) contained complete data on procedure codes and provider identifiers. Bivariate analysis, multivariable logistic regression, and our Bayesian hierarchical model all showed a large and statistically significant association between socioeconomic markers and antiemetic prophylaxis (ondansetron and dexamethasone). For Medicaid versus commercially insured patients, the odds ratio of receiving the antiemetic ondansetron is 0.85 in our Bayesian hierarchical mixed regression model, with a 95% Bayesian credible interval of 0.81–0.89 with similar inferences in classical (frequentist) regression models. CONCLUSIONS: Our analyses of NACOR anesthesia records raise concerns that patients with lower socioeconomic status may receive inferior anesthesia care provided by individual anesthesiologists, as indicated by less antiemetics administered. Effects persisted after we controlled for important patient characteristics and for procedure and provider influences. Findings were robust to sensitivity analyses. Our results challenge the notion that anesthesia providers do not contribute to health care disparities.