James F. Burgess

Why We Should Not Be Indifferent to Specification Choices for Difference-in-Differences.

OBJECTIVE To evaluate the effects of specification choices on the accuracy of estimates in difference-in-differences (DID) models. DATA SOURCES Process-of-care quality data from Hospital Compare between 2003 and 2009. STUDY DESIGN We performed a Monte Carlo simulation experiment to estimate the effect of an imaginary policy on quality. The experiment was performed for three different scenarios in which the probability of treatment was (1) unrelated to pre-intervention performance; (2) positively correlated with pre-intervention levels of performance; and (3) positively correlated with pre-intervention trends in performance. We estimated alternative DID models that varied with respect to the choice of data intervals, the comparison group, and the method of obtaining inference. We assessed estimator bias as the mean absolute deviation between estimated program effects and their true value. We evaluated the accuracy of inferences through statistical power and rates of false rejection of the null hypothesis. PRINCIPAL FINDINGS Performance of alternative specifications varied dramatically when the probability of treatment was correlated with pre-intervention levels or trends. In these cases, propensity score matching resulted in much more accurate point estimates. The use of permutation tests resulted in lower false rejection rates for the highly biased estimators, but the use of clustered standard errors resulted in slightly lower false rejection rates for the matching estimators. CONCLUSIONS When treatment and comparison groups differed on pre-intervention levels or trends, our results supported specifications for DID models that include matching for more accurate point estimates and models using clustered standard errors or permutation tests for better inference. Based on our findings, we propose a checklist for DID analysis.

A DEA based composite measure of quality and its associated data uncertainty interval for health care provider profiling and pay-for-performance

Composite measures calculated from individual performance indicators increasingly are used to profile and reward health care providers. We illustrate an innovative way of using Data Envelopment Analysis (DEA) to create a composite measure of quality for profiling facilities, informing consumers, and pay-for-performance programs. We compare DEA results to several widely used alternative approaches for creating composite measures: opportunity-based-weights (OBW, a form of equal weighting) and a Bayesian latent variable model (BLVM, where weights are driven by variances of the individual measures). Based on point estimates of the composite measures, to a large extent the same facilities appear in the top decile. However, when high performers are identified because the lower limits of their interval estimates are greater than the population average (or, in the case of the BLVM, the upper limits are less), there are substantial differences in the number of facilities identified: OBWs, the BLVM and DEA identify 25, 17 and 5 high-performers, respectively. With DEA, where every facility is given the flexibility to set its own weights, it becomes much harder to distinguish the high performers. In a pay-for-performance program, the different approaches result in very different reward structures: DEA rewards a small group of facilities a larger percentage of the payment pool than the other approaches. Finally, as part of the DEA analyses, we illustrate an approach that uses Monte Carlo resampling with replacement to calculate interval estimates by incorporating uncertainty in the data generating process for facility input and output data. This approach, which can be used when data generating processes are hierarchical, has the potential for wider use than in our particular application.

Population-level cost-effectiveness of implementing evidence-based practices into routine care

OBJECTIVE The objective of this research was to apply a new methodology (population-level cost-effectiveness analysis) to determine the value of implementing an evidence-based practice in routine care. DATA SOURCES/STUDY SETTING Data are from sequentially conducted studies: a randomized controlled trial and an implementation trial of collaborative care for depression. Both trials were conducted in the same practice setting and population (primary care patients prescribed antidepressants). STUDY DESIGN The study combined results from a randomized controlled trial and a pre-post-quasi-experimental implementation trial. DATA COLLECTION/EXTRACTION METHODS The randomized controlled trial collected quality-adjusted life years (QALYs) from survey and medication possession ratios (MPRs) from administrative data. The implementation trial collected MPRs and intervention costs from administrative data and implementation costs from survey. PRINCIPAL FINDINGS In the randomized controlled trial, MPRs were significantly correlated with QALYs (p = .03). In the implementation trial, patients at implementation sites had significantly higher MPRs (p = .01) than patients at control sites, and by extrapolation higher QALYs (0.00188). Total costs (implementation, intervention) were nonsignificantly higher (

Federal provision of health care: creating access for the underinsured.

63.76) at implementation sites. The incremental population-level cost-effectiveness ratio was

Dense Breast Notification Laws: Impact on Downstream Imaging After Screening Mammography

33,905.92/QALY (bootstrap interquartile range -

Longitudinal Analysis of Quality of Diabetes Care and Relational Climate in Primary Care

45,343.10/QALY to

Space-Time Cluster Analysis to Detect Innovative Clinical Practices: A Case Study of Aripiprazole in the Department of Veterans Affairs.

99,260.90/QALY). CONCLUSIONS The methodology was feasible to operationalize and gave reasonable estimates of implementation value.

Medical profiling: improving standards and risk adjustments using hierarchical models.

Access to health care for the underinsured in America is a major current policy issue. Federal provision of health care has not been evaluated seriously as part of the solution to the problem despite the presence of federal health care provided to veterans, the class of Americans most completely guaranteed universal access. We first explore the arguments for and against universal access, clarifying issues but yielding no definitive solution. Then the federal health care system for veterans is used as a model for exploring problems that must be solved in a universal access plan. The discussion focuses on the effects of competition for patients and health care resources on costs, innovation, regulation, and quality.

The early effects of Medicare's mandatory hospital pay-for-performance program.

Dense breast tissue is a common finding that decreases the sensitivity of mammography in detecting cancer. Many states have recently enacted dense breast notification (DBN) laws to provide patients with information to help them make better-informed decisions about their health. To test whether DBN legislation affected the probability of screening mammography follow-up by ultrasound and magnetic resonance imaging (MRI), we examined the proportion of times screening mammography was followed by ultrasound or MRI for a series of months pre- and post-legislation. The subjects were women aged 40 to 64 years, covered by private health insurance, undergoing screening mammography from 2007 to 2014. Except for Hawaii, Maryland, and New York, DBN legislation significantly increased the probability of ultrasound follow-up in all states that implemented DBN legislation before December 2014. It also increased the probability of MRI follow-up in California, North Carolina, Pennsylvania, and Texas. The financial and access consequences merit further study.

The effect of network arrangements on hospital pricing behavior

OBJECTIVE To assess the influence of relational climate on quality of diabetes care. DATA SOURCES/STUDY SETTING The study was conducted at the Department of Veterans Affairs (VA). The VA All Employee Survey (AES) was used to measure relational climate. Patient and facility characteristics were gathered from VA administrative datasets. STUDY DESIGN Multilevel panel data (2008-2012) with patients nested into clinics. DATA COLLECTION/EXTRACTION METHODS Diabetic patients were identified using ICD-9 codes and assigned to the clinic with the highest frequency of primary care visits. Multiple quality indicators were used, including an all-or-none process measure capturing guideline compliance, the actual number of tests and procedures, and three intermediate continuous outcomes (cholesterol, glycated hemoglobin, and blood pressure). PRINCIPAL FINDINGS The study sample included 327,805 patients, 212 primary care clinics, and 101 parent facilities in 2010. Across all study years, there were 1,568,180 observations. Clinics with the highest relational climate were 25 percent more likely to provide guideline-compliant care than those with the lowest relational climate (OR for a 1-unit increase: 1.02, p-value <.001). Among insulin-dependent diabetic veterans, this effect was twice as large. Contrary to that expected, relational climate did not influence intermediate outcomes. CONCLUSIONS Relational climate is positively associated with tests and procedures provision, but not with intermediate outcomes of diabetes care.