Elizabeth A. Stuart

Matching methods for causal inference: A review and a look forward.

When estimating causal effects using observational data, it is desirable to replicate a randomized experiment as closely as possible by obtaining treated and control groups with similar covariate distributions. This goal can often be achieved by choosing well-matched samples of the original treated and control groups, thereby reducing bias due to the covariates. Since the 1970s, work on matching methods has examined how to best choose treated and control subjects for comparison. Matching methods are gaining popularity in fields such as economics, epidemiology, medicine, and political science. However, until now the literature and related advice has been scattered across disciplines. Researchers who are interested in using matching methods-or developing methods related to matching-do not have a single place to turn to learn about past and current research. This paper provides a structure for thinking about matching methods and guidance on their use, coalescing the existing research (both old and new) and providing a summary of where the literature on matching methods is now and where it should be headed.

Multiple imputation by chained equations: what is it and how does it work?

Multivariate imputation by chained equations (MICE) has emerged as a principled method of dealing with missing data. Despite properties that make MICE particularly useful for large imputation procedures and advances in software development that now make it accessible to many researchers, many psychiatric researchers have not been trained in these methods and few practical resources exist to guide researchers in the implementation of this technique. This paper provides an introduction to the MICE method with a focus on practical aspects and challenges in using this method. A brief review of software programs available to implement MICE and then analyze multiply imputed data is also provided. Copyright

Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research

There is considerable interest in using propensity score (PS) statistical techniques to address questions of causal inference in psychological research. Many PS techniques exist, yet few guidelines are available to aid applied researchers in their understanding, use, and evaluation. In this study, the authors give an overview of available techniques for PS estimation and PS application. They also provide a way to help compare PS techniques, using the resulting measured covariate balance as the criterion for selecting between techniques. The empirical example for this study involves the potential causal relationship linking early-onset cannabis problems and subsequent negative mental health outcomes and uses data from a prospective cohort study. PS techniques are described and evaluated on the basis of their ability to balance the distributions of measured potentially confounding covariates for individuals with and without early-onset cannabis problems. This article identifies the PS techniques that yield good statistical balance of the chosen measured covariates within the context of this particular research question and cohort.

Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies

The propensity score is defined as a subjects probability of treatment selection, conditional on observed baseline covariates. Weighting subjects by the inverse probability of treatment received creates a synthetic sample in which treatment assignment is independent of measured baseline covariates. Inverse probability of treatment weighting (IPTW) using the propensity score allows one to obtain unbiased estimates of average treatment effects. However, these estimates are only valid if there are no residual systematic differences in observed baseline characteristics between treated and control subjects in the sample weighted by the estimated inverse probability of treatment. We report on a systematic literature review, in which we found that the use of IPTW has increased rapidly in recent years, but that in the most recent year, a majority of studies did not formally examine whether weighting balanced measured covariates between treatment groups. We then proceed to describe a suite of quantitative and qualitative methods that allow one to assess whether measured baseline covariates are balanced between treatment groups in the weighted sample. The quantitative methods use the weighted standardized difference to compare means, prevalences, higher‐order moments, and interactions. The qualitative methods employ graphical methods to compare the distribution of continuous baseline covariates between treated and control subjects in the weighted sample. Finally, we illustrate the application of these methods in an empirical case study. We propose a formal set of balance diagnostics that contribute towards an evolving concept of ‘best practice’ when using IPTW to estimate causal treatment effects using observational data.

Improving propensity score weighting using machine learning

Machine learning techniques such as classification and regression trees (CART) have been suggested as promising alternatives to logistic regression for the estimation of propensity scores. The authors examined the performance of various CART-based propensity score models using simulated data. Hypothetical studies of varying sample sizes (n=500, 1000, 2000) with a binary exposure, continuous outcome, and 10 covariates were simulated under seven scenarios differing by degree of non-linear and non-additive associations between covariates and the exposure. Propensity score weights were estimated using logistic regression (all main effects), CART, pruned CART, and the ensemble methods of bagged CART, random forests, and boosted CART. Performance metrics included covariate balance, standard error, per cent absolute bias, and 95 per cent confidence interval (CI) coverage. All methods displayed generally acceptable performance under conditions of either non-linearity or non-additivity alone. However, under conditions of both moderate non-additivity and moderate non-linearity, logistic regression had subpar performance, whereas ensemble methods provided substantially better bias reduction and more consistent 95 per cent CI coverage. The results suggest that ensemble methods, especially boosted CART, may be useful for propensity score weighting.

Intervention Targeting Development of Socially Synchronous Engagement in Toddlers with Autism Spectrum Disorder: A Randomized Controlled Trial

BACKGROUND Social and communication impairments are core deficits and prognostic indicators of autism. We evaluated the impact of supplementing a comprehensive intervention with a curriculum targeting socially synchronous behavior on social outcomes of toddlers with autism spectrum disorders (ASD). METHODS Fifty toddlers with ASD, ages 21 to 33 months, were randomized to one of two six-month interventions: Interpersonal Synchrony or Non-Interpersonal Synchrony. The interventions provided identical intensity (10 hours per week in classroom), student-to-teacher ratio, schedule, home-based parent training (1.5 hours per month), parent education (38 hours), and instructional strategies, except the Interpersonal Synchrony condition provided a supplementary curriculum targeting socially engaged imitation, joint attention, and affect sharing; measures of these were primary outcomes. Assessments were conducted pre-intervention, immediately post-intervention, and, to assess maintenance, at six-month follow-up. Random effects models were used to examine differences between groups over time. Secondary analyses examined gains in expressive language and nonverbal cognition, and time effects during the intervention and follow-up periods. RESULTS A significant treatment effect was found for socially engaged imitation (p = .02), with more than doubling (17% to 42%) of imitated acts paired with eye contact in the Interpersonal Synchrony group after the intervention. This skill was generalized to unfamiliar contexts and maintained through follow-up. Similar gains were observed for initiation of joint attention and shared positive affect, but between-group differences did not reach statistical significance. A significant time effect was found for all outcomes (p < .001); greatest change occurred during the intervention period, particularly in the Interpersonal Synchrony group. CONCLUSIONS This is the first ASD randomized trial involving toddlers to identify an active ingredient for enhancing socially engaged imitation. Adding social engagement targets to intervention improves short-term outcome at no additional cost to the intervention. The social, language, and cognitive gains in our participants provide evidence for plasticity of these developmental systems in toddlers with ASD. http://www.clinicaltrials.gov/ct2/show/NCT00106210?term = landa&rank = 3.

A Potential Outcomes View of Value-Added Assessment in Education.

There has been substantial interest in recent years in the performance and accountability of teachers and schools, partially due to the No Child Left Behind legislation, which requires states to develop a system of sanctions and rewards to hold districts and schools accountable for academic achievement. This focus has lead to an increase in “high-stakes” testing with publicized school rankings and test results. The papers by Ballou et al. (2004), McCaffrey et al. (2004) and Tekwe et al. (2004) approach the estimation of school and teacher effects through a variety of statistical models, known as “value-added” models in the education literature. There are many complex issues involved, and we applaud the authors for addressing this challenging topic.

Generalizing observational study results: Applying propensity score methods to complex surveys

OBJECTIVE To provide a tutorial for using propensity score methods with complex survey data. DATA SOURCES Simulated data and the 2008 Medical Expenditure Panel Survey. STUDY DESIGN Using simulation, we compared the following methods for estimating the treatment effect: a naïve estimate (ignoring both survey weights and propensity scores), survey weighting, propensity score methods (nearest neighbor matching, weighting, and subclassification), and propensity score methods in combination with survey weighting. Methods are compared in terms of bias and 95 percent confidence interval coverage. In Example 2, we used these methods to estimate the effect on health care spending of having a generalist versus a specialist as a usual source of care. PRINCIPAL FINDINGS In general, combining a propensity score method and survey weighting is necessary to achieve unbiased treatment effect estimates that are generalizable to the original survey target population. CONCLUSIONS Propensity score methods are an essential tool for addressing confounding in observational studies. Ignoring survey weights may lead to results that are not generalizable to the survey target population. This paper clarifies the appropriate inferences for different propensity score methods and suggests guidelines for selecting an appropriate propensity score method based on a researchers goal.

Generalizing Evidence From Randomized Clinical Trials to Target Populations The ACTG 320 Trial

Properly planned and conducted randomized clinical trials remain susceptible to a lack of external validity. The authors illustrate a model-based method to standardize observed trial results to a specified target population using a seminal human immunodeficiency virus (HIV) treatment trial, and they provide Monte Carlo simulation evidence supporting the method. The example trial enrolled 1,156 HIV-infected adult men and women in the United States in 1996, randomly assigned 577 to a highly active antiretroviral therapy and 579 to a largely ineffective combination therapy, and followed participants for 52 weeks. The target population was US people infected with HIV in 2006, as estimated by the Centers for Disease Control and Prevention. Results from the trial apply, albeit muted by 12%, to the target population, under the assumption that the authors have measured and correctly modeled the determinants of selection that reflect heterogeneity in the treatment effect. In simulations with a heterogeneous treatment effect, a conventional intent-to-treat estimate was biased with poor confidence limit coverage, but the proposed estimate was largely unbiased with appropriate confidence limit coverage. The proposed method standardizes observed trial results to a specified target population and thereby provides information regarding the generalizability of trial results.

Using Full Matching to Estimate Causal Effects in Nonexperimental Studies: Examining the Relationship between Adolescent Marijuana Use and Adult Outcomes.

Matching methods such as nearest neighbor propensity score matching are increasingly popular techniques for controlling confounding in nonexperimental studies. However, simple k:1 matching methods, which select k well-matched comparison individuals for each treated individual, are sometimes criticized for being overly restrictive and discarding data (the unmatched comparison individuals). The authors illustrate the use of a more flexible method called full matching. Full matching makes use of all individuals in the data by forming a series of matched sets in which each set has either 1 treated individual and multiple comparison individuals or 1 comparison individual and multiple treated individuals. Full matching has been shown to be particularly effective at reducing bias due to observed confounding variables. The authors illustrate this approach using data from the Woodlawn Study, examining the relationship between adolescent marijuana use and adult outcomes.