Beyond Overall Treatment Effects: Leveraging Covariates in Randomized Experiments Guided by Causal Structure

Abstract

Researchers using randomized controlled trials (RCTs) often subgroup or condition on auxiliary variables that are not the randomized treatment variable. There are many good reasons to condition on auxiliary variables—also referred to as control variables or covariates—in randomized experiments. In particular, designing and conducting RCTs is costly to researchers and subjects, and therefore it is important to derive greater value from RCT data; measuring not just the average treatment effect (ATE), but also finding more nuanced insights about the underlying theoretical mechanisms and generalizing the inferences. Unfortunately, there are many confusing and even contradictory guidelines on the use of subgroups or auxiliary variables in RCTs. We show how researchers can leverage covariates without biasing their causal inferences, by applying a few simple rules based on Judea Pearl’s causal diagramming framework. We demonstrate how to create a causal schema, through careful and deliberate operationalization of auxiliary covariates, in order to analyze the intermediate effects along a causal chain from the treatment to outcome; and we discuss some other ways to leverage covariates for theory development and generalization of findings from RCTs. We present a criterion for distinguishing pre-treatment and post-treatment variables that is based on directed acyclic graphs (DAGs). We provide a succinct set of guidelines to help readers begin to employ some essential techniques of DAG-based causal analysis. Finally, we provide a series of short tutorials (with accompanying simulated data and R scripts) to help readers explore the connections between RCT and observational contexts in causal diagramming. This commentary aims to raise awareness of the DAG methodology, explain its usefulness to experimental research, and encourage adoption in the IS community for studies using RCTs as well as observational data.