“Natural Experiments” in Health Care Research

Abstract

In “natural experiments,” the treatment or intervention is determined by variation not under the control of the researcher. These designs, used in economics and epidemiology to support inferences about causal relationships between interventions and outcomes, are useful tools to help improve the rigor of observational studies in health policy and medicine. Perhaps the first natural experiment in medicine was that of the English physician John Snow in the mid-nineteenth century. In 1854, a cholera outbreak struck Broad Street in London, killing hundreds. Studying case clusters, Snow discovered that neighborhoods supplied with water downstream of where sewage was discharged into the Thames River experienced high levels of disease, while neighborhoods receiving upstream water had low disease levels.1 Snow described the populations as similar in age, occupation, income, and social rank, divided into groups without choice, illustrating an essential component of natural experiments: similar but distinct populations that are exposed to a condition outside the researchers’ control, allowing for reasonable conclusions about the potential causal link between exposure and outcome. Randomized clinical trials (RCTs) have traditionally been viewed as the primary method for establishing causality in health care, but they have important limitations: they are expensive; it is not always possible to randomize patients; and their findings may not be generalizable to different patient populations or nonexperimental settings. When RCTs are not possible, medical and health policy researchers have turned to observational studies. In observational studies, however, individuals are not assigned to the intervention independently of potential confounding factors that could also influence outcomes, making it difficult to separate the treatment effect from other factors that may be associated with receiving the treatment. By contrast, natural experiments rely on variation in treatment exposure that may be unrelated to other factors associated with the outcomes. Suppose researchers are interested in examining the likelihood of long-term use and adverse outcomes for patients after an initial opioid prescription. An observational analysis might be confounded if the factors that influence a clinician’s decision to prescribe opioids (eg, cancer-related pain) also affect long-term outcomes (eg, opioid dependence). An RCT might resolve this issue but would be ethically and practically challenging. Instead, researchers could examine how long-term opioid use varies among opioid-naive individuals who, by chance, are exposed to physicians with a high propensity vs low propensity to prescribe opioids (eg, when assigned to the next available physician in an emergency department).2 In this scenario, the long-term outcomes following an initial opioid prescription could be identified by variation in the drug’s use associated with prescriber variation that is plausibly unrelated to variation in unobservable patient factors associated with both initial opioid use and long-term outcomes. Natural experiments use quasi-randomization, a method of allocation to study groups that is not truly random and is not assigned by a researcher, such as a specific date, age, or event. These study designs have an important feature: the similarity of the groups can be measured. Treatment and control groups should be similar in sociodemographic characteristics, comorbidities, prior health care utilization, and any other factors that might be associated with outcomes, but often this is not the case and adjustments are needed based on these observed variables. Natural experiments attempt to control for unobserved variables. When well implemented, natural experiments may be more informative than traditional observational studies that do not control for unobservable confounders, but are less informative than RCTs in establishing true cause and effect. With natural experiments, + Editorial