Thomas J. Leeper

The Generalizability of Survey Experiments

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazons Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

A Source of Bias in Public Opinion Stability

A long acknowledged but seldom addressed problem with political communication experiments concerns the use of captive participants. Study participants rarely have the opportunity to choose information themselves, instead receiving whatever information the experimenter provides. We relax this assumption in the context of an over-time framing experiment focused on opinions about health care policy. Our results dramatically deviate from extant understandings of over-time communication effects. Allowing individuals to choose information themselves—a common situation on many political issues—leads to the preeminence of early frames and the rejection of later frames. Instead of opinion decay, we find dogmatic adherence to opinions formed in response to the first frame to which participants were exposed (i.e., staunch opinion stability). The effects match those that occur when early frames are repeated multiple times. The results suggest that opinion stability may often reflect biased information seeking. Moreover, the findings have implications for a range of topics including the micro–macro disconnect in studies of public opinion, political polarization, normative evaluations of public opinion, the role of inequality considerations in the debate about health care, and, perhaps most importantly, the design of experimental studies of public opinion.

Doing What Others Do Norms, Science, and Collective Action on Global Warming

Does rhetoric highlighting social norms or mentioning science in a communication affect individuals’ beliefs about global warming and/or willingness to take action? We draw from framing theory and collective-interest models of action to motivate hypotheses that are tested in two large web-based survey-experiments using convenience samples. Our results show that attitudes about global warming, support for policies that would reduce carbon emissions, and behavioral intentions to take voluntary action are strongly affected by norm- and science-based interventions. This has implications for information campaigns targeting voluntary efforts to promote lifestyle changes that would reduce greenhouse gas emissions.

How does treatment self-selection affect inferences about political communication?

Ecological validity is vital to experimental research because designs that are too artificial may not speak to any real-world political phenomenon. One such concern is treatment self-selection: if individuals in the real world self-select treatments, such as political communications, how well does the sample average treatment effect estimate the effects of message exposure for those individuals who would — if given the choice — opt-in to and out of receiving treatment? This study shows that randomization masks effect heterogeneity between individuals who would select different messages if given the choice. Yet such selections are themselves complex, revealing additional challenges for realistically studying treatments prone to self-selection. The evidence of effect heterogeneity raises questions about the appropriateness of random assignment experiments for studying political communication and the results more broadly advance our understanding of citizens’ selection into and responses to communications when, as they often do, have choice over what messages to receive.