John G. Bullock

Yes, But What's the Mechanism? (Don't Expect an Easy Answer)

Psychologists increasingly recommend experimental analysis of mediation. This is a step in the right direction because mediation analyses based on nonexperimental data are likely to be biased and because experiments, in principle, provide a sound basis for causal inference. But even experiments cannot overcome certain threats to inference that arise chiefly or exclusively in the context of mediation analysis-threats that have received little attention in psychology. The authors describe 3 of these threats and suggest ways to improve the exposition and design of mediation tests. Their conclusion is that inference about mediators is far more difficult than previous research suggests and is best tackled by an experimental research program that is specifically designed to address the challenges of mediation analysis.

Enough Already about “Black Box” Experiments: Studying Mediation Is More Difficult than Most Scholars Suppose:

The question of how causal effects are transmitted is fascinating and inevitably arises whenever experiments are presented. Social scientists cannot be faulted for taking a lively interest in “mediation,” the process by which causal influences are transmitted. However, social scientists frequently underestimate the difficulty of establishing causal pathways in a rigorous empirical manner. We argue that the statistical methods currently used to study mediation are flawed and that even sophisticated experimental designs cannot speak to questions of mediation without the aid of strong assumptions. The study of mediation is more demanding than most social scientists suppose and requires not one experimental study but rather an extensive program of experimental research.

''Don't Know'' Means ''Don't Know'': DK Responses and the Public's Level of Political Knowledge

Does the public know much more about politics than conventionally thought? A number of studies have recently argued, on various grounds, that the ‘‘don’t know’’ (DK) and incorrect responses to traditionally designed and scored survey knowledge items conceal a good deal of knowledge. This paper examines these claims, focusing on the prominent and influential argument that discouraging DKs would reveal a substantially more knowledgeable public. Using two experimental surveys with national random samples, we show that discouraging DKs does little to affect our picture of how much the public knows about politics. For closed-ended items, the increase in correct responses is large but mainly illusory. For open-ended items, it is genuine but minor. We close by examining the other recent evidence for a substantially more knowledgeable public, showing that it too holds little water.

Partisan Bias and the Bayesian Ideal in the Study of Public Opinion

Bayes’ Theorem is increasingly used as a benchmark against which to judge the quality of citizens’ thinking, but some of its implications are not well understood. A common claim is that Bayesians must agree more as they learn and that the failure of partisans to do the same is evidence of bias in their responses to new information. Formal inspection of Bayesian learning models shows that this is a misunderstanding. Learning need not create agreement among Bayesians. Disagreement among partisans is never clear evidence of bias. And although most partisans are not Bayesians, their reactions to new information are surprisingly consistent with the ideal of Bayesian rationality.

LETHAL INCOMPETENCE: VOTERS, OFFICIALS, AND SYSTEMS

ABSTRACT The study of voter competence has made significant contributions to our understanding of politics, but at this point there are diminishing returns to the endeavor. There is little reason, in theory or in practice, to expect voter competence to improve dramatically enough to make much of a difference, but there is reason to think that officials’ competence can vary enough to make large differences. To understand variations in government performance, therefore, we would do better to focus on the abilities and performance of officials, not ordinary citizens.

Reference Rot: An Emerging Threat to Transparency in Political Science

Transparency of research is a large concern in political science, and the practice of publishing links to datasets and other online resources is one of the main methods by which political scientists promote transparency. But the method cannot work if the links don’t, and very often, they don’t. We show that most of the URLs ever published in the American Political Science Review no longer work as intended. The problem is severe in recent as well as in older articles; for example, more than one-fourth of links published in the APSR in 2013 were broken by the end of 2014. We conclude that “reference rot” limits the transparency and reproducibility of political science research. We also describe practices that scholars can adopt to combat the problem: when possible, they should archive data in trustworthy repositories, use links that incorporate persistent digital identifiers, and create archival versions of the webpages to which they link.