Allan Dafoe

Science Deserves Better: The Imperative to Share Complete Replication Files

Scientific knowledge is only as reliable as the empirical analysis on which it is based. For the majority of published statistical analyses, readers have to trust that the scholars correctly implemented the many stages of analysis between primary data collection and the presentation of results — including data cleaning, merging, recoding and transforming, analysis, and output. I advocate the adoption of a simple transparency maxim: good research involves publishing complete replication files, making every step of research as explicit and reproducible as is practical. Benefits of replication transparency include: raising the quality and refutability of scientific inferences; dissemination of useful data and code; greater freedom for scientists to explore each others’ results and data; and a richer scientific conversation. More transparent replication prac- tices are a scientific public good: the costs are small but borne by the authors; the benefits are great and shared by the broader scientific community and public. Strong norms of replication transparency, once established, are likely to be partly self-enforcing. Data is presented on replication practices and the replicability of published works in political science. I outline good replication practices for scholars, and offer recommendations to journals, universities, and funders.

On Technological Determinism A Typology, Scope Conditions, and a Mechanism

“Technological determinism” is predominantly employed as a critic’s term, used to dismiss certain classes of theoretical and empirical claims. Understood more productively as referring to claims that place a greater emphasis on the autonomous and social-shaping tendencies of technology, technological determinism is a valuable and prominent perspective. This article will advance our understanding of technological determinism through four contributions. First, I clarify some debates about technological determinism through an examination of the meaning of technology. Second, I parse the family of claims related to technological determinism. Third, I note that constructivist and determinist insights may each be valid given particular scope conditions, the most prominent of which is the scale of analysis. Finally, I propose a theoretical microfoundation for technological determinism—military–economic adaptationism—in which economic and military competition constrain sociotechnical evolution to deterministic paths. This theory is a special case of a general theory—sociotechnical selectionism—which can be regarded as also including (mild) constructivist theories as special cases. Greater understanding of, respect for, and engagement with technological determinism will enhance the study of technology and our ability to shape our sociotechnical systems.

Honor and War: Southern US Presidents and the Effects of Concern for Reputation

Reputation has long been considered central to international relations, but unobservability, strategic selection, and endogeneity have handicapped quantitative research. A rare source of haphazard variation in the cultural origins of leaders—the fact that one-third of US presidents were raised in the American South, a well-studied example of a culture of honor—provides an opportunity to identify the effects of heightened concern for reputation for resolve. A formal theory that yields several testable predictions while accounting for unobserved selection into disputes is offered. The theory is illustrated through a comparison of presidents John F. Kennedy and Lyndon B. Johnson and systematically tested using matching, permutation inference, and the nonparametric combination of tests. Interstate conflicts under Southern presidents are shown to be twice as likely to involve uses of force, last on average twice as long, and are three times more likely to end in victory for the United States than disputes under non-Southern presidents. Other characteristics of Southern presidencies do not seem able to account for this pattern of results. The results provide evidence that concern for reputation is an important cause of interstate conflict behavior.

Nonparametric Combination (NPC): A Framework for Testing Elaborate Theories

Social scientists are commonly advised to deduce and test all observable implications of their theories. We describe a principled framework for testing such “elaborate” theories: nonparametric combination. Nonparametric combination (NPC) assesses the joint probability of observing the theoretically predicted pattern of results under the sharp null of no effects. NPC accounts for the dependence among the component tests without relying on modeling assumptions or asymptotic approximations. Multiple-testing corrections are also easily implemented with NPC. As we demonstrate with four applications, NPC leverages theoretical knowledge into greater statistical power, which is particularly valuable for studies with strong research designs but small sample sizes. We implement these methods in a new R package, NPC.

Prescriptions for Temporal Dependence: First Do No Harm

Temporal dependence is often thought to be present in analyses of time-series (cross-sectional) data. Temporal dependence is the property that observations are not independent, conditional on an estimation strategy. Naive analysis of temporally dependent data will lead to incorrect, usually overconfident, standard errors, and may also lead to biased and inconsistent estimates of quantities of interest. A common response is to model the temporal dependence as a function of lags of the dependent variable (LDVs); in some sub-fields this is the universally adopted solution to temporal dependence. Contrary this practice, this paper demonstrates that the solution to temporal dependence always depends on assumptions about the data-generating process (DGP); the correct fix for temporal dependence cannot be diagnosed from the data alone. Beginning from a general nonparametric DGP, this paper articulates the assumptions required for LDVs to improve inference. These assumptions can be expressed as empirical commitments that analysts should articulate and justify. This paper examines the viability and assumptions required of other solutions, such as including exhaustive isolated mechanisms, instrumental variables, parametric assumptions, placebo tests (the Achen Diagnostic), and signing the biases. Absent confident justified beliefs about the DGP, scholars should routinely report estimates with and without LDVs. Sensitivity to temporal specification implies that a result can not be adequately understood without a better understanding of the temporal dynamics of the phenomenon of interest.

Nonparametric Identification of Causal Effects under Temporal Dependence

Social scientists routinely address temporal dependence by adopting a simple technical fix. However, the correct identification strategy for a causal effect depends on causal assumptions. These need to be explicated and justified; almost no studies do so. This article addresses this shortcoming by offering a precise general statement of the (nonparametric) causal assumptions required to identify causal effects under temporal dependence. In particular, this article clarifies when one should condition or not condition on lagged dependent variables (LDVs) to identify causal effects: one should not condition on LDVs, if there is no reverse causation and no outcome autocausation; one should condition on LDVs if there are no unobserved common causes of treatment and the lagged outcome, or no unobserved persistent causes of the outcome. When only one of these is true (with one exception), the incorrect decision will induce bias. Absent a well-justified identification strategy, inferences should be appropriately qualified.

Is There a War Party? Party Change, the Left–Right Divide, and International Conflict:

Are leaders from certain parties particularly likely to engage in military conflict? This question is difficult to answer because of selection bias. For example, countries may be more likely to elect right-wing leaders if their publics are more hawkish or if the international system is particularly dangerous. Put simply, who comes to power is not random, which makes causal inference difficult. We overcome this problem by using a regression discontinuity design. Specifically, we look at close presidential elections that were essentially “tossups” between two candidates. We find that electing right-wing candidates increases state aggression. We also find that electing candidates from challenger parties makes countries much more likely to initiate military disputes, particularly in the first year of the new leader’s term. This result is consistent with other studies that find that the likelihood of state aggression increases following major leadership transitions.

Democratic peace versus contractualism

If contractualism causes both democracy and peace, the correlation between democracy and peace is spurious. But the definition of contractualism is sufficiently unclear as to create doubts that it has such an impact. In addition, if democracy has an impact on contractualism, then, even with an otherwise perfectly specified model, controlling for contractualism will bias estimates of the effect of democracy on peace. International trade involves contracting behavior between strangers in different states. In addition, unlike contracting behavior within states, trade involves interactions in the same arena where interstate conflict needs to be deterred. Contractualist states have existed in significant numbers only recently. This limits contractualism’s ability to compete with democracy as a predictor of peace. Mousseau’s previous findings have been extraordinarily fragile. Finally, observational data should be complemented by multiple streams of evidence. Some experimental data support democratic peace theory; analogous experiments would almost certainly not provide evidence to support contractualism.