Cristobal Young

Redefine Statistical Significance

We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries.

Model Uncertainty and Robustness: A Computational Framework for Multimodel Analysis

Model uncertainty is pervasive in social science. A key question is how robust empirical results are to sensible changes in model specification. We present a new approach and applied statistical software for computational multimodel analysis. Our approach proceeds in two steps: First, we estimate the modeling distribution of estimates across all combinations of possible controls as well as specified functional form issues, variable definitions, standard error calculations, and estimation commands. This allows analysts to present their core, preferred estimate in the context of a distribution of plausible estimates. Second, we develop a model influence analysis showing how each model ingredient affects the coefficient of interest. This shows which model assumptions, if any, are critical to obtaining an empirical result. We demonstrate the architecture and interpretation of multimodel analysis using data on the union wage premium, gender dynamics in mortgage lending, and tax flight migration among U.S. states. These illustrate how initial results can be strongly robust to alternative model specifications or remarkably dependent on a knife-edge specification.

Millionaire Migration and Taxation of the Elite: Evidence from Administrative Data

A growing number of U.S. states have adopted “millionaire taxes” on top income-earners. This increases the progressivity of state tax systems, but it raises concerns about tax flight: elites migrating from high-tax to low-tax states, draining state revenues, and undermining redistributive social policies. Are top income-earners “transitory millionaires” searching for lower-tax places to live? Or are they “embedded elites” who are reluctant to migrate away from places where they have been highly successful? This question is central to understanding the social consequences of progressive taxation. We draw on administrative tax returns for all million-dollar income-earners in the United States over 13 years, tracking the states from which millionaires file their taxes. Our dataset contains 45 million tax records and provides census-scale panel data on top income-earners. We advance two core analyses: (1) state-to-state migration of millionaires over the long-term, and (2) a sharply-focused discontinuity analysis of millionaire population along state borders. We find that millionaire tax flight is occurring, but only at the margins of statistical and socioeconomic significance.

A Reply to “A Replication of “Millionaire Migration and State Taxation of Top Incomes Evidence from a Natural Experiment” (National Tax Journal 2011)”

This response discusses the findings and criticisms in Cohen, Lai, and Steindel (CLS). Despite the skeptical tone of their article, the CLS analysis confirms our core conclusion of a small (or very small) migration effect of the millionaire tax. The range of estimates reported by CLS, including the wrong-signed estimates they find, scarcely reaches beyond the 95 percent confidence interval originally reported by Young and Varner. The critical modeling choice made by CLS is to exclude the observed in-migration of millionaires in the years following the tax increase. Even this leaves small migration effects, with an implied revenue cost that is a small fraction of the additional revenues generated by the millionaire tax.

Time is a network good

Time is a network good: the value of time depends on whether others also have it. We can deepen our understanding of time from a comparison with other network goods like personal computers, Facebook, and communications technology that derive their value from widely shared usage. We review recent research on the importance of collective social time with family and friends, and the role that temporal coordination plays in enhancing community ties and subjective well-being. The standard workweek is one of the most taken-for-granted institutions that creates effortless social coordination of time. The weekend provides people with collective time off that facilitates social interaction and leads to remarkable gains in emotional well-being. A breakdown in the temporal coordination of the standard workweek can have a negative impact on individuals, families, and communities. Future directions for research emphasize the importance of recognizing the network properties of time and its implications for society at large.

Model Uncertainty and the Crisis in Science

The “crisis in science” today is rooted in genuine problems of model uncertainty and lack of transparency. Researchers estimate a large number of models in the course of their research but only publish a small number of preferred results. Authors have much influence on the results of an empirical study through their choices about model specification. I advance methods to quantify the influence of the author—or at least demonstrate the scope an author has to choose a preferred result. Multimodel analysis, combined with modern computational power, allows authors to present their preferred estimate alongside a distribution of estimates from many other plausible models. I demonstrate the method using new software and applied empirical examples. When evaluating research results, accounting for model uncertainty and model robustness is at least as important as statistical significance.

The Difference Between Causal Analysis and Predictive Models: Response to “Comment on Young and Holsteen (2017)”

The commenter’s proposal may be a reasonable method for addressing uncertainty in predictive modeling, where the goal is to predict y. In a treatment effects framework, where the goal is causal inference by conditioning-on-observables, the commenter’s proposal is deeply flawed. The proposal (1) ignores the definition of omitted-variable bias, thus systematically omitting critical kinds of controls; (2) assumes for convenience there are no bad controls in the model space, thus waving off the premise of model uncertainty; and (3) deletes virtually all alternative models to select a single model with the highest R 2. Rather than showing what model assumptions are necessary to support one’s preferred results, this proposal favors biased parameter estimates and deletes alternative results before anyone has a chance to see them. In a treatment effects framework, this is not model robustness analysis but simply biased model selection.

We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness

False positive findings are a growing problem in many research literatures. We argue that excessive false positives often stem from model uncertainty. There are many plausible ways of specifying a regression model, but researchers typically report only a few preferred estimates. This raises the concern that such research reveals only a small fraction of the possible results and may easily lead to nonrobust, false positive conclusions. It is often unclear how much the results are driven by model specification and how much the results would change if a different plausible model were used. Computational model robustness analysis addresses this challenge by estimating all possible models from a theoretically informed model space. We use large-scale random noise simulations to show (1) the problem of excess false positive errors under model uncertainty and (2) that computational robustness analysis can identify and eliminate false positives caused by model uncertainty. We also draw on a series of empirical applications to further illustrate issues of model uncertainty and estimate instability. Computational robustness analysis offers a method for relaxing modeling assumptions and improving the transparency of applied research.