Aris Spanos

Severe Testing as a Basic Concept in a Neyman-Pearson Philosophy of Induction

Despite the widespread use of key concepts of the Neyman–Pearson (N–P) statistical paradigm—type I and II errors, significance levels, power, confidence levels—they have been the subject of philosophical controversy and debate for over 60 years. Both current and long-standing problems of N–P tests stem from unclarity and confusion, even among N–P adherents, as to how a tests (pre-data) error probabilities are to be used for (post-data) inductive inference as opposed to inductive behavior. We argue that the relevance of error probabilities is to ensure that only statistical hypotheses that have passed severe or probative tests are inferred from the data. The severity criterion supplies a meta-statistical principle for evaluating proposed statistical inferences, avoiding classic fallacies from tests that are overly sensitive, as well as those not sensitive enough to particular errors and discrepancies. 1. Introduction and overview 1.1Behavioristic and inferential rationales for Neyman–Pearson (N–P) tests 1.2Severity rationale: induction as severe testing 1.3Severity as a meta-statistical concept: three required restrictions on the N–P paradigm2. Error statistical tests from the severity perspective 2.1N–P test T(α): type I, II error probabilities and power 2.2Specifying test T(α) using p-values3. Neymans post-data use of power 3.1Neyman: does failure to reject H warrant confirming H?4. Severe testing as a basic concept for an adequate post-data inference 4.1The severity interpretation of acceptance (SIA) for test T(α) 4.2The fallacy of acceptance (i.e., an insignificant difference): Ms Rosy 4.3Severity and power5. Fallacy of rejection: statistical vs. substantive significance 5.1Taking a rejection of H0 as evidence for a substantive claim or theory 5.2A statistically significant difference from H0 may fail to indicate a substantively important magnitude 5.3Principle for the severity interpretation of a rejection (SIR) 5.4Comparing significant results with different sample sizes in T(α): large n problem 5.5General testing rules for T(α), using the severe testing concept6. The severe testing concept and confidence intervals 6.1Dualities between one and two-sided intervals and tests 6.2Avoiding shortcomings of confidence intervals7. Beyond the N–P paradigm: pure significance, and misspecification tests8. Concluding comments: have we shown severity to be a basic concept in a N–P philosophy of induction? Introduction and overview 1.1Behavioristic and inferential rationales for Neyman–Pearson (N–P) tests 1.2Severity rationale: induction as severe testing 1.3Severity as a meta-statistical concept: three required restrictions on the N–P paradigm 1.1Behavioristic and inferential rationales for Neyman–Pearson (N–P) tests 1.2Severity rationale: induction as severe testing 1.3Severity as a meta-statistical concept: three required restrictions on the N–P paradigm Error statistical tests from the severity perspective 2.1N–P test T(α): type I, II error probabilities and power 2.2Specifying test T(α) using p-values 2.1N–P test T(α): type I, II error probabilities and power 2.2Specifying test T(α) using p-values Neymans post-data use of power 3.1Neyman: does failure to reject H warrant confirming H? 3.1Neyman: does failure to reject H warrant confirming H? Severe testing as a basic concept for an adequate post-data inference 4.1The severity interpretation of acceptance (SIA) for test T(α) 4.2The fallacy of acceptance (i.e., an insignificant difference): Ms Rosy 4.3Severity and power 4.1The severity interpretation of acceptance (SIA) for test T(α) 4.2The fallacy of acceptance (i.e., an insignificant difference): Ms Rosy 4.3Severity and power Fallacy of rejection: statistical vs. substantive significance 5.1Taking a rejection of H0 as evidence for a substantive claim or theory 5.2A statistically significant difference from H0 may fail to indicate a substantively important magnitude 5.3Principle for the severity interpretation of a rejection (SIR) 5.4Comparing significant results with different sample sizes in T(α): large n problem 5.5General testing rules for T(α), using the severe testing concept 5.1Taking a rejection of H0 as evidence for a substantive claim or theory 5.2A statistically significant difference from H0 may fail to indicate a substantively important magnitude 5.3Principle for the severity interpretation of a rejection (SIR) 5.4Comparing significant results with different sample sizes in T(α): large n problem 5.5General testing rules for T(α), using the severe testing concept The severe testing concept and confidence intervals 6.1Dualities between one and two-sided intervals and tests 6.2Avoiding shortcomings of confidence intervals 6.1Dualities between one and two-sided intervals and tests 6.2Avoiding shortcomings of confidence intervals Beyond the N–P paradigm: pure significance, and misspecification tests Concluding comments: have we shown severity to be a basic concept in a N–P philosophy of induction?

On theory testing in econometrics : Modeling with nonexperimental data

Abstract The paper has two main objectives. The first is to trace the roots of the traditional textbook approach to econometric modeling to two older statistical traditions, Fishers experimental design paradigm, and Gausss ‘theory of errors’, and argue that this approach is ill-suited for nonexperimental (observational) data because these older traditions are appropriate for modeling observable phenomena which can be viewed as generated by ‘nearly isolated’ systems. The second aim is to argue in favor of an alternative approach, which can be traced back to the Galton-Pearson biometric tradition, that takes account of the nonexperimental nature of economic data. It is argued that this approach often leads to more reliable empirical evidence and hence to a better evaluation of economic theories. The (weak) Efficient Market Hypothesis is used throughout to illustrate some of the arguments in the paper.

Revisiting data mining: ‘hunting’ with or without a license

The primary objective of this paper is to revisit a number of empirical modelling activities which are often characterized as data mining, in an attempt to distinguish between the problematic and the non-problematic cases. The key for this distinction is provided by the notion of error-statistical severity. It is argued that many unwarranted data mining activities often arise because of inherent weaknesses in the Traditional Textbook (TT) methodology. Using the Probabilistic Reduction (PR) approach to empirical modelling, it is argued that the unwarranted cases of data mining can often be avoided by dealing directly with the weaknesses of the TT approach. Moreover, certain empirical modelling activities, such as diagnostic testing and data snooping, constitute legitimate procedures in the context of the PR approach.

On Modeling Heteroskedasticity: The Student's t and Elliptical Linear Regression Models

This paper proposes a new approach to modeling heteroskedastidty which enables the modeler to utilize information conveyed by data plots in making informed decisions on the form and structure of heteroskedasticity. It extends the well-known normal/linear/homoskedastic models to a family of non-normal/linear/heteroskedastic models. The non-normality is kept within the bounds of the elliptically symmetric family of multivariate distributions (and in particular the Students t distribution) that lead to several forms of heteroskedasticity, including quadratic and exponential functions of the conditioning variables. The choice of the latter family is motivated by the fact that it enables us to model some of the main sources of heteroskedasticity: “thicktails,” individual heterogeneity, and nonlinear dependence. A common feature of the proposed class of regression models is that the weak exogeneity assumption is inappropriate. The estimation of these models, without the weak exogeneity assumption, is discussed, and the results are illustrated by using cross-section data on charitable contributions.

The simultaneous-equations model revisited: Statistical adequacy and identification

Abstract The paper proposes a reinterpretation of the simultaneous-equations model in order to allow for the nature and structure of the observed data to play a role in the empirical analysis of structural models without extenuating the importance of the theory. This suggests a revised view of the identification problem separating it into statistical and structural identification re.lative to a given data. The proposed re-interpretation provides a framework wherein structural modeling and the data-based specifications such as the VAR and the dynamic data-based reduced forms are compliments and not substitutes.

Methodology in Practice: Statistical Misspecification Testing

The growing availability of computer power and statistical software has greatly increased the ease with which practitioners apply statistical methods, but this has not been accompanied by attention to checking the assumptions on which these methods are based. At the same time, disagreements about inferences based on statistical research frequently revolve around whether the assumptions are actually met in the studies available, e.g., in psychology, ecology, biology, risk assessment. Philosophical scrutiny can help disentangle ‘practical’ problems of model validation, and conversely, a methodology of statistical model validation can shed light on a number of issues of interest to philosophers of science.

On normality and the linear regression model

The purpose of this note is to discuss the role of normality in the context of linear[zddot]homoskedastic regression models. A new characterization result, relating the joint normal distribution and the linear, homoskedastic regression, sheds some light on the role of normality in this context. It is shown that if the assumptions of (a) linearity and (b) homoskedasticity, are supplemented with the assumption of (c) linearity of the reverse regression, assumptions (a)-(c) are tantamount to assuming joint normality of the regressors and regressands, not just conditional normality.

On Rereading Haavelmo: A Retrospective View of Econometric Modeling

The main aim of the paper is to reevaluate the methodological contributions of Tinbergen and Haavelmo in the context of the current discussions on econometric modeling and propose a reformulation of the Haavelmo methodology. The paper argues that the textbook methodology constitutes a less flexible version of Tinbergens approach and apart from the probabilistic language, it has little in common with the methodology in Haavelmos 1944 monograph, commonly acknowledged as having founded modern econometrics. The methodology in this monograph includes several important elements which have either been discarded or never fully integrated within the textbook approach. A re-synthesis of these elements gives rise to an alternative methodological framework. This framework can be used to meet most of the objections to the textbook methodology and provides a framework in the context of which the recent methodological controversies can be evaluated.

Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability, and the Objectivity and Rationality of Science

Part I. Introduction and Background: 1. Philosophy of methodological practice Deborah Mayo 2. Error statistical philosophy Deborah Mayo and Aris Spanos Part II: 3. Severe testing, error statistics, and the growth of theoretical knowledge Deborah Mayo Part III: 4. Can scientific theories be warranted? Alan Chalmers 5. Can scientific theories be warranted with severity? Exchanges with Alan Chalmers Deborah Mayo Part IV: 6. Critical rationalism, explanation and severe tests Alan Musgrave 7. Towards progressive critical rationalism: exchanges with Alan Musgrave Deborah Mayo Part V: 8. Error, tests and theory-confirmation John Worrall 9. Has Worrall saved his theory (on ad hoc saves) in a non ad hoc manner? Exchanges with Worrall Deborah Mayo Part VI: 10. Mills sins, or Mayos errors? Peter Achinstein 11. Sins of the Bayesian epistemologist: exchanges with Achinstein Deborah Mayo Part VII: 12. Theory testing in economics and the error statistical perspective Aris Spanos Part VIII: 13. Frequentist statistics as a theory of inductive inference Deborah Mayo and David Cox 14. Objectivity and conditionality in Frequentist inference David Cox and Deborah Mayo 15. An error in the argument from WCP and S to the SLP Deborah Mayo 16. On a new philosophy of Frequentist inference: exchanges with Cox and Mayo Aris Spanos Part IX: 17. Explanation and truth Clark Glymour 18. Explanation and testing: exchanges with Glymour Deborah Mayo 19. Graphical causal modeling and error statistics: exchanges with Glymour Aris Spanos Part X: 20. Legal epistemology: the anomaly of affirmative defenses Larry Laudan 21. Error and the law: exchanges with Laudan Deborah Mayo.

Statistical Adequacy and the Testing of Trend Versus Difference Stationarity

Abstract The debate on whether macroeconomic series are trend or difference stationary, initiated by Nelson and Plosser [Nelson, C. R.; Plosser, C. I. (1982). Trends and random walks in macroeconomic time series: some evidence and implications. Journal of Monetary Economics10:139–162] remains unresolved. The main objective of the paper is to contribute toward a resolution of this issue by bringing into the discussion the problem of statistical adequacy. The paper revisits the empirical results of Nelson and Plosser [Nelson, C. R.; Plosser, C. I. (1982). Trends and random walks in macroeconomic time series: some evidence and implications. Journal of Monetary Economics10:139–162] and Perron [Perron, P. (1989). The great crash, the oil price shock, and the unit root hypothesis. Econometrica57:1361–1401] and shows that several of their estimated models are misspecified. Respecification with a view to ensuring statistical adequacy gives rise to heteroskedastic AR(k) models for some of the price series. Based on estimated models which are statistically adequate, the main conclusion of the paper is that the majority of the data series are trend stationary.