T. D. Stanley

Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection*

This study investigates the small‐sample performance of meta‐regression methods for detecting and estimating genuine empirical effects in research literatures tainted by publication selection. Publication selection exists when editors, reviewers or researchers have a preference for statistically significant results. Meta‐regression methods are found to be robust against publication selection. Even if a literature is dominated by large and unknown misspecification biases, precision‐effect testing and joint precision‐effect and meta‐significance testing can provide viable strategies for detecting genuine empirical effects. Publication biases are greatly reduced by combining two biased estimates, the estimated meta‐regression coefficient on precision (1/Se) and the unadjusted‐average effect.

Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study

BackgroundIn meta-analysis, the presence of funnel plot asymmetry is attributed to publication or other small-study effects, which causes larger effects to be observed in the smaller studies. This issue potentially mean inappropriate conclusions are drawn from a meta-analysis. If meta-analysis is to be used to inform decision-making, a reliable way to adjust pooled estimates for potential funnel plot asymmetry is required.MethodsA comprehensive simulation study is presented to assess the performance of different adjustment methods including the novel application of several regression-based methods (which are commonly applied to detect publication bias rather than adjust for it) and the popular Trim & Fill algorithm. Meta-analyses with binary outcomes, analysed on the log odds ratio scale, were simulated by considering scenarios with and without i) publication bias and; ii) heterogeneity. Publication bias was induced through two underlying mechanisms assuming the probability of publication depends on i) the study effect size; or ii) the p-value.ResultsThe performance of all methods tended to worsen as unexplained heterogeneity increased and the number of studies in the meta-analysis decreased. Applying the methods conditional on an initial test for the presence of funnel plot asymmetry generally provided poorer performance than the unconditional use of the adjustment method. Several of the regression based methods consistently outperformed the Trim & Fill estimators.ConclusionRegression-based adjustments for publication bias and other small study effects are easy to conduct and outperformed more established methods over a wide range of simulation scenarios.

Gender Wage Discrimination Bias? A Meta-Regression Analysis

This study provides a quantitative review of the empirical literature on gender wage discrimination. Although there is considerable agreement that gender wage discrimination exists, estimates of its magnitude vary widely. Our meta-regression analysis (MRA) reveals that the estimated gender gap has been steadily declining and the wage rate calculation to be crucial. Large biases are likely when researchers omit experience or fail to correct for selection bias. Finally, there appears to be significant gender bias in gender research. However, it is a virtuous variety where researchers tend to compensate for potential bias implicit in their gender membership.

Meta-regression approximations to reduce publication selection bias

Publication selection bias is a serious challenge to the integrity of all empirical sciences. We derive meta-regression approximations to reduce this bias. Our approach employs Taylor polynomial approximations to the conditional mean of a truncated distribution. A quadratic approximation without a linear term, precision-effect estimate with standard error (PEESE), is shown to have the smallest bias and mean squared error in most cases and to outperform conventional meta-analysis estimators, often by a great deal. Monte Carlo simulations also demonstrate how a new hybrid estimator that conditionally combines PEESE and the Egger regression intercept can provide a practical solution to publication selection bias. PEESE is easily expanded to accommodate systematic heterogeneity along with complex and differential publication selection bias that is related to moderator variables. By providing an intuitive reason for these approximations, we can also explain why the Egger regression works so well and when it does not. These meta-regression methods are applied to several policy-relevant areas of research including antidepressant effectiveness, the value of a statistical life, the minimum wage, and nicotine replacement therapy.

Meta-Analysis Of Economics Research Reporting Guidelines

Meta‐regression analysis (MRA) can provide objective and comprehensive summaries of economics research. Their use has grown rapidly over the last few decades. To improve transparency and to raise the quality of MRA, the meta‐analysis of economics research‐network (MAER‐Net) has created the below reporting guidelines. Future meta‐analyses in economics will be expected to follow these guidelines or give valid reasons why a meta‐analysis must deviate from them.

Declining Bias and Gender Wage Discrimination? A Meta-Regression Analysis

This paper extends, tests, and revises a previous meta-regression analysis of the gender wage gap (Stanley and Jarrell 1998). We find that there remains a strong, though dampened, tendency for discrimination estimates to fall, and male researchers still report significantly larger amounts of wage discrimination against women. This extensive research base, containing 104 estimates, suggests that there is less need to correct for selection bias—an indirect sign of lessened discrimination. There is evidence that gender research is changing and improving. Although gender wage discrimination has lessened, the research base still finds a significant gender wage inequality.

New wine in old bottles: A meta-analysis of Ricardian equivalence

A quantitative review, or meta-analysis, of 28 empirical studies of the Ricardian equivalence theorem (RET) gives persuasive testament of its falsity. Although there is great dissonance concerning the validity of Ricardian equivalence among contributors and reviewers alike, my meta-analysis reveals that the testing record, as a whole, entails a strong empirical rejection of RET. This conclusion is further strengthened by the fact that both a studys degrees of freedom and its proper econometric specification increase the likelihood of rejection. A meta-regression analysis identifies nine study characteristics that help explain the large study-to-study variation found among reported RET test results (R2 = 83%).

Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity

is growing concern and mounting evidence of selectivity in empirical economics. Most empirical economic literatures have a truncated distribution of results. The aim of this paper is to explore the link between publication selectivity and theory contests. This link is confirmed through the analysis of 87 distinct empirical economics literatures, involving more than three and a half thousand separate empirical studies, using objective measures of both selectivity and contests. Our meta–meta‐analysis shows that publication selection is widespread, but not universal. It distorts scientific inference with potentially adverse effects on policy making, but competition and debate between rival theories reduces this selectivity and thereby improves economic inference.

PICTURE THIS: A SIMPLE GRAPH THAT REVEALS MUCH ADO ABOUT RESEARCH

Funnel graphs provide a simple, yet highly effective, means to identify key features of an empirical literature. This paper illustrates the use of funnel graphs to detect publication selection bias, identify the existence of genuine empirical effects and discover potential moderator variables that can help to explain the wide variation routinely found among reported research findings. Applications include union–productivity effects, water price elasticities, common currency-trade effects, minimum-wage employment effects, efficiency wages and the price elasticity of prescription drugs.

Neither fixed nor random: weighted least squares meta‐analysis

This study challenges two core conventional meta-analysis methods: fixed effect and random effects. We show how and explain why an unrestricted weighted least squares estimator is superior to conventional random-effects meta-analysis when there is publication (or small-sample) bias and better than a fixed-effect weighted average if there is heterogeneity. Statistical theory and simulations of effect sizes, log odds ratios and regression coefficients demonstrate that this unrestricted weighted least squares estimator provides satisfactory estimates and confidence intervals that are comparable to random effects when there is no publication (or small-sample) bias and identical to fixed-effect meta-analysis when there is no heterogeneity. When there is publication selection bias, the unrestricted weighted least squares approach dominates random effects; when there is excess heterogeneity, it is clearly superior to fixed-effect meta-analysis. In practical applications, an unrestricted weighted least squares weighted average will often provide superior estimates to both conventional fixed and random effects.