Do pooled estimates from meta-analyses of observational epidemiology studies contribute to causal inference?

Abstract

INTRODUCTION The purpose of metaanalyses that generate pooled risk estimates is generally to inform causal inference, compiling evidence to address an aetiological question. Logically, drawing on information from all relevant studies is likely to be superior to relying on any single study. Metaanalysis applies an objective, quantitative method to integrate evidence across studies. Objectivity in identifying studies, evaluating their relevance, assessing the quality of the methods and extracting results has clear benefits over informal, subjective approaches. It guards against arbitrary selection of studies and allows for replication of at least parts of the review protocol (eg, literature identification). Clearly, metaanalysis has great appeal in the research community. A cursory examination of PubMed searching on ‘epidemiology’ and ‘metaanalysis’ yielded the expected pattern of proliferation—fewer than 100 publications per year prior to 1990, around 400 per year in 2000, 800 per year in 2005, 2000 per year in 2010, 5600 in 2015 and over 6000 per year starting in 2019. In the 1990s, there was intense debate over the merits and demerits of metaanalysis in observational epidemiology, with some arguing for abandoning this approach entirely 2 and others expressing reservations based largely on the heterogeneity of study methods. The role of metaanalysis in causal inference specifically also has been considered. 8 Interestingly, the debate appeared to end over 20 years ago without a clear resolution, yet metaanalysis became the default approach to summarising and evaluating evidence. We suggest that the debate should be reopened and make the case that the negative features of this approach often outweigh its benefits. The primary competitor to generating pooled estimates through metaanalysis is some variant of expert review. Obviously, pooled estimates through metaanalysis and expert reviews may be divergent, deviating in either direction—seemingly clear evidence for an effect based on metaanalysis that is not accounting for important limitations or little evidence in support of an effect from metaanalysis that fails to recognise there are a subset of superior studies that lead to a different conclusion. When the inferences diverge, it is possible that the forced objectivity of the metaanalysis reveals truth that contrasts with the subjective, biased assessment of experts. But the simplification and compromises required to generate pooled estimates may fail to capture important underlying methodological issues that are apparent to expert evaluators. Apparent consistency in results may reflect consistent biases, or there may be a small subset of highly informative studies that are overwhelmed by a large number of weaker ones in a pooled estimate. Examining the impact of study methods on results calls for deep expertise in the subject being evaluated, and metaanalysis is not a reliable substitute for evidence review and synthesis by experts.