Fujian Song

Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses

Abstract Objective: To determine the validity of adjusted indirect comparisons by using data from published meta-analyses of randomised trials. Design: Direct comparison of different interventions in randomised trials and adjusted indirect comparison in which two interventions were compared through their relative effect versus a common comparator. The discrepancy between the direct and adjusted indirect comparison was measured by the difference between the two estimates. Data sources: Database of abstracts of reviews of effectiveness (1994-8), the Cochrane database of systematic reviews, Medline, and references of retrieved articles. Results: 44 published meta-analyses (from 28 systematic reviews) provided sufficient data. In most cases, results of adjusted indirect comparisons were not significantly different from those of direct comparisons. A significant discrepancy (P<0.05) was observed in three of the 44 comparisons between the direct and the adjusted indirect estimates. There was a moderate agreement between the statistical conclusions from the direct and adjusted indirect comparisons (κ 0.51). The direction of discrepancy between the two estimates was inconsistent. Conclusions: Adjusted indirect comparisons usually but not always agree with the results of head to head randomised trials. When there is no or insufficient direct evidence from randomised trials, the adjusted indirect comparison may provide useful or supplementary information on the relative efficacy of competing interventions. The validity of the adjusted indirect comparisons depends on the internal validity and similarity of the included trials. What is already known on this topic Many competing interventions have not been compared in randomised trials Indirect comparison of competing interventions has been carried out in systematic reviews, often implicitly Indirect comparison adjusted by a common control can partially take account of prognostic characteristics of patients in different trials What this study adds Results of adjusted indirect comparison usually, but not always, agree with those of head to head randomised trials The validity of adjusted indirect comparisons depends on the internal validity and similarity of the trials involved

Dissemination and publication of research findings: an updated review of related biases.

OBJECTIVES To identify and appraise empirical studies on publication and related biases published since 1998; to assess methods to deal with publication and related biases; and to examine, in a random sample of published systematic reviews, measures taken to prevent, reduce and detect dissemination bias. DATA SOURCES The main literature search, in August 2008, covered the Cochrane Methodology Register Database, MEDLINE, EMBASE, AMED and CINAHL. In May 2009, PubMed, PsycINFO and OpenSIGLE were also searched. Reference lists of retrieved studies were also examined. REVIEW METHODS In Part I, studies were classified as evidence or method studies and data were extracted according to types of dissemination bias or methods for dealing with it. Evidence from empirical studies was summarised narratively. In Part II, 300 systematic reviews were randomly selected from MEDLINE and the methods used to deal with publication and related biases were assessed. RESULTS Studies with significant or positive results were more likely to be published than those with non-significant or negative results, thereby confirming findings from a previous HTA report. There was convincing evidence that outcome reporting bias exists and has an impact on the pooled summary in systematic reviews. Studies with significant results tended to be published earlier than studies with non-significant results, and empirical evidence suggests that published studies tended to report a greater treatment effect than those from the grey literature. Exclusion of non-English-language studies appeared to result in a high risk of bias in some areas of research such as complementary and alternative medicine. In a few cases, publication and related biases had a potentially detrimental impact on patients or resource use. Publication bias can be prevented before a literature review (e.g. by prospective registration of trials), or detected during a literature review (e.g. by locating unpublished studies, funnel plot and related tests, sensitivity analysis modelling), or its impact can be minimised after a literature review (e.g. by confirmatory large-scale trials, updating the systematic review). The interpretation of funnel plot and related statistical tests, often used to assess publication bias, was often too simplistic and likely misleading. More sophisticated modelling methods have not been widely used. Compared with systematic reviews published in 1996, recent reviews of health-care interventions were more likely to locate and include non-English-language studies and grey literature or unpublished studies, and to test for publication bias. CONCLUSIONS Dissemination of research findings is likely to be a biased process, although the actual impact of such bias depends on specific circumstances. The prospective registration of clinical trials and the endorsement of reporting guidelines may reduce research dissemination bias in clinical research. In systematic reviews, measures can be taken to minimise the impact of dissemination bias by systematically searching for and including relevant studies that are difficult to access. Statistical methods can be useful for sensitivity analyses. Further research is needed to develop methods for qualitatively assessing the risk of publication bias in systematic reviews, and to evaluate the effect of prospective registration of studies, open access policy and improved publication guidelines.

Cholesterol lowering and mortality: the importance of considering initial level of risk.

OBJECTIVE--To investigate the level of risk of death from coronary heart disease above which cholesterol lowering treatment produces net benefits. DESIGN--Meta-analysis of results of randomised controlled trials of cholesterol lowering treatments. METHODS--Published and unpublished data from all identified randomised controlled trials of cholesterol lowering treatments with six months or more follow up and with at least one death were included in the meta-analysis. The analyses were stratified by the rate of death from coronary heart disease in the control arms of the trials. MAIN OUTCOME MEASURES--Death from all causes, from coronary heart disease, and from causes other than coronary heart disease. RESULTS--In the pooled analysis, net benefit in terms of total mortality from cholesterol lowering was seen only for trials including patients at very high initial risk of coronary heart disease (odds ratio 0.74; 95% confidence interval 0.60 to 0.92). In a medium risk group no net effect was seen, and in the low risk group there were adverse treatment effects (1.22; 1.06 to 1.42). In a weighted regression analysis a significant (p < 0.001) trend of increasing benefit with increasing initial risk of coronary heart disease was shown. Raised mortality from causes other than coronary heart disease was seen in trials of drug treatment (1.21; 1.05 to 1.39) but not in the trials of non-drug treatments (1.02; 0.88 to 1.19). Cumulative meta-analysis showed that these results seem to have been stable as new trials appeared. CONCLUSION--Currently evaluated cholesterol lowering drugs seem to produce mortality benefits in only a small proportion of patients at very high risk of death from coronary heart disease. Population cholesterol screening could waste resources and even result in net harm in substantial groups of patients. Overall risk of coronary heart disease should be the main focus of clinical guidelines, and a cautious approach to the use of cholesterol lowering drugs should be advocated. Future trials should aim to clarify the level of risk above which treatment is of net benefit.

Selective serotonin reuptake inhibitors: meta-analysis of efficacy and acceptability.

OBJECTIVE--To examine the evidence for using selective serotonin reuptake inhibitors instead of tricyclic antidepressants in the first line treatment of depression. DESIGN--Meta-analysis of 63 randomised controlled trials comparing the efficacy and acceptability of selective serotonin reuptake inhibitors with those of tricyclic and related antidepressants. MAIN OUTCOME MEASURES--Improvement in mean scores on Hamilton depression rating scale for 53 randomised controlled trials. Pooled drop out rates from the 58 trials which reported drop out by treatment group. RESULTS--Among the 20 studies reporting standard deviation for the Hamilton score no difference was found in efficacy between serotonin reuptake inhibitors and tricyclic and related antidepressants (standardised mean difference 0.004, 95% confidence interval -0.096 to 0.105). The difference remained insignificant when the remaining 33 studies that used the 17 item and 21 item Hamilton score were included by ascribing weighted standard deviations. The odds ratio for drop out rate in patients receiving serotonin reuptake inhibitors compared with those receiving tricyclic antidepressants was 0.95 (0.86 to 1.07). Similar proportions in both groups cited lack of efficacy as the reason for dropping out but slightly more patients in the tricyclic group cited side effects (18.8% v 15.4% in serotonin reuptake group). CONCLUSIONS--Routine use of selective serotonin reuptake inhibitors as the first line treatment of depressive illness may greatly increase cost with only questionable benefit.

Increasing value and reducing waste: addressing inaccessible research

The methods and results of health research are documented in study protocols, full study reports (detailing all analyses), journal reports, and participant-level datasets. However, protocols, full study reports, and participant-level datasets are rarely available, and journal reports are available for only half of all studies and are plagued by selective reporting of methods and results. Furthermore, information provided in study protocols and reports varies in quality and is often incomplete. When full information about studies is inaccessible, billions of dollars in investment are wasted, bias is introduced, and research and care of patients are detrimentally affected. To help to improve this situation at a systemic level, three main actions are warranted. First, academic institutions and funders should reward investigators who fully disseminate their research protocols, reports, and participant-level datasets. Second, standards for the content of protocols and full study reports and for data sharing practices should be rigorously developed and adopted for all types of health research. Finally, journals, funders, sponsors, research ethics committees, regulators, and legislators should endorse and enforce policies supporting study registration and wide availability of journal reports, full study reports, and participant-level datasets.

Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews

Objective To investigate basic assumptions and other methodological problems in the application of indirect comparison in systematic reviews of competing healthcare interventions. Design Survey of published systematic reviews. Inclusion criteria Systematic reviews published between 2000 and 2007 in which an indirect approach had been explicitly used. Data extraction Identified reviews were assessed for comprehensiveness of the literature search, method for indirect comparison, and whether assumptions about similarity and consistency were explicitly mentioned. Results The survey included 88 review reports. In 13 reviews, indirect comparison was informal. Results from different trials were naively compared without using a common control in six reviews. Adjusted indirect comparison was usually done using classic frequentist methods (n=49) or more complex methods (n=18). The key assumption of trial similarity was explicitly mentioned in only 40 of the 88 reviews. The consistency assumption was not explicit in most cases where direct and indirect evidence were compared or combined (18/30). Evidence from head to head comparison trials was not systematically searched for or not included in nine cases. Conclusions Identified methodological problems were an unclear understanding of underlying assumptions, inappropriate search and selection of relevant trials, use of inappropriate or flawed methods, lack of objective and validated methods to assess or improve trial similarity, and inadequate comparison or inappropriate combination of direct and indirect evidence. Adequate understanding of basic assumptions underlying indirect and mixed treatment comparison is crucial to resolve these methodological problems. Appendix 1 PubMed search strategy Appendix 2 Characteristics of identified reports Appendix 3 Identified studies References of included studies

Modelling publication bias in meta-analysis: a review

Meta-analysis is now a widely used technique for summarizing evidence from multiple studies. Publication bias, the bias induced by the fact that research with statistically significant results is potentially more likely to be submitted and published than work with null or non-significant results, poses a thereat to the validity of such analyses. The implication of this is that combining only the identified published studies uncritically may lead to an incorrect, usually over optimistic, conclusion. How publication bias should be addressed when carrying out a meta-analysis is currently a hotly debated subject. While statistical methods to test for its presence are starting be used, they do not address the problem of how to proceed if publication bias is suspected. This paper provides a review of methods, which can be employed as a sensitivity analysis to assess the likely impact of publication bias on a meta-analysis. It is hoped that this will raise awareness of such methods, and promote their use and development, as well as provide an agenda for future research.

Accuracy of outpatient endometrial biopsy in the diagnosis of endometrial cancer: a systematic quantitative review

Objective To determine the accuracy of outpatient endometrial biopsy in diagnosing endometrial cancer in women with abnormal uterine bleeding.

Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions

OBJECTIVE To investigate discrepancies between direct comparison and adjusted indirect comparison in meta-analyses of new versus conventional pharmaceutical interventions. STUDY DESIGN AND SETTING Results of direct comparison were compared with results of adjusted indirect comparison in three meta-analyses of new versus conventional drugs. The three case studies are (1) bupropion versus nicotine replacement therapy for smoking cessation, (2) risperidone versus haloperidol for schizophrenia, and (3) fluoxetine versus imipramine for depressive disorders. RESULTS In all the three cases, effects of new drugs estimated by head-to-head trials tend to be greater than that by adjusted indirect comparisons. The observed discrepancies could not be satisfactorily explained by the play of chance or by bias and heterogeneity in adjusted indirect comparison. This observation, along with analysis of possible systematic bias in the direct comparisons, suggested that the indirect method might have produced less biased results. Simulations found that adjusted indirect comparison may counterbalance bias under certain circumstances. CONCLUSION Adjusted indirect comparison could be used to cross-examine the validity and applicability of results from head-to-head randomized trials. The hypothesis that adjusted indirect comparison may provide less biased results than head-to-head randomized trials needs to be investigated by further research.

Methods for Exploring Heterogeneity in Meta-Analysis:

In meta-analysis, when the difference in results between studies is greater than would be expected by chance, one needs to investigate whether the observed variation in results across studies is associated with clinical and/or methodological differences between studies. This article reviews methods used in meta-analysis for exploring heterogeneity, including statistical tests for homogeneity, methods for visually displaying results of primary studies, methods for reducing heterogeneity, methods for investigating sources of heterogeneity, and identification of moderator variables or effect modifiers. The investigation of sources of heterogeneity in meta-analysis is by nature exploratory, and therefore its results should always be interpreted with caution. However, careful investigation of heterogeneity may provide an important second level of evidence that can be useful in suggesting direction of future research. Sometimes, it may provide clinically important results by indicating who might benefit more or less from a treatment or how an intervention should be applied.