Kristian Thorlund

The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations

Systematic reviews and meta-analyses are fundamental tools for the generation of reliable summaries of health care information for clinicians, decision makers, and patients. Systematic reviews provide information on clinical benefits and harms of interventions, inform the development of clinical recommendations, and help to identify future research needs. In 1999 and 2009, respectively, groups developed the Quality of Reporting of Meta-Analyses (QUOROM) statement (1) and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement (2, 3) to improve the reporting of systematic reviews and meta-analyses. Both statements have been widely used, and coincident with their adoption, the quality of reporting of systematic reviews has improved (4, 5). Systematic reviews and meta-analyses often address the comparative effectiveness of multiple treatment alternatives. Because randomized trials that evaluate the benefits and harms of multiple interventions simultaneously are difficult to perform, comparative effectiveness reviews typically involve many studies that have addressed only a subset of the possible treatment comparisons. Traditionally, meta-analyses have usually compared only 2 interventions at a time, but the need to summarize a comprehensive and coherent set of comparisons based on all of the available evidence has led more recently to synthesis methods that address multiple interventions. These methods are commonly referred to as network meta-analysis, mixed treatment comparisons meta-analysis, or multiple treatments meta-analysis (68). In recent years, there has been a notable increase in the publication of articles using these methods (9). On the basis of our recent overview (10) of reporting challenges in the field, as well as findings from our Delphi exercise involving researchers and journal editors, we believe that reporting guidance for such analyses is sorely needed. In this article, we describe the process of developing specific advice for the reporting of systematic reviews that incorporate network meta-analyses, and we present the guidance generated from this process. Development of the PRISMA Network Meta-analysis Extension Statement We followed an established approach for this work (11). We formed a steering committee (consisting of Drs. Hutton, Salanti, Moher, Caldwell, Chaimani, Schmid, Thorlund, and Altman); garnered input from 17 journal editors, reporting guideline authors, and researchers with extensive experience in systematic reviews and network meta-analysis; and performed an overview of existing reviews of the reporting quality of network meta-analyses to identify candidate elements important to report in network meta-analyses (10). We also implemented an online Delphi survey of authors of network meta-analyses in mid-2013 (215 invited; response rate, 114 [53%]) by using Fluid Surveys online software (Fluidware, Ottawa, Ontario, Canada) to determine consensus items for which either a new checklist item or an elaboration statement would be required, and to identify specific topics requiring further discussion. Next, we held a 1-day, face-to-face meeting to discuss the structure of the extension statement, topics requiring further consideration, and publication strategy. After this meeting, members of the steering committee and some of the meeting participants were invited to contribute specific components for this guidance. All participants reviewed drafts of the report. Scope of This Extension Statement This document provides reporting guidance primarily intended for authors, peer reviewers, and editors. It may also help clinicians, technology assessment practitioners, and patients understand and interpret network meta-analyses. We also aim to help readers develop a greater understanding of core concepts, terminology, and issues associated with network meta-analysis. This document is not intended to be prescriptive about how network meta-analyses should be conducted or interpreted; considerable literature addressing such matters is available (6, 1251). Instead, we seek to provide guidance on important information to be included in reports of systematic reviews that address networks of multiple treatment comparisons. For specific checklist items where we have suggested modification of instructions from the PRISMA statement, we have included examples of potential approaches for reporting different types of information. However, modified approaches to those presented here may also be feasible. How to Use This Document This document describes modifications of checklist items from the original PRISMA statement for systematic reviews incorporating network meta-analyses. It also describes new checklist items that are important for transparent reporting of such reviews. We present an integrated checklist of 32 items, along with elaborations that demonstrate good reporting practice. The elaboration (Appendix,) describes each item and presents examples for new or modified items. Although new items have been added in what was deemed the most logical place in the core PRISMA checklist, we do not prescribe an order in which these must be addressed. The elaboration also includes 5 boxes that highlight methodological considerations for network meta-analysis. The Table presents the PRISMA network analysis checklist that authors may use for tracking inclusion of key elements in reports of network meta-analyses. The checklist has been structured to present core PRISMA items and modifications of these items where needed, as well as new checklist items specific to network meta-analysis. Checklist items are designated New Item in the main text if they address a particular aspect of reporting that is novel to network meta-analyses; these are labeled S1 through S5. The heading Addition indicates discussion of an issue that was covered by the original PRISMA statement but requires additional considerations for reviews incorporating network meta-analyses. Examples with elaborations have been provided for checklist items in these 2 categories. Table.Checklist of Items to Include When Reporting a Systematic Review Involving a Network Meta-analysis What Is a Treatment Network? Systematic reviews comparing the benefits and harms of multiple treatments are more complex than those comparing only 2 treatments. To present their underlying evidence base, reviews involving a network meta-analysis commonly include a graph of the network to summarize the numbers of studies that compared the different treatments and the numbers of patients who have been studied for each treatment (Figure 1). This network graph consists of nodes (points representing the competing interventions) and edges (adjoining lines between the nodes that show which interventions have been compared among the included studies). The sizes of the nodes and the thicknesses of the edges in network graphs typically represent the amounts of respective evidence for specific nodes and comparisons. Sometimes, additional edges are added to distinguish comparisons that may be part of multigroup studies that compare more than 2 treatments. Figure 1. Overview of a network graph. A network graph presenting the evidence base for a hypothetical review of 4 interventions is shown. Treatments are represented by nodes and head-to-head studies between treatments are represented by edges. The sizes of edges and nodes are used to visually depict the available numbers of studies comparing interventions and the numbers of patients studied with each treatment. The graphs also allow readers to note particular features of the shape of a treatment network. This includes the identification of closed loops in the network; a closed loop is present in a treatment network when 3 or more comparators are connected to each other through a polygon, as in Figure 1 for treatments A, B, and C. This shows that treatments A, B, and C have all been compared against each other in existing studies, and thus each comparison in the closed loop (AB, AC, BC) is informed by both direct and indirect evidence (see the Box for definitions of direct and indirect evidence and Figure 2 for a graphical representation of terms in the Box). Box. Terminology: Reviews With Networks of Multiple Treatments Terms are discussed further in the Box. Top. Adjusted indirect treatment comparison of treatments B and C based on studies that used a common comparator, treatment A. Middle. A network of 8 treatments and a common comparator, with a mix of comparisons against the control treatment and a subset of all possible comparisons between active treatments. Bottom. A treatment network similar to that shown in the middle panel, but with study data available for an additional 4 comparisons in the network which form closed loops. Figure 2. Graphical overview of the terminologies that are related to the study of treatment networks. Discussion All phases of the clinical research cycle generate considerable waste, from posing irrelevant questions to inappropriate study methods, bad reporting, and inadequate dissemination of the completed report. Poor reporting is not an esoteric issue. It can introduce biased estimates of an interventions effectiveness and thus affect patient care and decision making. Journals regularly publish new evidence regarding some aspect of inadequate reporting (52). Improving the completeness and transparency of reporting research is a low-hanging fruit to help reduce waste, and possibly explains the rise in developing reporting guidelines (53, 54) and such initiatives as the EQUATOR Network. The PRISMA statement was aimed at improving the reporting of traditional pairwise systematic reviews and meta-analyses; it has been endorsed by hundreds of journals and editorial groups. Some extensions have been developed, including PRISMA for reporting abstracts (55) and equity (56). Other extensions are in various stages of development, including those for individual patientdat

Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses

OBJECTIVESnTo evaluate meta-analyses with trial sequential analysis (TSA). TSA adjusts for random error risk and provides the required number of participants (information size) in a meta-analysis. Meta-analyses not reaching information size are analyzed with trial sequential monitoring boundaries analogous to interim monitoring boundaries in a single trial.nnnSTUDY DESIGN AND SETTINGnWe applied TSA on meta-analyses performed in Cochrane Neonatal reviews. We calculated information sizes and monitoring boundaries with three different anticipated intervention effects of 30% relative risk reduction (TSA(30%)), 15% (TSA(15%)), or a risk reduction suggested by low-bias risk trials of the meta-analysis corrected for heterogeneity (TSA(LBHIS)).nnnRESULTSnA total of 174 meta-analyses were eligible; 79 out of 174 (45%) meta-analyses were statistically significant (P<0.05). In the significant meta-analyses, TSA(30%) showed firm evidence in 61%. TSA(15%) and TSA(LBHIS) found firm evidence in 33% and 73%, respectively. The remaining significant meta-analyses had potentially spurious evidence of effect. In the 95 statistically nonsignificant (P>or=0.05) meta-analyses, TSA(30%) showed absence of evidence in 80% (insufficient information size). TSA(15%) and TSA(LBHIS) found that 95% and 91% had absence of evidence. The remaining nonsignificant meta-analyses had evidence of lack of effect.nnnCONCLUSIONnTSA reveals insufficient information size and potentially false positive results in many meta-analyses.

Estimating required information size by quantifying diversity in random-effects model meta-analyses

BackgroundThere is increasing awareness that meta-analyses require a sufficiently large information size to detect or reject an anticipated intervention effect. The required information size in a meta-analysis may be calculated from an anticipated a priori intervention effect or from an intervention effect suggested by trials with low-risk of bias.MethodsInformation size calculations need to consider the total model variance in a meta-analysis to control type I and type II errors. Here, we derive an adjusting factor for the required information size under any random-effects model meta-analysis.ResultsWe devise a measure of diversity (D2) in a meta-analysis, which is the relative variance reduction when the meta-analysis model is changed from a random-effects into a fixed-effect model. D2 is the percentage that the between-trial variability constitutes of the sum of the between-trial variability and a sampling error estimate considering the required information size. D2 is different from the intuitively obvious adjusting factor based on the common quantification of heterogeneity, the inconsistency (I2), which may underestimate the required information size. Thus, D2 and I2 are compared and interpreted using several simulations and clinical examples. In addition we show mathematically that diversity is equal to or greater than inconsistency, that is D2 ≥ I2, for all meta-analyses.ConclusionWe conclude that D2 seems a better alternative than I2 to consider model variation in any random-effects meta-analysis despite the choice of the between trial variance estimator that constitutes the model. Furthermore, D2 can readily adjust the required information size in any random-effects model meta-analysis.

Apparently conclusive meta-analyses may be inconclusive—Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses

BACKGROUNDnRandom error may cause misleading evidence in meta-analyses. The required number of participants in a meta-analysis (i.e. information size) should be at least as large as an adequately powered single trial. Trial sequential analysis (TSA) may reduce risk of random errors due to repetitive testing of accumulating data by evaluating meta-analyses not reaching the information size with monitoring boundaries. This is analogous to sequential monitoring boundaries in a single trial.nnnMETHODSnWe selected apparently conclusive (P </= 0.05) Cochrane neonatal meta-analyses. We applied heterogeneity-adjusted and unadjusted TSA on these meta-analyses by calculating the information size, the monitoring boundaries, and the cumulative Z-statistic after each trial. We identified the proportion of meta-analyses that did not reach the required information size and the proportion of these meta-analyses in which the Z-curve did not cross the monitoring boundaries.nnnRESULTSnOf 54 apparently conclusive meta-analyses, 39 (72%) did not reach the heterogeneity-adjusted information size required to accept or reject an intervention effect of 25% relative risk reduction. Of these 39, 19 meta-analyses (49%) were considered inconclusive, because the cumulative Z-curve did not cross the monitoring boundaries. The median number of participants required to reach the required information size was 1591 (range, 339-6149). TSA without heterogeneity adjustment largely confirmed these results.nnnCONCLUSIONSnMany apparently conclusive Cochrane neonatal meta-analyses may become inconclusive when the statistical analyses take into account the risk of random error due to repetitive testing.

Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?

BACKGROUNDnResults from apparently conclusive meta-analyses may be false. A limited number of events from a few small trials and the associated random error may be under-recognized sources of spurious findings. The information size (IS, i.e. number of participants) required for a reliable and conclusive meta-analysis should be no less rigorous than the sample size of a single, optimally powered randomized clinical trial. If a meta-analysis is conducted before a sufficient IS is reached, it should be evaluated in a manner that accounts for the increased risk that the result might represent a chance finding (i.e. applying trial sequential monitoring boundaries).nnnMETHODSnWe analysed 33 meta-analyses with a sufficient IS to detect a treatment effect of 15% relative risk reduction (RRR). We successively monitored the results of the meta-analyses by generating interim cumulative meta-analyses after each included trial and evaluated their results using a conventional statistical criterion (alpha = 0.05) and two-sided Lan-DeMets monitoring boundaries. We examined the proportion of false positive results and important inaccuracies in estimates of treatment effects that resulted from the two approaches.nnnRESULTSnUsing the random-effects model and final data, 12 of the meta-analyses yielded P > alpha = 0.05, and 21 yielded P </= alpha = 0.05. False positive interim results were observed in 3 out of 12 meta-analyses with P > alpha = 0.05. The monitoring boundaries eliminated all false positives. Important inaccuracies in estimates were observed in 6 out of 21 meta-analyses using the conventional P </= alpha = 0.05 and 0 out of 21 using the monitoring boundaries.nnnCONCLUSIONSnEvaluating statistical inference with trial sequential monitoring boundaries when meta-analyses fall short of a required IS may reduce the risk of false positive results and important inaccurate effect estimates.

Peginterferon alpha‐2a is associated with higher sustained virological response than peginterferon alfa‐2b in chronic hepatitis C: Systematic review of randomized trials

A combination of weekly pegylated interferon (peginterferon) alpha and daily ribavirin represents the standard of care for the treatment of chronic hepatitis C according to current guidelines. It is not established which of the two licensed products (peginterferon alpha‐2a or peginterferon alfa‐2b) is most effective. We performed a systematic review of head‐to‐head randomized trials to assess the benefits and harms of the two treatments. We searched the Cochrane Central Register of Controlled Trials, MEDLINE, EMBASE, and LILACS through July 2009. Using standardized forms, two reviewers independently extracted data from each eligible trial report. We statistically combined data using a random effects meta‐analysis according to the intention‐to‐treat principle. We identified 12 randomized clinical trials, including 5,008 patients, that compared peginterferon alpha‐2a plus ribavirin versus peginterferon alfa‐2b plus ribavirin. Overall, peginterferon alpha‐2a significantly increased the number of patients who achieved a sustained virological response (SVR) versus peginterferon alfa‐2b (47% versus 41%; risk ratio 1.11, 95% confidence interval 1.04–1.19; P = 0.004 [eight trials]). Subgroup analyses of risk of bias, viral genotype, and treatment history yielded similar results. The meta‐analysis of adverse events leading to treatment discontinuation included 11 trials and revealed no significant differences between the two peginterferons. Conclusion: Current evidence suggests that peginterferon alpha‐2a is associated with higher SVR than peginterferon alfa‐2b. However, the paucity of evidence on adverse events curbs the decision to definitively recommend one peginterferon over the other, because any potential benefit must outweigh the risk of harm. (HEPATOLOGY 2010.)

Systematic review: glucocorticosteroids for alcoholic hepatitis – a Cochrane Hepato-Biliary Group systematic review with meta-analyses and trial sequential analyses of randomized clinical trials

Backgroundu2002 Glucocorticosteroids versus placebo or no intervention for patients with alcoholic hepatitis have been evaluated for more than 35 years. However, the results of randomized trials and meta‐analyses differ substantially.

Clinical benefit of steroid use in patients undergoing cardiopulmonary bypass: a meta-analysis of randomized trials

We sought to establish the efficacy and safety of prophylactic steroids in adult patients undergoing cardiopulmonary bypass (CPB). We performed a meta-analysis of randomized trials reporting the effects of prophylactic steroids on clinical outcomes after CPB. Outcomes examined were mortality, myocardial infarction, neurological events, new onset atrial fibrillation, transfusion requirements, postoperative bleeding, duration of ventilation, intensive care unit (ICU) stay, hospital stay, wound complications, gastrointestinal complications, and infectious complications. We included 44 trials randomizing 3205 patients. Steroids reduced new onset atrial fibrillation [relative risk (RR) 0.71, 95% confidence interval (CI) 0.59 to 0.87], postoperative bleeding [weighted mean difference (WMD) -99.6 mL, 95% CI -149.8 to -49.3], and duration of ICU stay (WMD -0.23 days, 95% CI -0.40 to -0.07). Length of hospital stay was also reduced (WMD -0.59 days, 95% CI -1.17 to -0.02), but this result was less robust. A trend towards reduction in mortality was observed (RR 0.73, 95% CI 0.45 to 1.18). Randomized trials suggest that perioperative steroids have significant clinical benefit in CPB patients by decreasing the risk of new onset atrial fibrillation, while results are encouraging for reducing bleeding, length of stay, and mortality. These data do not raise major safety concerns, however, a sufficiently powered trial is warranted to confirm or refute these findings.

Attention should be given to multiplicity issues in systematic reviews.

OBJECTIVEnThe objective of this paper is to describe the problem of multiple comparisons in systematic reviews and to provide some guidelines on how to deal with it in practice.nnnSTUDY DESIGN AND SETTINGnWe describe common reasons for multiplicity in systematic reviews, and present some examples. We provide guidance on how to deal with multiplicity when it is unavoidable.nnnRESULTSnWe identified six common reasons for multiplicity in systematic reviews: multiple outcomes, multiple groups, multiple time points, multiple effect measures, subgroup analyses, and multiple looks at accumulating data. The existing methods to deal with multiplicity in single trials can not always be applied in systematic reviews.nnnCONCLUSIONnThere is no simple and completely satisfactory solution to the problem of multiple comparisons in systematic reviews. More research is required to develop multiple comparison procedures for use in systematic reviews. Authors and consumers of systematic reviews should give serious attention to multiplicity in systematic reviews when presenting, interpreting and using the results of these reports.

Cryotherapy for hepatocellular carcinoma

BACKGROUNDnHepatocellular carcinoma is the most common primary malignant cancer of the liver. Evidence for the role of cryotherapy in the treatment of hepatocellular carcinoma is controversial.nnnOBJECTIVESnThe aim of this review is to evaluate the potential benefits and harms of cryotherapy for the treatment of hepatocellular carcinoma.nnnSEARCH STRATEGYnWe searched The Cochrane Hepato-Biliary Group Controlled Trials Register, the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library, MEDLINE, EMBASE, and LILACS until June 2009. We identified further studies by searching national and topic-specific databases, bibliographies, conference abstracts, journals, and grey literature. Furthermore, we reviewed the reference lists and contacted the principal authors of the identified studies.nnnSELECTION CRITERIAnRandomised clinical trials (irrespective of language or publication status) comparing cryotherapy with or without co-intervention(s) to placebo, no treatment, or other control interventions were considered for the review. Due to the absence of randomised clinical trials, we searched for quasi-randomised studies as well as prospective cohort studies and retrospective cohort studies.nnnDATA COLLECTION AND ANALYSISnTwo authors independently identified and assessed studies for their fulfilment of the inclusion criteria. Dichotomous data were expressed as risk ratio (RR) with 95% confidence intervals (CI). We performed the review following the recommendations of The Cochrane Collaboration.nnnMAIN RESULTSnWe were unable to identify any randomised clinical trials. We were also unable to identify quasi-randomised trials. Instead, we identified two prospective cohort studies and two retrospective cohort studies. However, only one of these studies could be included for the assessment of benefit as the study results were stratified according to both the type of hepatic malignancy (primary or secondary) and the intervention group. This retrospective study compared percutaneous cryotherapy with percutaneous radiofrequency. The remaining studies were excluded for the analyses of benefit but included for the assessment of harm. Both severe and non-severe adverse events were reported, but the true nature and extent of harm was difficult to asses.nnnAUTHORS CONCLUSIONSnAt present, there is no evidence to recommend or refute cryotherapy for patients with hepatocellular carcinoma. Randomised clinical trials with low-risk of bias may help in defining the role of cryotherapy in the treatment of hepatocellular carcinoma.