Sarah Donegan

Indirect Comparisons: A Review of Reporting and Methodological Quality

Background The indirect comparison of two interventions can be valuable in many situations. However, the quality of an indirect comparison will depend on several factors including the chosen methodology and validity of underlying assumptions. Published indirect comparisons are increasingly more common in the medical literature, but as yet, there are no published recommendations of how they should be reported. Our aim is to systematically review the quality of published indirect comparisons to add to existing empirical data suggesting that improvements can be made when reporting and applying indirect comparisons. Methodology/Findings Reviews applying statistical methods to indirectly compare the clinical effectiveness of two interventions using randomised controlled trials were eligible. We searched (1966–2008) Database of Abstracts and Reviews of Effects, The Cochrane library, and Medline. Full review publications were assessed for eligibility. Specific criteria to assess quality were developed and applied. Forty-three reviews were included. Adequate methodology was used to calculate the indirect comparison in 41 reviews. Nineteen reviews assessed the similarity assumption using sensitivity analysis, subgroup analysis, or meta-regression. Eleven reviews compared trial-level characteristics. Twenty-four reviews assessed statistical homogeneity. Twelve reviews investigated causes of heterogeneity. Seventeen reviews included direct and indirect evidence for the same comparison; six reviews assessed consistency. One review combined both evidence types. Twenty-five reviews urged caution in interpretation of results, and 24 reviews indicated when results were from indirect evidence by stating this term with the result. Conclusions This review shows that the underlying assumptions are not routinely explored or reported when undertaking indirect comparisons. We recommend, therefore, that the quality of indirect comparisons should be improved, in particular, by assessing assumptions and reporting the assessment methods applied. We propose that the quality criteria applied in this article may provide a basis to help review authors carry out indirect comparisons and to aid appropriate interpretation.

Is Dengue Vector Control Deficient in Effectiveness or Evidence?: Systematic Review and Meta-analysis

Background Although a vaccine could be available as early as 2016, vector control remains the primary approach used to prevent dengue, the most common and widespread arbovirus of humans worldwide. We reviewed the evidence for effectiveness of vector control methods in reducing its transmission. Methodology/Principal Findings Studies of any design published since 1980 were included if they evaluated method(s) targeting Aedes aegypti or Ae. albopictus for at least 3 months. Primary outcome was dengue incidence. Following Cochrane and PRISMA Group guidelines, database searches yielded 960 reports, and 41 were eligible for inclusion, with 19 providing data for meta-analysis. Study duration ranged from 5 months to 10 years. Studies evaluating multiple tools/approaches (23 records) were more common than single methods, while environmental management was the most common method (19 studies). Only 9/41 reports were randomized controlled trials (RCTs). Two out of 19 studies evaluating dengue incidence were RCTs, and neither reported any statistically significant impact. No RCTs evaluated effectiveness of insecticide space-spraying (fogging) against dengue. Based on meta-analyses, house screening significantly reduced dengue risk, OR 0.22 (95% CI 0.05–0.93, p = 0.04), as did combining community-based environmental management and water container covers, OR 0.22 (95% CI 0.15–0.32, p<0.0001). Indoor residual spraying (IRS) did not impact significantly on infection risk (OR 0.67; 95% CI 0.22–2.11; p = 0.50). Skin repellents, insecticide-treated bed nets or traps had no effect (p>0.5), but insecticide aerosols (OR 2.03; 95% CI 1.44–2.86) and mosquito coils (OR 1.44; 95% CI 1.09–1.91) were associated with higher dengue risk (p = 0.01). Although 23/41 studies examined the impact of insecticide-based tools, only 9 evaluated the insecticide susceptibility status of the target vector population during the study. Conclusions/Significance This review and meta-analysis demonstrate the remarkable paucity of reliable evidence for the effectiveness of any dengue vector control method. Standardised studies of higher quality to evaluate and compare methods must be prioritised to optimise cost-effective dengue prevention.

Assessing key assumptions of network meta-analysis: a review of methods

BACKGROUND Homogeneity and consistency assumptions underlie network meta-analysis (NMA). Methods exist to assess the assumptions but they are rarely and poorly applied. We review and illustrate methods to assess homogeneity and consistency. METHODS Eligible articles focussed on indirect comparison or NMA methodology. Articles were sought by hand-searching and scanning references (March 2013). Assumption assessment methods described in the articles were reviewed, and applied to compare anti-malarial drugs. RESULTS 116 articles were included. Methods to assess homogeneity were: comparing characteristics across trials; comparing trial-specific treatment effects; using hypothesis tests or statistical measures; applying fixed-effect and random-effects pair-wise meta-analysis; and investigating treatment effect-modifiers. Methods to assess consistency were: comparing characteristics; investigating treatment effect-modifiers; comparing outcome measurements in the referent group; node-splitting; inconsistency modelling; hypothesis tests; back transformation; multidimensional scaling; a two-stage approach; and a graph-theoretical method. For the malaria example, heterogeneity existed for some comparisons that was unexplained by investigating treatment effect-modifiers. Inconsistency was detected using node-splitting and inconsistency modelling. It was unclear whether the covariates explained the inconsistency. CONCLUSIONS Presently, we advocate applying existing assessment methods collectively to gain the best understanding possible regarding whether assumptions are reasonable. In our example, consistency was questionable; therefore the NMA results may be unreliable.

Combining individual patient data and aggregate data in mixed treatment comparison meta‐analysis: Individual patient data may be beneficial if only for a subset of trials

BACKGROUND Individual patient data (IPD) meta-analysis is the gold standard. Aggregate data (AD) and IPD can be combined using conventional pairwise meta-analysis when IPD cannot be obtained for all relevant studies. We extend the methodology to combine IPD and AD in a mixed treatment comparison (MTC) meta-analysis. METHODS The proposed random-effects MTC models combine IPD and AD for a dichotomous outcome. We study the benefits of acquiring IPD for a subset of trials when assessing the underlying consistency assumption by including treatment-by-covariate interactions in the model. We describe three different model specifications that make increasingly stronger assumptions regarding the interactions. We illustrate the methodology through application to real data sets to compare drugs for treating malaria by using the outcome unadjusted treatment success at day 28. We compare results from AD alone, IPD alone and all data. RESULTS When IPD contributed (i.e. either using IPD alone or combining IPD and AD), the chains converged, and we identified statistically significant regression coefficients for the interactions. Using IPD alone, we were able to compare only three of the six treatments of interest. When models were fitted to AD, the treatment effects and regression coefficients for the interactions were far more imprecise, and the chains did not converge. CONCLUSIONS The models combining IPD and AD encapsulated all available evidence. When exploring interactions, it can be beneficial to obtain IPD for a subset of trials and to combine IPD with additional AD.

The diagnostic accuracy of the GenoType® MTBDRsl assay for the detection of resistance to second-line anti-tuberculosis drugs

Background Accurate and rapid tests for tuberculosis (TB) drug resistance are critical for improving patient care and decreasing the transmission of drug-resistant TB. Genotype®MTBDRsl (MTBDRsl) is the only commercially-available molecular test for detecting resistance in TB to the fluoroquinolones (FQs; ofloxacin, moxifloxacin and levofloxacin) and the second-line injectable drugs (SLIDs; amikacin, kanamycin and capreomycin), which are used to treat patients with multidrug-resistant (MDR-)TB. Objectives To obtain summary estimates of the diagnostic accuracy ofMTBDRsl for FQ resistance, SLID resistance and extensively drug-resistant TB (XDR-TB; defined asMDR-TB plus resistance to a FQand a SLID) when performed (1) indirectly (ie on culture isolates confirmed as TB positive) and (2) directly (ie on smear-positive sputum specimens). To compare summary estimates of the diagnostic accuracy of MTBDRsl for FQ resistance, SLID resistance and XDR-TB by type of testing (indirect versus direct testing). The populations of interest were adults with drug-susceptible TB or drug-resistant TB. The settings of interest were intermediate and central laboratories. Search methods We searched the following databases without any language restriction up to 30 January 2014: Cochrane Infectious Diseases Group Specialized Register; MEDLINE; EMBASE; ISI Web of Knowledge; MEDION; LILACS; BIOSIS; SCOPUS; the metaRegister of Controlled Trials; the search portal of the World Health Organization International Clinical Trials Registry Platform; and ProQuest Dissertations & Theses A&I. Selection criteria We included all studies that determined MTBDRsl accuracy against a defined reference standard (culture-based drug susceptibility testing (DST), genetic testing or both).We included cross-sectional and diagnostic case-control studies.We excluded unpublished data and conference proceedings. Data collection and analysis For each study, two review authors independently extracted data using a standardized form and assessed study quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. We performed meta-analyses to estimate the pooled sensitivity and specificity of MTBDRsl for FQ resistance, SLID resistance, and XDR-TB. We explored the influence of different reference standards. We performed the majority of analyses using a bivariate random-effects model against culture-based DST as the reference standard. Main results We included 21 unique studies: 14 studies reported the accuracy of MTBDRsl when done directly, five studies when done indirectly and two studies that did both. Of the 21 studies, 15 studies (71%) were cross-sectional and 11 studies (58%) were located in lowincome or middle-income countries. All studies but two were written in English. Nine (43%) of the 21 included studies had a high risk of bias for patient selection. At least half of the studies had low risk of bias for the other QUADAS-2 domains. As a test for FQ resistance measured against culture-based DST, the pooled sensitivity of MTBDRsl when performed indirectly was 83.1% (95% confidence interval (CI) 78.7% to 86.7%) and the pooled specificity was 97.7% (95% CI 94.3% to 99.1%), respectively (16 studies, 1766 participants; 610 confirmed cases of FQ-resistant TB; moderate quality evidence).When performed directly, the pooled sensitivity was 85.1% (95% CI 71.9% to 92.7%) and the pooled specificity was 98.2% (95% CI 96.8% to 99.0%), respectively (seven studies, 1033 participants; 230 confirmed cases of FQ-resistant TB; moderate quality evidence). For indirect testing for FQ resistance, four (0.2%) of 1766MTBDRsl results were indeterminate, whereas for direct testing 20 (1.9%) of 1033 wereMTBDRsl indeterminate (P < 0.001). As a test for SLID resistance measured against culture-based DST, the pooled sensitivity of MTBDRsl when performed indirectly was 76.9% (95% CI 61.1% to 87.6%) and the pooled specificity was 99.5% (95% CI 97.1% to 99.9%), respectively (14 studies, 1637 participants; 414 confirmed cases of SLID-resistant TB; moderate quality evidence). For amikacin resistance, the pooled sensitivity and specificity were 87.9% (95% CI 82.1% to 92.0%) and 99.5% (95% CI 97.5% to 99.9%), respectively. For kanamycin resistance, the pooled sensitivity and specificity were 66.9% (95% CI 44.1% to 83.8%) and 98.6% (95% CI 96.1% to 99.5%), respectively. for capreomycin resistance, the pooled sensitivity and specificity were 79.5% (95% CI 58.3% to 91.4%) and 95.8% (95% CI 93.4% to 97.3%), respectively. When performed directly, the pooled sensitivity for SLID resistance was 94.4% (95% CI 25.2% to 99.9%) and the pooled specificity was 98.2% (95% CI 88.9% to 99.7%), respectively (six studies, 947 participants; 207 confirmed cases of SLIDresistant TB, 740 SLID susceptible cases of TB; very low quality evidence). For indirect testing for SLID resistance, three (0.4%) of 774 MTBDRsl results were indeterminate, whereas for direct testing 53 (6.1%) of 873 were MTBDRsl indeterminate (P < 0.001). As a test for XDR-TB measured against culture-based DST, the pooled sensitivity of MTBDRsl when performed indirectly was 70.9% (95%CI 42.9%to 88.8%) and the pooled specificitywas 98.8%(95%CI 96.1%to 99.6%), respectively (eight studies, 880 participants; 173 confirmed cases of XDR-TB; low quality evidence). Authors’ conclusions In adults with TB, a positiveMTBDRsl result for FQ resistance, SLID resistance, or XDR-TB can be treated with confidence. However, MTBDRsl does not detect approximately one in five cases of FQ-resistant TB, and does not detect approximately one in four cases of SLID-resistant TB. Of the three SLIDs, MTBDRsl has the poorest sensitivity for kanamycin resistance. MTBDRsl will miss between one in four and one in three cases of XDR-TB. The diagnostic accuracy of MTBDRsl is similar when done using either culture isolates or smear-positive sputum. As the location of the resistance causing mutations can vary on a strain-by-strain basis, further research is required on test accuracy in different settings and, if genetic sequencing is used as a reference standard, it should examine all resistancedetermining regions. Given the confidence one can have in a positive result, and the ability of the test to provide results within a matter of days, MTBDRsl may be used as an initial test for second-line drug resistance. However, when the test reports a negative result, clinicians may still wish to carry out conventional testing. Plain Language Summary The rapid test GenoType® MTBDRsl for testing resistance to second-line TB drugs Background Different drugs are available to treat people with tuberculosis (TB), but resistance to these drugs is a growing problem. People with drug-resistant TB are more likely to die than people with drug-susceptible TB. People with drug-resistant TB require “second-line” TB drugs that, compared with “first-line” TB drugs used to treat drug-susceptible TB, cause more side effects and must be taken for longer. Extensively drug-resistant TB (XDR-TB) is a type of TB that is resistant to almost all TB drugs. A rapid and accurate test could identify people with drug-resistant TB, likely improve patient care, and reduce the spread of drug-resistant TB. Test evaluated by this review GenoType® MTBDRsl (MTBDRsl) is the only rapid test that detects resistance to second-line fluoroquinolone drugs and the secondline injectable drugs. The test also detects XDR-TB. MTBDRsl can be performed on TB bacteria grown by culture from sputum, which takes a long time (indirect testing), or immediately on sputum (direct testing). Main results We examined evidence available up to 30 January 2014 and included 21 studies, 11 of which were in low-income or middle-income countries. What do these results mean? Fluoroquinolone drugs By indirect testing, the test detected 83% of people with fluoroquinolone resistance and rarely gave a positive result for people without resistance. In a population of 1000 people,where 170 have fluoroquinolone resistance,MTBDRsl will correctly identify 141 people with fluoroquinolone resistance and miss 29 people. In this same population of 1000 people, where 830 people do not have fluoroquinolone resistance, the test will correctly classify 811 people as not having fluoroquinolone resistance and misclassify 19 people as having resistance (moderate quality evidence). By direct testing, the test detected 85% of people with fluoroquinolone resistance and rarely gave a positive result for people without resistance (moderate quality evidence). Second-line injectable drugs By indirect testing, the test detected 77%of people with second-line injectable drug resistance and rarely gave a positive result for people without resistance. In a population of 1000 people, where 230 have second-line injectable drug resistance, MTBDRsl will correctly identify 177 people with second-line injectable drug resistance and miss 53 people. In this same population of 1000 people, where 770 do not have second-line injectable drug resistance, the test will correctly classify 766 people as not having second-line injectable drug resistance and misclassify four people as having resistance (moderate quality evidence). By direct testing, the test detected 94% of people with second-line injectable drug resistance and rarely gave a positive result for people without resistance (very low quality evidence). XDR-TB By indirect testing, the test detected 71% of people with XDR-TB and rarely gave a positive result for people without XDR-TB. In a population of 1000 people, where 80 have XDR-TB, MTBDRsl will correctly identify 57 people with XDR-TB and miss 23 people. In this same population of 1000 people, where 920 do not have XDR-TB, the test will correctly classify 909 people as not having XDRTB and misclassify 11 people as having XDR-TB (low quality evidence). There was insufficient evidence to determine the accuracy of MTBDRsl by direct testing for XDR-TB. Conclusions The results s

Assessing the consistency assumption by exploring treatment by covariate interactions in mixed treatment comparison meta-analysis: individual patient-level covariates versus aggregate trial-level covariates.

Mixed treatment comparison (MTC) meta-analysis allows several treatments to be compared in a single analysis while utilising direct and indirect evidence. Treatment by covariate interactions can be included in MTC models to explore how the covariate modifies the treatment effects. If interactions exist, the assumptions underlying MTCs may be invalidated. For conventional pair-wise meta-analysis, important benefits regarding the investigation of such interactions, gained from using individual patient data (IPD) rather than aggregate data (AD), have been described. We aim to compare IPD MTC models including patient-level covariates with AD MTC models including study-level covariates. IPD and AD random-effects MTC models for dichotomous outcomes are specified. Three assumptions are made regarding the interactions (i.e. independent, exchangeable and common interactions). The models are applied to a dataset to compare four drugs for treating malaria (i.e. amodiaquine-artesunate, dihydroartemisinin-piperaquine (DHAPQ), artemether-lumefantrine and chlorproguanil-dapsone plus artesunate) using the outcome unadjusted treatment success at day 28. The treatment effects and regression coefficients for interactions from the IPD models were more precise than those from AD models. Using IPD, assuming independent or exchangeable interactions, the regression coefficient for chlorproguanil-dapsone plus artesunate versus DHAPQ was statistically significant and assuming common interactions, the common coefficient was significant; whereas using AD, no coefficients were significant. Using IPD, DHAPQ was the best drug; whereas using AD, the best drug varied. Using AD models, there was no evidence that the consistency assumption was invalid; whereas, the assumption was questionable based on the IPD models. The AD analyses were misleading.

Treatment outcome after failure of a first antiepileptic drug

Objectives: We assessed the likelihood of 12-month seizure remission and treatment failure after failure of a first antiepileptic drug, and identified factors influencing these outcomes. Methods: SANAD (Standard and New Antiepileptic Drug) was a randomized controlled trial comparing monotherapy with standard and new antiepileptic drugs. Patients were followed up to study completion, even if they were switched from their randomized treatment. After a first treatment failure, we assessed the probability of 12-month seizure remission and treatment failure. Prognostic modeling identified predictors of these outcomes. Results: Forty-four percent of patients in the SANAD trial had a first treatment failure. Seventy-five percent of these subsequently achieved 12-month remission by 6 years of follow-up. Significant prognostic factors included sex, age at treatment failure, time on randomized treatment at treatment failure, neurologic insult, total number of tonic-clonic seizures at treatment failure, reason for treatment failure, seizure type, and CT/MRI scan result. After a first treatment failure, young patients without tonic-clonic seizures, with a normal CT/MRI scan and failing their treatment because of unacceptable adverse events, had the highest likelihood of 12-month remission. Approximately 50% of patients who failed a first treatment also failed their second. Significant prognostic factors included total number of tonic-clonic seizures at first treatment failure, reason for first treatment failure, and CT/MRI scan result. Patients with tonic-clonic seizures and failing because of inadequate seizure control had the highest risk of a second treatment failure. Conclusions: A high proportion of patients will achieve 12-month remission after a first treatment failure. Clinical factors can stratify patients according to likely outcome.

A systematic review of placebo-controlled trials of topiramate: How useful is a multiple-indications review for evaluating the adverse events of an antiepileptic drug?

Topiramate (TPM) is an antiepileptic drug that is also used for other indications (e.g., migraine). Adverse event (AE) data from epilepsy trials could be supplemented by data from trials in other indications. Combining data across trials and indications is a novel method for evaluating AEs. We conducted a multiple‐indications review by systematically reviewing randomized placebo‐controlled trials of TPM, to compare the nervous system AEs of TPM in epilepsy with those in other indications.

Exploring Treatment by Covariate Interactions Using Subgroup Analysis and Meta-Regression in Cochrane Reviews: A Review of Recent Practice

Background Treatment by covariate interactions can be explored in reviews using interaction analyses (e.g., subgroup analysis). Such analyses can provide information on how the covariate modifies the treatment effect and is an important methodological approach for personalising medicine. Guidance exists regarding how to apply such analyses but little is known about whether authors follow the guidance. Methods Using published recommendations, we developed criteria to assess how well interaction analyses were designed, applied, interpreted, and reported. The Cochrane Database of Systematic Reviews was searched (8th August 2013). We applied the criteria to the most recently published review, with an accessible protocol, for each Cochrane Review Group. We excluded review updates, diagnostic test accuracy reviews, withdrawn reviews, and overviews of reviews. Data were summarised regarding reviews, covariates, and analyses. Results Each of the 52 included reviews planned or did interaction analyses; 51 reviews (98%) planned analyses and 33 reviews (63%) applied analyses. The type of analysis planned and the type subsequently applied (e.g., sensitivity or subgroup analysis) was discrepant in 24 reviews (46%). No review reported how or why each covariate had been chosen; 22 reviews (42%) did state each covariate a priori in the protocol but no review identified each post-hoc covariate as such. Eleven reviews (21%) mentioned five covariates or less. One review reported planning to use a method to detect interactions (i.e., interaction test) for each covariate; another review reported applying the method for each covariate. Regarding interpretation, only one review reported whether an interaction was detected for each covariate and no review discussed the importance, or plausibility, of the results, or the possibility of confounding for each covariate. Conclusions Interaction analyses in Cochrane Reviews can be substantially improved. The proposed criteria can be used to help guide the reporting and conduct of analyses.

Cluster Randomised Trials in Cochrane Reviews: Evaluation of Methodological and Reporting Practice

Objective Systematic reviews can include cluster-randomised controlled trials (C-RCTs), which require different analysis compared with standard individual-randomised controlled trials. However, it is not known whether review authors follow the methodological and reporting guidance when including these trials. The aim of this study was to assess the methodological and reporting practice of Cochrane reviews that included C-RCTs against criteria developed from existing guidance. Methods Criteria were developed, based on methodological literature and personal experience supervising review production and quality. Criteria were grouped into four themes: identifying, reporting, assessing risk of bias, and analysing C-RCTs. The Cochrane Database of Systematic Reviews was searched (2nd December 2013), and the 50 most recent reviews that included C-RCTs were retrieved. Each review was then assessed using the criteria. Results The 50 reviews we identified were published by 26 Cochrane Review Groups between June 2013 and November 2013. For identifying C-RCTs, only 56% identified that C-RCTs were eligible for inclusion in the review in the eligibility criteria. For reporting C-RCTs, only eight (24%) of the 33 reviews reported the method of cluster adjustment for their included C-RCTs. For assessing risk of bias, only one review assessed all five C-RCT-specific risk-of-bias criteria. For analysing C-RCTs, of the 27 reviews that presented unadjusted data, only nine (33%) provided a warning that confidence intervals may be artificially narrow. Of the 34 reviews that reported data from unadjusted C-RCTs, only 13 (38%) excluded the unadjusted results from the meta-analyses. Conclusions The methodological and reporting practices in Cochrane reviews incorporating C-RCTs could be greatly improved, particularly with regard to analyses. Criteria developed as part of the current study could be used by review authors or editors to identify errors and improve the quality of published systematic reviews incorporating C-RCTs.