Anne Ws Rutjes

Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study.

Objective To examine the presence and extent of small study effects in clinical osteoarthritis research. Design Meta-epidemiological study. Data sources 13 meta-analyses including 153 randomised trials (41 605 patients) that compared therapeutic interventions with placebo or non-intervention control in patients with osteoarthritis of the hip or knee and used patients’ reported pain as an outcome. Methods We compared estimated benefits of treatment between large trials (at least 100 patients per arm) and small trials, explored funnel plots supplemented with lines of predicted effects and contours of significance, and used three approaches to estimate treatment effects: meta-analyses including all trials irrespective of sample size, meta-analyses restricted to large trials, and treatment effects predicted for large trials. Results On average, treatment effects were more beneficial in small than in large trials (difference in effect sizes −0.21, 95% confidence interval −0.34 to −0.08, P=0.001). Depending on criteria used, six to eight funnel plots indicated small study effects. In six of 13 meta-analyses, the overall pooled estimate suggested a clinically relevant, significant benefit of treatment, whereas analyses restricted to large trials and predicted effects in large trials yielded smaller non-significant estimates. Conclusions Small study effects can often distort results of meta-analyses. The influence of small trials on estimated treatment effects should be routinely assessed.

The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study

Objective To examine whether excluding patients from the analysis of randomised trials are associated with biased estimates of treatment effects and higher heterogeneity between trials. Design Meta-epidemiological study based on a collection of meta-analyses of randomised trials. Data sources 14 meta-analyses including 167 trials that compared therapeutic interventions with placebo or non-intervention control in patients with osteoarthritis of the hip or knee and used patient reported pain as an outcome. Methods Effect sizes were calculated from differences in means of pain intensity between groups at the end of follow-up, divided by the pooled standard deviation. Trials were combined by using random effects meta-analysis. Estimates of treatment effects were compared between trials with and trials without exclusions from the analysis, and the impact of restricting meta-analyses to trials without exclusions was assessed. Results 39 trials (23%) had included all patients in the analysis. In 128 trials (77%) some patients were excluded from the analysis. Effect sizes from trials with exclusions tended to be more beneficial than those from trials without exclusions (difference −0.13, 95% confidence interval −0.29 to 0.04). However, estimates of bias between individual meta-analyses varied considerably (τ2=0.07). Tests of interaction between exclusions from the analysis and estimates of treatment effects were positive in five meta-analyses. Stratified analyses indicated that differences in effect sizes between trials with and trials without exclusions were more pronounced in meta-analyses with high between trial heterogeneity, in meta-analyses with large estimated treatment benefits, and in meta-analyses of complementary medicine. Restriction of meta-analyses to trials without exclusions resulted in smaller estimated treatment benefits, larger P values, and considerable decreases in between trial heterogeneity. Conclusion Excluding patients from the analysis in randomised trials often results in biased estimates of treatment effects, but the extent and direction of bias is unpredictable. Results from intention to treat analyses should always be described in reports of randomised trials. In systematic reviews, the influence of exclusions from the analysis on estimated treatment effects should routinely be assessed.

Uses and misuses of the STROBE statement: bibliographic study

Objectives Appropriate reporting is central to the application of findings from research to clinical practice. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) recommendations consist of a checklist of 22 items that provide guidance on the reporting of cohort, case–control and cross-sectional studies, in order to facilitate critical appraisal and interpretation of results. STROBE was published in October 2007 in several journals including The Lancet, BMJ, Annals of Internal Medicine and PLoS Medicine. Within the framework of the revision of the STROBE recommendations, the authors examined the context and circumstances in which the STROBE statement was used in the past. Design The authors searched the Web of Science database in August 2010 for articles which cited STROBE and examined a random sample of 100 articles using a standardised, piloted data extraction form. The use of STROBE in observational studies and systematic reviews (including meta-analyses) was classified as appropriate or inappropriate. The use of STROBE to guide the reporting of observational studies was considered appropriate. Inappropriate uses included the use of STROBE as a tool to assess the methodological quality of studies or as a guideline on how to design and conduct studies. Results The authors identified 640 articles that cited STROBE. In the random sample of 100 articles, about half were observational studies (32%) or systematic reviews (19%). Comments, editorials and letters accounted for 15%, methodological articles for 8%, and recommendations and narrative reviews for 26% of articles. Of the 32 observational studies, 26 (81%) made appropriate use of STROBE, and three uses (10%) were considered inappropriate. Among 19 systematic reviews, 10 (53%) used STROBE inappropriately as a tool to assess study quality. Conclusions The STROBE reporting recommendations are frequently used inappropriately in systematic reviews and meta-analyses as an instrument to assess the methodological quality of observational studies.

The haematocrit and platelet target in polycythemia vera

Polycythemia vera (PV) is a chronic myeloproliferative disorder whose major morbidity and mortality are thrombohaemorragic events and progression to acute leukaemia or myelofibrosis. Whether the haematocrit and platelet count predict such complications remains unclear. The European Collaboration on Low‐dose Aspirin in Polycythemia Vera prospective study included 1638 PV patients. A total of 164 deaths (10%), 145 (8·85%) major thrombosis and 226 (13·8%) total thrombosis were encountered during 4393 person‐years follow‐up (median 2·8 years). In time‐dependent multivariable analysis, a haematocrit in the evaluable range of 40–55% was neither associated with the occurrence of thrombotic events, mortality nor with haematological progression in the studied population. The haematocrit of patients in the highest and lowest deciles at baseline was maintained within a narrow interval of haematocrit values ranging from 40% to 47% throughout follow‐up. High platelet count was associated with a lower progression rate to acute leukaemia/myelofibrosis, whereas it had no significant relationship with thrombotic events or mortality. Our findings do not suggest that the range of haematocrit (<55%) and platelet counts (<600 × 109/l) we encountered in our population had an impact on the outcome of PV patients treated by current therapeutic strategies.

Accuracy of diagnostic tests for clinically suspected upper extremity deep vein thrombosis: a systematic review

Summary.  Background: The best available test for the diagnosis of upper extremity deep venous thrombosis (UEDVT) is contrast venography. The aim of this systematic review was to assess whether the diagnostic accuracy of other tests for clinically suspected UEDVT is high enough to justify their use in clinical practise and to evaluate if any test can replace venography. Methods: MEDLINE and EMBASE databases were searched from inception to June 2009. Two reviewers independently evaluated study eligibility, extracted data, and assessed study quality. Results: We identified 17 papers, reporting on 793 patients. Overall, the methodological quality was poor, sample sizes were small, and large between‐study differences were observed in spectrum and design. The summary estimates of sensitivity (95% confidence interval) were 97% (90–100%) for compression ultrasonography, 84% (72–97%) for Doppler ultrasonography, 91% (85–97%) for Doppler ultrasonography with compression, and 85% (72–99%) for phleboreography. The corresponding summary estimates of specificity were, respectively, 96% (87–100%), 94% (86–100%), 93% (80–100%), and 87% (71–100%). Clinical findings, a clinical score, D‐dimer, magnetic resonance imaging, rheography and plethysmography were evaluated in one study each, involving a median number of 46 patients (range 21–214). Sensitivity and specificity ranged from 0% to 100% and from 14% to 100%. Conclusions: Methodological limitations, large between‐study differences and small sample sizes limit the evidence of tests for clinically suspected UEDVT. Compression ultrasonography may be an acceptable alternative to venography. The addition of (color) Doppler does not seem to improve the accuracy. Adequately designed studies are warranted to confirm these findings.

A systematic review classifies sources of bias and variation in diagnostic test accuracy studies

OBJECTIVE To classify the sources of bias and variation and to provide an updated summary of the evidence of the effects of each source of bias and variation. STUDY DESIGN AND SETTING We conducted a systematic review of studies of any design with the main objective of addressing bias or variation in the results of diagnostic accuracy studies. We searched MEDLINE, EMBASE, BIOSIS, the Cochrane Methodology Register, and Database of Abstracts of Reviews of Effects (DARE) from 2001 to October 2011. Citation searches based on three key papers were conducted, and studies from our previous review (search to 2001) were eligible. One reviewer extracted data on the study design, objective, sources of bias and/or variation, and results. A second reviewer checked the extraction. RESULTS We summarized the number of studies providing evidence of an effect arising from each source of bias and variation on the estimates of sensitivity, specificity, and overall accuracy. CONCLUSIONS We found consistent evidence for the effects of case-control design, observer variability, availability of clinical information, reference standard, partial and differential verification bias, demographic features, and disease prevalence and severity. Effects were generally stronger for sensitivity than for specificity. Evidence for other sources of bias and variation was limited.

Reporting standards for studies of diagnostic test accuracy in dementia: The STARDdem Initiative

Objective: To provide guidance on standards for reporting studies of diagnostic test accuracy for dementia disorders. Methods: An international consensus process on reporting standards in dementia and cognitive impairment (STARDdem) was established, focusing on studies presenting data from which sensitivity and specificity were reported or could be derived. A working group led the initiative through 4 rounds of consensus work, using a modified Delphi process and culminating in a face-to-face consensus meeting in October 2012. The aim of this process was to agree on how best to supplement the generic standards of the STARD statement to enhance their utility and encourage their use in dementia research. Results: More than 200 comments were received during the wider consultation rounds. The areas at most risk of inadequate reporting were identified and a set of dementia-specific recommendations to supplement the STARD guidance were developed, including better reporting of patient selection, the reference standard used, avoidance of circularity, and reporting of test-retest reliability. Conclusion: STARDdem is an implementation of the STARD statement in which the original checklist is elaborated and supplemented with guidance pertinent to studies of cognitive disorders. Its adoption is expected to increase transparency, enable more effective evaluation of diagnostic tests in Alzheimer disease and dementia, contribute to greater adherence to methodologic standards, and advance the development of Alzheimer biomarkers.

The importance of allocation concealment and patient blinding in osteoarthritis trials: A meta-epidemiologic study

OBJECTIVE To evaluate the association of adequate allocation concealment and patient blinding with estimates of treatment benefits in osteoarthritis trials. METHODS We performed a meta-epidemiologic study of 16 meta-analyses with 175 trials that compared therapeutic interventions with placebo or nonintervention control in patients with hip or knee osteoarthritis. We calculated effect sizes from the differences in means of pain intensity between groups at the end of followup divided by the pooled SD and compared effect sizes between trials with and trials without adequate methodology. RESULTS Effect sizes tended to be less beneficial in 46 trials with adequate allocation concealment compared with 112 trials with inadequate or unclear concealment of allocation (difference -0.15; 95% confidence interval [95% CI] -0.31, 0.02). Selection bias associated with inadequate or unclear concealment of allocation was most pronounced in meta-analyses with large estimated treatment benefits (P for interaction < 0.001), meta-analyses with high between-trial heterogeneity (P = 0.009), and meta-analyses of complementary medicine (P = 0.019). Effect sizes tended to be less beneficial in 64 trials with adequate blinding of patients compared with 58 trials without (difference -0.15; 95% CI -0.39, 0.09), but differences were less consistent and disappeared after accounting for allocation concealment. Detection bias associated with a lack of adequate patient blinding was most pronounced for nonpharmacologic interventions (P for interaction < 0.001). CONCLUSION Results of osteoarthritis trials may be affected by selection and detection bias. Adequate concealment of allocation and attempts to blind patients will minimize these biases.

Methods to convert continuous outcomes into odds ratios of treatment response and numbers needed to treat: meta-epidemiological study

BACKGROUND Clinicians find standardized mean differences (SMDs) calculated from continuous outcomes difficult to interpret. Our objective was to determine the performance of methods in converting SMDs or means to odds ratios of treatment response and numbers needed to treat (NNTs) as more intuitive measures of treatment effect. METHODS Meta-epidemiological study of large-scale trials (≥ 100 patients per group) comparing active treatment with placebo, sham or non-intervention control. Trials had to use pain or global symptoms as continuous outcomes and report both the percentage of patients with treatment response and mean pain or symptom scores per group. For each trial, we calculated odds ratios of observed treatment response and NNTs and approximated these estimates from SMDs or means using all five currently available conversion methods by Hasselblad and Hedges (HH), Cox and Snell (CS), Furukawa (FU), Suissa (SU) and Kraemer and Kupfer (KK). We compared observed and approximated values within trials by deriving pooled ratios of odds ratios (RORs) and differences in NNTs. ROR <1 and positive differences in NNTs imply that approximations are more conservative than estimates calculated from observed treatment response. As measures of agreement, we calculated intraclass correlation coefficients. RESULTS A total of 29 trials in 13 654 patients were included. Four out of five methods were suitable (HH, CS, FU, SU), with RORs between 0.92 for SU [95% confidence interval (95% CI), 0.86-0.99] and 0.97 for HH (95% CI, 0.91-1.04) and differences in NNTs between 0.5 (95% CI, -0.1 to -1.6) and 1.3 (95% CI, 0.4-2.1). Intraclass correlation coefficients were ≥ 0.90 for these four methods, but ≤ 0.76 for the fifth method by KK (P for differences ≤ 0.027). CONCLUSIONS The methods by HH, CS, FU and SU are suitable to convert summary treatment effects calculated from continuous outcomes into odds ratios of treatment response and NNTs, whereas the method by KK is unsuitable.

Comparing pneumococcal conjugate vaccine schedules based on 3 and 2 primary doses: systematic review and meta-analysis

BACKGROUND Pneumococcal conjugate vaccines (PCV) were first licensed for use with 3 primary doses in infancy and a booster dose. The evidence for the effects of different schedules was examined in this systematic review and meta-analysis. METHODS We searched 12 databases and trial registers up to March 2010. We selected randomised controlled trials (RCTs), cohort and case-control studies making direct comparisons between PCV schedules with (2p) or (3p) primary doses, with (+1) or without (+0) a booster dose. We extracted data on clinical, nasopharyngeal carriage and immunological outcomes and used meta-analysis to combine results where appropriate. RESULTS Seropositivity levels (antibody concentration ≥0.35 μg/ml) following 3p and 2p PCV schedules were high for most serotypes (5 RCTs). Differences between schedules were generally small and tended to favour 3p schedules, particularly for serotypes 6B and 23F; between-study heterogeneity was high. Seropositivity levels following 3p+1 and 2p+1 schedules were similar but small differences favouring 3p+1 schedules were seen for serotypes 6B and 23F. We did not identify any RCTs reporting clinical outcomes for these comparisons. In 2 RCTs there was weak evidence of a reduction in carriage of S. pneumoniae serotypes included in the vaccine when 3p+0 schedules were compared to 2p+0 at 6 months of age. CONCLUSIONS Most data about the relative effects of different PCV schedules relate to immunological outcomes. Both 3p and 2p schedules result in high levels of seropositivity. The clinical relevance of differences in immunological outcomes between schedules is not known. There is an absence of clinical outcome data from RCTs with direct comparisons of any 2p with any 3p PCV schedule.