Janus Christian Jakobsen

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods

BackgroundThresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.MethodsMethodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedure was developed, based mainly on The Cochrane Collaboration methodology and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines.ResultsWe propose an eight-step procedure for better validation of meta-analytic results in systematic reviews (1) Obtain the 95% confidence intervals and the P-values from both fixed-effect and random-effects meta-analyses and report the most conservative results as the main results. (2) Explore the reasons behind substantial statistical heterogeneity using subgroup and sensitivity analyses (see step 6). (3) To take account of problems with multiplicity adjust the thresholds for significance according to the number of primary outcomes. (4) Calculate required information sizes (≈ the a priori required number of participants for a meta-analysis to be conclusive) for all outcomes and analyse each outcome with trial sequential analysis. Report whether the trial sequential monitoring boundaries for benefit, harm, or futility are crossed. (5) Calculate Bayes factors for all primary outcomes. (6) Use subgroup analyses and sensitivity analyses to assess the potential impact of bias on the review results. (7) Assess the risk of publication bias. (8) Assess the clinical significance of the statistically significant review results.ConclusionsIf followed, the proposed eight-step procedure will increase the validity of assessments of intervention effects in systematic reviews of randomised clinical trials.

Trial Sequential Analysis in systematic reviews with meta-analysis

BackgroundMost meta-analyses in systematic reviews, including Cochrane ones, do not have sufficient statistical power to detect or refute even large intervention effects. This is why a meta-analysis ought to be regarded as an interim analysis on its way towards a required information size. The results of the meta-analyses should relate the total number of randomised participants to the estimated required meta-analytic information size accounting for statistical diversity. When the number of participants and the corresponding number of trials in a meta-analysis are insufficient, the use of the traditional 95% confidence interval or the 5% statistical significance threshold will lead to too many false positive conclusions (type I errors) and too many false negative conclusions (type II errors).MethodsWe developed a methodology for interpreting meta-analysis results, using generally accepted, valid evidence on how to adjust thresholds for significance in randomised clinical trials when the required sample size has not been reached.ResultsThe Lan-DeMets trial sequential monitoring boundaries in Trial Sequential Analysis offer adjusted confidence intervals and restricted thresholds for statistical significance when the diversity-adjusted required information size and the corresponding number of required trials for the meta-analysis have not been reached. Trial Sequential Analysis provides a frequentistic approach to control both type I and type II errors. We define the required information size and the corresponding number of required trials in a meta-analysis and the diversity (D2) measure of heterogeneity. We explain the reasons for using Trial Sequential Analysis of meta-analysis when the actual information size fails to reach the required information size. We present examples drawn from traditional meta-analyses using unadjusted naïve 95% confidence intervals and 5% thresholds for statistical significance. Spurious conclusions in systematic reviews with traditional meta-analyses can be reduced using Trial Sequential Analysis. Several empirical studies have demonstrated that the Trial Sequential Analysis provides better control of type I errors and of type II errors than the traditional naïve meta-analysis.ConclusionsTrial Sequential Analysis represents analysis of meta-analytic data, with transparent assumptions, and better control of type I and type II errors than the traditional meta-analysis using naïve unadjusted confidence intervals.

The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials

BackgroundThresholds for statistical significance are insufficiently demonstrated by 95% confidence intervals or P-values when assessing results from randomised clinical trials. First, a P-value only shows the probability of getting a result assuming that the null hypothesis is true and does not reflect the probability of getting a result assuming an alternative hypothesis to the null hypothesis is true. Second, a confidence interval or a P-value showing significance may be caused by multiplicity. Third, statistical significance does not necessarily result in clinical significance. Therefore, assessment of intervention effects in randomised clinical trials deserves more rigour in order to become more valid.MethodsSeveral methodologies for assessing the statistical and clinical significance of intervention effects in randomised clinical trials were considered. Balancing simplicity and comprehensiveness, a simple five-step procedure was developed.ResultsFor a more valid assessment of results from a randomised clinical trial we propose the following five-steps: (1) report the confidence intervals and the exact P-values; (2) report Bayes factor for the primary outcome, being the ratio of the probability that a given trial result is compatible with a ‘null’ effect (corresponding to the P-value) divided by the probability that the trial result is compatible with the intervention effect hypothesised in the sample size calculation; (3) adjust the confidence intervals and the statistical significance threshold if the trial is stopped early or if interim analyses have been conducted; (4) adjust the confidence intervals and the P-values for multiplicity due to number of outcome comparisons; and (5) assess clinical significance of the trial results.ConclusionsIf the proposed five-step procedure is followed, this may increase the validity of assessments of intervention effects in randomised clinical trials.

Selective serotonin reuptake inhibitors versus placebo in patients with major depressive disorder. A systematic review with meta-analysis and Trial Sequential Analysis.

BackgroundThe evidence on selective serotonin reuptake inhibitors (SSRIs) for major depressive disorder is unclear.MethodsOur objective was to conduct a systematic review assessing the effects of SSRIs versus placebo, ‘active’ placebo, or no intervention in adult participants with major depressive disorder. We searched for eligible randomised clinical trials in The Cochrane Library’s CENTRAL, PubMed, EMBASE, PsycLIT, PsycINFO, Science Citation Index Expanded, clinical trial registers of Europe and USA, websites of pharmaceutical companies, the U.S. Food and Drug Administration (FDA), and the European Medicines Agency until January 2016. All data were extracted by at least two independent investigators. We used Cochrane systematic review methodology, Trial Sequential Analysis, and calculation of Bayes factor. An eight-step procedure was followed to assess if thresholds for statistical and clinical significance were crossed. Primary outcomes were reduction of depressive symptoms, remission, and adverse events. Secondary outcomes were suicides, suicide attempts, suicide ideation, and quality of life.ResultsA total of 131 randomised placebo-controlled trials enrolling a total of 27,422 participants were included. None of the trials used ‘active’ placebo or no intervention as control intervention. All trials had high risk of bias. SSRIs significantly reduced the Hamilton Depression Rating Scale (HDRS) at end of treatment (mean difference −1.94 HDRS points; 95% CI −2.50 to −1.37; P < 0.00001; 49 trials; Trial Sequential Analysis-adjusted CI −2.70 to −1.18); Bayes factor below predefined threshold (2.01*10−23). The effect estimate, however, was below our predefined threshold for clinical significance of 3 HDRS points. SSRIs significantly decreased the risk of no remission (RR 0.88; 95% CI 0.84 to 0.91; P < 0.00001; 34 trials; Trial Sequential Analysis adjusted CI 0.83 to 0.92); Bayes factor (1426.81) did not confirm the effect). SSRIs significantly increased the risks of serious adverse events (OR 1.37; 95% CI 1.08 to 1.75; P = 0.009; 44 trials; Trial Sequential Analysis-adjusted CI 1.03 to 1.89). This corresponds to 31/1000 SSRI participants will experience a serious adverse event compared with 22/1000 control participants. SSRIs also significantly increased the number of non-serious adverse events. There were almost no data on suicidal behaviour, quality of life, and long-term effects.ConclusionsSSRIs might have statistically significant effects on depressive symptoms, but all trials were at high risk of bias and the clinical significance seems questionable. SSRIs significantly increase the risk of both serious and non-serious adverse events. The potential small beneficial effects seem to be outweighed by harmful effects.Systematic review registrationPROSPERO CRD42013004420.

Effects of cognitive therapy versus interpersonal psychotherapy in patients with major depressive disorder: a systematic review of randomized clinical trials with meta-analyses and trial sequential analyses

BACKGROUND Major depressive disorder afflicts an estimated 17% of individuals during their lifetime at tremendous suffering and cost. Cognitive therapy and interpersonal psychotherapy are treatment options, but their effects have only been limitedly compared in systematic reviews. METHOD Using Cochrane systematic review methodology we compared the benefits and harm of cognitive therapy versus interpersonal psychotherapy for major depressive disorder. Trials were identified by searching the Cochrane Librarys CENTRAL, Medline via PubMed, EMBASE, Psychlit, PsycInfo, and Science Citation Index Expanded until February 2010. Continuous outcome measures were assessed by mean difference and dichotomous outcomes by odds ratio. We conducted trial sequential analysis to control for random errors. RESULTS We included seven trials randomizing 741 participants. All trials had high risk of bias. Meta-analysis of the four trials reporting data at cessation of treatment on the Hamilton Rating Scale for Depression showed no significant difference between the two interventions [mean difference -1.02, 95% confidence interval (CI) -2.35 to 0.32]. Meta-analysis of the five trials reporting data at cessation of treatment on the Beck Depression Inventory showed comparable results (mean difference -1.29, 95% CI -2.73 to 0.14). Trial sequential analysis indicated that more data are needed to definitively settle the question of a differential effect. None of the included trial reported on adverse events. CONCLUSIONS Randomized trials with low risk of bias and low risk of random errors are needed, although the effects of cognitive therapy and interpersonal psychotherapy do not seem to differ significantly regarding depressive symptoms. Future trials should report on adverse events.

The Necessity of Randomized Clinical Trials

Aims: The hierarchy of evidence-based medicine determines the inferential powers of different clinical research designs. We want to address the difficult question if observational evidence under some circumstances can validate intervention effects. Methodology: Assessment of previous argumentation aiming at a clear conclusion for future decision-making. Results: We present five arguments demonstrating the fundamental need of randomized clinical trials to sufficiently validate intervention effects. Furthermore, we argue that hindrances to the conduct of randomized clinical trials can be lessened through education, collaboration, infrastructure, and other measures. Our arguments validate why the randomized clinical trial should and must be the study design evaluating interventions. By choosing the randomized clinical trial as the primary study design, effective preventive, prognostic, diagnostic, and therapeutic interventions will reach more patients earlier. Conclusion: Clinical experience or observational studies should never be used as the sole basis for assessment of intervention effects — randomized clinical trials are always needed. Therefore, always randomize the first patient as Thomas C Chalmers suggested in 1977. Observational studies should primarily be used for quality control after treatments are included in clinical practice. Policy Paper

Evidence-based clinical practice: Overview of threats to the validity of evidence and how to minimise them

Using the best quality of clinical research evidence is essential for choosing the right treatment for patients. How to identify the best research evidence is, however, difficult. In this narrative review we summarise these threats and describe how to minimise them. Pertinent literature was considered through literature searches combined with personal files. Treatments should generally not be chosen based only on evidence from observational studies or single randomised clinical trials. Systematic reviews with meta-analysis of all identifiable randomised clinical trials with Grading of Recommendations Assessment, Development and Evaluation (GRADE) assessment represent the highest level of evidence. Even though systematic reviews are trust worthier than other types of evidence, all levels of the evidence hierarchy are under threats from systematic errors (bias); design errors (abuse of surrogate outcomes, composite outcomes, etc.); and random errors (play of chance). Clinical research infrastructures may help in providing larger and better conducted trials. Trial Sequential Analysis may help in deciding when there is sufficient evidence in meta-analyses. If threats to the validity of clinical research are carefully considered and minimised, research results will be more valid and this will benefit patients and heath care systems.

The effects of cognitive therapy versus 'no intervention' for major depressive disorder.

Background Major depressive disorder afflicts an estimated 17% of individuals during their lifetimes at tremendous suffering and costs. Cognitive therapy may be an effective treatment option for major depressive disorder, but the effects have only had limited assessment in systematic reviews. Methods/Principal Findings We used The Cochrane systematic review methodology with meta-analyses and trial sequential analyses of randomized trials comparing the effects of cognitive therapy versus ‘no intervention’ for major depressive disorder. Participants had to be older than 17 years with a primary diagnosis of major depressive disorder to be eligible. Altogether, we included 12 trials randomizing a total of 669 participants. All 12 trials had high risk of bias. Meta-analysis on the Hamilton Rating Scale for Depression showed that cognitive therapy significantly reduced depressive symptoms (four trials; mean difference −3.05 (95% confidence interval (Cl), −5.23 to −0.87; P<0.006)) compared with ‘no intervention’. Trial sequential analysis could not confirm this result. Meta-analysis on the Beck Depression Inventory showed that cognitive therapy significantly reduced depressive symptoms (eight trials; mean difference on −4.86 (95% CI −6.44 to −3.28; P = 0.00001)). Trial sequential analysis on these data confirmed the result. Only a few trials reported on ‘no remission’, suicide inclination, suicide attempts, suicides, and adverse events without significant differences between the compared intervention groups. Discussion Cognitive therapy might be an effective treatment for depression measured on Hamilton Rating Scale for Depression and Beck Depression Inventory, but these outcomes may be overestimated due to risks of systematic errors (bias) and random errors (play of chance). Furthermore, the effects of cognitive therapy on no remission, suicidality, adverse events, and quality of life are unclear. There is a need for randomized trials with low risk of bias, low risk of random errors, and longer follow-up assessing both benefits and harms with clinically relevant outcome measures.

The Effects of Cognitive Therapy Versus ‘Treatment as Usual’ in Patients with Major Depressive Disorder

Background Major depressive disorder afflicts an estimated 17% of individuals during their lifetimes at tremendous suffering and costs. Cognitive therapy may be an effective treatment option for major depressive disorder, but the effects have only had limited assessment in systematic reviews. Methods/Principal Findings Cochrane systematic review methodology, with meta-analyses and trial sequential analyses of randomized trials, are comparing the effects of cognitive therapy versus ‘treatment as usual’ for major depressive disorder. To be included the participants had to be older than 17 years with a primary diagnosis of major depressive disorder. Altogether, we included eight trials randomizing a total of 719 participants. All eight trials had high risk of bias. Four trials reported data on the 17-item Hamilton Rating Scale for Depression and four trials reported data on the Beck Depression Inventory. Meta-analysis on the data from the Hamilton Rating Scale for Depression showed that cognitive therapy compared with ‘treatment as usual’ significantly reduced depressive symptoms (mean difference −2.15 (95% confidence interval −3.70 to −0.60; P<0.007, no heterogeneity)). However, meta-analysis with both fixed-effect and random-effects model on the data from the Beck Depression Inventory (mean difference with both models −1.57 (95% CL −4.30 to 1.16; P = 0.26, I2 = 0) could not confirm the Hamilton Rating Scale for Depression results. Furthermore, trial sequential analysis on both the data from Hamilton Rating Scale for Depression and Becks Depression Inventory showed that insufficient data have been obtained. Discussion Cognitive therapy might not be an effective treatment for major depressive disorder compared with ‘treatment as usual’. The possible treatment effect measured on the Hamilton Rating Scale for Depression is relatively small. More randomized trials with low risk of bias, increased sample sizes, and broader more clinically relevant outcomes are needed.

The effect of interpersonal psychotherapy and other psychodynamic therapies versus 'treatment as usual' in patients with major depressive disorder.

Background Major depressive disorder afflicts an estimated 17% of individuals during their lifetimes at tremendous suffering and costs. Interpersonal psychotherapy and other psychodynamic therapies may be effective interventions for major depressive disorder, but the effects have only had limited assessment in systematic reviews. Methods/Principal Findings Cochrane systematic review methodology with meta-analysis and trial sequential analysis of randomized trials comparing the effect of psychodynamic therapies versus ‘treatment as usual’ for major depressive disorder. To be included the participants had to be older than 17 years with a primary diagnosis of major depressive disorder. Altogether, we included six trials randomizing a total of 648 participants. Five trials assessed ‘interpersonal psychotherapy’ and only one trial assessed ‘psychodynamic psychotherapy’. All six trials had high risk of bias. Meta-analysis on all six trials showed that the psychodynamic interventions significantly reduced depressive symptoms on the 17-item Hamilton Rating Scale for Depression (mean difference −3.12 (95% confidence interval −4.39 to −1.86;P<0.00001), no heterogeneity) compared with ‘treatment as usual’. Trial sequential analysis confirmed this result. Discussion We did not find convincing evidence supporting or refuting the effect of interpersonal psychotherapy or psychodynamic therapy compared with ‘treatment as usual’ for patients with major depressive disorder. The potential beneficial effect seems small and effects on major outcomes are unknown. Randomized trials with low risk of systematic errors and low risk of random errors are needed.