Signe Flottorp

Grading quality of evidence and strength of recommendations.

Abstract Users of clinical practice guidelines and other recommendations need to know how much confidence they can place in the recommendations. Systematic and explicit methods of making judgments can reduce errors and improve communication. We have developed a system for grading the quality of evidence and the strength of recommendations that can be applied across a wide range of interventions and contexts. In this article we present a summary of our approach from the perspective of a guideline user. Judgments about the strength of a recommendation require consideration of the balance between benefits and harms, the quality of the evidence, translation of the evidence into specific circumstances, and the certainty of the baseline risk. It is also important to consider costs (resource utilisation) before making a recommendation. Inconsistencies among systems for grading the quality of evidence and the strength of recommendations reduce their potential to facilitate critical appraisal and improve communication of these judgments. Our system for guiding these complex judgments balances the need for simplicity with the need for full and transparent consideration of all important issues. Clinical guidelines are only as good as the evidence and judgments they are based on. The GRADE approach aims to make it easier for users to assess the judgments behind recommendations

Systems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approaches The GRADE Working Group

BackgroundA number of approaches have been used to grade levels of evidence and the strength of recommendations. The use of many different approaches detracts from one of the main reasons for having explicit approaches: to concisely characterise and communicate this information so that it can easily be understood and thereby help people make well-informed decisions. Our objective was to critically appraise six prominent systems for grading levels of evidence and the strength of recommendations as a basis for agreeing on characteristics of a common, sensible approach to grading levels of evidence and the strength of recommendations.MethodsSix prominent systems for grading levels of evidence and strength of recommendations were selected and someone familiar with each system prepared a description of each of these. Twelve assessors independently evaluated each system based on twelve criteria to assess the sensibility of the different approaches. Systems used by 51 organisations were compared with these six approaches.ResultsThere was poor agreement about the sensibility of the six systems. Only one of the systems was suitable for all four types of questions we considered (effectiveness, harm, diagnosis and prognosis). None of the systems was considered usable for all of the target groups we considered (professionals, patients and policy makers). The raters found low reproducibility of judgements made using all six systems. Systems used by 51 organisations that sponsor clinical practice guidelines included a number of minor variations of the six systems that we critically appraised.ConclusionsAll of the currently used approaches to grading levels of evidence and the strength of recommendations have important shortcomings.

Systems for grading the quality of evidence and the strength of recommendations II: Pilot study of a new system

BackgroundSystems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.MethodsA GRADE evidence profile consists of two tables: a quality assessment and a summary of findings. Twelve evidence profiles were used in this pilot study. Each evidence profile was made based on information available in a systematic review. Seventeen people were given instructions and independently graded the level of evidence and strength of recommendation for each of the 12 evidence profiles. For each example judgements were collected, summarised and discussed in the group with the aim of improving the proposed grading system. Kappas were calculated as a measure of chance-corrected agreement for the quality of evidence for each outcome for each of the twelve evidence profiles. The seventeen judges were also asked about the ease of understanding and the sensibility of the approach. All of the judgements were recorded and disagreements discussed.ResultsThere was a varied amount of agreement on the quality of evidence for the outcomes relating to each of the twelve questions (kappa coefficients for agreement beyond chance ranged from 0 to 0.82). However, there was fair agreement about the relative importance of each outcome. There was poor agreement about the balance of benefits and harms and recommendations. Most of the disagreements were easily resolved through discussion. In general we found the GRADE approach to be clear, understandable and sensible. Some modifications were made in the approach and it was agreed that more information was needed in the evidence profiles.ConclusionJudgements about evidence and recommendations are complex. Some subjectivity, especially regarding recommendations, is unavoidable. We believe our system for guiding these complex judgements appropriately balances the need for simplicity with the need for full and transparent consideration of all important issues.

A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice

BackgroundDeterminants of practice are factors that might prevent or enable improvements. Several checklists, frameworks, taxonomies, and classifications of determinants of healthcare professional practice have been published. In this paper, we describe the development of a comprehensive, integrated checklist of determinants of practice (the TICD checklist).MethodsWe performed a systematic review of frameworks of determinants of practice followed by a consensus process. We searched electronic databases and screened the reference lists of key background documents. Two authors independently assessed titles and abstracts, and potentially relevant full text articles. We compiled a list of attributes that a checklist should have: comprehensiveness, relevance, applicability, simplicity, logic, clarity, usability, suitability, and usefulness. We assessed included articles using these criteria and collected information about the theory, model, or logic underlying how the factors (determinants) were selected, described, and grouped, the strengths and weaknesses of the checklist, and the determinants and the domains in each checklist. We drafted a preliminary checklist based on an aggregated list of determinants from the included checklists, and finalized the checklist by a consensus process among implementation researchers.ResultsWe screened 5,778 titles and abstracts and retrieved 87 potentially relevant papers in full text. Several of these papers had references to papers that we also retrieved in full text. We also checked potentially relevant papers we had on file that were not retrieved by the searches. We included 12 checklists. None of these were completely comprehensive when compared to the aggregated list of determinants and domains. We developed a checklist with 57 potential determinants of practice grouped in seven domains: guideline factors, individual health professional factors, patient factors, professional interactions, incentives and resources, capacity for organisational change, and social, political, and legal factors. We also developed five worksheets to facilitate the use of the checklist.ConclusionsBased on a systematic review and a consensus process we developed a checklist that aims to be comprehensive and to build on the strengths of each of the 12 included checklists. The checklist is accompanied with five worksheets to facilitate its use in implementation research and quality improvement projects.

Antivirals for treatment of influenza: a systematic review and meta-analysis of observational studies.

BACKGROUND Systematic reviews of randomized, controlled trials in patients with influenza suggest a lack of evidence about the effects of antiviral therapy on several patient-important outcomes of influenza. PURPOSE To systematically review observational studies for benefits and harms of oseltamivir, zanamivir, amantadine, or rimantadine in the treatment of influenza. DATA SOURCES MEDLINE, EMBASE, the Cochrane Central Register of Controlled Trials, CINAHL, SIGLE, the Chinese Biomedical Literature Database, Panteleimon, and LILACS up to November 2010; contact with pharmaceutical companies; and reference lists. STUDY SELECTION Observational studies in any language that compared single antiviral therapy with no therapy or other antiviral therapy, or that had no comparator, for influenza or influenza-like illness. DATA EXTRACTION Two independent investigators extracted data. Confidence in the estimates of the obtained effects (quality of evidence) was assessed by using the Grading of Recommendations Assessment, Development, and Evaluation approach. DATA SYNTHESIS 74 studies fulfilled the inclusion criteria. Meta-analyses of the few studies providing effects with adjustment for confounders suggest that, in high-risk populations, oral oseltamivir may reduce mortality (odds ratio, 0.23 [95% CI, 0.13 to 0.43]; low-quality evidence), hospitalization (odds ratio, 0.75 [CI, 0.66 to 0.89]; low-quality evidence), and duration of symptoms (33 hours [CI, 21 to 45 hours]; very low-quality evidence) compared with no treatment. Earlier treatment with oseltamivir was generally associated with better outcomes. Inhaled zanamivir may lead to shorter symptom duration (23 hours [CI, 17 to 28 hours]; moderate-quality evidence) and fewer hospitalizations (odds ratio, 0.66 [CI, 0.37 to 1.18]) but more complications than no treatment. Direct comparison of oral oseltamivir and inhaled zanamivir suggests no important differences in key outcomes. Data from 1 study suggest that oral amantadine may reduce mortality and pneumonia associated with influenza A. No included study evaluated rimantadine. LIMITATIONS Mortality was assessed in high-risk patients, and generalizability is limited. The overall body of evidence is limited by risk for confounding and selection, reporting, and publication bias. CONCLUSION Therapy with oral oseltamivir and inhaled zanamivir may provide a net benefit over no treatment of influenza. However, as with the randomized trials, the confidence in the estimates of the effects for decision making is low to very low. PRIMARY FUNDING SOURCES: World Health Organization and McMaster University.

Supporting the delivery of cost-effective interventions in primary health-care systems in low-income and middle-income countries: an overview of systematic reviews

Strengthening health systems is a key challenge to improving the delivery of cost-effective interventions in primary health care and achieving the vision of the Alma-Ata Declaration. Effective governance, financial and delivery arrangements within health systems, and effective implementation strategies are needed urgently in low-income and middle-income countries. This overview summarises the evidence from systematic reviews of health systems arrangements and implementation strategies, with a particular focus on evidence relevant to primary health care in such settings. Although evidence is sparse, there are several promising health systems arrangements and implementation strategies for strengthening primary health care. However, their introduction must be accompanied by rigorous evaluations. The evidence base needs urgently to be strengthened, synthesised, and taken into account in policy and practice, particularly for the benefit of those who have been excluded from the health care advances of recent decades.

Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat

Abstract Objective:To assess the effectiveness of tailored interventions to implement guidelines for urinary tract infections in women and sore throat Design:Unblinded, cluster randomised pretest-post-test trial Setting:142 general practices in Norway Participants:72 practices received interventions to implement guidelines for urinary tract infection and 70 practices received interventions to implement guidelines for sore throat, serving as controls for each other. 59 practices in the urinary tract infection group and 61 practices in the sore throat group completed the study. Outcomes were measured in 16 939 consultations for sore throat and 9887 consultations for urinary tract infection. Interventions:Interventions were developed to overcome identified barriers to implementing the guidelines. The main components of the tailored interventions were patient educational material, computer based decision support and reminders, an increase in the fee for telephone consultations, and interactive courses for general practitioners and practice assistants Main outcome measures:Changes in rates of use of antibiotics, laboratory tests, and telephone consultations Results:Patients in the sore throat group were 3% less likely to receive antibiotics after the intervention. Women with symptoms of urinary tract infection in the intervention group were 5.1% less likely to have a laboratory test ordered. No significant differences were found between the groups for the other outcomes. Large variation was found across the included practicesin the rates of antibiotic prescription, use of laboratory tests and telephone consultations, and in the extent of change for all three outcome measures Conclusions:Passively delivered, complex interventions targeted at identified barriers to change had little effect in changing practice

Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results

BackgroundHealthcare decision makers face challenges when using guidelines, including understanding the quality of the evidence or the values and preferences upon which recommendations are made, which are often not clear.MethodsGRADE is a systematic approach towards assessing the quality of evidence and the strength of recommendations in healthcare. GRADE also gives advice on how to go from evidence to decisions. It has been developed to address the weaknesses of other grading systems and is now widely used internationally. The Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) consortium (http://www.decide-collaboration.eu/), which includes members of the GRADE Working Group and other partners, will explore methods to ensure effective communication of evidence-based recommendations targeted at key stakeholders: healthcare professionals, policymakers, and managers, as well as patients and the general public. Surveys and interviews with guideline producers and other stakeholders will explore how presentation of the evidence could be improved to better meet their information needs. We will collect further stakeholder input from advisory groups, via consultations and user testing; this will be done across a wide range of healthcare systems in Europe, North America, and other countries. Targeted communication strategies will be developed, evaluated in randomized trials, refined, and assessed during the development of real guidelines.DiscussionResults of the DECIDE project will improve the communication of evidence-based healthcare recommendations. Building on the work of the GRADE Working Group, DECIDE will develop and evaluate methods that address communication needs of guideline users. The project will produce strategies for communicating recommendations that have been rigorously evaluated in diverse settings, and it will support the transfer of research into practice in healthcare systems globally.

Growing Literature, Stagnant Science? Systematic Review, Meta-Regression and Cumulative Analysis of Audit and Feedback Interventions in Health Care

ABSTRACTBACKGROUNDThis paper extends the findings of the Cochrane systematic review of audit and feedback on professional practice to explore the estimate of effect over time and examine whether new trials have added to knowledge regarding how optimize the effectiveness of audit and feedback.METHODSWe searched the Cochrane Central Register of Controlled Trials, MEDLINE, and EMBASE for randomized trials of audit and feedback compared to usual care, with objectively measured outcomes assessing compliance with intended professional practice. Two reviewers independently screened articles and abstracted variables related to the intervention, the context, and trial methodology. The median absolute risk difference in compliance with intended professional practice was determined for each study, and adjusted for baseline performance. The effect size across studies was recalculated as studies were added to the cumulative analysis. Meta-regressions were conducted for studies published up to 2002, 2006, and 2010 in which characteristics of the intervention, the recipients, and trial risk of bias were tested as predictors of effect size.RESULTSOf the 140 randomized clinical trials (RCTs) included in the Cochrane review, 98 comparisons from 62 studies met the criteria for inclusion. The cumulative analysis indicated that the effect size became stable in 2003 after 51 comparisons from 30 trials. Cumulative meta-regressions suggested new trials are contributing little further information regarding the impact of common effect modifiers. Feedback appears most effective when: delivered by a supervisor or respected colleague; presented frequently; featuring both specific goals and action-plans; aiming to decrease the targeted behavior; baseline performance is lower; and recipients are non-physicians.DISCUSSIONThere is substantial evidence that audit and feedback can effectively improve quality of care, but little evidence of progress in the field. There are opportunity costs for patients, providers, and health care systems when investigators test quality improvement interventions that do not build upon, or contribute toward, extant knowledge.

Methodologies for assessing telemedicine: A systematic review of reviews

BACKGROUND AND OBJECTIVES Previous reviews have expressed concerns about the quality of telemedicine studies. There is debate about shortcomings and appropriate methodologies. The aim of this review of systematic reviews of telemedicine is to summarize methodologies used in telemedicine research, discuss knowledge gaps and recommendations and suggest methodological approaches for further research. METHODS We conducted a review of systematic reviews of telemedicine according to a protocol listing explicit methods, selection criteria, data collection and quality assessment procedures. We included reviews where authors explicitly addressed and made recommendations for assessment methodologies. We did a qualitative analysis of the reviews included, sensitized by two broad methodological positions; positivist and naturalistic approaches. The analysis focused on methodologies used in the primary studies included in the reviews as reported by the review authors, and methodological recommendations made by the review authors. RESULTS We identified 1593 titles/abstracts. We included 50 reviews that explicitly addressed assessment methodologies. One group of reviews recommended larger and more rigorously designed controlled studies to assess the impacts of telemedicine; a second group proposed standardisation of populations, and/or interventions and outcome measures to reduce heterogeneity and facilitate meta-analysis; a third group recommended combining quantitative and qualitative research methods; and others applying different naturalistic approaches including methodologies addressing mutual adaptations of services and users; politically driven action research and formative research aimed at collaboration to ensure capacity for improvement of services in natural settings. CONCLUSIONS Larger and more rigorous studies are crucial for the production of evidence of effectiveness of unambiguous telemedicine services for pre defined outcome measures. Summative methodologies acknowledging telemedicine as complex innovations and outcomes as partly contingent on values, meanings and contexts are also important. So are formative, naturalistic methodologies that acknowledge telemedicine as ongoing collaborative achievements and engage with stakeholders, including patients to produce and conceptualise new and effective telemedicine innovations.