Suzanne Hill

Grading quality of evidence and strength of recommendations.

Abstract Users of clinical practice guidelines and other recommendations need to know how much confidence they can place in the recommendations. Systematic and explicit methods of making judgments can reduce errors and improve communication. We have developed a system for grading the quality of evidence and the strength of recommendations that can be applied across a wide range of interventions and contexts. In this article we present a summary of our approach from the perspective of a guideline user. Judgments about the strength of a recommendation require consideration of the balance between benefits and harms, the quality of the evidence, translation of the evidence into specific circumstances, and the certainty of the baseline risk. It is also important to consider costs (resource utilisation) before making a recommendation. Inconsistencies among systems for grading the quality of evidence and the strength of recommendations reduce their potential to facilitate critical appraisal and improve communication of these judgments. Our system for guiding these complex judgments balances the need for simplicity with the need for full and transparent consideration of all important issues. Clinical guidelines are only as good as the evidence and judgments they are based on. The GRADE approach aims to make it easier for users to assess the judgments behind recommendations

Systems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approaches The GRADE Working Group

BackgroundA number of approaches have been used to grade levels of evidence and the strength of recommendations. The use of many different approaches detracts from one of the main reasons for having explicit approaches: to concisely characterise and communicate this information so that it can easily be understood and thereby help people make well-informed decisions. Our objective was to critically appraise six prominent systems for grading levels of evidence and the strength of recommendations as a basis for agreeing on characteristics of a common, sensible approach to grading levels of evidence and the strength of recommendations.MethodsSix prominent systems for grading levels of evidence and strength of recommendations were selected and someone familiar with each system prepared a description of each of these. Twelve assessors independently evaluated each system based on twelve criteria to assess the sensibility of the different approaches. Systems used by 51 organisations were compared with these six approaches.ResultsThere was poor agreement about the sensibility of the six systems. Only one of the systems was suitable for all four types of questions we considered (effectiveness, harm, diagnosis and prognosis). None of the systems was considered usable for all of the target groups we considered (professionals, patients and policy makers). The raters found low reproducibility of judgements made using all six systems. Systems used by 51 organisations that sponsor clinical practice guidelines included a number of minor variations of the six systems that we critically appraised.ConclusionsAll of the currently used approaches to grading levels of evidence and the strength of recommendations have important shortcomings.

Systems for grading the quality of evidence and the strength of recommendations II: Pilot study of a new system

BackgroundSystems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.MethodsA GRADE evidence profile consists of two tables: a quality assessment and a summary of findings. Twelve evidence profiles were used in this pilot study. Each evidence profile was made based on information available in a systematic review. Seventeen people were given instructions and independently graded the level of evidence and strength of recommendation for each of the 12 evidence profiles. For each example judgements were collected, summarised and discussed in the group with the aim of improving the proposed grading system. Kappas were calculated as a measure of chance-corrected agreement for the quality of evidence for each outcome for each of the twelve evidence profiles. The seventeen judges were also asked about the ease of understanding and the sensibility of the approach. All of the judgements were recorded and disagreements discussed.ResultsThere was a varied amount of agreement on the quality of evidence for the outcomes relating to each of the twelve questions (kappa coefficients for agreement beyond chance ranged from 0 to 0.82). However, there was fair agreement about the relative importance of each outcome. There was poor agreement about the balance of benefits and harms and recommendations. Most of the disagreements were easily resolved through discussion. In general we found the GRADE approach to be clear, understandable and sensible. Some modifications were made in the approach and it was agreed that more information was needed in the evidence profiles.ConclusionJudgements about evidence and recommendations are complex. Some subjectivity, especially regarding recommendations, is unavoidable. We believe our system for guiding these complex judgements appropriately balances the need for simplicity with the need for full and transparent consideration of all important issues.

WHO Rapid Advice Guidelines for pharmacological management of sporadic human infection with avian influenza A (H5N1) virus

Summary Recent spread of avian influenza A (H5N1) virus to poultry and wild birds has increased the threat of human infections with H5N1 virus worldwide. Despite international agreement to stockpile antivirals, evidence-based guidelines for their use do not exist. WHO assembled an international multidisciplinary panel to develop rapid advice for the pharmacological management of human H5N1 virus infection in the current pandemic alert period. A transparent methodological guideline process on the basis of the Grading Recommendations, Assessment, Development and Evaluation (GRADE) approach was used to develop evidence-based guidelines. Our development of specific recommendations for treatment and chemoprophylaxis of sporadic H5N1 infection resulted from the benefits, harms, burden, and cost of interventions in several patient and exposure groups. Overall, the quality of the underlying evidence for all recommendations was rated as very low because it was based on small case series of H5N1 patients, on extrapolation from preclinical studies, and high quality studies of seasonal influenza. A strong recommendation to treat H5N1 patients with oseltamivir was made in part because of the severity of the disease. Similarly, strong recommendations were made to use neuraminidase inhibitors as chemoprophylaxis in high-risk exposure populations. Emergence of other novel influenza A viral subtypes with pandemic potential, or changes in the pathogenicity of H5N1 virus strains, will require an update of these guidelines and WHO will be monitoring this closely.

The role of value for money in public insurance coverage decisions for drugs in Australia: a retrospective analysis 1994-2004

Objective . To analyze the relative influence of factors in decisions for public insurance coverage of new drugs in Australia. Data Sources . Evidence presented at meetings of the Australian Pharmaceutical Benefits Advisory Committee (PBAC) that makes recommendations on coverage of drugs under Pharmaceutical Benefits Scheme. Study Selection . All major submissions to the PBAC between February 1994 and December 2004 (n = 858) if one of the outcomes measured was life year gained (n=138) or quality-adjusted life years (QALYs) gained (n=116). Results . Clinical significance, cost-effectiveness, cost to government, and severity of disease were significant influences on decisions. Compared to the average submission, clinical significance increased the probability of recommending coverage by 0.21 (95% confidence interval [CI] 0.02 to 0.40), whereas a drug in a life-threatening condition had an increased probability of being recommended for coverage of 0.38 (0.06 to 0.69). An increase in

Response rate or time to progression as predictors of survival in trials of metastatic colorectal cancer or non-small-cell lung cancer: a meta-analysis

A10,000 from a mean incremental cost per QALY of

GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence

A46,400 reduced the probability of listing by 0.06 (95% CI 0.04 to 0.1). Conclusions . The PBAC provides an example of the long-term stability and coherence of evidence-based coverage and pricing decisions for drugs that weighs up the evidence on clinical effectiveness, clinical need, and value for money. There is no evidence of a fixed public threshold value of life years or QALYs, but willingness to pay is clearly related to the characteristics of the clinical condition, perceived confidence in the evidence of effectiveness and its relevance, as well as total cost to government.

Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results

BACKGROUND The duration and cost of cancer clinical trials could be reduced if a surrogate endpoint were used in place of survival. We did a meta-analysis to assess the extent to which two surrogates, tumour response and time to progression, are predictive of mortality in metastatic colorectal cancer and non-small-cell lung cancer. METHODS Summary data (median time to progression, proportion of patients responding to treatment, and median overall survival) from randomised trials of first-line treatment in colorectal cancer (146 trials) and lung cancer (191 trials) were identified. Data were extracted and analysed by linear regression. We used prediction bands for trials with 250, 500, and 750 patients to identify the surrogate threshold effect that would predict a significant difference in survival. FINDINGS Response to treatment and time to progression correlated with improvement in survival for both lung cancer (p<0.0001 and p=0.0003, respectively) and colorectal cancer (p<0.0001 for both). To predict a significant survival gain in colorectal cancer trials, an improvement of 20% in the number of patients responding to treatment was required in trials with 750 patients, increasing to 26% in trials with 500 patients and 38% in trials with 250 patients. In lung cancer trials, the same prediction required differences in response of 18% for 750 patients, 21% for 500 patients, and 30% for 250 patients. For time to progression for both cancer types, the incremental gain needed to predict a survival improvement was a median of 1.8 months for trials with 750 patients, 2.2 months for 500 patients, and 3.3 months for 250 patients. INTERPRETATION Irrespective of trial size, large differences in tumour response rate are needed to predict a significant survival benefit. If surrogates are chosen as the primary endpoint in a clinical trial, time to progression is the preferred measure because more modest and achievable differences are needed for a significant survival benefit. Trials in metastatic lung cancer and colorectal cancer should measure survival as their primary outcome unless the surrogate outcome difference is anticipated to exceed the threshold effect size.

The Effects of Information Framing on the Practices of Physicians

OBJECTIVES In this article, we describe how to include considerations about resource utilization when making recommendations according to the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach. STUDY DESIGN AND SETTINGS We focus on challenges with rating the confidence in effect estimates (quality of evidence) and incorporating resource use into evidence profiles and Summary of Findings (SoF) tables. RESULTS GRADE recommends that important differences in resource use between alternative management strategies should be included along with other important outcomes in the evidence profile and SoF table. Key steps in considering resources in making recommendations with GRADE are the identification of items of resource use that may differ between alternative management strategies and that are potentially important to decision makers, finding evidence for the differences in resource use, making judgments regarding confidence in effect estimates using the same criteria used for health outcomes, and valuing the resource use in terms of costs for the specific setting for which recommendations are being made. CONCLUSIONS With our framework, decision makers will have access to concise summaries of recommendations, including ratings of the quality of economic evidence, and better understand the implications for clinical decision making.

Challenges in developing evidence-based recommendations using the GRADE approach: the case of mental, neurological, and substance use disorders.

BackgroundHealthcare decision makers face challenges when using guidelines, including understanding the quality of the evidence or the values and preferences upon which recommendations are made, which are often not clear.MethodsGRADE is a systematic approach towards assessing the quality of evidence and the strength of recommendations in healthcare. GRADE also gives advice on how to go from evidence to decisions. It has been developed to address the weaknesses of other grading systems and is now widely used internationally. The Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) consortium (http://www.decide-collaboration.eu/), which includes members of the GRADE Working Group and other partners, will explore methods to ensure effective communication of evidence-based recommendations targeted at key stakeholders: healthcare professionals, policymakers, and managers, as well as patients and the general public. Surveys and interviews with guideline producers and other stakeholders will explore how presentation of the evidence could be improved to better meet their information needs. We will collect further stakeholder input from advisory groups, via consultations and user testing; this will be done across a wide range of healthcare systems in Europe, North America, and other countries. Targeted communication strategies will be developed, evaluated in randomized trials, refined, and assessed during the development of real guidelines.DiscussionResults of the DECIDE project will improve the communication of evidence-based healthcare recommendations. Building on the work of the GRADE Working Group, DECIDE will develop and evaluate methods that address communication needs of guideline users. The project will produce strategies for communicating recommendations that have been rigorously evaluated in diverse settings, and it will support the transfer of research into practice in healthcare systems globally.