Tessa Tan-Torres Edejer

Grading quality of evidence and strength of recommendations.

Abstract Users of clinical practice guidelines and other recommendations need to know how much confidence they can place in the recommendations. Systematic and explicit methods of making judgments can reduce errors and improve communication. We have developed a system for grading the quality of evidence and the strength of recommendations that can be applied across a wide range of interventions and contexts. In this article we present a summary of our approach from the perspective of a guideline user. Judgments about the strength of a recommendation require consideration of the balance between benefits and harms, the quality of the evidence, translation of the evidence into specific circumstances, and the certainty of the baseline risk. It is also important to consider costs (resource utilisation) before making a recommendation. Inconsistencies among systems for grading the quality of evidence and the strength of recommendations reduce their potential to facilitate critical appraisal and improve communication of these judgments. Our system for guiding these complex judgments balances the need for simplicity with the need for full and transparent consideration of all important issues. Clinical guidelines are only as good as the evidence and judgments they are based on. The GRADE approach aims to make it easier for users to assess the judgments behind recommendations

Systems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approaches The GRADE Working Group

BackgroundA number of approaches have been used to grade levels of evidence and the strength of recommendations. The use of many different approaches detracts from one of the main reasons for having explicit approaches: to concisely characterise and communicate this information so that it can easily be understood and thereby help people make well-informed decisions. Our objective was to critically appraise six prominent systems for grading levels of evidence and the strength of recommendations as a basis for agreeing on characteristics of a common, sensible approach to grading levels of evidence and the strength of recommendations.MethodsSix prominent systems for grading levels of evidence and strength of recommendations were selected and someone familiar with each system prepared a description of each of these. Twelve assessors independently evaluated each system based on twelve criteria to assess the sensibility of the different approaches. Systems used by 51 organisations were compared with these six approaches.ResultsThere was poor agreement about the sensibility of the six systems. Only one of the systems was suitable for all four types of questions we considered (effectiveness, harm, diagnosis and prognosis). None of the systems was considered usable for all of the target groups we considered (professionals, patients and policy makers). The raters found low reproducibility of judgements made using all six systems. Systems used by 51 organisations that sponsor clinical practice guidelines included a number of minor variations of the six systems that we critically appraised.ConclusionsAll of the currently used approaches to grading levels of evidence and the strength of recommendations have important shortcomings.

Systems for grading the quality of evidence and the strength of recommendations II: Pilot study of a new system

BackgroundSystems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.MethodsA GRADE evidence profile consists of two tables: a quality assessment and a summary of findings. Twelve evidence profiles were used in this pilot study. Each evidence profile was made based on information available in a systematic review. Seventeen people were given instructions and independently graded the level of evidence and strength of recommendation for each of the 12 evidence profiles. For each example judgements were collected, summarised and discussed in the group with the aim of improving the proposed grading system. Kappas were calculated as a measure of chance-corrected agreement for the quality of evidence for each outcome for each of the twelve evidence profiles. The seventeen judges were also asked about the ease of understanding and the sensibility of the approach. All of the judgements were recorded and disagreements discussed.ResultsThere was a varied amount of agreement on the quality of evidence for the outcomes relating to each of the twelve questions (kappa coefficients for agreement beyond chance ranged from 0 to 0.82). However, there was fair agreement about the relative importance of each outcome. There was poor agreement about the balance of benefits and harms and recommendations. Most of the disagreements were easily resolved through discussion. In general we found the GRADE approach to be clear, understandable and sensible. Some modifications were made in the approach and it was agreed that more information was needed in the evidence profiles.ConclusionJudgements about evidence and recommendations are complex. Some subjectivity, especially regarding recommendations, is unavoidable. We believe our system for guiding these complex judgements appropriately balances the need for simplicity with the need for full and transparent consideration of all important issues.

Generalized cost-effectiveness analysis for national-level priority-setting in the health sector

Cost-effectiveness analysis (CEA) is potentially an important aid to public health decision-making but, with some notable exceptions, its use and impact at the level of individual countries is limited. A number of potential reasons may account for this, among them technical shortcomings associated with the generation of current economic evidence, political expediency, social preferences and systemic barriers to implementation. As a form of sectoral CEA, Generalized CEA sets out to overcome a number of these barriers to the appropriate use of cost-effectiveness information at the regional and country level. Its application via WHO-CHOICE provides a new economic evidence base, as well as underlying methodological developments, concerning the cost-effectiveness of a range of health interventions for leading causes of, and risk factors for, disease.The estimated sub-regional costs and effects of different interventions provided by WHO-CHOICE can readily be tailored to the specific context of individual countries, for example by adjustment to the quantity and unit prices of intervention inputs (costs) or the coverage, efficacy and adherence rates of interventions (effectiveness). The potential usefulness of this information for health policy and planning is in assessing if current intervention strategies represent an efficient use of scarce resources, and which of the potential additional interventions that are not yet implemented, or not implemented fully, should be given priority on the grounds of cost-effectiveness.Health policy-makers and programme managers can use results from WHO-CHOICE as a valuable input into the planning and prioritization of services at national level, as well as a starting point for additional analyses of the trade-off between the efficiency of interventions in producing health and their impact on other key outcomes such as reducing inequalities and improving the health of the poor.

Disseminating health information in developing countries: the role of the internet

During a visit to India in March President Clinton watched a woman enter a village health centre, call up a web page on the computer, and get information on how to care for her baby.1 It is possible that this baby will have better health because of the availability of information on the internet. However, this possibility is underpinned by several assumptions, and the potential of information and communication technologies still has not been harnessed systematically to bring about important improvements in the health of populations, particularly among those who are poor and isolated in developing countries. This article explores the potential of advances in information and communication technologies to disseminate information and describes the availability of access to technology in developing countries. It also discusses issues of accuracy and the relevance of content. #### Summary points Information and communication technologies have not been harnessed systematically to improve the health of populations in developing countries These technologies empower those who use information by providing them with a choice of information to be accessed in their own time and by allowing them to put their own information on the web The current digital divide is more dramatic than any other inequity in health or income The quality of health information available on the web is inconsistent, and the visibility of research from developing countries is limited The way forward is to exploit the full interactivity of the internet, which allows rapid feedback and change to continuously mould information into useful knowledge Sources for this paper were obtained by searching Medline using the term dissemination and from visits to websites and links to related organisations known to be working in the areas of health research and information and communication technologies in developing countries. Information was also provided by key informants working in international health …

Cost effectiveness analysis of strategies for child health in developing countries.

Abstract Objective To determine the costs and effectiveness of selected child health interventions—namely, case management of pneumonia, oral rehydration therapy, supplementation or fortification of staple foods with vitamin A or zinc, provision of supplementary food with counselling on nutrition, and immunisation against measles. Design Cost effectiveness analysis. Data sources Efficacy data came from published systematic reviews and before and after evaluations of programmes. For resource inputs, quantities came from literature and expert opinion, and prices from the World Health Organization Choosing Interventions that are Cost Effective (WHO-CHOICE) database Results Cost effectiveness ratios clustered in three groups, with fortification with zinc or vitamin A as the most cost effective intervention, and provision of supplementary food and counselling on nutrition as the least cost effective. Between these were oral rehydration therapy, case management of pneumonia, vitamin A or zinc supplementation, and measles immunisation. Conclusions On the grounds of cost effectiveness, micronutrients and measles immunisation should be provided routinely to all children, in addition to oral rehydration therapy and case management of pneumonia for those who are sick. The challenge of malnutrition is not well addressed by existing interventions. This article is part of a series examining the cost effectiveness of strategies to achieve the millennium development goals for health

Methods to assess the costs and health effects of interventions for improving health in developing countries.

Assessment of the cost effectiveness of interventions designed to achieve the millennium development goals for health is complex. The methods must be capable of showing the efficiency with which current and possible new resources are used, and incorporating interactions between concurrent interventions and the effect of expanding coverage on unit costs.1 They should also allow valid comparisons across a wide range of interventions. Here we describe how the standardised cost effectiveness methods used in the World Health Organizations Choosing Interventions that are Cost Effective (CHOICE) project have tackled these issues. The analysis was performed for 14 regions classified by WHO according to their epidemiological grouping (table A on bmj.com). The regional results (except if not relevant to the disease area, for example, malaria) are available at www.who.int/choice, but the papers in this series give details for just two regions: Afr-E, which includes countries in sub-Saharan Africa with high child mortality and very high adult mortality, and Sear-D, which comprises countries in South East Asia with high child and adult mortality.2–6 The term intervention is defined to include any preventive, promotive, curative, or rehabilitative action that improves health. Interventions are analysed individually and then in combinations or packages that could be undertaken together (box 1), taking into account interactions in costs or effectiveness, or both. Interventions were chosen for analysis either because they are commonly used or because disease control experts have advocated their introduction. In each case, some evidence was needed that the intervention could be effective. The list is not exhaustive, and excluding an intervention does not imply it is cost ineffective. All interventions and combinations are assessed assuming they are implemented for 10 years starting in 2000, the year the Millennium Declaration was signed. Good policy making would then require a reassessment of …

Cost-effectiveness thresholds: pros and cons.

Abstract Cost–effectiveness analysis is used to compare the costs and outcomes of alternative policy options. Each resulting cost–effectiveness ratio represents the magnitude of additional health gained per additional unit of resources spent. Cost–effectiveness thresholds allow cost–effectiveness ratios that represent good or very good value for money to be identified. In 2001, the World Health Organization’s Commission on Macroeconomics in Health suggested cost–effectiveness thresholds based on multiples of a country’s per-capita gross domestic product (GDP). In some contexts, in choosing which health interventions to fund and which not to fund, these thresholds have been used as decision rules. However, experience with the use of such GDP-based thresholds in decision-making processes at country level shows them to lack country specificity and this – in addition to uncertainty in the modelled cost–effectiveness ratios – can lead to the wrong decision on how to spend health-care resources. Cost–effectiveness information should be used alongside other considerations – e.g. budget impact and feasibility considerations – in a transparent decision-making process, rather than in isolation based on a single threshold value. Although cost–effectiveness ratios are undoubtedly informative in assessing value for money, countries should be encouraged to develop a context-specific process for decision-making that is supported by legislation, has stakeholder buy-in, for example the involvement of civil society organizations and patient groups, and is transparent, consistent and fair.

Evaluation of current strategies and future priorities for improving health in developing countries

This article is the last in a series examining the cost effectiveness of strategies to achieve the millennium development goals for health

Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis

This Guidance for Priority Setting in Health Care (GPS-Health), initiated by the World Health Organization, offers a comprehensive map of equity criteria that are relevant to health care priority setting and should be considered in addition to cost-effectiveness analysis. The guidance, in the form of a checklist, is especially targeted at decision makers who set priorities at national and sub-national levels, and those who interpret findings from cost-effectiveness analysis. It is also targeted at researchers conducting cost-effectiveness analysis to improve reporting of their results in the light of these other criteria.The guidance was develop through a series of expert consultation meetings and involved three steps: i) methods and normative concepts were identified through a systematic review; ii) the review findings were critically assessed in the expert consultation meetings which resulted in a draft checklist of normative criteria; iii) the checklist was validated though an extensive hearing process with input from a range of relevant stakeholders.The GPS-Health incorporates criteria related to the disease an intervention targets (severity of disease, capacity to benefit, and past health loss); characteristics of social groups an intervention targets (socioeconomic status, area of living, gender; race, ethnicity, religion and sexual orientation); and non-health consequences of an intervention (financial protection, economic productivity, and care for others).