Sarah Rosenbaum

Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results

BackgroundHealthcare decision makers face challenges when using guidelines, including understanding the quality of the evidence or the values and preferences upon which recommendations are made, which are often not clear.MethodsGRADE is a systematic approach towards assessing the quality of evidence and the strength of recommendations in healthcare. GRADE also gives advice on how to go from evidence to decisions. It has been developed to address the weaknesses of other grading systems and is now widely used internationally. The Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) consortium (http://www.decide-collaboration.eu/), which includes members of the GRADE Working Group and other partners, will explore methods to ensure effective communication of evidence-based recommendations targeted at key stakeholders: healthcare professionals, policymakers, and managers, as well as patients and the general public. Surveys and interviews with guideline producers and other stakeholders will explore how presentation of the evidence could be improved to better meet their information needs. We will collect further stakeholder input from advisory groups, via consultations and user testing; this will be done across a wide range of healthcare systems in Europe, North America, and other countries. Targeted communication strategies will be developed, evaluated in randomized trials, refined, and assessed during the development of real guidelines.DiscussionResults of the DECIDE project will improve the communication of evidence-based healthcare recommendations. Building on the work of the GRADE Working Group, DECIDE will develop and evaluate methods that address communication needs of guideline users. The project will produce strategies for communicating recommendations that have been rigorously evaluated in diverse settings, and it will support the transfer of research into practice in healthcare systems globally.

GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction

#### Summary points Healthcare decision making is complex. Decision-making processes and the factors (criteria) that decision makers should consider vary for different types of decisions, including clinical recommendations, coverage decisions, and health system or public health recommendations or decisions.1 2 3 4 However, some criteria are relevant for all of these decisions, including the anticipated effects of the options being considered, the certainty of the evidence for those effects (also referred to as quality of evidence or confidence in effect estimates), and the costs and feasibility of the options. Decision makers must make judgments about each relevant factor, informed by the best evidence that is available to them. Often, the processes that decision makers use, the criteria that they consider and the evidence that they …

User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews.

OBJECTIVE To develop a Summary of Findings (SoF) table for use in Cochrane reviews that is understandable and useful for health professionals, acceptable to Cochrane Collaboration stakeholders, and feasible to implement. STUDY DESIGN AND SETTING We gathered stakeholder feedback on the format and content of an SoF table from an advisory group of more than 50 participants and their constituencies through e-mail consultations. We conducted user tests using a think-aloud protocol method, collecting feedback from 21 health professionals and researchers in Norway and the UK. We analyzed the feedback, defined problem areas, and generated new solutions in brainstorming workshops. RESULTS Stakeholders were concerned about precision in the data representation and about production feasibility. User testing revealed unexpected comprehension problems, mainly confusion about what the different numbers referred to (class reference). Resolving the tension between achieving table precision and table simplicity became the main focus of the working group. CONCLUSION User testing led to a table more useful and understandable for clinical audiences. We arrived at an SoF table that was acceptable to the stakeholders and in principle feasible to implement technically. Some challenges remain, including presenting continuous outcomes and technical/editorial implementation.

Presenting the Results of Cochrane Systematic Reviews to a Consumer Audience: A Qualitative Study

Objective. To develop and obtain feedback about a summary format for Cochrane reviews that is accessible to a consumer audience, without oversimplification or incorrect presentation. Methods. We developed 3 versions of a Plain Language Summary (PLS) format of a Cochrane Systematic Review. Using a semi-structured interview guide we tested these versions among 34 members of the public in Norway, Argentina, Canada, and Australia. The authors analyzed feedback, identified problems, and generated new solutions before retesting to produce a final version of a Plain Language Summary format. Results. Participants preferred results presented as words, supplemented by numbers in a table. There was a lack of understanding regarding the difference between a review and an individual study, that the effect is rarely an exact number, that evidence can be of low or high quality, and that level of quality is a separate issue from intervention effect. Participants also found it difficult to move between presentations of dichotomous and continuous outcomes. Rephrasing the introduction helped participants grasp the concept of a review. Confidence intervals were largely ignored or misunderstood. Our attempts to explain them were only partially successful. Text modifiers (‘‘probably,’’ ‘‘may’’) to convey different levels of quality were only partially understood, whereas symbols with explanations were more helpful. Participants often understood individual information elements about effect size and quality of these results, but did not always actively merge these elements. Conclusion. Through testing and iteration the authors identified and addressed several problems, using explanations, rephrasing, and symbols to present scientific concepts. Other problems remain, including how best to present confidence intervals and continuous outcomes. Future research should also test information elements in combination rather than in isolation. The new Plain Language Summary format is being evaluated in a randomized controlled trial.

GRADE Guidelines : 16. GRADE evidence to decision frameworks for tests in clinical practice and public health

OBJECTIVES To describe the grading of recommendations assessment, development and evaluation (GRADE) interactive evidence to decision (EtD) frameworks for tests and test strategies for clinical, public health, or coverage decisions. STUDY DESIGN AND SETTING As part of the GRADE Working Groups DECIDE project, we conducted workshops, user testing with systematic review authors, guideline developers and other decision makers, and piloted versions of the EtD framework. RESULTS EtD frameworks for tests share the structure, explicitness, and transparency of other EtD frameworks. They require specifying the purpose of the test, linked or related management, and the key outcomes of concern for different test results and subsequent management. The EtD criteria address test accuracy and assessments of the certainty of the additional evidence necessary for decision making. When there is no direct evidence of test effects on patient-important outcomes, formal or informal modeling is needed to estimate effects. We describe the EtD criteria based on examples developed with GRADEpro (www.gradepro.org), GRADEs software that also allows development and dissemination of interactive summary of findings tables. CONCLUSION EtD frameworks for developing recommendations and making decisions about tests lay out the sequential steps in reviewing and assessing the different types of evidence that need to be linked.

Policymakers' and other stakeholders' perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: An international survey

BackgroundThe DECIDE framework was developed to support evidence-informed health system decisions through evidence summaries tailored to health policymakers. The objective of this study was to determine policymakers’ perceptions regarding the criteria in the DECIDE framework and how best to summarise and present evidence to support health system decisions.MethodsWe conducted an online survey of a diverse group of stakeholders with health system decision experience from 15 countries and the World Health Organization. We asked about perceptions of criteria relevant to making health system decisions, use of evidence, grading systems, and evidence summaries.ResultsWe received 112 responses (70% response rate). Most respondents had healthcare (85%) and research (79%) experience. They (99%) indicated that systematic consideration of the available evidence would help to improve health system decision-making processes and supported the use of evidence from other countries (94%) and grading systems (81%). All ten criteria in the DECIDE framework were rated as important in the decision-making process. Respondents had divergent views regarding whether the same (38%) or different (45%) grading systems should be used across different types of health decisions. All components of our evidence summary were rated as important by over 90% of respondents.ConclusionsSurvey respondents were supportive of the DECIDE framework for health system decisions and the use of succinct summaries of the estimated size of effects and the quality of evidence. It is uncertain whether the findings of this survey represent the views of policymakers with little or no healthcare and research experience.

A summary to communicate evidence from systematic reviews to the public improved understanding and accessibility of information: a randomized controlled trial.

OBJECTIVES To evaluate a new format of a summary, which presents research from synthesized evidence to patients and the public. STUDY DESIGN AND SETTING We conducted a randomized controlled trial in 143 members of the public from five countries (Canada, Norway, Spain, Argentina, and Italy). Participants received either a new summary format (a plain language summary [PLS]) or the current format used in Cochrane systematic reviews. The new PLS presents information about the condition and intervention, a narrative summary of results, and a table of results with absolute numbers for effects of the intervention and quality of the evidence using Grading of Recommendations Assessment, Development, and Evaluation. RESULTS With the new PLS, more participants understood the benefits and harms and quality of evidence (53% vs. 18%, P < 0.001); more answered each of the five questions correctly (P ≤ 0.001 for four questions); and they answered more questions correctly, median 3 (interquartile range [IQR]: 1-4) vs. 1 (IQR: 0-1), P < 0.001). Better understanding was independent of education level. More participants found information in the new PLS reliable, easy to find, easy to understand, and presented in a way that helped make decisions. Overall, participants preferred the new PLS. CONCLUSION This new PLS format for patients and the public is a promising tool to translate evidence from synthesized research.

Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format

OBJECTIVES The current format of summary of findings (SoFs) tables for presenting effect estimates and associated quality of evidence improve understanding and assist users finding key information in systematic reviews. Users of SoF tables have demanded alternative formats to express findings from systematic reviews. STUDY DESIGN AND SETTING We conducted a randomized controlled trial among systematic review users to compare the relative merits of a new format with the current formats of SoF tables regarding understanding, accessibility of information, satisfaction, and preference. Our primary goal was to show that the new format is not inferior to the current format. RESULTS Of 390 potentially eligible subjects, 290 were randomized. Of seven items testing understanding, three showed similar results, two showed small differences favoring the new format, and two (understanding risk difference and quality of the evidence associated with a treatment effect) showed large differences favoring the new format [63% (95% confidence interval {CI}: 55, 71) and 62% (95% CI: 52, 71) more correct answers, respectively]. Respondents rated information in the alternative format as more accessible overall and preferred the new format over the current format. CONCLUSIONS While providing at least similar levels of understanding for some items and increased understanding for others, users prefer the new format of SoF tables.

Development of a Novel, Multilayered Presentation Format for Clinical Practice Guidelines

BACKGROUND Bridging the gap between clinical research and everyday health-care practice requires effective communication strategies. To address current shortcomings in conveying practice recommendations and supporting evidence, we are creating and testing presentation formats for clinical practice guidelines (CPGs). METHODS We carried out multiple cycles of brainstorming and sketching, developing a prototype. Physicians participating in the user testing viewed CPG formats linked to clinical scenarios and engaged in semistructured interviews applying a think-aloud method for exploring important aspects of user experience. RESULTS We developed a multilayered presentation format that allows clinicians to successively view more in-depth information. Starting with the recommendations, clinicians can, on demand, access a rationale and a key information section containing statements on quality of the evidence, balance between desirable and undesirable consequences, values and preferences, and resource considerations. We collected feedback from 27 stakeholders and performed user testing with 47 practicing physicians from six countries. Advisory group feedback and user testing of the first version revealed problems with conceptual understanding of underlying CPG methodology, as well as difficulties with the complexity of the layout and content. Extensive revisions made before the second round of user testing resulted in most participants expressing overall satisfaction with the final presentation format. CONCLUSIONS We have developed an electronic, multilayered, CPG format that enhances the usability of CPGs for frontline clinicians. We have implemented the format in electronic guideline tools that guideline organizations can now use when authoring and publishing their guidelines.

Key concepts that people need to understand to assess claims about treatment effects.

People are confronted with claims about the effects of treatments and health policies daily. Our objective was to develop a list of concepts that may be important for people to understand when assessing claims about treatment effects.