Robbie Foy

What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review

BackgroundImproving the quality of health care requires a range of evidence-based activities. Audit and feedback is commonly used as a quality improvement tool in the UK National Health Service [NHS]. We set out to assess whether current guidance and systematic review evidence can sufficiently inform practical decisions about how to use audit and feedback to improve quality of care.MethodsWe selected an important chronic disease encountered in primary care: diabetes mellitus. We identified recommendations from National Institute for Clinical Excellence (NICE) guidance on conducting audit and generated questions which would be relevant to any attempt to operationalise audit and feedback in a healthcare service setting. We explored the extent to which a systematic review of audit and feedback could provide practical guidance about whether audit and feedback should be used to improve quality of diabetes care and, if so, how audit and feedback could be optimised.ResultsNational guidance suggests the importance of securing the right organisational conditions and processes. Review evidence suggests that audit and feedback can be effective in changing healthcare professional practice. However, the available evidence says relatively little about the detail of how to use audit and feedback most efficiently.ConclusionAudit and feedback will continue to be an unreliable approach to quality improvement until we learn how and when it works best. Conceptualising audit and feedback within a theoretical framework offers a way forward.

Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting

BackgroundProcess evaluations are recommended to open the ‘black box’ of complex interventions evaluated in trials, but there is limited guidance to help researchers design process evaluations. Much current literature on process evaluations of complex interventions focuses on qualitative methods, with less attention paid to quantitative methods. This discrepancy led us to develop our own framework for designing process evaluations of cluster-randomised controlled trials.MethodsWe reviewed recent theoretical and methodological literature and selected published process evaluations; these publications identified a need for structure to help design process evaluations. We drew upon this literature to develop a framework through iterative exchanges, and tested this against published evaluations.ResultsThe developed framework presents a range of candidate approaches to understanding trial delivery, intervention implementation and the responses of targeted participants. We believe this framework will be useful to others designing process evaluations of complex intervention trials. We also propose key information that process evaluations could report to facilitate their identification and enhance their usefulness.ConclusionThere is no single best way to design and carry out a process evaluation. Researchers will be faced with choices about what questions to focus on and which methods to use. The most appropriate design depends on the purpose of the process evaluation; the framework aims to help researchers make explicit their choices of research questions and methods.Trial registrationClinicaltrials.gov NCT01425502

Advancing the science of patient safety

Despite a decades worth of effort, patient safety has improved slowly, in part because of the limited evidence base for the development and widespread dissemination of successful patient safety practices. The Agency for Healthcare Research and Quality sponsored an international group of experts in patient safety and evaluation methods to develop criteria to improve the design, evaluation, and reporting of practice research in patient safety. This article reports the findings and recommendations of this group, which include greater use of theory and logic models, more detailed descriptions of interventions and their implementation, enhanced explanation of desired and unintended outcomes, and better description and measurement of context and of how context influences interventions. Using these criteria and measuring and reporting contexts will improve the science of patient safety.

Attributes of clinical recommendations that influence change in practice following audit and feedback.

The object of this study was to determine which attributes of clinical practice recommendations influence changes in clinical practice following audit and feedback. This was an observational study using multilevel modeling to examine the relationship between attributes of clinical practice recommendations and compliance with the recommendations before and after audit and feedback. Sixteen hospital gynecology units in Scotland participated in a national audit project. Clinical practice recommendations covering selected gynecological topics were developed and data collected to assess baseline (preintervention) compliance. Summaries of performance were fed back to consultant gynecologists in each hospital and follow-up (postintervention) data were collected. Trained audit assistants used standardized forms to abstract data from case notes. Compliance data were available at baseline and follow-up for a total of 42 clinical practice recommendations. Altogether, 4,664 case notes contributed to baseline data and 4,382 to follow-up data. Thirteen attributes describing clinical practice recommendations were developed, based upon previous work, and pretested. A panel of seven consultant gynecologists rated the extent to which each of the 42 recommendations possessed each of the 13 attributes. The main outcome measures were the association of each attribute with compliance and with changes in clinical practice. Recommendations compatible with clinician values and not requiring changes to fixed routines were independently associated with greater compliance at baseline and follow-up. However, recommendations incompatible with clinician values were independently associated with greater change in practice following audit and feedback. Attributes of recommendations may influence the effectiveness of audit and feedback in secondary care. Recommendations seen as incompatible with clinician values are associated with lower compliance but greater behavioral change following audit and feedback.

What context features might be important determinants of the effectiveness of patient safety practice interventions

Background Differences in contexts (eg, policies, healthcare organisation characteristics) may explain variations in the effects of patient safety practice (PSP) implementations. However, knowledge of which contextual features are important determinants of PSP effectiveness is limited and consensus is lacking on a taxonomy of which contexts matter. Methods Iterative, formal discussions were held with a 22-member technical expert panel composed of experts or leaders in patient safety, healthcare systems, and methods. First, potentially important contextual features were identified, focusing on five PSPs. Then, two surveys were conducted to determine the context likely to influence PSP implementations. Results The panel reached a consensus on a taxonomy of four broad domains of contextual features important for PSP implementations: safety culture, teamwork and leadership involvement; structural organisational characteristics (eg, size, organisational complexity or financial status); external factors (eg, financial or performance incentives or PSP regulations); and availability of implementation and management tools (eg, training organisational incentives). Panelists also tended to rate specific patient safety culture, teamwork and leadership contexts as high priority for assessing their effects on PSP implementations, but tended to rate specific organisational characteristic contexts as high priority only for use in PSP evaluations. Panelists appeared split on whether specific external factors and implementation/management tools were important for assessment or only description. Conclusion This work can guide research commissioners and evaluators on the contextual features of PSP implementations that are important to report or evaluate. It represents a first step towards developing guidelines on contexts in PSP implementation evaluations. However, the science of context measurement needs maturing.

Meta-analysis: Effect of Interactive Communication Between Collaborating Primary Care Physicians and Specialists

BACKGROUND Whether collaborative care models that enable interactive communication (timely, 2-way exchange of pertinent clinical information directly between primary care and specialist physicians) improve patient outcomes is uncertain. PURPOSE To assess the effects of interactive communication between collaborating primary care physicians and key specialists on outcomes for patients receiving ambulatory care. DATA SOURCES PubMed, PsycInfo, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, and Web of Science through June 2008 and secondary references, with no language restriction. STUDY SELECTION Studies that evaluated the effects of interactive communication between collaborating primary care physicians and specialists on outcomes for patients with diabetes, psychiatric conditions, or cancer. DATA EXTRACTION Contextual, intervention, and outcome data from 23 studies were extracted by one reviewer and checked by another. Study quality was assessed with a 13-item checklist. Disagreement was resolved by consensus. Main outcomes for analysis were selected by reviewers who were blinded to study results. DATA SYNTHESIS Meta-analysis indicated consistent effects across 11 randomized mental health studies (pooled effect size, -0.41 [95% CI, -0.73 to -0.10]), 7 nonrandomized mental health studies (pooled effect size, -0.47 [CI, -0.84 to -0.09]), and 5 nonrandomized diabetes studies (pooled effect size, -0.64 [CI, -0.93 to -0.34]). These findings remained robust to sensitivity analyses. Meta-regression indicated studies that included interventions to enhance the quality of information exchange had larger effects on patient outcomes than those that did not (-0.84 vs. -0.27; P = 0.002). LIMITATIONS Because collaborative interventions were inherently multifaceted, the efficacy of interactive communication by itself cannot be established. Inclusion of study designs with lower internal validity increased risk for bias. No studies involved oncologists. CONCLUSION Consistent and clinically important effects suggest a potential role of interactive communication for improving the effectiveness of primary care-specialist collaboration. PRIMARY FUNDING SOURCE RAND Healths Comprehensive Assessment of Reform Options Initiative, the Veterans Affairs Center for the Study of Provider Behavior, The Commonwealth Fund, and the Health Foundation.

No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention

BackgroundAudit and feedback interventions in healthcare have been found to be effective, but there has been little progress with respect to understanding their mechanisms of action or identifying their key ‘active ingredients.’DiscussionGiven the increasing use of audit and feedback to improve quality of care, it is imperative to focus further research on understanding how and when it works best. In this paper, we argue that continuing the ‘business as usual’ approach to evaluating two-arm trials of audit and feedback interventions against usual care for common problems and settings is unlikely to contribute new generalizable findings. Future audit and feedback trials should incorporate evidence- and theory-based best practices, and address known gaps in the literature.SummaryWe offer an agenda for high-priority research topics for implementation researchers that focuses on reviewing best practices for designing audit and feedback interventions to optimize effectiveness.

Appropriate disclosure of a diagnosis of dementia: identifying the key behaviours of 'best practice'

BackgroundDespite growing evidence that many people with dementia want to know their diagnosis, there is wide variation in attitudes of professionals towards disclosure. The disclosure of the diagnosis of dementia is increasingly recognised as being a process rather than a one-off behaviour. However, the different behaviours that contribute to this process have not been comprehensively defined. No intervention studies to improve diagnostic disclosure in dementia have been reported to date. As part of a larger study to develop an intervention to promote appropriate disclosure, we sought to identify important disclosure behaviours and explore whether supplementing a literature review with other methods would result in the identification of new behaviours.MethodsTo identify a comprehensive list of behaviours in disclosure we conducted a literature review, interviewed people with dementia and informal carers, and used a consensus process involving health and social care professionals. Content analysis of the full list of behaviours was carried out.ResultsInterviews were conducted with four people with dementia and six informal carers. Eight health and social care professionals took part in the consensus panel. From the interviews, consensus panel and literature review 220 behaviours were elicited, with 109 behaviours over-lapping. The interviews and consensus panel elicited 27 behaviours supplementary to the review. Those from the interviews appeared to be self-evident but highlighted deficiencies in current practice and from the panel focused largely on balancing the needs of people with dementia and family members. Behaviours were grouped into eight categories: preparing for disclosure; integrating family members; exploring the patients perspective; disclosing the diagnosis; responding to patient reactions; focusing on quality of life and well-being; planning for the future; and communicating effectively.ConclusionThis exercise has highlighted the complexity of the process of disclosing a diagnosis of dementia in an appropriate manner. It confirms that many of the behaviours identified in the literature (often based on professional opinion rather than empirical evidence) also resonate with people with dementia and informal carers. The presence of contradictory behaviours emphasises the need to tailor the process of disclosure to individual patients and carers. Our combined methods may be relevant to other efforts to identify and define complex clinical practices for further study.

Perspectives of commissioners and cancer specialists in prioritising new cancer drugs: impact of the evidence threshold

The provision of high quality cancer services is a major priority for the NHS.1 However, greater budgetary pressures are being placed on specialist hospitals and health authorities by several factors: Christie Hospital NHS Trust in Manchester provides specialist cancer care to a population of 4.5million people from North Wales to Cumbria. The trusts drug budget of £4 million accounted for 18% of the total income from patient services in 1996-7,creating an obvious focus for annual negotiations with commissioning health authorities. Historical underfunding of the trusts drug budget, relative to that of other oncology centres, further supported the trusts case for increased resources. Clinicians highlighted the paradox whereby only 1% of the annual NHS drug budget (£58 million) was allocated to cytotoxic drugs for treating a disease that would affect at least one in three of the population, while £200 million was spent on treating constipation.2–4 The trust was concerned that individual health authorities within its catchment area would negotiate different contracts, resulting in inequitable access to new developments, as has previously occurred with cancer treatments.5 A Greater Manchester consortium of six health authorities was established, partly as a response to this and partly to develop more efficient commissioning. The consortium, responsible for purchasing around two thirds of services from the hospital, also advised neighbouring health authorities in the catchment area. It soon became clear that the trusts proposed developments could not be met in full from available resources and the consortium agreed to set priorities in funding drug developments. This paper describes our joint experiences in prioritising drug proposals for 1997-8. ### Summary points

A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems

BackgroundImplementing new practices requires changes in the behaviour of relevant actors, and this is facilitated by understanding of the determinants of current and desired behaviours. The Theoretical Domains Framework (TDF) was developed by a collaboration of behavioural scientists and implementation researchers who identified theories relevant to implementation and grouped constructs from these theories into domains. The collaboration aimed to provide a comprehensive, theory-informed approach to identify determinants of behaviour. The first version was published in 2005, and a subsequent version following a validation exercise was published in 2012. This guide offers practical guidance for those who wish to apply the TDF to assess implementation problems and support intervention design. It presents a brief rationale for using a theoretical approach to investigate and address implementation problems, summarises the TDF and its development, and describes how to apply the TDF to achieve implementation objectives. Examples from the implementation research literature are presented to illustrate relevant methods and practical considerations.MethodsResearchers from Canada, the UK and Australia attended a 3-day meeting in December 2012 to build an international collaboration among researchers and decision-makers interested in the advancing use of the TDF. The participants were experienced in using the TDF to assess implementation problems, design interventions, and/or understand change processes. This guide is an output of the meeting and also draws on the authors’ collective experience. Examples from the implementation research literature judged by authors to be representative of specific applications of the TDF are included in this guide.ResultsWe explain and illustrate methods, with a focus on qualitative approaches, for selecting and specifying target behaviours key to implementation, selecting the study design, deciding the sampling strategy, developing study materials, collecting and analysing data, and reporting findings of TDF-based studies. Areas for development include methods for triangulating data, e.g. from interviews, questionnaires and observation and methods for designing interventions based on TDF-based problem analysis.ConclusionsWe offer this guide to the implementation community to assist in the application of the TDF to achieve implementation objectives. Benefits of using the TDF include the provision of a theoretical basis for implementation studies, good coverage of potential reasons for slow diffusion of evidence into practice and a method for progressing from theory-based investigation to intervention.