Annette Boaz

Social science and the evidence-based policy movement

There is a growing interest in ‘ evidence-based policy making’ in the UK. However, there remains some confusion about what evidence-based policy making actually means. This paper outlines some of the models used to understand how evidence is thought to shape or inform policy in order to explore the assumptions underlying ‘evidence-based policy making.’ By way of example, it considers the process of evidence seeking and in particular the systematic review as a presumed ‘gold standard’ of the EBP movement. It highlights some of the opportunities and challenges represented in this approach for policy research. The final part of the paper outlines some questions of capacity that need to be addressed if the social sciences are to make a more effective contribution to policy debate in Britain.

Effective implementation of research into practice: an overview of systematic reviews of the health literature.

BackgroundThe gap between research findings and clinical practice is well documented and a range of interventions has been developed to increase the implementation of research into clinical practice.FindingsA review of systematic reviews of the effectiveness of interventions designed to increase the use of research in clinical practice. A search for relevant systematic reviews was conducted of Medline and the Cochrane Database of Reviews 1998-2009. 13 systematic reviews containing 313 primary studies were included. Four strategy types are identified: audit and feedback; computerised decision support; opinion leaders; and multifaceted interventions. Nine of the reviews reported on multifaceted interventions. This review highlights the small effects of single interventions such as audit and feedback, computerised decision support and opinion leaders. Systematic reviews of multifaceted interventions claim an improvement in effectiveness over single interventions, with effect sizes ranging from small to moderate. This review found that a number of published systematic reviews fail to state whether the recommended practice change is based on the best available research evidence.ConclusionsThis overview of systematic reviews updates the body of knowledge relating to the effectiveness of key mechanisms for improving clinical practice and service development. Multifaceted interventions are more likely to improve practice than single interventions such as audit and feedback. This review identified a small literature focusing explicitly on getting research evidence into clinical practice. It emphasizes the importance of ensuring that primary studies and systematic reviews are precise about the extent to which the reported interventions focus on changing practice based on research evidence (as opposed to other information codified in guidelines and education materials).

Enhancing the evidence base for health impact assessment

Health impact assessment differs from other purposes for which evidence is collated in a number of ways, including: the focus on complex interventions or policy and their diverse effects on determinants of health; the need for evidence on the reversibility of adverse factors damaging to health; the diversity of the evidence in terms of relevant disciplines, study designs, quality criteria and sources of information; the broad range of stakeholders involved; the short timescale and limited resources generally available; the pragmatic need to inform decision makers regardless of the quality of the evidence. These have implications for commissioning and conducting reviews. Methods must be developed to: facilitate comprehensive searching across a broad range of disciplines and information sources; collate appropriate quality criteria to assess a range of study designs; synthesise different kinds of evidence; and facilitate timely stakeholder involvement. Good practice standards for reviews are needed to reduce the risk of poor quality recommendations. Advice to decision makers must make explicit limitations resulting from absent, conflicting, or poor quality evidence.

Assessing the impact of research on policy: a literature review

Understanding the impact of research on policy is a vital, and often overlooked, element of policy-making. A systematic literature review was conducted to examine methods for evaluating the impact of research on policy outcomes. The review focused in particular on strategic policy levels (rather than implementation) and waste, environment and pollution policy. The review draws on an international literature, although it is limited to English language publications. The findings identify the different methods used, the advantages and disadvantages of different approaches and the methods that are most effective (particularly in terms of cost). The field of research impact evaluation is currently experiencing rapid development, and there is scope to develop and apply new conceptual frameworks and innovative methods for evaluation, and to conduct and publish more research impact evaluations.

Evidence and policy in six European countries: diverse approaches and common challenges

This issue of Evidence & Policy contributes to these efforts to learn from international developments in the evidence and policy field. It draws together papers from a seminar series that considered developments in six European countries: Iceland, Ireland, the Netherlands, Norway, Scotland and Sweden. The seminar series was funded through the New Opportunities for Research Funding Agency Co operation in Europe (NORFACE) initiative,2 a partnership between 14 research councils to increase cooperation in research and research policy in Europe, and we gratefully acknowledge their support.

The Perilous Road from Evidence to Policy: Five Journeys Compared

Comprehensive reviews of the available research are generally considered to be the cornerstone of contemporary efforts to establish evidence- based policy. This article provides an examination of the potential of this stratagem using the case study of ‘mentoring’ programmes. Mentoring initiatives (and allied schemes such as ‘coaching counselling peer education and so on) are to be found in every corner of public policy. Researchers have been no less energetic producing a huge body of evidence on the process and outcomes of such interventions. Reviewers accordingly have plenty to get their teeth into and by now there are numerous reports offering review-based advice on the benefits of mentoring. The article asks whether the sum total of these efforts as represented by five contemporary reviews is a useful tool for guiding policy and practice. Our analysis is a cause for some pessimism. We note a propensity for delivering unequivocal policy verdicts on the basis of ambiguous evidence. Even more disconcertingly the five reviews head off on different judgmental tangents one set of recommendations appearing to gainsay the next. The article refrains from recommending the ejection of evidence baby and policy bathwater but suggests that much closer attention needs to be paid to the explanatory scope of systematic reviews. (authors)

Research Activity and the Association with Mortality

Introduction The aims of this study were to describe the key features of acute NHS Trusts with different levels of research activity and to investigate associations between research activity and clinical outcomes. Methods National Institute for Health Research (NIHR) Comprehensive Clinical Research Network (CCRN) funding and number of patients recruited to NIHR Clinical Research Network (CRN) portfolio studies for each NHS Trusts were used as markers of research activity. Patient-level data for adult non-elective admissions were extracted from the English Hospital Episode Statistics (2005-10). Risk-adjusted mortality associations between Trust structures, research activity and, clinical outcomes were investigated. Results Low mortality Trusts received greater levels of funding and recruited more patients adjusted for size of Trust (n = 35, 2,349 £/bed [95% CI 1,855–2,843], 5.9 patients/bed [2.7–9.0]) than Trusts with expected (n = 63, 1,110 £/bed, [864–1,357] p<0.0001, 2.6 patients/bed [1.7–3.5] p<0.0169) or, high (n = 42, 930 £/bed [683–1,177] p = 0.0001, 1.8 patients/bed [1.4–2.1] p<0.0005) mortality rates. The most research active Trusts were those with more doctors, nurses, critical care beds, operating theatres and, made greater use of radiology. Multifactorial analysis demonstrated better survival in the top funding and patient recruitment tertiles (lowest vs. highest (odds ratio & 95% CI: funding 1.050 [1.033–1.068] p<0.0001, recruitment 1.069 [1.052–1.086] p<0.0001), middle vs. highest (funding 1.040 [1.024–1.055] p<0.0001, recruitment 1.085 [1.070–1.100] p<0.0001). Conclusions Research active Trusts appear to have key differences in composition than less research active Trusts. Research active Trusts had lower risk-adjusted mortality for acute admissions, which persisted after adjustment for staffing and other structural factors.

Research, Policy and Practice – Worlds Apart?

The relationship between research, policy and practice remains a contested area. This article explores pressures for researchers to make their work more useful and relevant to policy and practice, and for practitioners to undertake research. Whilst there are clearly areas of mutual interest and benefit, we argue that the research, policy and practice communities also have distinct traditions, skills and obligations which should be recognised and valued rather than artificially suppressed. Narrow conceptions of research utility constrain the debate about what each community has to offer and how best to communicate with each other across borders.

Using a national archive of patient experience narratives to promote local patient-centered quality improvement: an ethnographic process evaluation of ‘accelerated’ experience-based co-design

Objectives To evaluate an accelerated form of experience-based co-design (EBCD), a type of participatory action research in which patients and staff work together to improve quality; to observe how acceleration affected the process and outcomes of the intervention. Methods An ethnographic process evaluation of an adapted form of EBCD was conducted, including observations, interviews, questionnaires and documentary analysis. Whilst retaining all components of EBCD, the adapted approach replaced local patient interviews with secondary analysis of a national archive of patient experience narratives to create national trigger films; shortened the timeframe; and employed local improvement facilitators. It was tested in intensive care and lung cancer in two English National Health Service (NHS) hospitals. A total of 96 clinical staff (primarily nursing and medical), and 63 patients and family members participated in co-design activities. Results The accelerated approach proved acceptable to staff and patients; using films of national rather than local narratives did not adversely affect local NHS staff engagement, and may have made the process less threatening or challenging. Local patients felt the national films generally reflected important themes although a minority felt they were more negative than their own experience. However, they served their purpose of ‘triggering’ discussion between patients and staff, and the resulting 48 co-design (improvement) activities across the four pathways were similar to those in EBCD, but achieved more quickly and at lower cost. Conclusions Accelerated EBCD offers a rigorous and relatively cost-effective patient-centered quality improvement approach.

Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review

Objective There is a widely held assumption that engagement by clinicians and healthcare organisations in research improves healthcare performance at various levels, but little direct empirical evidence has previously been collated. The objective of this study was to address the question: Does research engagement (by clinicians and organisations) improve healthcare performance? Methods An hourglass-shaped review was developed, consisting of three stages: (1) a planning and mapping stage; (2) a focused review concentrating on the core question of whether or not research engagement improves healthcare performance; and (3) a wider (but less systematic) review of papers identified during the two earlier stages, focusing on mechanisms. Results Of the 33 papers included in the focused review, 28 identified improvements in health services performance. Seven out of these papers reported some improvement in health outcomes, with others reporting improved processes of care. The wider review demonstrated that mechanisms such as collaborative and action research can encourage some progress along the pathway from research engagement towards improved healthcare performance. Organisations that have deliberately integrated the research function into organisational structures demonstrate how research engagement can, among other factors, contribute to improved healthcare performance. Conclusions Current evidence suggests that there is an association between the engagement of individuals and healthcare organisations in research and improvements in healthcare performance. The mechanisms through which research engagement might improve healthcare performance overlap and rarely act in isolation, and their effectiveness often depends on the context in which they operate.