Stefanie Ettelt

Health services research in Europe and its use for informing policy

Objective: As an applied field of research, health services research (HSR) is undertaken primarily to inform health care policy. However, the relationship between policy and HSR is not straightforward. Many policies are developed without taking account of available evidence. Our objective was to provide an overview of the infrastructure and capacity of HSR in Europe, and to explore the links between HSR and policy in European countries. Methods: A survey of health policy experts in 34 European countries undertaken between September 2009 and June 2010, covering: funding and prioritizing HSR; production and producers of HSR; health policy users of HSR; activities to promote the use of HSR; and barriers and facilitators to using HSR in policy-making. Experts were selected based on their professional expertise and reputation in HSR and/or health policy. Results: Information was returned from 30 of the 34 countries. Infrastructure and capacity for HSR varied widely across countries as did the mechanisms likely to promote the use of research for national policy-making. In many cases, HSR was not separately identified from other health-related research. Government funding for HSR varied between zero and over €170 million per year (in 2004). Few countries had mechanisms to match HSR priorities with those of policy-makers. Although most countries provided some relevant training, multidisciplinary training was available in few countries. Few mechanisms supporting the use of research in policy were reported. One exception was coverage decisions for new drugs and technologies where formal accountability for the use of health technology assessments appeared to be most developed. Overall, respondents struggled to locate information on many aspects of HSR, particularly its use in decision-making. Conclusion: HSR is unevenly developed across Europe. There is considerable scope to build the infrastructure and to take steps to improve the use of HSR in policy-making. There is also a need for research, as opposed to expert opinion, on how HSR is undertaken and used.

Assessing quality in cross-country comparisons of health systems and policies: Towards a set of generic quality criteria

There is a growing body of cross-country comparisons in health systems and policy research. However, there is little consensus as to how to assess its quality. This is partly due to the fact that cross-country comparison constitutes a diverse inter-disciplinary field of study, with much variation in the motives for research, foci and levels of analyses, and methodological approaches. Inspired by the views of subject area experts and using the distinction between variable-based and case-based research, we briefly review the main different types of cross-country comparisons in health systems and policy research to identify pertinent quality issues. From this, we identify the following generic quality criteria for cross-country comparisons: (1) appropriate use of theory, (2) explicit selection of comparator countries, (3) rigour of the comparative design, (4) attention to the complexity of cross-national comparison, (5) rigour of the research methods, and (6) contribution to knowledge. This list may not be exclusive though publication and discussion of the list of criteria should help raise awareness in this field of what constitutes high quality research. In turn, this should be helpful for those planning, undertaking, or commissioning cross-country comparative research.

The Multiple Purposes of Policy Piloting and Their Consequences: Three Examples from National Health and Social Care Policy in England

In England, policy piloting has become firmly established in almost all areas of public policy and is seen as good practice in establishing ‘what works’. However, equating piloting with evaluation can risk oversimplifying the relationship between piloting and policy-making. Using three case studies from health and social care – the Partnerships for Older People Projects (POPP) pilots, the Individual Budgets pilots and the Whole System Demonstrators (WSD) – the paper identifies multiple purposes of piloting, of which piloting for generating evidence of effectiveness was only one. Importantly, piloting was also aimed at promoting policy change and driving implementation, both in pilot sites and nationally. Indeed, policy makers appeared to be using pilots mainly to promote government policy, using evaluation as a strategy to strengthen the legitimacy of their decisions and to convince critical audiences. These findings highlight the ambiguous nature of piloting and thus question the extent to which piloting contributes to the agenda of evidence-based policy-making.

Policy experiments: Investigating effectiveness or confirming direction?

In England, ‘policy experiments’ are largely synonymous with the use of randomized controlled trials (RCTs) to test whether one policy ‘works’ better than another. While advocacy of the use of RCTs in public policy presents this as relatively straightforward, even common sense, the reality is different, as shown through analysis of three high profile policy pilots and their evaluations undertaken in health and social care in England in the mid/late 2000s. The RCTs were expected to confirm the direction of policy by resolving any remaining uncertainty about the effectiveness of the chosen path and their existence was used largely as instruments of persuasion. The findings from the analysis of the three pilots confirm the continuing relevance of Campbell’s 1969 insight that governments struggle to experiment in the scientific sense and explain the limited effect of these policy experiments on policy decisions.

Policy learning from abroad - Why it is more difficult than it seems

This article explores the process of policy learning from abroad from a knowledge utilisation perspective, using examples of health policy making in the Department of Health in England. It argues that information about policy abroad is often heterogeneous and difficult to obtain systematically and therefore does not fit easily with notions of evidence-based policy making. While some officials interviewed for this study did regard policy examples from other countries as a substitute for evidence, especially in areas in which research evidence was insufficient, others appeared to be less confident about itsvalidity and generalisability. Department of Health officials reported a great variability in strategies to obtain such information, with processes often constrained by pressures on time and resources. They were also highly selective in exploring policy examples from abroad, with most respondents stating that they were largely interested in generating ideas to address domestic policy problems, often relating to details of policy. The iterative process of using this information thus raises questions about the extent to which looking abroad contributed to genuine policy learning.

The Politics of Evidence Use in Health Policy Making in Germany—the Case of Regulating Hospital Minimum Volumes

This article examines the role of scientific evidence in informing health policy decisions in Germany, using minimum volumes policy as a case study. It argues that scientific evidence was used strategically at various stages of the policy process both by individual corporatist actors and by the Federal Joint Committee as the regulator. Minimum volumes regulation was inspired by scientific evidence suggesting a positive relationship between service volume and patient outcomes for complex surgical interventions. Federal legislation was introduced in 2002 to delegate the selection of services and the setting of volumes to corporatist decision makers. Yet, despite being represented in the Federal Joint Committee, hospitals affected by its decisions took the Committee to court to seek legal redress and prevent policy implementation. Evidence has been key to support, and challenge, decisions about minimum volumes, including in court. The analysis of the role of scientific evidence in minimum volumes regulation in Germany highlights the dynamic relationship between evidence use and the political and institutional context of health policy making, which in this case is characterized by the legislative nature of policy making, corporatism, and the role of the judiciary in reviewing policy decisions.

Conclusion: Reflecting on Studying Evidence Use from a Public Policy Perspective

This final chapter reflects on the insights provided by the case studies that constitute this volume, focussing on the key questions which informed the broader research programme from which the cases arise: what does it mean to use evidence effectively to inform health policymaking?; and how do political and institutional factors shape the specific forms of evidence use that arise in different contexts? It discusses the circumstances in which instrumental or strategic uses of evidence were seen; as well as synthesising insights about the ways in which political contestation, the framing and construction of policy problems and their solutions, and different institutional arrangements for policy delivery shape evidence use in different policy contexts. The chapter concludes by considering future areas of research in this field to provide further insights into the use of evidence for health policymaking from a public policy perspective.

Evidence Use and the Institutions of the State: The Role of Parliament and the Judiciary

This chapter explores the role of parliaments and the judiciary in shaping evidence use in health policy making. Most analyses of the role of scientific evidence focus on the executive, i.e. national governments and ministries of health, as the key state actors in health policy and health system governance. This chapter shifts attention to the other two powers within the state, the legislative and the judiciary. Using the examples analysed in this book the chapter examines how parliaments can use evidence to inform legislative processes and to hold governments to account, although there are substantial differences between countries and political systems. However, there was little suggestion that such approaches were undertaken systematically. In cases in which policies are brought to court, judges may have to deal with scientific evidence within a country’s legal and constitutional framework, again with significant differences between national legal practices.

Ministries of Health and the Stewardship of Health Evidence

This chapter describes how Ministries of Health have been mandated to act as stewards of populations’ health according to the World Health Organization. We argue that this mandate extends to them having (at least partial) responsibility for ensuring relevant evidence informs policy decisions. Yet this requires consideration of the evidence advisory systems serving Ministry needs, particularly whether or how such systems work to provide relevant information in a timely manner to key decision points in the policy process. Insights from our six cases are presented to illustrate the structural and practical differences which exist between evidence advisory systems and how, at certain times, key health decisions may in fact lie outside ministerial authority. These divergent experiences highlight a range of analytical challenges when considering the provision of evidence to inform health decisions from an institutional perspective.

Studying Evidence Use for Health Policymaking from a Policy Perspective

Individuals working within the health sector widely embrace the idea of using evidence to achieve their goals of improving individual and population health. Yet while these actors embrace an ideal form of rational-instrumental evidence use under the banner of ‘evidence based policymaking’, they often struggle to understand when, why, or how evidence is used in policy processes. This chapter sets out the conceptual framework employed in this volume to study the use of evidence within policymaking from a public policy perspective. It explores the importance of both political contestation and institutional context to understand when and how evidence will be used within policy processes. The chapter then outlines the structure of this book and the focus of subsequent chapters, highlighting how each of these talks to these themes.