Sara Van Belle

Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research

This overview aims to stimulate conceptual and practical discussions to help unlock the full potential of realist evaluation in health systems research. Based on a structured literature search, this review maps how the concepts of realist evaluation are applied in health systems research and which methodological problems are encountered. We found a great diversity in the depth of application of the philosophical concepts, use of terminology and scope of application in the research process. Terms of theory-driven evaluation, theories of change and realist evaluation are often used interchangeably. Diverging views exist regarding the nature of ‘mechanism’ and the difference between mechanism and essential context condition. A lack of methodological guidance was highlighted by many authors. Realist evaluation is slowly gaining traction in health systems research, but more clarity is needed concerning the definitions of mechanisms and context and how the configuration of context, mechanism and intervention can be described and assessed.

Realist RCTs of complex interventions - An oxymoron

Bonell et al. discuss the challenges of carrying out randomised controlled trials (RCTs) to evaluate complex interventions in public health, and consider the role of realist evaluation in enhancing this design (Bonell, Fletcher, Morton, Lorenc, & Moore, 2012). They argue for a synergistic, rather than oppositional relationship between realist and randomised evaluation and that it is possible to benefit from the insights provided by realist evaluation without relinquishing the RCT as the best means of examining intervention causality. We present counter-arguments to their analysis of realist evaluation and their recommendations for realist RCTs. Bonell et al. are right to question whether and how (quasi-)experimental designs can be improved to better evaluate complex public health interventions. However, the paper does not explain how a research design that is fundamentally built upon a positivist ontological and epistemological position can be meaningfully adapted to allow it to be used from within a realist paradigm. The recommendations for realist RCTs do not sufficiently take into account important elements of complexity that pose major challenges for the RCT design. They also ignore key tenets of the realist evaluation approach. We propose that the adjective realist should continue to be used only for studies based on a realist philosophy and whose analytic approach follows the established principles of realist analysis. It seems more correct to call the approach proposed by Bonell and colleagues theory informed RCT, which indeed can help in enhancing RCTs.

How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa.

BackgroundThis paper presents the development of a study design built on the principles of theory-driven evaluation. The theory-driven evaluation approach was used to evaluate an adolescent sexual and reproductive health intervention in Mali, Burkina Faso and Cameroon to improve continuity of care through the creation of networks of social and health care providers.Methods/designBased on our experience and the existing literature, we developed a six-step framework for the design of theory-driven evaluations, which we applied in the ex-post evaluation of the networking component of the intervention. The protocol was drafted with the input of the intervention designer. The programme theory, the central element of theory-driven evaluation, was constructed on the basis of semi-structured interviews with designers, implementers and beneficiaries and an analysis of the interventions logical framework.DiscussionThe six-step framework proved useful as it allowed for a systematic development of the protocol. We describe the challenges at each step. We found that there is little practical guidance in the existing literature, and also a mix up of terminology of theory-driven evaluation approaches. There is a need for empirical methodological development in order to refine the tools to be used in theory driven evaluation. We conclude that ex-post evaluations of programmes can be based on such an approach if the required information on context and mechanisms is collected during the programme.

Studying complex interventions: reflections from the FEMHealth project on evaluating fee exemption policies in West Africa and Morocco

BackgroundThe importance of complexity in health care policy-making and interventions, as well as research and evaluation is now widely acknowledged, but conceptual confusion reigns and few applications of complexity concepts in research design have been published. Taking user fee exemption policies as an entry point, we explore the methodological consequences of complexity’ for health policy research and evaluation. We first discuss the difference between simple, complicated and complex and introduce key concepts of complex adaptive systems theory. We then apply these to fee exemption policies.DesignWe describe how the FEMHealth research project attempts to address the challenges of complexity in its evaluation of fee exemption policies for maternal care. We present how the development of a programme theory for fee exemption policies was used to structure the overall design. This allowed for structured discussions on the hypotheses held by the researchers and helped to structure, integrate and monitor the sub-studies. We then show how the choice of data collection methods and tools for each sub-study was informed by the overall design.DiscussionApplying key concepts from complexity theory proved useful in broadening our view on fee exemption policies and in developing the overall research design. However, we encountered a number of challenges, including maintaining adaptiveness of the design during the evaluation, and ensuring cohesion in the disciplinary diversity of the research teams. Whether the programme theory can fulfil its claimed potential to help making sense of the findings is yet to be tested. Experience from other studies allows for some moderate optimism. However, the biggest challenge complexity throws at health system researchers may be to deal with the unknown unknowns and the consequence that complex issues can only be understood in retrospect. From a complexity theory point of view, only plausible explanations can be developed, not predictive theories. Yet here, theory-driven approaches may help.

Towards an agenda for implementation science in global health : there is nothing more practical than good (social science) theories

### Summary boxnnRidde recently called for a commitment to implementation science and for ‘better quality research in order to have a greater understanding of how to implement health interventions’.1 He argued for better use of theory and implementation science studies that contribute to theory. This is indeed of paramount importance, especially for interventions in complex settings such as health systems. In practice, however, few researchers take up the challenge; all too often, implementation studies adopt a mixed methods approach that is a-theoretical.nnIf there is a lack of theory in the field of implementation science, it is not for want of theories. In the case of health governance, for instance, theory development and empirical research testing such theories have been taking place in the fields of economics and political science. However, there has been little, if any, systematic crossing over of recent governance theories to the field of health policy and systems research. This does not only apply to health (system) governance; research on health worker motivation and strategic decision-making at operational level, for instance, could use a healthy injection of recent theories and methods from other disciplines.nnIn applied research fields, such as global environmental studies, there are more signs of effective cross-pollination. The study of global climate change is an example of how the integration of different paradigms yields new theoretical insights that drive empirical research; for example, the late Elinor Ostroms conceptualisation of polycentricity in efforts to address global …

Can “realist” randomised controlled trials be genuinely realist?

In this paper, we respond to a paper by Jamal and colleagues published in Trials in October 2015 and take an opportunity to continue the much-needed debate about what applied scientific realism is. The paper by Jamal et al. is useful because it exposes the challenges of combining a realist evaluation approach (as developed by Pawson and Tilley) with the randomised controlled trial (RCT) design.We identified three fundamental differences that are related to paradigmatic differences in the treatment of causation between post-positivist and realist logic: (1) the construct of mechanism, (2) the relation between mediators and moderators on one hand and mechanisms and contexts on the other hand, and (3) the variable-oriented approach to analysis of causation versus the configurational approach.We show how Jamal et al. consider mechanisms as observable, external treatments and how their approach reduces complex causal processes to variables. We argue that their proposed RCT design cannot provide a truly realist understanding. Not only does the proposed realist RCT design not deal with the RCT’s inherent inability to “unpack” complex interventions, it also does not enable the identification of the dynamic interplay among the intervention, actors, context, mechanisms and outcomes, which is at the core of realist research. As a result, the proposed realist RCT design is not, as we understand it, genuinely realist in nature.

An exploration of group-based HIV/AIDS treatment and care models in Sub-Saharan Africa using a realist evaluation (Intervention-Context-Actor-Mechanism-Outcome) heuristic tool: a systematic review

IntroductionIt is increasingly acknowledged that differentiated care models hold potential to manage large volumes of patients on antiretroviral therapy (ART). Various group-based models of ART service delivery aimed at decongesting local health facilities, encouraging patient retention in care, and enhancing adherence to medication have been implemented across sub-Saharan Africa. Evidence from the literature suggests that these models of ART service delivery are more effective than corresponding facility-based care and superior to individual-based models. Nevertheless, there is little understanding of how these care models work to achieve their intended outcomes. The aim of this study was to review the theories explicating how and why group-based ART models work using a realist evaluation framework.MethodsA systematic review of the literature on group-based ART support models in sub-Saharan Africa was conducted. We searched the Google Scholar and PubMed databases and supplemented these with a reference chase of the identified articles. We applied a theory-driven approach—narrative synthesis—to synthesise the data. Data were analysed using the thematic content analysis method and synthesised according to aspects of the Intervention-Context-Actor-Mechanism-Outcome heuristic-analytic tool—a realist evaluation theory building tool.ResultsTwelve articles reporting primary studies on group-based models of ART service delivery were included in the review. The six studies that employed a quantitative study design failed to identify aspects of the context and mechanisms that work to trigger the outcomes of group-based models. While the other four studies that applied a qualitative and the two using ufeffa mixed methods design identified some of the aspects of the context and mechanisms that could trigger the outcomes of group-based ART models, these studies did not explain the relationship(s) between the theory elements and how they interact to produce the outcome(s).ConclusionAlthough we could distill various components of the Intervention-Context-Actor-Mechanism-Outcome analytic tool from different studies exploring group-based programmes, we could not, however, identify a salient programme theory based on the Intervention-Context-Actor-Mechanism-Outcome heuristic analysis. The scientific community, policy makers and programme implementers would benefit more if explanatory findings of how, why, for whom and in what circumstances programmes work are presented rather than just reporting on the outcomesxa0of the interventions.

Towards Developing an Initial Programme Theory: Programme Designers and Managers Assumptions on the Antiretroviral Treatment Adherence Club Programme in Primary Health Care Facilities in the Metropolitan Area of Western Cape Province, South Africa.

Background The antiretroviral adherence club intervention was rolled out in primary health care facilities in the Western Cape province of South Africa to relieve clinic congestion, and improve retention in care, and treatment adherence in the face of growing patient loads. We adopted the realist evaluation approach to evaluate what aspects of antiretroviral club intervention works, for what sections of the patient population, and under which community and health systems contexts, to inform guidelines for scaling up of the intervention. In this article, we report on a step towards the development of a programme theory—the assumptions of programme designers and health service managers with regard to how and why the adherence club intervention is expected to achieve its goals and perceptions on how it has done so (or not). Methods We adopted an exploratory qualitative research design. We conducted a document review of 12 documents on the design and implementation of the adherence club intervention, and key informant interviews with 12 purposively selected programme designers and managers. Thematic content analysis was used to identify themes attributed to the programme actors, context, mechanisms, and outcomes. Using the context-mechanism-outcome configurational tool, we provided an explanatory focus of how the adherence club intervention is roll-out and works guided by the realist perspective. Results We classified the assumptions of the adherence club designers and managers into the rollout, implementation, and utilisation of the adherence club programme, constructed around the providers, management/operational staff, and patients, respectively. Two rival theories were identified at the patient-perspective level. We used these perspectives to develop an initial programme theory of the adherence club intervention, which will be tested in a later phase. Conclusion The perspectives of the programme designers and managers provided an important step towards developing an initial programme theory, which will guide our realist evaluation of the adherence club programme in South Africa.

What can we learn on public accountability from non-health disciplines: a meta-narrative review

Objective In health, accountability has since long been acknowledged as a central issue, but it remains an elusive concept. The literature on accountability spans various disciplines and research traditions, with differing interpretations. There has been little transfer of ideas and concepts from other disciplines to public health and global health. In the frame of a study of accountability of (international) non-governmental organisations in local health systems, we carried out a meta-narrative review to address this gap. Our research questions were: (1) What are the main approaches to accountability in the selected research traditions? (2) How is accountability defined? (3) Which current accountability approaches are relevant for the organisation and regulation of local health systems and its multiple actors? Setting The search covered peer-reviewed journals, monographs and readers published between 1992 and 2012 from political science, public administration, organisational sociology, ethics and development studies. 34 papers were selected and analysed. Results Our review confirms the wide range of approaches to the conceptualisation of accountability. The definition of accountability used by the authors allows the categorisation of these approaches into four groups: the institutionalist, rights-based, individual choice and collective action group. These four approaches can be considered to be complementary. Conclusions We argue that in order to effectively achieve public accountability, accountability strategies are to be complementary and synergistic.

The challenge of complexity in evaluating health policies and programs: the case of women’s participatory groups to improve antenatal outcomes

BackgroundDuring the last years, randomized designs have been promoted as the cornerstone of evidence-based policymaking. Also in the field of community participation, Random Control Trials (RCTs) have been the dominant design, used for instance to examine the contribution of community participation to health improvement. We aim at clarifying why RCTs and related (quasi-) experimental designs may not be the most appropriate approach to evaluate such complex programmes.ResultsWe argue that the current methodological debate could be more fruitful if it would start from the position that the choice of designs should fit the nature of the program and research questions rather than be driven by methodological preferences. We present how realist evaluation, a theory-driven approach to research and evaluation, is a relevant methodology that could be used to assess whether and how community participation works. Using the realist evaluation approach to examine the relationship between participation and action of women groups and antenatal outcomes would enable evaluators to examine in detail the underlying mechanisms which influence actual practices and outcomes, as well as the context conditions required to make it work.ConclusionsRealist research in fact allows opening the black boxes of “community” and “participation” in order to examine the role they play in ensuring cost-effective, sustainable interventions. This approach yields important information for policy makers and programme managers considering how such programs could be implemented in their own setting.