Ana Manzano

Protocol—the RAMESES II study: developing guidance and reporting standards for realist evaluation

Introduction Realist evaluation is an increasingly popular methodology in health services research. For realist evaluations (RE) this project aims to: develop quality and reporting standards and training materials; build capacity for undertaking and critically evaluating them; produce resources and training materials for lay participants, and those seeking to involve them. Methods To achieve our aims, we will: (1) Establish management and governance infrastructure; (2) Recruit an interdisciplinary Delphi panel of 35 participants with diverse relevant experience of RE; (3) Summarise current literature and expert opinion on best practice in RE; (4) Run an online Delphi panel to generate and refine items for quality and reporting standards; (5) Capture ‘real world’ experiences and challenges of RE—for example, by providing ongoing support to realist evaluations, hosting the RAMESES JISCmail list on realist research, and feeding problems and insights from these into the deliberations of the Delphi panel; (6) Produce quality and reporting standards; (7) Collate examples of the learning and training needs of researchers, students, reviewers and lay members in relation to RE; (8) Develop, deliver and evaluate training materials for RE and deliver training workshops; and (9) Develop and evaluate information and resources for patients and other lay participants in RE (eg, draft template information sheets and model consent forms) and; (10) Disseminate training materials and other resources. Planned outputs: (1) Quality and reporting standards and training materials for RE. (2) Methodological support for RE. (3) Increase in capacity to support and evaluate RE. (4) Accessible, plain-English resources for patients and the public participating in RE. Discussion The realist evaluation is a relatively new approach to evaluation and its overall place in the is not yet fully established. As with all primary research approaches, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency.

The craft of interviewing in realist evaluation

Pawson and Tilley suggest a unique strategy for conducting qualitative interviews within realist evaluation studies. Theories are placed before the interviewee for them to comment on with a view to providing refinement. The subject matter of the interview is the researcher’s theory and interviewees confirm, falsify, and refine this theory. This relationship – described as a teacher–learner cycle – is integral to realist evaluations. This article provides an overview of how interview techniques have been applied in realist evaluations in the last decade as well as suggesting two guiding principles. The first one relates to the design of realist studies and the second one explains how to ask questions like a realist, and proposes three different phases in realist interviews: theory gleaning, theory refining and theory consolidation. The article aims to contribute to a growing understanding of the practical and epistemological challenges presented by primary data collection in realist evaluation.

Exploring complex causal pathways between urban renewal, health and health inequality using a theory-driven realist approach.

INTRODUCTION Urban populations are growing and to accommodate these numbers, cities are becoming more involved in urban renewal programs to improve the physical, social and economic conditions in different areas. This paper explores some of the complexities surrounding the link between urban renewal, health and health inequalities using a theory-driven approach. METHODS We focus on an urban renewal initiative implemented in Barcelona, the Neighbourhoods Law, targeting Barcelonas (Spain) most deprived neighbourhoods. We present evidence from two studies on the health evaluation of the Neighbourhoods Law, while drawing from recent urban renewal literature, to follow a four-step process to develop a program theory. We then use two specific urban renewal interventions, the construction of a large central plaza and the repair of streets and sidewalks, to further examine this link. DISCUSSION In order for urban renewal programs to affect health and health inequality, neighbours must use and adapt to the changes produced by the intervention. However, there exist barriers that can result in negative outcomes including factors such as accessibility, safety and security. CONCLUSION This paper provides a different perspective to the field that is largely dominated by traditional quantitative studies that are not always able to address the complexities such interventions provide. Furthermore, the framework and discussions serve as a guide for future research, policy development and evaluation.

Can “realist” randomised controlled trials be genuinely realist?

In this paper, we respond to a paper by Jamal and colleagues published in Trials in October 2015 and take an opportunity to continue the much-needed debate about what applied scientific realism is. The paper by Jamal et al. is useful because it exposes the challenges of combining a realist evaluation approach (as developed by Pawson and Tilley) with the randomised controlled trial (RCT) design.We identified three fundamental differences that are related to paradigmatic differences in the treatment of causation between post-positivist and realist logic: (1) the construct of mechanism, (2) the relation between mediators and moderators on one hand and mechanisms and contexts on the other hand, and (3) the variable-oriented approach to analysis of causation versus the configurational approach.We show how Jamal et al. consider mechanisms as observable, external treatments and how their approach reduces complex causal processes to variables. We argue that their proposed RCT design cannot provide a truly realist understanding. Not only does the proposed realist RCT design not deal with the RCT’s inherent inability to “unpack” complex interventions, it also does not enable the identification of the dynamic interplay among the intervention, actors, context, mechanisms and outcomes, which is at the core of realist research. As a result, the proposed realist RCT design is not, as we understand it, genuinely realist in nature.

Evaluating deceased organ donation: a programme theory approach.

PURPOSE Organ donation and transplantation services represent a microcosm of modern healthcare organisations. They are complex adaptive systems. They face perpetual problems of matching supply and demand. They operate under fierce time and resource constraints. And yet they have received relatively little attention from a systems perspective. The purpose of this paper is to consider some of the fundamental issues in evaluating, improving and policy reform in such complex systems. DESIGN/METHODOLOGY/APPROACH The paper advocates an approach based on programme theory evaluation. FINDINGS The paper explains how the death to donation to transplantation process depends on the accumulation of series of embedded, institutional sub-processes. Evaluators need to be concerned with this whole system rather than with its discrete parts or sectors. Policy makers may expect disappointment if they seek to improve donation rates by applying nudges or administrative reforms at a single point in the implementation chain. ORIGINALITY/VALUE These services represent concentrated, perfect storms of complexity and the paper offers guidance to practitioners with bio-medical backgrounds on how such services might be evaluated and improved. For the methodological audience the paper caters for the burgeoning interest in programme theory evaluation while illustrating the design phase of this research strategy.

Study protocol: realist evaluation of effectiveness and sustainability of a community health workers programme in improving maternal and child health in Nigeria

BackgroundAchievement of improved maternal and child health (MCH) outcomes continues to be an issue of international priority, particularly for sub-Saharan African countries such as Nigeria. Evidence suggests that the use of Community Health Workers (CHWs) can be effective in broadening access to, and coverage of, health services and improving MCH outcomes in such countries.Methods/designIn this paper, we report the methodology for a 5-year study which aims to evaluate the context, processes, outcomes and longer-term sustainability of a Nigerian CHW scheme. Evaluation of complex interventions requires a comprehensive understanding of intervention context, mechanisms and outcomes. The multidisciplinary and mixed-method realist approach will facilitate such evaluation. A favourable policy environment within which the study is conducted will ensure the successful uptake of results into policy and practice.A realist evaluation provides an overall methodological framework for this multidisciplinary and mixed methods research, which will be undertaken in Anambra state. The study will draw upon health economics, social sciences and statistics. The study comprises three steps: (1) initial theory development; (2) theory validation and (3) theory refinement and development of lessons learned. Specific methods for data collection will include in-depth interviews and focus group discussions with purposefully identified key stakeholders (managers, service providers and service users), document reviews, analyses of quantitative data from the CHW programme and health information system, and a small-scale survey. The impact of the programme on key output and outcome indicators will be assessed through an interrupted time-series analysis (ITS) of monthly quantitative data from health information system and programme reports. Ethics approvals for this study were obtained from the University of Leeds and the University of Nigeria.DiscussionThis study will provide a timely and important contribution to health systems strengthening specifically within Anambra state in southeast Nigeria but also more widely across Nigeria. This paper should be of interest to researchers who are interested in adapting and applying robust methodologies for assessing complex health system interventions. The paper will also be useful to policymakers and practitioners who are interested in commissioning and engaging in such complex evaluations to inform policies and practices.

A Systematic Review of Discrete-Choice Experiments and Conjoint Analysis Studies in People with Multiple Sclerosis

BackgroundMultiple sclerosis (MS) is a chronic disabling, inflammatory, and degenerative disease of the central nervous system that, in most cases, requires long-term disease-modifying treatment (DMT). The drugs used vary in efficacy and adverse effect profiles. Several studies have used attribute-based stated-preference methods, primarily to investigate patient preferences for initiating or escalating DMT.ObjectivesTo conduct a systematic review of attribute-based stated-preference studies in people with MS to identify common methods employed and to assess study quality, with reference to the specific challenges of this disease area.MethodsWe conducted a systematic search for studies related to attribute-based stated-preference and MS in multiple databases, including Cochrane and MEDLINE. Studies were included if they were published in a peer-reviewed journal, were on the topic of MS, and used a survey methodology that measured stated preferences for attributes of a whole. Analysis was conducted using narrative synthesis and summary statistics. Study quality was judged against the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) conjoint analysis checklist.ResultsWe identified 16 relevant articles reporting 17 separate studies, all but one focusing on DMTs. Most studies were discrete-choice experiments. Study quality was generally high, but we recommend the following: (1) that consideration of sample sizes be improved, (2) that survey design choices be justified and documented, (3) that qualitative approaches for attribute and level selection be incorporated to better involve patients, and (4) that reporting of experimental practice be improved. The effects of DMTs on reproduction and the impact of how risk and uncertainty are presented were identified as neglected research topics. The ISPOR conjoint analysis checklist was found to be unsuitable for the assessment of study quality.ConclusionAttribute-based stated preference is a useful method with which to examine the preferences of people with MS in their choice of DMT. However, further research embracing the methodological recommendations identified, particularly greater use of qualitative methods in attribute development, is needed.

History and Cultural Diversity

This chapter will discuss different ways gender has been understood historically, and across different cultures. Historical data is reviewed using the case study approach. Identity categories are analysed, illustrating how relatively recently gender was not commonly or instinctively binarised. Pre-dating contemporary language, variation in gender expression (sometimes related to sexuality) is explored through examples of English Mollies, Italian Femminielli, Roman eunuchs, and Albanian Sworn Virgins, who were all positioned as “other” from men and women, without necessarily being marginalised. The chapter will also explore the range of North American two-spirit people, the Buginese people of Indonesia who have a five-gender system, and other examples including the South American Machi and Indian Hijra.