Gill Westhorp

RAMESES publication standards: realist syntheses

BackgroundThere is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas -for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The projects aim is to produce preliminary publication standards for realist systematic reviews.MethodsWe (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards.ResultsWe identified 35 published realist syntheses, provided real-time support to 9 on-going syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within three rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%.ConclusionThis project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses.To encourage dissemination of the RAMESES publication standards, this article is co-published in the Journal of Advanced Nursing and is freely accessible on Wiley Online Library (http://www.wileyonlinelibrary.com/journal/jan). Please see related article http://www.biomedcentral.com/1741-7015/11/20 and http://www.biomedcentral.com/1741-7015/11/22

Realist methods in medical education research: what are they and what can they contribute?

Medical Education 2012: 46: 89–96

Using complexity-consistent theory for evaluating complex systems:

This article draws on two recent traditions in evaluation methodology, one grounded in complexity theory and the other in a realist philosophy of science. Sometimes seen as incompatible, it is argued here that complexity theory and realist evaluation are ‘natural bedfellows’. Because of their similarities and differences, evaluators can usefully draw on both perspectives within the same evaluation. One way to do so is to draw on key concepts from each in the selection and use of substantive theory. The article introduces the concept of ‘complexity-consistent’ substantive theories and suggests that this is useful for the evaluation of policies and programs in complex adaptive systems. It demonstrates how substantive theories can be analysed in complexity terms, how multiple theories can be ‘layered’ to reflect multiple levels of systems, and how such theories can be used within evaluation design and analysis.

Can “realist” randomised controlled trials be genuinely realist?

In this paper, we respond to a paper by Jamal and colleagues published in Trials in October 2015 and take an opportunity to continue the much-needed debate about what applied scientific realism is. The paper by Jamal et al. is useful because it exposes the challenges of combining a realist evaluation approach (as developed by Pawson and Tilley) with the randomised controlled trial (RCT) design.We identified three fundamental differences that are related to paradigmatic differences in the treatment of causation between post-positivist and realist logic: (1) the construct of mechanism, (2) the relation between mediators and moderators on one hand and mechanisms and contexts on the other hand, and (3) the variable-oriented approach to analysis of causation versus the configurational approach.We show how Jamal et al. consider mechanisms as observable, external treatments and how their approach reduces complex causal processes to variables. We argue that their proposed RCT design cannot provide a truly realist understanding. Not only does the proposed realist RCT design not deal with the RCT’s inherent inability to “unpack” complex interventions, it also does not enable the identification of the dynamic interplay among the intervention, actors, context, mechanisms and outcomes, which is at the core of realist research. As a result, the proposed realist RCT design is not, as we understand it, genuinely realist in nature.

Developing complexity-consistent theory in a realist investigation

This is the second of a two part article examining the use of complexity-consistent theory in a realist investigation. The first article (Evaluation 18[4]) introduced the idea of complexity-consistent theory. It argued that complexity-consistent theories are likely to be useful for understanding complex processes of change and therefore useful in many kinds of evaluation. Further, it argued that theories can be organized within a hierarchy to reflect the different levels of reality involved in a change program. This was described as ‘layering’ theories. This second article describes the use of these concepts in a realist investigation. The purpose of the investigation was to develop a realist, middle-range theory to explain how and why some early intervention programs which ‘work’ on average for disadvantaged families do not work for the most disadvantaged. The investigation combined a small-scale realist evaluation of a family support program in Adelaide, South Australia with a modified form of realist synthesis. The work was undertaken as part of a PhD, supervised by Professor Nick Tilley, co-author of Realistic Evaluation.

Using realist action research for service redesign

This case demonstrates the integration of realist action research and co-design to address the complex social problem of long-term reliance on welfare benefits. Realist action research combines a realist philosophy of science and the questions that flow from it with an action research cycle. Realist approaches to evaluation and planning seek to explain for whom in what contexts and how impacts are generated or might be generated. Action research seeks to solve real world problems, trialling solutions until a ‘best fit’ solution is reached. The article describes the principles underpinning the methodology and the research cycles through which the project worked – situation analysis; prioritizing; co-design; trialling and further refining ideas for change. It demonstrates the development and testing of program theory for one service innovation. It also reflects on the experience and potential benefits of this approach.