Sonia Dalkin

What's in a mechanism? Development of a key concept in realist evaluation

BackgroundThe idea that underlying, generative mechanisms give rise to causal regularities has become a guiding principle across many social and natural science disciplines. A specific form of this enquiry, realist evaluation is gaining momentum in the evaluation of complex social interventions. It focuses on ‘what works, how, in which conditions and for whom’ using context, mechanism and outcome configurations as opposed to asking whether an intervention ‘works’. Realist evaluation can be difficult to codify and requires considerable researcher reflection and creativity. As such there is often confusion when operationalising the method in practice. This article aims to clarify and further develop the concept of mechanism in realist evaluation and in doing so aid the learning of those operationalising the methodology.DiscussionUsing a social science illustration, we argue that disaggregating the concept of mechanism into its constituent parts helps to understand the difference between the resources offered by the intervention and the ways in which this changes the reasoning of participants. This in turn helps to distinguish between a context and mechanism. The notion of mechanisms ‘firing’ in social science research is explored, with discussions surrounding how this may stifle researchers’ realist thinking. We underline the importance of conceptualising mechanisms as operating on a continuum, rather than as an ‘on/off’ switch.SummaryThe discussions in this article will hopefully progress and operationalise realist methods. This development is likely to occur due to the infancy of the methodology and its recent increased profile and use in social science research. The arguments we present have been tested and are explained throughout the article using a social science illustration, evidencing their usability and value.

Functionality and feedback: a protocol for a realist synthesis of the collation, interpretation and utilisation of PROMs data to improve patient care

Introduction The feedback and public reporting of PROMs data aims to improve the quality of care provided to patients. Existing systematic reviews have found it difficult to draw overall conclusions about the effectiveness of PROMs feedback. We aim to execute a realist synthesis of the evidence to understand by what means and in what circumstances the feedback of PROMs data leads to the intended service improvements. Methods and analysis Realist synthesis involves (stage 1) identifying the ideas, assumptions or ‘programme theories’ which explain how PROMs feedback is supposed to work and in what circumstances and then (stage 2) reviewing the evidence to determine the extent to which these expectations are met in practice. For stage 1, six provisional ‘functions’ of PROMs feedback have been identified to structure our review (screening, monitoring, patient involvement, demand management, quality improvement and patient choice). For each function, we will identify the different programme theories that underlie these different goals and develop a logical map of the respective implementation processes. In stage 2, we will identify studies that will provide empirical tests of each component of the programme theories to evaluate the circumstances in which the potential obstacles can be overcome and whether and how the unintended consequences of PROMs feedback arise. We will synthesise this evidence to (1) identify the implementation processes which support or constrain the successful collation, interpretation and utilisation of PROMs data; (2) identify the implementation processes through which the unintended consequences of PROMs data arise and those where they can be avoided. Ethics and dissemination The study will not require NHS ethics approval. We have secured ethical approval for the study from the University of Leeds (LTSSP-019). We will disseminate the findings of the review through a briefing paper and dissemination event for National Health Service stakeholders, conferences and peer reviewed publications.

An exploratory randomised controlled trial using short messaging service to facilitate insulin administration in young adults with type 1 diabetes

This exploratory randomised controlled trial examined the effectiveness of a novel short messaging service intervention underpinned by the theory of planned behaviour (TPB) in improving insulin administration in young adults with type 1 diabetes and the role of moderating variables. Those in the intervention condition (N = 8) received one daily text message underpinned by TPB constructs: Attitudes, subjective norms, perceived behavioural control and intention. Those in the control condition (N = 10) received weekly general health messages. Self-reported insulin administration was the main outcome measure; conscientiousness and consideration of future consequences (CFC) were measured as potential moderators. Analyses of covariance revealed no main effects of condition for morning and afternoon injections but a marginally significant effect for evening injections (p = .08). This main effect was qualified by significant interactions of condition with conscientiousness (p = .001), CFC (p = .007) and a three-way interaction among condition, conscientiousness and CFC (p = .009). Exploration of the interactions indicated the intervention significantly improved evening injection rates only in the low conscientiousness and low CFC groups. This effect was particularly strong among those low in both conscientiousness and CFC. Further investigation is warranted, using more objective measures of insulin adherence in a larger sample.

Understanding integrated care pathways in palliative care using realist evaluation: a mixed methods study protocol

Introduction Policy- and evidence-based guidelines have highlighted the need for improved palliative and end-of-life care. However, there is still evidence of individuals dying undignified deaths with little pain control, therefore inflicting unnecessary suffering. New commissioning powers have enabled a 2-year pilot of an innovative integrated care pathway (ICP) designed to improve arrangements for individuals with life-limiting illnesses requiring palliative care. A novel feature of the ICP is its focus on palliative care over the last 6 months of life, aiming to intervene early to prepare for and ensure a good death. What is not known is if this pathway works, how it works and who it works for. Methods and analysis A realist evaluation and a complex analytical framework will investigate and discover context, mechanism and outcome conjectures and configurations of the ICP and thus facilitate exploration of how it works and who it works for. A mixed methods approach will be used with small sample sizes to capture the breadth of the ICP. Phase 1 will identify if the pathway works through analysis of NHS Morbidity Information Query and Export Syntax data, locality Death Audit data and the Quality of Dying and Death Questionnaire. Phase 2 employs soft systems methodology with data from focus groups with health professionals to identify how the pathway works. Phase 3 uses the Miller Behavioural Style Scale and interviews with palliative care patients and bereaved relatives to analyse communication in palliative care. Ethics and dissemination Ethical approval has been granted from the NHS local ethics committee (REC reference number: 11/NE/0318). Research & Development approval has been gained from four different trusts, and relevant voluntary organisations and the local council have been informed about the research. This protocol illustrates the complexity inherent in evaluating a palliative care ICP. Identification of whether the pathway works, how it works and who it works for will be beneficial to all practices and other care providers involved as it will give objective data on the impact of the ICP. Results will be disseminated throughout the study for continuous quality improvement of the ICP. Outcomes from each data collection phase will be disseminated separately if analysis warrants it; all data collection will be utilised in the realist evaluation. The research provides a potential for the dissemination of the pathway to other localities through the transferable knowledge it will generate, from its focus on the contexts that are crucial for successful implementation, the mechanisms that facilitate implementation and the outcomes achieved.

Reducing inequalities in care for patients with non-malignant diseases: Insights from a realist evaluation of an integrated palliative care pathway

Background: The need for palliative care is growing internationally with an increasing prevalence of non-malignant diseases. The integrated care pathway was implemented in primary care by multidisciplinary teams from 2009 in a locality in the North East of England. Fourteen general practitioner practices provided data for the study. Aim: To find whether, how, and under what circumstances palliative care registrations are made for patients with non-malignant diseases in primary care. Design: General practitioner practice data were analysed statistically and qualitative data were collected from health care professionals and members of relevant organisations. Findings: A mixed-effects logistic model indicated a significant difference beyond the 0.1% level (p < 0.001) in registrations between the malignant and non-malignant groups in 2011, with an odds ratio of 0.09 (=exp(−2.4266)), indicating that patients in the non-malignant group are around 11 times (1/0.09) less likely to be registered than patients in the malignant group. However, patients with non-malignant diseases were significantly more likely to be registered in 2012 than in 2011 with an odds ratio of 1.46, significant beyond the 1% level. Qualitative analyses indicate that health care professionals find registering patients with non-malignant diseases stressful, yet feel that their confidence in treating this population is increasing. Conclusion: The integrated care pathway began to enable the reduction in inequalities in care by identifying, registering and managing an increasing number of palliative patients with non-malignant diseases. Consensual and inclusive definitions of palliative care were developed in order to legitimise the registration of such patients.

Care planning: what works, for whom and in what circumstances? A rapid realist review

Care planning has been described as a “better conversation” that helps people with long-term conditions to be in control of planning their care. Each person with long-term conditions faces individual challenges and each health care setting is fundamentally different, so there is a need for empirical testing of the specific mechanisms through which care planning may lead to health improvements. A rapid realist review was conducted to unearth underpinning mechanisms leading to outcomes in particular contexts. These are expressed in the form of realist theories, which are developed and refined through the review process. Fifty-one full text studies were included in the review. Seven program theories were iteratively tested and refined. A detailed description of what care planning is and what it should look like in practice has been achieved in the form of realist theories.

How do aggregated patient-reported outcome measures data stimulate health care improvement? A realist synthesis:

Objectives Internationally, there has been considerable debate about the role of data in supporting quality improvement in health care. Our objective was to understand how, why and in what circumstances the feedback of aggregated patient-reported outcome measures data improved patient care. Methods We conducted a realist synthesis. We identified three main programme theories underlying the use of patient-reported outcome measures as a quality improvement strategy and expressed them as nine ‘if then’ propositions. We identified international evidence to test these propositions through searches of electronic databases and citation tracking, and supplemented our synthesis with evidence from similar forms of performance data. We synthesized this evidence through comparing the mechanisms and impact of patient-reported outcome measures and other performance data on quality improvement in different contexts. Results Three programme theories were identified: supporting patient choice, improving accountability and enabling providers to compare their performance with others. Relevant contextual factors were extent of public disclosure, use of financial incentives, perceived credibility of the data and the practicality of the results. Available evidence suggests that patients or their agents rarely use any published performance data when selecting a provider. The perceived motivation behind public reporting is an important determinant of how providers respond. When clinicians perceived that performance indicators were not credible but were incentivized to collect them, gaming or manipulation of data occurred. Outcome data do not provide information on the cause of poor care: providers needed to integrate and interpret patient-reported outcome measures and other outcome data in the context of other data. Lack of timeliness of performance data constrains their impact. Conclusions Although there is only limited research evidence to support some widely held theories of how aggregated patient-reported outcome measures data stimulate quality improvement, several lessons emerge from interventions sharing the same programme theories to help guide the increasing use of these measures.

Exploring the use of Soft Systems Methodology with realist approaches: A novel way to map programme complexity and develop and refine programme theory:

As the use of realist approaches gains momentum, there is a growing interest in how systems approaches can complement realist thinking. In this article, we discuss how the epistemology of Soft Systems Methodology is compatible with realist approaches. Both Soft Systems Methodology and realist approaches emphasize the necessity to engage stakeholders; through models, the description of contingencies and exploring the intricacies of how complex programmes really work. We outline the key elements of realist approaches and Soft Systems Methodology, and report on two novel case studies. Drawing on our own experiences, we make the case that, used in conjunction with a realist approach, Soft Systems Methodology can provide a useful tool to a) map programme complexity, and b) develop and refine stakeholders programme theories, thus increasing the transparency, reliability, validity and accuracy of the theory building and refining process in realist approaches. We highlight Soft Systems Methodology as a novel companion to realist approaches and detail the first case studies of its use.

OP09 For whom and in what circumstances does the use of patient reported outcome measures (PROMs) improve patient care? A realist synthesis

Background It is suggested that the use of patient reported outcome measure (PROMs) can enhance patients’ consultations with clinicians and improve clinical management. However, existing systematic reviews have found it difficult to reach firm conclusions about the impact of PROMs feedback on the process and outcomes of patient care, largely due to the heterogeneity and complexity of the intervention itself and the wide range of indicators used to assess its impact. This suggests a lack of consensus or even clarity about how this interventions is expected to work. Methods We conducted a realist synthesis to explore the contexts in which and processes through which PROMs enable patients to share concerns with clinicians and change clinicians’ communication practices within the consultation. We identified the ideas and assumptions (program theories) underlying how PROMs use was intended to work and developed an overall model to act as a framework for the review. Electronic databases were searched and backwards and forward citation tracking were carried out on key systematic reviews. We selected papers relevant to testing our theories and 36 papers were included. Results PROMs completion prompts patients to engage in self-reflection and identify then prioritise issues of importance to them. Whether PROMs supported or constrained patients in sharing issues with clinicians depended on the structure of the PROM and existing clinician-patient relationship. Patients valued PROMs but preferred to share information with clinicians when they had established a trusting relationship with them. Clinicians preferred to develop rapport through their verbal interactions with patients. Clinicians perceived standardised PROMs constrained their relationship with patients and were difficult to incorporate into the flow of consultations. Clinicians avoided using them or adapted the PROM, which may compromise their validity. Individualised PROMs supported consultations by allowing patients to ‘tell their story’ but were less useful as an outcome measure to measure change over time. Discussion It is the process of PROMs completion which helps patients to reflect on their health and raise issues with clinicians. The structure of the PROM was a key determinant of the extent to which the use of PROMs supported or constrained the clinician-patient relationship. PROMs may support the care of individual patients through acting as a ‘conversation opener’ rather than as a standardised, quantified summary of patients’ problems.

Open communication strategies between a triad of ‘experts’ facilitates death in usual place of residence: A realist evaluation

Background: In order to meet policy drivers on death in usual place of residence, it is key to understand how shared decision-making can be facilitated in practice. An integrated care pathway was implemented in primary care in the North East of England to facilitate death in usual place of residence. Aim: To understand how, for whom and in which circumstances death in usual place of residence is facilitated. Design: A mixed method realist evaluation was employed. Local primary care practice death audit data were analysed to identify outcomes using a mixed effects logistic regression model. Focus groups and interviews with staff of the integrated care pathway and bereaved relatives were analysed to identify the related contexts and mechanisms. Setting/participants: Death audit data of 4182 patients were readily available from 14 general practitioner practices. Three focus groups were conducted with primary and secondary care staff, voluntary sector organisations and care home representatives. Interviews with bereaved relatives were carried out in participants’ homes (n = 5). Results: A mixed effects logistic regression model indicated a significant effect of year on death in usual place of residence when compared to a model without year using an analysis of deviance (p = 0.016). Qualitative analysis suggested that this outcome was achieved when a triad of ‘experts’ (comprising patient, family members/family carers/formal carers and healthcare professionals) used open communication strategies. Conclusion: An empirically supported theory of how, for whom and in which circumstances death in usual place of residence happens is provided, which has important implications for both policy and practice.