Aileen Grant

Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting

BackgroundProcess evaluations are recommended to open the ‘black box’ of complex interventions evaluated in trials, but there is limited guidance to help researchers design process evaluations. Much current literature on process evaluations of complex interventions focuses on qualitative methods, with less attention paid to quantitative methods. This discrepancy led us to develop our own framework for designing process evaluations of cluster-randomised controlled trials.MethodsWe reviewed recent theoretical and methodological literature and selected published process evaluations; these publications identified a need for structure to help design process evaluations. We drew upon this literature to develop a framework through iterative exchanges, and tested this against published evaluations.ResultsThe developed framework presents a range of candidate approaches to understanding trial delivery, intervention implementation and the responses of targeted participants. We believe this framework will be useful to others designing process evaluations of complex intervention trials. We also propose key information that process evaluations could report to facilitate their identification and enhance their usefulness.ConclusionThere is no single best way to design and carry out a process evaluation. Researchers will be faced with choices about what questions to focus on and which methods to use. The most appropriate design depends on the purpose of the process evaluation; the framework aims to help researchers make explicit their choices of research questions and methods.Trial registrationClinicaltrials.gov NCT01425502

Preferences of Community Pharmacists for Extended Roles in Primary Care: A Survey and Discrete Choice Experiment

AbstractBackground: Major changes in the roles and responsibilities of pharmacists across the world are occurring. A new Scottish Community Pharmacy contract was introduced in April 2006, following the introduction of a similar contract in England in 2005. This contract encourages greater involvement in medicines management and other clinical cognitive roles, whilst retaining a supply function. Objective: To use a discrete choice experiment (DCE) to examine the strength of preference of community pharmacists for existing and potential new roles, prior to the introduction of the new contract. Study design: The DCE was a component of a larger questionnaire, which assessed demography, workload, attitudes to, and satisfaction with, proposed new roles, and current levels of activity. Attributes and levels for the DCE were based on the recent policy document for Scotland, The Right Medicine, and informed consensus, respectively. Scenarios were organised into pairs, and pharmacists were asked “Which job would you prefer?” The questionnaire was mailed to all pharmacists working in the community setting in Scotland (n = 1621), as identified from a telephone survey. The questionnaire was totally anonymous, and two reminders were sent. Results: There was an overall response rate of 56.4% (914/1621). Community pharmacists preferred to work in an extended pharmacy team, to have strong integration with secondary care, and to provide a minor illness advice service. In 2003, they would forgo an annual income of £3443, £2183 and £2798, respectively to achieve this. However, overall, the pharmacists preferred more income to less. Repeat dispensing, chronic disease management, offering health promotion services, and the number of prescriptions dispensed per month were not significant predictors of job choice. Conclusion: Community pharmacists placed the highest value on organisational aspects of their work, and having a first contact primary care role. Although total income was important, there were indications that they would be prepared to forgo income to attain their preferred job.

A cluster randomised stepped wedge trial to evaluate the effectiveness of a multifaceted information technology-based intervention in reducing high-risk prescribing of non-steroidal anti-inflammatory drugs and antiplatelets in primary medical care: The DQIP study protocol

BackgroundHigh-risk prescribing of non-steroidal anti-inflammatory drugs (NSAIDs) and antiplatelet agents accounts for a significant proportion of hospital admissions due to preventable adverse drug events. The recently completed PINCER trial has demonstrated that a one-off pharmacist-led information technology (IT)-based intervention can significantly reduce high-risk prescribing in primary care, but there is evidence that effects decrease over time and employing additional pharmacists to facilitate change may not be sustainable.Methods/designWe will conduct a cluster randomised controlled with a stepped wedge design in 40 volunteer general practices in two Scottish health boards. Eligible practices are those that are using the INPS Vision clinical IT system, and have agreed to have relevant medication-related data to be automatically extracted from their electronic medical records. All practices (clusters) that agree to take part will receive the data-driven quality improvement in primary care (DQIP) intervention, but will be randomised to one of 10 start dates. The DQIP intervention has three components: a web-based informatics tool that provides weekly updated feedback of targeted prescribing at practice level, prompts the review of individual patients affected, and summarises each patients relevant risk factors and prescribing; an outreach visit providing education on targeted prescribing and training in the use of the informatics tool; and a fixed payment of 350 GBP (560 USD; 403 EUR) up front and a small payment of 15 GBP (24 USD; 17 EUR) for each patient reviewed in the 12 months of the intervention. We hypothesise that the DQIP intervention will reduce a composite of nine previously validated measures of high-risk prescribing. Due to the nature of the intervention, it is not possible to blind practices, the core research team, or the data analyst. However, outcome assessment is entirely objective and automated. There will additionally be a process and economic evaluation alongside the main trial.DiscussionThe DQIP intervention is an example of a potentially sustainable safety improvement intervention that builds on the existing National Health Service IT-infrastructure to facilitate systematic management of high-risk prescribing by existing practice staff. Although the focus in this trial is on Non-steroidal anti-inflammatory drugs and antiplatelets, we anticipate that the tested intervention would be generalisable to other types of prescribing if shown to be effective.Trial registrationClinicalTrials.gov, dossier number: NCT01425502

An ethnographic exploration of influences on prescribing in general practice: why is there variation in prescribing practices?

BackgroundPrescribing is a core activity for general practitioners, yet significant variation in the quality of prescribing has been reported. This suggests there may be room for improvement in the application of the current best research evidence. There has been substantial investment in technologies and interventions to address this issue, but effect sizes so far have been small to moderate. This suggests that prescribing is a decision-making process that is not sufficiently understood. By understanding more about prescribing processes and the implementation of research evidence, variation may more easily be understood and more effective interventions proposed.MethodsAn ethnographic study in three Scottish general practices with diverse organizational characteristics. Practices were ranked by their performance against Audit Scotland prescribing quality indicators, incorporating established best research evidence. Two practices of high prescribing quality and one practice of low prescribing quality were recruited. Participant observation, formal and informal interviews, and a review of practice documentation were employed.ResultsPractices ranked as high prescribing quality consistently made and applied macro and micro prescribing decisions, whereas the low-ranking practice only made micro prescribing decisions. Macro prescribing decisions were collective, policy decisions made considering research evidence in light of the average patient, one disease, condition, or drug. Micro prescribing decisions were made in consultation with the patient considering their views, preferences, circumstances and other conditions (if necessary).Although micro prescribing can operate independently, the implementation of evidence-based, quality prescribing was attributable to an interdependent relationship. Macro prescribing policy enabled prescribing decisions to be based on scientific evidence and applied consistently where possible. Ultimately, this influenced prescribing decisions that occur at the micro level in consultation with patients.ConclusionGeneral practitioners in the higher prescribing quality practices made two different ‘types’ of prescribing decision; macro and micro. Macro prescribing informs micro prescribing and without a macro basis to draw upon the low-ranked practice had no effective mechanism to engage with, reflect on and implement relevant evidence. Practices that recognize these two levels of decision making about prescribing are more likely to be able to implement higher quality evidence.

Pro’s and con’s of the stepped wedge design in cluster randomised trials of quality improvement interventions: two current examples

The stepped wedge design, under which all trial participants receive the intervention but the order in which the intervention is received is randomised, is potentially useful to rigorously evaluate organisational interventions to improve quality and safety. We use two examples of cluster-randomised stepped-wedge trials (DQIP and GP-POLY) to illustrate advantages and disadvantages of the design in evaluations of complex prescribing improvement interventions in primary care. DQIP is nearing completion and GP-POLY will start in 2013. The intervention in both DQIP and GP-Poly involves outreach visits by researchers for education and informatics tool training, making sequential roll out a logistic necessity. The stepped wedge allows for this by design, but trial durations may be prolonged compared to parallel-arm trials and other designs, and arranging initiation visits to fit with randomisation schedules is challenging. Since all participants receive the intervention and there are multiple repeated measurements, practice sample size requirements in DQIP and GP-POLY were reduced compared to a parallel-arm design, but power calculations are more complex. Recruitment may be improved by offering the intervention to all participants, but creates potential problems for retention and avoiding contamination in practices with long lags between recruitment and intervention start. Because of the vulnerability of stepped wedge trials to time varying confounding, avoiding changes in intervention delivery to successive cohorts is important and needs careful planning. The stepped wedge design is attractive for cluster randomised trials of quality improvement interventions, especially when staggering of intervention delivery is inevitable, but presents challenges for implementation that need careful planning. Oral presentation presented at the 2nd Clinical Trials Methodology Conference 2013: Methodology matters, 18-19 November 2013, Edinburgh, UK.

Developing a complex intervention to improve prescribing safety in primary care: mixed methods feasibility and optimisation pilot study

Objectives (A) To measure the extent to which different candidate outcome measures identified high-risk prescribing that is potentially changeable by the data-driven quality improvement in primary care (DQIP) intervention.(B) To explore the value of reviewing identified high-risk prescribing to clinicians.(C) To optimise the components of the DQIP intervention. Design Mixed method study. Setting General practices in two Scottish Health boards. Participants 4 purposively sampled general practices of varying size and socioeconomic deprivation. Outcome measures Prescribing measures targeting (1) high-risk use of the non-steroidal anti-inflammatory drugs (NSAIDs) and antiplatelets; (2) ‘Asthma control’ and (3) ‘Antithrombotics in atrial fibrillation (AF)’. Intervention The prescribing measures were used to identify patients for review by general practices. The ability of the measures to identify potentially changeable high-risk prescribing was measured as the proportion of patients reviewed where practices identified a need for action. Field notes were recorded from meetings between researchers and staff and key staff participated in semistructured interviews exploring their experience of the piloted intervention processes. Results Practices identified a need for action in 68%, 25% and 18% of patients reviewed for prescribing measures (1), (2) and (3), respectively. General practitioners valued being prompted to review patients, and perceived that (1) ‘NSAID and antiplatelet’ and (2) ‘antithrombotics in AF’ were the most important to act on. Barriers to initial and ongoing engagement and to sustaining improvements in prescribing were identified. Conclusions ‘NSAIDs and antiplatelets’ measures were selected as the most suitable outcome measures for the DQIP trial, based on evidence of this prescribing being more easily changeable. In response to the barriers identified, the intervention was designed to include a financial incentive, additional ongoing feedback on progress and reprompting review of patients, whose high-risk prescribing was restarted after a decision to stop. Trial registration number Clinicaltrials.gov NCT01425502.

Acceptability and perceived barriers and facilitators to creating a national research register to enable 'direct to patient' enrolment into research: the Scottish Health Research Register (SHARE).

BackgroundDifficulties with recruitment pose a major, increasingly recognised challenge to the viability of research. We sought to explore whether a register of volunteers interested in research participation, with data linkage to electronic health records to identify suitable research participants, would prove acceptable to healthcare staff, patients and researchers.MethodsWe undertook a qualitative study in which a maximum variation sampling approach was adopted. Focus groups and interviews were conducted with patients, general practitioners (GP), practice managers and health service researchers in two Scottish health boards. Analysis was primarily thematic to identify a range of issues and concerns for all stakeholder groups.ResultsThe concept of a national research register was, in general, acceptable to all stakeholder groups and was widely regarded as beneficial for research and for society. Patients, however, highlighted a number of conditions which should be met in the design of a register to expedite confidence and facilitate recruitment. They also gave their perceptions on how a register should operate and be promoted, favouring a range of media. GPs and practice managers were primarily concerned with the security and confidentiality of patient data and the impact a register may have on their workload. Researchers were supportive of the initiative seeing advantages in more rapid access to a wider pool of patients. They did raise concerns that GPs may be able to block access to personal patient data held in general practice clinical systems and that the register may not be representative of the whole population.ConclusionsThis work suggests that patients, healthcare staff and researchers have a favourable view of the potential benefits of a national register to identify people who are potentially eligible and willing to participate in health related research. It has highlighted a number of issues for the developers to incorporate in the design of research registers.

Study protocol of a mixed-methods evaluation of a cluster randomized trial to improve the safety of NSAID and antiplatelet prescribing: data-driven quality improvement in primary care

BackgroundTrials of complex interventions are criticized for being ‘black box’, so the UK Medical Research Council recommends carrying out a process evaluation to explain the trial findings. We believe it is good practice to pre-specify and publish process evaluation protocols to set standards and minimize bias. Unlike protocols for trials, little guidance or standards exist for the reporting of process evaluations. This paper presents the mixed-method process evaluation protocol of a cluster randomized trial, drawing on a framework designed by the authors.Methods/designThis mixed-method evaluation is based on four research questions and maps data collection to a logic model of how the data-driven quality improvement in primary care (DQIP) intervention is expected to work. Data collection will be predominately by qualitative case studies in eight to ten of the trial practices, focus groups with patients affected by the intervention and quantitative analysis of routine practice data, trial outcome and questionnaire data and data from the DQIP intervention.DiscussionWe believe that pre-specifying the intentions of a process evaluation can help to minimize bias arising from potentially misleading post-hoc analysis. We recognize it is also important to retain flexibility to examine the unexpected and the unintended. From that perspective, a mixed-methods evaluation allows the combination of exploratory and flexible qualitative work, and more pre-specified quantitative analysis, with each method contributing to the design, implementation and interpretation of the other.As well as strengthening the study the authors hope to stimulate discussion among their academic colleagues about publishing protocols for evaluations of randomized trials of complex interventions.Data-driven quality improvement in primary care trial registrationClinicalTrials.gov: NCT01425502

Process evaluation of the data-driven quality improvement in primary care (DQIP) trial: active and less active ingredients of a multi-component complex intervention to reduce high-risk primary care prescribing

BackgroundTwo to 4% of emergency hospital admissions are caused by preventable adverse drug events. The estimated costs of such avoidable admissions in England were £530 million in 2015. The data-driven quality improvement in primary care (DQIP) intervention was designed to prompt review of patients vulnerable from currently prescribed non-steroidal anti-inflammatory drugs (NSAIDs) and anti-platelets and was found to be effective at reducing this prescribing. A process evaluation was conducted parallel to the trial, and this paper reports the analysis which aimed to explore response to the intervention delivered to clusters in relation to participants’ perceptions about which intervention elements were active in changing their practice.MethodsData generation was by in-depth interview with key staff exploring participant’s perceptions of the intervention components. Analysis was iterative using the framework technique and drawing on normalisation process theory.ResultsAll the primary components of the intervention were perceived as active, but at different stages of implementation: financial incentives primarily supported recruitment; education motivated the GPs to initiate implementation; the informatics tool facilitated sustained implementation. Participants perceived the primary components as interdependent. Intervention subcomponents also varied in whether and when they were active. For example, run charts providing feedback of change in prescribing over time were ignored in the informatics tool, but were motivating in some practices in the regular e-mailed newsletter. The high-risk NSAID and anti-platelet prescribing targeted was accepted as important by all interviewees, and this shared understanding was a key wider context underlying intervention effectiveness.ConclusionsThis was a novel use of process evaluation data which examined whether and how the individual intervention components were effective from the perspective of the professionals delivering changed care to patients. These findings are important for reproducibility and roll-out of the intervention.Trial registrationClinicalTrials.gov, NCT01425502.

Learning from errors: what is the return on investment from training medical students in incident review?

Prescribing errors made by junior doctors are rare only due to lack of knowledge or technical skills. Final Year medical students were given the opportunity to investigate medication incidents involving junior doctors. However long-term sustainability of this activity requires evidence about return on investment for NHS Tayside who assist in supporting medical students. Kirkpatricks model for evaluating training programmes has been adapted for evaluation of medical education. We address all four levels of Kirkpatricks model for training medical students in incident reporting, discuss other models and emphasis the importance of return on investment. Examples of organizational learning highlighted in the pilot study suggest that incident review by medical students can provide fresh insights into achieving the “not-seeking-to-blame” culture. Incident Review is now a core activity for all final year medical students at the University of Dundee. Plans to engage with NHS Tayside in an iterative learning cycle that begins and ends with evaluation are underway.