Deborah DiLiberto

The practice of 'doing' evaluation: lessons learned from nine complex intervention trials in action

BackgroundThere is increasing recognition among trialists of the challenges in understanding how particular ‘real-life’ contexts influence the delivery and receipt of complex health interventions. Evaluations of interventions to change health worker and/or patient behaviours in health service settings exemplify these challenges. When interpreting evaluation data, deviation from intended intervention implementation is accounted for through process evaluations of fidelity, reach, and intensity. However, no such systematic approach has been proposed to account for the way evaluation activities may deviate in practice from assumptions made when data are interpreted.MethodsA collective case study was conducted to explore experiences of undertaking evaluation activities in the real-life contexts of nine complex intervention trials seeking to improve appropriate diagnosis and treatment of malaria in varied health service settings. Multiple sources of data were used, including in-depth interviews with investigators, participant-observation of studies, and rounds of discussion and reflection.Results and discussionFrom our experiences of the realities of conducting these evaluations, we identified six key ‘lessons learned’ about ways to become aware of and manage aspects of the fabric of trials involving the interface of researchers, fieldworkers, participants and data collection tools that may affect the intended production of data and interpretation of findings. These lessons included: foster a shared understanding across the study team of how individual practices contribute to the study goals; promote and facilitate within-team communications for ongoing reflection on the progress of the evaluation; establish processes for ongoing collaboration and dialogue between sub-study teams; the importance of a field research coordinator bridging everyday project management with scientific oversight; collect and review reflective field notes on the progress of the evaluation to aid interpretation of outcomes; and these approaches should help the identification of and reflection on possible overlaps between the evaluation and intervention.ConclusionThe lessons we have drawn point to the principle of reflexivity that, we argue, needs to become part of standard practice in the conduct of evaluations of complex interventions to promote more meaningful interpretations of the effects of an intervention and to better inform future implementation and decision-making.

Impact of introduction of rapid diagnostic tests for malaria on antibiotic prescribing: analysis of observational and randomised studies in public and private healthcare settings

Objectives To examine the impact of use of rapid diagnostic tests for malaria on prescribing of antimicrobials, specifically antibiotics, for acute febrile illness in Africa and Asia. Design Analysisof nine preselected linked and codesigned observational and randomised studies (eight cluster or individually randomised trials and one observational study). Setting Public and private healthcare settings, 2007-13, in Afghanistan, Cameroon, Ghana, Nigeria, Tanzania, and Uganda. Participants 522 480 children and adults with acute febrile illness. Interventions Rapid diagnostic tests for malaria. Main outcome measures Proportions of patients for whom an antibiotic was prescribed in trial groups who had undergone rapid diagnostic testing compared with controls and in patients with negative test results compared with patients with positive results. A secondary aim compared classes of antibiotics prescribed in different settings. Results Antibiotics were prescribed to 127 052/238 797 (53%) patients in control groups and 167 714/283 683 (59%) patients in intervention groups. Antibiotics were prescribed to 40% (35 505/89 719) of patients with a positive test result for malaria and to 69% (39 400/57 080) of those with a negative result. All but one study showed a trend toward more antibiotic prescribing in groups who underwent rapid diagnostic tests. Random effects meta-analysis of the trials showed that the overall risk of antibiotic prescription was 21% higher (95% confidence interval 7% to 36%) in intervention settings. In most intervention settings, patients with negative test results received more antibiotic prescriptions than patients with positive results for all the most commonly used classes: penicillins, trimethoprim-sulfamethoxazole (one exception), tetracyclines, and metronidazole. Conclusions Introduction of rapid diagnostic tests for malaria to reduce unnecessary use of antimalarials—a beneficial public health outcome—could drive up untargeted use of antibiotics. That 69% of patients were prescribed antibiotics when test results were negative probably represents overprescription.This included antibiotics from several classes, including those like metronidazole that are seldom appropriate for febrile illness, across varied clinical, health system, and epidemiological settings. It is often assumed that better disease specific diagnostics will reduce antimicrobial overuse, but they might simply shift it from one antimicrobial class to another. Current global implementation of malaria testing might increase untargeted antibiotic use and must be examined.

Tools for mass screening of G6PD deficiency: validation of the WST8/1-methoxy-PMS enzymatic assay in Uganda

BackgroundThe distribution of the enzymopathy glucose-6-phosphate dehydrogenase (G6PD) deficiency is linked to areas of high malaria endemicity due to its association with protection from disease. G6PD deficiency is also identified as the cause of severe haemolysis following administration of the anti-malarial drug primaquine and further use of this drug will likely require identification of G6PD deficiency on a population level. Current conventional methods for G6PD screening have various disadvantages for field use.MethodsThe WST8/1-methoxy PMS method, recently adapted for field use, was validated using a gold standard enzymatic assay (R&D Diagnostics Ltd ®) in a study involving 235 children under five years of age, who were recruited by random selection from a cohort study in Tororo, Uganda. Blood spots were collected by finger-prick onto filter paper at routine visits, and G6PD activity was determined by both tests. Performance of the WST8/1-methoxy PMS test under various temperature, light, and storage conditions was evaluated.ResultsThe WST8/1-methoxy PMS assay was found to have 72% sensitivity and 98% specificity when compared to the commercial enzymatic assay and the AUC was 0.904, suggesting good agreement. Misclassifications were at borderline values of G6PD activity between mild and normal levels, or related to outlier haemoglobin values (<8.0 gHb/dl or >14 gHb/dl) associated with ongoing anaemia or recent haemolytic crises. Although severe G6PD deficiency was not found in the area, the test enabled identification of low G6PD activity. The assay was found to be highly robust for field use; showing less light sensitivity, good performance over a wide temperature range, and good capacity for medium-to-long term storage.ConclusionsThe WST8/1-methoxy PMS assay was comparable to the currently used standard enzymatic test, and offers advantages in terms of cost, storage, portability and use in resource-limited settings. Such features make this test a potential key tool for deployment in the field for point of care assessment prior to primaquine administration in malaria-endemic areas. As with other G6PD tests, outlier haemoglobin levels may confound G6PD level estimation.

Community case management of malaria: exploring support, capacity and motivation of community medicine distributors in Uganda.

Background In Uganda, community services for febrile children are expanding from presumptive treatment of fever with anti-malarials through the home-based management of fever (HBMF) programme, to include treatment for malaria, diarrhoea and pneumonia through Integrated Community Case Management (ICCM). To understand the level of support available, and the capacity and motivation of community health workers to deliver these expanded services, we interviewed community medicine distributors (CMDs), who had been involved in the HBMF programme in Tororo district, shortly before ICCM was adopted. Methods Between October 2009 and April 2010, 100 CMDs were recruited to participate by convenience sampling. The survey included questionnaires to gather information about the CMDs’ work experience and to assess knowledge of fever case management, and in-depth interviews to discuss experiences as CMDs including motivation, supervision and relationships with the community. All questionnaires and knowledge assessments were analysed. Summary contact sheets were made for each of the 100 interviews and 35 were chosen for full transcription and analysis. Results CMDs faced multiple challenges including high patient load, limited knowledge and supervision, lack of compensation, limited drugs and supplies, and unrealistic expectations of community members. CMDs described being motivated to volunteer for altruistic reasons; however, the main benefits of their work appeared related to ‘becoming someone important’, with the potential for social mobility for self and family, including building relationships with health workers. At the time of the survey, over half of CMDs felt demotivated due to limited support from communities and the health system. Conclusions Community health worker programmes rely on the support of communities and health systems to operate sustainably. When this support falls short, motivation of volunteers can wane. If community interventions, in increasingly complex forms, are to become the solution to improving access to primary health care, greater attention to what motivates individuals, and ways to strengthen health system support are required.

The PROCESS study: a protocol to evaluate the implementation, mechanisms of effect and context of an intervention to enhance public health centres in Tororo, Uganda

BackgroundDespite significant investments into health improvement programmes in Uganda, health indicators and access to healthcare remain poor across the country. The PRIME trial aims to evaluate the impact of a complex intervention delivered in public health centres on health outcomes of children and management of malaria in rural Uganda. The intervention consists of four components: Health Centre Management; Fever Case Management; Patient- Centered Services; and support for supplies of malaria diagnostics and antimalarial drugs.MethodsThe PROCESS study will use mixed methods to evaluate the processes, mechanisms of change, and context of the PRIME intervention by addressing five objectives. First, to develop a comprehensive logic model of the intervention, articulating the project’s hypothesised pathways to trial outcomes. Second, to evaluate the implementation of the intervention, including health worker training, health centre management tools, and the supply of artemether-lumefantrine (AL) and rapid diagnostic tests (RDTs) for malaria. Third, to understand mechanisms of change of the intervention components, including testing hypotheses and interpreting realities of the intervention, including resistance, in context. Fourth, to develop a contextual record over time of factors that may have affected implementation of the intervention, mechanisms of change, and trial outcomes, including factors at population, health centre and district levels. Fifth, to capture broader expected and unexpected impacts of the intervention and trial activities among community members, health centre workers, and private providers. Methods will include intervention logic mapping, questionnaires, recorded consultations, in-depth interviews, focus group discussions, and contextual data documentation.DiscussionThe findings of this PROCESS study will be interpreted alongside the PRIME trial results. This will enable a greater ability to generalise the findings of the main trial. The investigators will attempt to assess which methods are most informative in such evaluations of complex interventions in low-resource settings.Trial registrationClinicaltrials.gov, NCT01024426

Aspirations for quality health care in Uganda: How do we get there?

BackgroundDespite significant investments and reforms, health care remains poor for many in Africa. To design an intervention to improve access and quality of health care at health facilities in eastern Uganda, we aimed to understand local priorities for qualities in health care, and factors that enable or prevent these qualities from being enacted.MethodsIn 2009 to 2010, we carried out 69 in-depth interviews and 6 focus group discussions with 65 health workers at 17 health facilities, and 10 focus group discussions with 113 community members in Tororo District, Uganda.ResultsHealth-care workers and seekers valued technical, interpersonal and resource qualities in their aspirations for health care. However, such qualities were frequently not enacted, and our analysis suggests that meeting aspirations required social and financial resources to negotiate various power structures.ConclusionsWe argue that achieving aspirations for qualities valued in health care will require a genuine reorientation of focus by health workers and their managers toward patients, through renewed respect and support for these providers as professionals.

Strengthening patient-centred communication in rural Ugandan health centres: A theory-driven evaluation within a cluster randomized trial

This article describes a theory-driven evaluation of one component of an intervention to improve the quality of health care at Ugandan public health centres. Patient-centred services have been advocated widely, but such approaches have received little attention in Africa. A cluster randomized trial is evaluating population-level outcomes of an intervention with multiple components, including ‘patient-centred services.’ A process evaluation was designed within this trial to articulate and evaluate the implementation and programme theories of the intervention. This article evaluates one hypothesized mechanism of change within the programme theory: the impact of the Patient Centred Services component on health-worker communication. The theory-driven approach extended to evaluation of the outcome measures. The study found that the proximal outcome of patient-centred communication was rated 10 percent higher (p < 0.008) by care seekers consulting with the health workers who were at the intervention health centres compared with those at control health centres. This finding will strengthen interpretation of more distal trial outcomes.

Behind the scenes of the PRIME intervention: designing a complex intervention to improve malaria care at public health centres in Uganda

Background In Uganda, health system challenges limit access to good quality healthcare and contribute to slow progress on malaria control. We developed a complex intervention (PRIME), which was designed to improve quality of care for malaria at public health centres. Objective Responding to calls for increased transparency, we describe the PRIME interventions design process, rationale, and final content and reflect on the choices and challenges encountered during the design of this complex intervention. Design To develop the intervention, we followed a multistep approach, including the following: 1) formative research to identify intervention target areas and objectives; 2) prioritization of intervention components; 3) review of relevant evidence; 4) development of intervention components; 5) piloting and refinement of workshop modules; and 6) consolidation of the PRIME intervention theories of change to articulate why and how the intervention was hypothesized to produce desired outcomes. We aimed to develop an intervention that was evidence-based, grounded in theory, and appropriate for the study context; could be evaluated within a randomized controlled trial; and had the potential to be scaled up sustainably. Results The process of developing the PRIME intervention package was lengthy and dynamic. The final intervention package consisted of four components: 1) training in fever case management and use of rapid diagnostic tests for malaria (mRDTs); 2) workshops in health centre management; 3) workshops in patient-centred services; and 4) provision of mRDTs and antimalarials when stocks ran low. Conclusions The slow and iterative process of intervention design contrasted with the continually shifting study context. We highlight the considerations and choices made at each design stage, discussing elements we included and why, as well as those that were ultimately excluded. Reflection on and reporting of ‘behind the scenes’ accounts of intervention design may improve the design, assessment, and generalizability of complex interventions and their evaluations.Background In Uganda, health system challenges limit access to good quality healthcare and contribute to slow progress on malaria control. We developed a complex intervention (PRIME), which was designed to improve quality of care for malaria at public health centres. Objective Responding to calls for increased transparency, we describe the PRIME interventions design process, rationale, and final content and reflect on the choices and challenges encountered during the design of this complex intervention. Design To develop the intervention, we followed a multistep approach, including the following: 1) formative research to identify intervention target areas and objectives; 2) prioritization of intervention components; 3) review of relevant evidence; 4) development of intervention components; 5) piloting and refinement of workshop modules; and 6) consolidation of the PRIME intervention theories of change to articulate why and how the intervention was hypothesized to produce desired outcomes. We aimed to develop an intervention that was evidence-based, grounded in theory, and appropriate for the study context; could be evaluated within a randomized controlled trial; and had the potential to be scaled up sustainably. Results The process of developing the PRIME intervention package was lengthy and dynamic. The final intervention package consisted of four components: 1) training in fever case management and use of rapid diagnostic tests for malaria (mRDTs); 2) workshops in health centre management; 3) workshops in patient-centred services; and 4) provision of mRDTs and antimalarials when stocks ran low. Conclusions The slow and iterative process of intervention design contrasted with the continually shifting study context. We highlight the considerations and choices made at each design stage, discussing elements we included and why, as well as those that were ultimately excluded. Reflection on and reporting of ‘behind the scenes’ accounts of intervention design may improve the design, assessment, and generalizability of complex interventions and their evaluations.

Improving prescribing practices with rapid diagnostic tests (RDTs): synthesis of 10 studies to explore reasons for variation in malaria RDT uptake and adherence

Objectives The overuse of antimalarial drugs is widespread. Effective methods to improve prescribing practice remain unclear. We evaluated the impact of 10 interventions that introduced rapid diagnostic tests for malaria (mRDTs) on the use of tests and adherence to results in different contexts. Design A comparative case study approach, analysing variation in outcomes across different settings. Setting Studies from the ACT Consortium evaluating mRDTs with a range of supporting interventions in 6 malaria endemic countries. Providers were governmental or non-governmental healthcare workers, private retail sector workers or community volunteers. Each study arm in a distinct setting was considered a case. Participants 28 cases from 10 studies were included, representing 148 461 patients seeking care for suspected malaria. Interventions The interventions included different mRDT training packages, supervision, supplies and community sensitisation. Outcome measures Analysis explored variation in: (1) uptake of mRDTs (% febrile patients tested); (2) provider adherence to positive mRDTs (% Plasmodium falciparum positive prescribed/given Artemisinin Combination Treatment); (3) provider adherence to negative mRDTs (% P. falciparum negative not prescribed/given antimalarial). Results Outcomes varied widely across cases: 12–100% mRDT uptake; 44–98% adherence to positive mRDTs; 27–100% adherence to negative mRDTs. Providers appeared more motivated to perform well when mRDTs and intervention characteristics fitted with their own priorities. Goodness of fit of mRDTs with existing consultation and diagnostic practices appeared crucial to maximising the impact of mRDTs on care, as did prior familiarity with malaria testing; adequate human resources and supplies; possible alternative treatments for mRDT-negative patients; a more directive intervention approach and local preferences for ACTs. Conclusions Basic training and resources are essential but insufficient to maximise the potential of mRDTs in many contexts. Programme design should respond to assessments of provider priorities, expectations and capacities. As mRDTs become established, the intensity of supporting interventions required seems likely to reduce.

The Impact of an Intervention to Improve Malaria Care in Public Health Centers on Health Indicators of Children in Tororo, Uganda (PRIME): A Cluster-Randomized Trial

Optimizing quality of care for malaria and other febrile illnesses is a complex challenge of major public health importance. To evaluate the impact of an intervention aiming to improve malaria case management on the health of community children, a cluster-randomized trial was conducted from 2010–2013 in Tororo, Uganda, where malaria transmission is high. Twenty public health centers were included; 10 were randomized in a 1:1 ratio to intervention or control. Households within 2 km of health centers provided the sampling frame for the evaluation. The PRIME intervention included training in fever case management using malaria rapid diagnostic tests (mRDTs), patient-centered services, and health center management; plus provision of mRDTs and artemether–lumefantrine. Cross-sectional community surveys were conducted at baseline and endline (N = 8,766), and a cohort of children was followed for approximately 18 months (N = 992). The primary outcome was prevalence of anemia (hemoglobin < 11.0 g/dL) in children under 5 years of age in the final community survey. The intervention was delivered successfully; however, no differences in prevalence of anemia or parasitemia were observed between the study arms in the final community survey or the cohort. In the final survey, prevalence of anemia in children under 5 years of age was 62.5% in the intervention versus 63.1% in control (adjusted risk ratio = 1.01; 95% confidence interval = 0.91–1.13; P = 0.82). The PRIME intervention, focusing on training and commodities, did not produce the expected health benefits in community children in Tororo. This challenges common assumptions that improving quality of care and access to malaria diagnostics will yield health gains.