Imelda Bates

Evaluating health research capacity building: An evidence-based tool.

Bates and colleagues describe the development of a tool to assess capacity-building programs in health research, which they used in Kumasi, Ghana.

Indicators of sustainable capacity building for health research: analysis of four African case studies.

AbstractBackgroundDespite substantial investment in health capacity building in developing countries, evaluations of capacity building effectiveness are scarce. By analysing projects in Africa that had successfully built sustainable capacity, we aimed to identify evidence that could indicate that capacity building was likely to be sustainable.MethodsFour projects were selected as case studies using pre-determined criteria, including the achievement of sustainable capacity. By mapping the capacity building activities in each case study onto a framework previously used for evaluating health research capacity in Ghana, we were able to identify activities that were common to all projects. We used these activities to derive indicators which could be used in other projects to monitor progress towards building sustainable research capacity.ResultsIndicators of sustainable capacity building increased in complexity as projects matured and included- early engagement of stakeholders; explicit plans for scale up; strategies for influencing policies; quality assessments (awareness and experiential stages) - improved resources; institutionalisation of activities; innovation (expansion stage) - funding for core activities secured; management and decision-making led by southern partners (consolidation stage).Projects became sustainable after a median of 66 months. The main challenges to achieving sustainability were high turnover of staff and stakeholders, and difficulties in embedding new activities into existing systems, securing funding and influencing policy development.ConclusionsOur indicators of sustainable capacity building need to be tested prospectively in a variety of projects to assess their usefulness. For each project the evidence required to show that indicators have been achieved should evolve with the project and they should be determined prospectively in collaboration with stakeholders.

Frameworks for evaluating health research capacity strengthening: a qualitative study

BackgroundHealth research capacity strengthening (RCS) projects are often complex and hard to evaluate. In order to inform health RCS evaluation efforts, we aimed to describe and compare key characteristics of existing health RCS evaluation frameworks: their process of development, purpose, target users, structure, content and coverage of important evaluation issues. A secondary objective was to explore what use had been made of the ESSENCE framework, which attempts to address one such issue: harmonising the evaluation requirements of different funders.MethodsWe identified and analysed health RCS evaluation frameworks published by seven funding agencies between 2004 and 2012, using a mixed methods approach involving structured qualitative analyses of documents, a stakeholder survey and consultations with key contacts in health RCS funding agencies.ResultsThe frameworks were intended for use predominantly by the organisations themselves, and most were oriented primarily towards funders’ internal organisational performance requirements. The frameworks made limited reference to theories that specifically concern RCS. Generic devices, such as logical frameworks, were typically used to document activities, outputs and outcomes, but with little emphasis on exploring underlying assumptions or contextual constraints. Usage of the ESSENCE framework appeared limited.ConclusionsWe believe that there is scope for improving frameworks through the incorporation of more accessible information about how to do evaluation in practice; greater involvement of stakeholders, following evaluation capacity building principles; greater emphasis on explaining underlying rationales of frameworks; and structuring frameworks so that they separate generic and project-specific aspects of health RCS evaluation. The third and fourth of these improvements might assist harmonisation.

Indicators for tracking programmes to strengthen health research capacity in lower- and middle-income countries: a qualitative synthesis

BackgroundThe monitoring and evaluation of health research capacity strengthening (health RCS) commonly involves documenting activities and outputs using indicators or metrics. We sought to catalogue the types of indicators being used to evaluate health RCS and to assess potential gaps in quality and coverage.MethodsWe purposively selected twelve evaluations to maximize diversity in health RCS, funders, countries, and approaches to evaluation. We explored the quality of the indicators and extracted them into a matrix across individual, institutional, and national/regional/network levels, based on a matrix in the ESSENCE Planning, Monitoring and Evaluation framework. We synthesized across potential impact pathways (activities to outputs to outcomes) and iteratively checked our findings with key health RCS evaluation stakeholders.ResultsEvaluations varied remarkably in the strengths of their evaluation designs. The validity of indicators and potential biases were documented in a minority of reports. Indicators were primarily of activities, outputs, or outcomes, with little on their inter-relationships. Individual level indicators tended to be more quantitative, comparable, and attentive to equity considerations. Institutional and national–international level indicators were extremely diverse. Although linkage of activities through outputs to outcomes within evaluations was limited, across the evaluations we were able to construct potential pathways of change and assemble corresponding indicators.ConclusionsOpportunities for improving health RCS evaluations include work on indicator measurement properties and development of indicators which better encompass relationships with knowledge users. Greater attention to evaluation design, prospective indicator measurement, and systematic linkage of indicators in keeping with theories of change could provide more robust evidence on outcomes of health RCS.

Implementing a national health research for development platform in a low-income country - a review of Malawi's Health Research Capacity Strengthening Initiative.

BackgroundNational health research for development (R4D) platforms in lower income countries (LICs) are few. The Health Research Capacity Strengthening Initiative (HRCSI, 2008–2013) was a national systems-strengthening programme in Malawi involved in national priority setting, decision-making on funding, and health research actor mobilization.MethodsWe adopted a retrospective mixed-methods evaluation approach, starting with information gleaned from reports (HRCSI and Malawian) and databases (HRCSI). A framework of a health research system (actors and components) guided report review and interview guide development. From a list of 173 individuals involved in HRCSI, 30 interviewees were selected within categories of stakeholders. Interviews were conducted face-to-face or via telephone/Skype over 1 month, documented with extensive notes. Analysis of emerging themes was iterative among co-evaluators, with synthesis according to the implementation stage.ResultsMajor HRCSI outputs included (1) National research priority-setting: through the production of themed background papers by Malawian health researchers and broad consultation, HRCSI led the development of a National Health Research Agenda (2012–2016), widely regarded as one of HRCSI’s foremost achievements. (2) Institutional research capacity: there was an overwhelming view that HRCSI had produced a step-change in the number of high calibre scientists in Malawi and in fostering research interest among young Malawians, providing support for around 56 MSc and PhD students, and over 400 undergraduate health-related projects. (3) Knowledge sharing: HRCSI supported research dissemination through national and institutional meetings by sponsoring attendance at conferences and through close relationships with individuals in the print media for disseminating information. (4) Sustainability: From 2011–2013, HRCSI significantly improved research systems, processes and leadership in Malawi, but further strengthening was needed for HRCSI to be effectively integrated into government structures and sustained long-term.Overall, HRCSI carried out many components relevant to a national health research system coordinating platform, and became competent at managing over half of 12 areas of performance for research councils. Debate about its location and challenges to sustainability remain open questions.ConclusionsMore experimentation in the setting-up of national health R4D platforms to promote country ‘ownership’ is needed, accompanied by evaluation processes that facilitate learning and knowledge exchange of better practices among key actors in health R4D systems.

Promoting sustainable research partnerships: a mixed-method evaluation of a United Kingdom–Africa capacity strengthening award scheme

BackgroundResearch partnerships between high-income countries (HICs) and low- or middle-income countries (LMICs) are a leading model in research capacity strengthening activities. Although numerous frameworks and guiding principles for effective research partnerships exist, few include the perspective of the LMIC partner. This paper draws out lessons for establishing and maintaining successful research collaborations, based on partnership dynamics, from the perspectives of both HIC and LMIC stakeholders through the evaluation of a research capacity strengthening partnership award scheme.MethodsA mixed-method retrospective evaluation approach was used. Initially, a cross-sectional survey was administered to all award holders, which focused on partnership outputs and continuation. Fifty individuals were purposively selected to participate in interviews or focus group discussions from 12 different institutions in HICs and LMICs; the sample included the research investigators, research assistants, laboratory scientists and post-doctoral students. The evaluation collected data on critical elements of research partnership dynamics such as research outputs, nature of the partnership, future plans and research capacity. Quantitative data were analysed descriptively and qualitative data were analysed using an iterative framework approach.ResultsThe majority of United Kingdom and African award holders stated they would like to pursue future collaborations together. Key aspects within partnerships that appeared to influence this were; the perceived benefits of the partnership at the individual and institutional level such as publication of papers or collaborative grants; ability to influence ‘research culture’ and instigate critical thinking among mid-career researchers; previous working relationships, for example supervisor-student relationships; and equity within partnerships linked to partnership formation and experience of United Kingdom partners within LMICs. Factors which may hinder development of long term partnerships were also identified such as financial control or differing expectations of partners.ConclusionsThis paper provides evidence of what encourages international research partnerships for capacity strengthening to continue past award tenure, from the perspective of researchers in high and LMICs. Although every partnership is unique and individual experiences subjective, this paper provides extension and support of key principles and mechanisms that can contribute to successful research partnerships between researchers.

Tackling the tensions in evaluating capacity strengthening for health research in low- and middle-income countries

Strengthening research capacity in low- and middle-income countries is one of the most effective ways of advancing their health and development but the complexity and heterogeneity of health research capacity strengthening (RCS) initiatives means it is difficult to evaluate their effectiveness. Our study aimed to enhance understanding about these difficulties and to make recommendations about how to make health RCS evaluations more effective. Through discussions and surveys of health RCS funders, including the ESSENCE on Health Research initiative, we identified themes that were important to health RCS funders and used these to guide a systematic analysis of their evaluation reports. Eighteen reports, produced between 2000 and 2013, representing 12 evaluations, were purposefully selected from 54 reports provided by the funders to provide maximum variety. Text from the reports was extracted independently by two authors against a pre-designed framework. Information about the health RCS approaches, tensions and suggested solutions was re-constructed into a narrative. Throughout the process contacts in the health RCS funder agencies were involved in helping us to validate and interpret our results. The focus of the health RCS evaluations ranged from individuals and institutions to national, regional and global levels. Our analysis identified tensions around how much stakeholders should participate in an evaluation, the appropriate balance between measuring and learning and between a focus on short-term processes vs longer-term impact and sustainability. Suggested solutions to these tensions included early and ongoing stakeholder engagement in planning and evaluating health RCS, modelling of impact pathways and rapid assimilation of lessons learned for continuous improvement of decision making and programming. The use of developmental approaches could improve health RCS evaluations by addressing common tensions and promoting sustainability. Sharing learning about how to do robust and useful health RCS evaluations should happen alongside, not after, health RCS efforts.

Advancing the science of health research capacity strengthening in low-income and middle-income countries: a scoping review of the published literature, 2000–2016

Objectives Substantial development assistance and research funding are invested in health research capacity strengthening (HRCS) interventions in low-income and middle-income countries, yet the effectiveness, impact and value for money of these investments are not well understood. A major constraint to evidence-informed HRCS intervention has been the disparate nature of the research effort to date. This review aims to map and critically analyse the existing HRCS effort to better understand the level, type, cohesion and conceptual sophistication of the current evidence base. The overall goal of this article is to advance the development of a unified, implementation-focused HRCS science. Methods We used a scoping review methodology to identify peer-reviewed HRCS literature within the following databases: PubMed, Global Health and Scopus. HRCS publications available in English between the period 2000 and 2016 were included. 1195 articles were retrieved of which 172 met the final inclusion criteria. A priori thematic analysis of all included articles was completed. Content analysis of identified HRCS definitions was conducted. Results The number of HRCS publications increased exponentially between 2000 and 2016. Most publications during this period were perspective, opinion or commentary pieces; however, original research publications were the primary publication type since 2013. Twenty-five different definitions of research capacity strengthening were identified, of which three aligned with current HRCS guidelines. Conclusions The review findings indicate that an HRCS research field with a focus on implementation science is emerging, although the conceptual and empirical bases are not yet sufficiently advanced to effectively inform HRCS programme planning. Consolidating an HRCS implementation science therefore presents as a viable option that may accelerate the development of a useful evidence base to inform HRCS programme planning. Identifying an agreed operational definition of HRCS, standardising HRCS-related terminology, developing a needs-based HRCS-specific research agenda and synthesising currently available evidence may be useful first steps.

Qualitative study to develop processes and tools for the assessment and tracking of African institutions’ capacity for operational health research

Objectives Research is key to achieving global development goals. Our objectives were to develop and test an evidence-informed process for assessing health research management and support systems (RMSS) in four African universities and for tracking interventions to address capacity gaps. Setting Four African universities. Participants 83 university staff and students from 11 cadres. Intervention/methods A literature-informed ‘benchmark’ was developed and used to itemise all components of a university’s health RMSS. Data on all components were collected during site visits to four African universities using interview guides, document reviews and facilities observation guides. Gaps in RMSS capacity were identified against the benchmark and institutional action plans developed to remedy gaps. Progress against indicators was tracked over 15 months and common challenges and successes identified. Results Common gaps in operational health research capacity included no accessible research strategy, a lack of research e-tracking capability and inadequate quality checks for proposal submissions and contracts. Feedback indicated that the capacity assessment was comprehensive and generated practical actions, several of which were no-cost. Regular follow-up helped to maintain focus on activities to strengthen health research capacity in the face of challenges. Conclusions Identification of each institutions’ strengths and weaknesses against an evidence-informed benchmark enabled them to identify gaps in in their operational health research systems, to develop prioritised action plans, to justify resource requests to fulfil the plans and to track progress in strengthening RMSS. Use of a standard benchmark, approach and tools enabled comparisons across institutions which has accelerated production of evidence about the science of research capacity strengthening. The tools could be used by institutions seeking to understand their strengths and to address gaps in research capacity. Research capacity gaps that were common to several institutions could be a ‘smart’ investment for governments and health research funders.

A systematic approach for reviewing research capacity within Zimbabwe's national blood service

Blood services need to undertake research to improve their strategic goals, operational effectiveness and promote evidence‐based policies. NBSZ has a long history of active research and undertook a systematic review of its research capacity to guide its new research strategy. In the absence of a published approach for research capacity assessment for national blood services, a framework to assess research capacity in African universities was used.