Sanjeev Sridharan

Do health improvement programmes fit with MRC guidance on evaluating complex interventions

Although planning of new health policy could be improved to enable more robust evaluation, Mhairi Mackenzie and colleagues argue that randomised controlled trials are not always suitable or practical

Ten Steps to Making Evaluation Matter.

This paper proposes ten steps to make evaluations matter. The ten steps are a combination of the usual recommended practice such as developing program theory and implementing rigorous evaluation designs with a stronger focus on more unconventional steps including developing learning frameworks, exploring pathways of evaluation influence, and assessing spread and sustainability. Consideration of these steps can lead to a focused dialogue between program planners and evaluators and can result in more rigorously planned programs. The ten steps can also help in developing and implementing evaluation designs that have greater potential for policy and programmatic influence. The paper argues that there is a need to go beyond a formulaic approach to program evaluation design that often does not address the complexity of the programs. The complexity of the program will need to inform the design of the evaluation. The ten steps that are described in this paper are heavily informed by a Realist approach to evaluation. The Realist approach attempts to understand what is it about a program that makes it work.

Developing a Stakeholder-Driven Anticipated Timeline of Impact for Evaluation of Social Programs

The authors present a stakeholder-driven method, the earliest anticipated timeline of impact, which is designed to assess stakeholder expectations for the earliest time frame in which social programs are likely to affect outcomes. The utility of the anticipated timeline of impact is illustrated using an example from an evaluation of a comprehensive community initiative in which such a timeline was developed using the concept-mapping methodology. The benefits of such a timeline, including for planning programs and evaluations, are explored. Some potential problems that might arise when developing a stakeholder-driven timeline of impact are also discussed.

Does context matter for the relationship between deprivation and all-cause mortality? The West vs. the rest of Scotland

BackgroundA growing body of research emphasizes the importance of contextual factors on health outcomes. Using postcode sector data for Scotland (UK), this study tests the hypothesis of spatial heterogeneity in the relationship between area-level deprivation and mortality to determine if contextual differences in the West vs. the rest of Scotland influence this relationship. Research into health inequalities frequently fails to recognise spatial heterogeneity in the deprivation-health relationship, assuming that global relationships apply uniformly across geographical areas. In this study, exploratory spatial data analysis methods are used to assess local patterns in deprivation and mortality. Spatial regression models are then implemented to examine the relationship between deprivation and mortality more formally.ResultsThe initial exploratory spatial data analysis reveals concentrations of high standardized mortality ratios (SMR) and deprivation (hotspots) in the West of Scotland and concentrations of low values (coldspots) for both variables in the rest of the country. The main spatial regression result is that deprivation is the only variable that is highly significantly correlated with all-cause mortality in all models. However, in contrast to the expected spatial heterogeneity in the deprivation-mortality relationship, this relation does not vary between regions in any of the models. This result is robust to a number of specifications, including weighting for population size, controlling for spatial autocorrelation and heteroskedasticity, assuming a non-linear relationship between mortality and socio-economic deprivation, separating the dependent variable into male and female SMRs, and distinguishing between West, North and Southeast regions. The rejection of the hypothesis of spatial heterogeneity in the relationship between socio-economic deprivation and mortality complements prior research on the stability of the deprivation-mortality relationship over time.ConclusionsThe homogeneity we found in the deprivation-mortality relationship across the regions of Scotland and the absence of a contextualized effect of region highlights the importance of taking a broader strategic policy that can combat the toxic impacts of socio-economic deprivation on health. Focusing on a few specific places (e.g. 15% of the poorest areas) to concentrate resources might be a good start but the impact of socio-economic deprivation on mortality is not restricted to a few places. A comprehensive strategy that can be sustained over time might be needed to interrupt the linkages between poverty and mortality.

Towards an evidence base of theory-driven evaluations: Some questions for proponents of theory-driven evaluation

This article discusses the further development of theory-driven evaluation approaches that are informed by contribution analysis. Using an illustrative example of an ongoing dance/physical activity programme for health promotion, a number of challenges are identified when applying a theory-driven evaluation approach. These challenges are reformulated as questions that need to be answered to make further progress with theory-driven evaluation including contribution analysis. Questions include: What is a ‘good enough’ programme theory? How does one arrive at expectations of programme impacts? How does the programme theory incorporate heterogeneous mechanisms that programme recipients might need? What does causality mean for complex interventions? What are structures that can facilitate learning from evaluations? How does the application of theory-driven evaluation approaches help generate an ‘ecology of evidence’? Discussion of these questions leads to a ‘roadmap’ for how contribution analysis might be further tested and refined.

Where theory and practice of global health intersect: the developmental history of a Canadian global health initiative

Objective This paper examines the scope of practice of global health, drawing on the practical experience of a global health initiative of the Government of Canada - the Teasdale-Corti Global Health Research Partnership Program. A number of challenges in the practical application of theoretical definitions and understandings of global health are addressed. These challenges are grouped under five areas that form essential characteristics of global health: equity and egalitarian North-South partnerships, interdisciplinary scope, focus on upstream determinants of health, global conceptualization, and global health as an area of both research and practice. Design Information in this paper is based on the results of an external evaluation of the program, which involved analysis of project proposals and technical reports, surveys with grantees and interviews with grantees and program designers, as well as case studies of three projects and a review of relevant literature. Results The philosophy and recent definitions of global health represent a significant and important departure from the international health paradigm. However, the practical applicability of this maturing area of research and practice still faces significant systemic and structural impediments that, if not acknowledged and addressed, will continue to undermine the development of global health as an effective means to addressing health inequities globally and to better understanding, and acting upon, upstream determinants of health toward health for all. Conclusions While it strives to redress global inequities, global health continues to be a construct that is promoted, studied, and dictated mostly by Northern institutions and scholars. Until practical mechanisms are put in place for truly egalitarian partnerships between North and South for both the study and practice of global health, the emerging philosophy of global health cannot be effectively put into practice.Objective This paper examines the scope of practice of global health, drawing on the practical experience of a global health initiative of the Government of Canada – the Teasdale-Corti Global Health Research Partnership Program. A number of challenges in the practical application of theoretical definitions and understandings of global health are addressed. These challenges are grouped under five areas that form essential characteristics of global health: equity and egalitarian North–South partnerships, interdisciplinary scope, focus on upstream determinants of health, global conceptualization, and global health as an area of both research and practice. Design Information in this paper is based on the results of an external evaluation of the program, which involved analysis of project proposals and technical reports, surveys with grantees and interviews with grantees and program designers, as well as case studies of three projects and a review of relevant literature. Results The philosophy and recent definitions of global health represent a significant and important departure from the international health paradigm. However, the practical applicability of this maturing area of research and practice still faces significant systemic and structural impediments that, if not acknowledged and addressed, will continue to undermine the development of global health as an effective means to addressing health inequities globally and to better understanding, and acting upon, upstream determinants of health toward health for all. Conclusions While it strives to redress global inequities, global health continues to be a construct that is promoted, studied, and dictated mostly by Northern institutions and scholars. Until practical mechanisms are put in place for truly egalitarian partnerships between North and South for both the study and practice of global health, the emerging philosophy of global health cannot be effectively put into practice.

Towards a performance measurement system for health equity in a local health integration network

While there is a growing literature on building performance measurement systems for health equities, this literature for the most part has not dealt with the challenges of coordinating the various parts of the system, the heterogeneous nature of such systems, or how evaluations and measurement can themselves improve performance. This paper describes the initial steps taken to build a performance measurement system to coordinate health equity across 18 hospitals led by the Toronto Central Local Health Integration Network, which is a regional health authority serving a population of more than 2.5 million residents (near in population to Chicago and Rome) and the most socially diverse urban network in Ontario, Canada. This paper also describes some principles that can help inform a performance measurement system. The innovative aspect of this paper is that these principles were developed through feedback by the hospitals.

Delivering a national programme of anticipatory care in primary care: a qualitative study.

BACKGROUND Primary prevention often occurs against a background of inequalities in health and health care. Addressing this requires practitioners and systems to acknowledge the contribution of health-related and social determinants and to deal with the lack of interconnectedness between health and social service providers. Recognising this, the Scottish Government has implemented a national programme of anticipatory care targeting individuals aged 45-64 years living in areas of socioeconomic deprivation and at high risk of cardiovascular disease. This programme is called Keep Well. AIM To explore the issues and tensions underpinning the implementation of a national programme of anticipatory care. DESIGN AND SETTING A qualitative study in five Wave 1 Keep Well pilot sites, located in urban areas of Scotland, and involving 79 general practices. METHOD Annual semi-structured interviews were conducted with 74 key stakeholders operating at national government level, local pilot level and within general practices, resulting in 118 interviews. Interview transcripts were analysed using the framework approach. RESULTS Four underlying tensions were identified. First, those between a patient-focused general-practice approach versus a population-level health-improvement approach, linking disparate health and social services; secondly, medical approaches versus wider social approaches; thirdly, a population-wide approach versus individual targeting; and finally, reactive versus anticipatory care. CONCLUSION Implementing an anticipatory care programme to address inequalities in cardiovascular disease identified several tensions, which need to be understood and resolved in order to inform the development of such approaches in general practice and to develop systems that reduce the degree of fragmentation across health and social services.

Addressing Health Equities in Social Epidemiology: Learning from Evaluation(s)

This chapter examines how evaluations and evaluative thinking can help in the social epidemiologic study of complex interventions. There is increasing interest within the field of social epidemiology in studying interventions, as well as increasing pressure from funders and decision makers to make research more relevant for addressing social problems. Within the field of evaluation, there is a parallel move towards embracing the study of complex interventions − the very kinds of interventions that will almost invariably be the focus of social epidemiology. Using the example of interventions that seek to address health inequities in urban settings, we introduce a framework of steps through which evaluations can impact such health inequities. Rather than discussing a series of tools and methods, we use these steps to describe the importance of thinking evaluatively in addressing complex social problems. Specifically, we highlight a realist approach to evaluation. This approach focuses not only on whether an intervention works, but also on how it works, for whom and under what conditions (Pawson and Tilley 1997). This perspective marks a significant departure from traditions of other branches of epidemiology, such as clinical epidemiology, where the whether question is paramount and the how question is less important, often because of the uniformity and simplicity of interventions (e.g., administration of a drug). Research within epidemiology on social interventions has been relatively uncommon to date, and this chapter seeks to provide some guidance to expanding the literature on the health effects social interventions by engaging with cutting-edge theory on thinking evaluatively.

Making Evaluations Work : A Commentary on the Case Study.

This article describes the design and evaluation approaches to address the complexity posed by systems change initiatives. The role of evaluations in addressing the following issues is briefly reviewed: moving from strategic planning to implementation, impacts on system-level coordination, anticipated timeline of impact, and individual level impacts. The role of evaluations in enhancing multiple types of learning is also summarized.