Diana Silver

Integrating a Comparison Group Design into a Theory of Change Evaluation: The Case of the Urban Health Initiative

Abstract This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation’s Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).

HEALTH PROMOTION IN THE CITY. A STRUCTURED REVIEW OF THE LITERATURE ON INTERVENTIONS TO PREVENT HEART DISEASE, SUBSTANCE ABUSE, VIOLENCE AND HIV INFECTION IN US METROPOLITAN AREAS, 1980-1995

To achieve its national public health goals, the US must improve the health of low-income urban populations. To contribute to this process, this study reviewed published reports of health promotion interventions designed to prevent heart disease, HIV infection, substance abuse, and violence in US cities. The studys objectives were to describe the target populations, settings, and program characteristics of these interventions and to assess the extent to which these programs followed accepted principles for health promotion. Investigators searched five computerized databases and references of selected articles for articles published in peer-reviewed journals between 1980 and 1995. Selected articles listed as a main goal primary prevention of one of four index conditions; were carried out within a US city; included sufficient information to characterize the intervention; and organized at least 25% of its activities within a community setting. In general, programs reached a diverse population of low-income city residents in a variety of settings, employed multiple strategies, and recognized at least some of the principles of effective health promotion. Most programs reported a systematic evaluation. However, many programs did not involve participants in planning, intervene to change underlying social causes, last more than a year, or tailor for the subpopulations they targeted, limiting their potential effectiveness. Few programs addressed the unique characteristics of urban communities.

‘ They hear “Africa” and they think that there can’t be any good services ’ – perceived context in cross-national learning: a qualitative study of the barriers to Reverse Innovation

BackgroundCountry-of-origin of a product can negatively influence its rating, particularly if the product is from a low-income country. It follows that how non-traditional sources of innovation, such as low-income countries, are perceived is likely to be an important part of a diffusion process, particularly given the strong social and cognitive boundaries associated with the healthcare professions.MethodsBetween September and December 2014, we conducted eleven in-depth face-to-face or telephone interviews with key informants from innovation, health and social policy circles, experts in international comparative policy research and leaders in Reverse Innovation in the United States. Interviews were open-ended with guiding probes into the barriers and enablers to Reverse Innovation in the US context, specifically also to understand whether, in their experience translating or attempting to translate innovations from low-income contexts into the US, the source of the innovation matters in the adopter context. Interviews were recorded, transcribed and analyzed thematically using the process of constant comparison.ResultsOur findings show that innovations from low-income countries tend to be discounted early on because of prior assumptions about the potential for these contexts to offer solutions to healthcare problems in the US. Judgments are made about the similarity of low-income contexts with the US, even though this is based oftentimes on flimsy perceptions only. Mixing levels of analysis, local and national, leads to country-level stereotyping and missed opportunities to learn from low-income countries.ConclusionsOur research highlights that prior expectations, invoked by the Low-income country cue, are interfering with a transparent and objective learning process. There may be merit in adopting some techniques from the cognitive psychology and marketing literatures to understand better the relative importance of source in healthcare research and innovation diffusion. Counter-stereotyping techniques and decision-making tools may be useful to help decision-makers evaluate the generalizability of research findings objectively and transparently. We suggest that those interested in Reverse Innovation should reflect carefully on the value of disclosing the source of the innovation that is being proposed, if doing so is likely to invoke negative stereotypes.

Finding the impact in a messy intervention: Using an integrated design to evaluate a comprehensive citywide health initiative

This article uses the evaluation of the Robert Wood Johnson Foundation’s (RWJF) Urban Health Initiative (UHI), a 10-year effort to improve health and safety outcomes in distressed cities, to demonstrate the strength of an evaluation design that integrates theory of change and quasi-experimental approaches, including the use of comparison cities. This paper focuses on the later stages of implementation and, especially, our methods for estimating program impacts. While the theory of change was used to make preliminary identification of intended outcomes, we used the sites’ plans and early implementation to refine this list and revisit our strategy for estimating impacts. Using our integrated design, differences between program and comparison cities are considered impacts only if they were predicted by program theory, local plans for action, and early implementation. We find small, measurable changes in areas of greatest programmatic effort. We discuss the importance of the integrated design in identifying impacts.

Patterns of alcohol consumption and related behaviors in Brazil: evidence from the 2013 National Health Survey (PNS 2013)

This study uses data from a nationally representative household survey (the 2013 National Health Survey, n = 62,986) to describe patterns of alcohol consumption and related behaviors among Brazilian adults. Analyses include descriptive and multivariable Poisson regression for self-reports in the past 30 days of: drinking any alcohol, binge drinking, binge drinking 4 or more times, and driving after drinking (DD); as well as age of alcohol consumption initiation. Results show that current drinking prevalence was 26%, with an average age of initiation of 18.7 years. Binge drinking was reported by 51% of drinkers, 43% of whom reported binge drinking 4 or more times. Drinking and driving was reported by nearly one quarter of those who drive a car/motorcycle. Current drinking was more likely among males, ages 25–34, single, urban, and those with more education. Binge drinking was more likely among males, older age groups, and people who started drinking before 18. Drinking and driving was higher among males, those with more education, and rural residents. Those who binge-drink were nearly 70% more likely to report DD. All behaviors varied significantly among Brazilian states. Given their potential health consequences, the levels of injurious alcohol behaviors observed here warrant increased attention from Brazilian policymakers and civil society.

Improving State Health Policy Assessment: An Agenda for Measurement and Analysis

We examine the scope of inquiry into the measurement and assessment of the state public health policy environment. We argue that there are gains to be made by looking systematically at policies both within and across health domains. We draw from the public health and public policy literature to develop the concepts of interdomain and intradomain policy comprehensiveness and illustrate how these concepts can be used to enhance surveillance of the current public health policy environment, improve understanding of the adoption of new policies, and enhance evaluations of the impact of such policies on health outcomes.

Changes in health care inequity in Brazil between 2008 and 2013

BackgroundBrazil has made progress towards a more equitable distribution of health care, but gains may be threatened by economic instability resulting from the 2008 global financial crisis. This study measured predictors of health care utilization and changes in horizontal inequity between 2008 and 2013.MethodData were from two nationally representative surveys that measured a variety of sociodemographic, health behaviors and health care indicators. We used Poisson regression models to estimate adjusted prevalence ratios and the Horizontal Equity Index (HEI) standardized by health needs to measure inequity in the utilization of doctor and dentist visits, hospitalizations and reporting of a usual source of care (USC) for those 18 and older. To estimate the HEI, we ranked the population from the poorest to the richest using a wealth index. We also decomposed the HEI into its different components and assessed changes from 2008 to 2013.ResultsThe population proportion with doctor and dentist visits in the past year and a USC increased between 2008 and 2013, while hospitalizations declined. In 2013, pro-rich inequity in doctor visits increased significantly while the distribution of hospitalizations shifted from pro-rich in 2008 to neutral in 2013. Dentist visits were highly pro-rich and USC was slightly pro-rich; the distribution of dentist visits and USC did not change over time. Health need was a strong predictor of health care utilization regardless of the type of coverage (public or private). Education, wealth, and private health plans were associated with the pro-rich orientation of doctor and dentist visits. Private health plans contributed to the pro-rich orientation of all outcomes, while the Family Health Strategy contributed to the pro-poor orientation of all outcomes.ConclusionThe results of this study support the claim that Brazil’s population continued to see absolute gains in access to care despite recent economic crises. However, gains in equity have slowed and may even decline if investments are not maintained as the country enters deeper financial and political crises.

Compliance with minimum price and legal age for cigarette purchase laws: evidence from NYC in advance of raising purchase age to 21

Background New York City (NYC) raised the minimum purchase age for cigarettes from 18 to 21 on 1 August 2014. The new law is intended to decrease current smoking rates and smoking initiation among the citys youth. Assessment of compliance with existing cigarette sales and tax laws could aid in determining what may be needed for successful implementation of the citys new law. Purpose To assess compliance with minimum sales price and purchase age laws in NYC, before change in law. Methods Ten trained field investigators purchased cigarettes from different types of retailers throughout all five NYC boroughs, resulting in 421 purchases. Investigators noted whether they were asked for identification and the price of their purchase. Multivariable logistic and Ordinary Least Squares regression techniques were used to assess predictors of retailer compliance with sales price and minimum purchase age laws. Results In 29% of purchases, investigators did not have to produce identification (p<0.05) to purchase cigarettes. Only 3.1% of sales were at prices lower than the minimum sales price. City borough was significantly associated with purchase without identification (p<0.001) and mean sales price (p<0.024). Vendor type (independent vs chain) was significantly related to investigators being able to purchase cigarettes without identification (p<0.001). Conclusions Variation in compliance with existing laws suggests that more active monitoring of compliance with the new minimum legal purchase age will be required in order to realise the new laws public health potential.

Detecting causality in policy diffusion processes

A universal question in network science entails learning about the topology of interaction from collective dynamics. Here, we address this question by examining diffusion of laws across US states. We propose two complementary techniques to unravel determinants of this diffusion process: information-theoretic union transfer entropy and event synchronization. In order to systematically investigate their performance on law activity data, we establish a new stochastic model to generate synthetic law activity data based on plausible networks of interactions. Through extensive parametric studies, we demonstrate the ability of these methods to reconstruct networks, varying in size, link density, and degree heterogeneity. Our results suggest that union transfer entropy should be preferred for slowly varying processes, which may be associated with policies attending to specific local problems that occur only rarely or with policies facing high levels of opposition. In contrast, event synchronization is effective for faster enactment rates, which may be related to policies involving Federal mandates or incentives. This study puts forward a data-driven toolbox to explain the determinants of legal activity applicable to political science, across dynamical systems, information theory, and complex networks.

Variation in U.S. Traffic Safety Policy Environments and Motor Vehicle Fatalities 1980–2010

OBJECTIVE To examine the impact of variation in state laws governing traffic safety on motor vehicle fatalities. STUDY DESIGN Repeated cross sectional time series design. METHODS Fixed effects regression models estimate the relationship between state motor vehicle fatality rates and the strength of the state law environment for 50 states, 1980-2010. The strength of the state policy environment is measured by calculating the proportion of a set of 27 evidence-based laws in place each year. The effect of alcohol consumption on motor vehicle fatalities is estimated using a subset of alcohol laws as instrumental variables. RESULTS Once other risk factors are controlled in statistical models, states with stronger regulation of safer driving and driver/passenger protections had significantly lower motor vehicle fatality rates for all ages. Alcohol consumption was strongly associated with higher MVC death rates, as were state unemployment rates. CONCLUSIONS Encouraging laggard states to adopt the full range of available laws could significantly reduce preventable traffic-related deaths in the U.S. - especially those among younger individuals. Estimating the relationship between different policy environments and health outcomes can quantify the result of policy gaps.