Paul Jacob Robyn

Design of an impact evaluation using a mixed methods model – an explanatory assessment of the effects of results-based financing mechanisms on maternal healthcare services in Malawi

BackgroundIn this article we present a study design to evaluate the causal impact of providing supply-side performance-based financing incentives in combination with a demand-side cash transfer component on equitable access to and quality of maternal and neonatal healthcare services. This intervention is introduced to selected emergency obstetric care facilities and catchment area populations in four districts in Malawi. We here describe and discuss our study protocol with regard to the research aims, the local implementation context, and our rationale for selecting a mixed methods explanatory design with a quasi-experimental quantitative component.DesignThe quantitative research component consists of a controlled pre- and post-test design with multiple post-test measurements. This allows us to quantitatively measure ‘equitable access to healthcare services’ at the community level and ‘healthcare quality’ at the health facility level. Guided by a theoretical framework of causal relationships, we determined a number of input, process, and output indicators to evaluate both intended and unintended effects of the intervention. Overall causal impact estimates will result from a difference-in-difference analysis comparing selected indicators across intervention and control facilities/catchment populations over time.To further explain heterogeneity of quantitatively observed effects and to understand the experiential dimensions of financial incentives on clients and providers, we designed a qualitative component in line with the overall explanatory mixed methods approach. This component consists of in-depth interviews and focus group discussions with providers, service user, non-users, and policy stakeholders. In this explanatory design comprehensive understanding of expected and unexpected effects of the intervention on both access and quality will emerge through careful triangulation at two levels: across multiple quantitative elements and across quantitative and qualitative elements.DiscussionCombining a traditional quasi-experimental controlled pre- and post-test design with an explanatory mixed methods model permits an additional assessment of organizational and behavioral changes affecting complex processes. Through this impact evaluation approach, our design will not only create robust evidence measures for the outcome of interest, but also generate insights on how and why the investigated interventions produce certain intended and unintended effects and allows for a more in-depth evaluation approach.

Provider payment in community-based health insurance schemes in developing countries: a systematic review

Objectives Community-based health insurance (CBI) is a common mechanism to generate financial resources for health care in developing countries. We review for the first time provider payment methods used in CBI in developing countries and their impact on CBI performance. Methods We conducted a systematic review of the literature on provider payment methods used by CBI in developing countries published up to January 2010. Results Information on provider payment was available for a total of 32 CBI schemes in 34 reviewed publications: 17 schemes in South Asia, 10 in sub-Saharan Africa, 4 in East Asia and 1 in Latin America. Various types of provider payment were applied by the CBI schemes: 17 used fee-for-service, 12 used salaries, 9 applied a coverage ceiling, 7 used capitation and 6 applied a co-insurance. The evidence suggests that provider payment impacts CBI performance through provider participation and support for CBI, population enrolment and patient satisfaction with CBI, quantity and quality of services provided and provider and patient retention. Lack of provider participation in designing and choosing a CBI payment method can lead to reduced provider support for the scheme. Conclusion CBI schemes in developing countries have used a wide range of provider payment methods. The existing evidence suggests that payment methods are a key determinant of CBI performance and sustainability, but the strength of this evidence is limited since it is largely based on observational studies rather than on trials or on quasi-experimental research. According to the evidence, provider payment can affect provider participation, satisfaction and retention in CBI; the quantity and quality of services provided to CBI patients; patient demand of CBI services; and population enrollment, risk pooling and financial sustainability of CBI. CBI schemes should carefully consider how their current payment methods influence their performance, how changes in the methods could improve performance, and how such effects could be assessed with scientific rigour to increase the strength of evidence on this topic.

Developing attributes and attribute-levels for a discrete choice experiment on micro health insurance in rural Malawi

BackgroundDiscrete choice experiments (DCEs) are attribute-driven experimental techniques used to elicit stakeholders’ preferences to support the design and implementation of policy interventions. The validity of a DCE, therefore, depends on the appropriate specification of the attributes and their levels. There have been recent calls for greater rigor in implementing and reporting on the processes of developing attributes and attribute-levels for discrete choice experiments (DCEs). This paper responds to such calls by carefully reporting a systematic process of developing micro health insurance attributes and attribute-levels for the design of a DCE in rural Malawi.MethodsConceptual attributes and attribute-levels were initially derived from a literature review which informed the design of qualitative data collection tools to identify context specific attributes and attribute-levels. Qualitative data was collected in August-September 2012 from 12 focus group discussions with community residents and 8 in-depth interviews with health workers. All participants were selected according to stratified purposive sampling. The material was tape-recorded, fully transcribed, and coded by three researchers to identify context-specific attributes and attribute-levels. Expert opinion was used to scale down the attributes and levels. A pilot study confirmed the appropriateness of the selected attributes and levels for a DCE.ResultsFirst, a consensus, emerging from an individual level analysis of the qualitative transcripts, identified 10 candidate attributes. Levels were assigned to all attributes based on data from transcripts and knowledge of the Malawian context, derived from literature. Second, through further discussions with experts, four attributes were discarded based on multiple criteria. The 6 remaining attributes were: premium level, unit of enrollment, management structure, health service benefit package, transportation coverage and copayment levels. A final step of revision and piloting confirmed that the retained attributes satisfied the credibility criteria of DCE attributes.ConclusionThis detailed description makes our attribute development process transparent, and provides the reader with a basis to assess the rigor of this stage of constructing the DCE. This paper contributes empirical evidence to the limited methodological literature on attributes and levels development for DCE, thereby providing further empirical guidance on the matter, specifically within rural communities of low- and middle-income countries.

Econometric Analysis to Evaluate the Effect of Community-based Health Insurance on Reducing Informal Self-Care in Burkina Faso

Objective This study examines the role of community-based health insurance (CBHI) in influencing health-seeking behaviour in Burkina Faso, West Africa. Community-based health insurance was introduced in Nouna district, Burkina Faso, in 2004 with the goal to improve access to contracted providers based at primary- and secondary-level facilities. The paper specifically examines the effect of CBHI enrolment on reducing the prevalence of seeking modern and traditional methods of self-treatment as the first choice in care among the insured population. Methods Three stages of analysis were adopted to measure this effect. First, propensity score matching was used to minimize the observed baseline differences between the insured and uninsured populations. Second, through matching the average treatment effect on the treated, the effect of insurance enrolment on health-seeking behaviour was estimated. Finally, multinomial logistic regression was applied to model demand for available health care options, including no treatment, traditional self-treatment, modern self-treatment, traditional healers and facility-based care. Results For the first choice in care sought, there was no significant difference in the prevalence of self-treatment among the insured and uninsured populations, reaching over 55% for each group. When comparing the alternative option of no treatment, CBHI played no significant role in reducing the demand for self-care (either traditional or modern) or utilization of traditional healers, while it did significantly increase consumption of facility-based care. The average treatment effect on the treated was insignificant for traditional self-care, modern self-care and traditional healer, but was significant with a positive effect for use of facility care. Discussion While CBHI does have a positive impact on facility care utilization, its effect on reducing the prevalence of self-care is limited. The policy recommendations for improving the CBHI scheme’s responsiveness to population health care demand should incorporate community-based initiatives that offer attractive and appropriate alternatives to self-care.

Does health insurance improve health?: Evidence from a randomized community-based insurance rollout in rural Burkina Faso.

From 2004 to 2006, a community-based health insurance (CBI) scheme was rolled out in Nouna District, Burkina Faso, with the objective of improving access to health services and population health. We explore the random timing of the insurance rollout generated by the stepped wedge cluster-randomized design to evaluate the welfare and health impact of the insurance program. Our results suggest that the insurance had limited effects on average out-of-pocket expenditures in the target areas, but substantially reduced the likelihood of catastrophic health expenditure. The introduction of the insurance scheme did not have any effect on health outcomes for children and young adults, but appears to have increased mortality among individuals aged 65 and older. The negative health effects of the program appear to be primarily driven by the adverse provider incentives generated by the scheme and the resulting decline in the quality of care received by patients.

Health insurance and health-seeking behavior: Evidence from a randomized community-based insurance rollout in rural Burkina Faso

In 2004, a community-based health insurance (CBI) scheme was introduced in Nouna district, Burkina Faso, with the primary objective of improving access to facility-based health services. In order to overcome self-selection issues in the analysis of the behavioral effects of insurance, we combine four waves of the Nouna Health District Household Survey into a panel data set, and use the randomized timing of insurance rollout to estimate the causal effect of insurance coverage on health-seeking behavior. While we find a generally positive association between CBI affiliation and treatment seeking, we cannot reject the null that the introduction of health insurance does not have any effect on treatment seeking in general, and utilization of facility-based professional care, in particular. Low levels of health care provider satisfaction, poor perceived quality of care by enrollees, and ambiguity in the coverage level of the CBI benefit package appear to have contributed to these weak results. Our findings imply that the basic notion of insurance mechanically increasing facility-based professional care is not necessarily true empirically, and likely contingent on a large number of contextual factors affecting health-seeking behavior within households and communities.

Health worker preferences for community-based health insurance payment mechanisms: a discrete choice experiment

BackgroundIn 2004, a community-based health insurance scheme (CBI) was introduced in Nouna health district, Burkina Faso. Since its inception, coverage has remained low and dropout rates high. One important reason for low coverage and high dropout is that health workers do not support the CBI scheme because they are dissatisfied with the provider payment mechanism of the CBI.MethodsA discrete choice experiment (DCE) was used to examine CBI provider payment attributes that influence health workers’ stated preferences for payment mechanisms. The DCE was conducted among 176 health workers employed at one of the 34 primary care facilities or the district hospital in Nouna health district. Conditional logit models with main effects and interactions terms were used for analysis.ResultsReimbursement of service fees (adjusted odds ratio (aOR) 1.49, p < 0.001) and CBI contributions for medical supplies and equipment (aOR 1.47, p < 0.001) had the strongest effect on whether the health workers chose a given provider payment mechanism. The odds of selecting a payment mechanism decreased significantly if the mechanism included (i) results-based financing (RBF) payments made through the local health management team (instead of directly to the health workers (aOR 0.86, p < 0.001)) or (ii) RBF payments based on CBI coverage achieved in the health worker’s facility relative to the coverage achieved at other facilities (instead of payments based on the numbers of individuals or households enrolled at the health worker’s facility (aOR 0.86, p < 0.001)).ConclusionsProvider payment mechanisms can crucially determine CBI performance. Based on the results from this DCE, revised CBI payment mechanisms were introduced in Nouna health district in January 2011, taking into consideration health worker preferences on how they are paid.

Does enrollment status in community-based insurance lead to poorer quality of care? Evidence from Burkina Faso

IntroductionIn 2004, a community-based health insurance (CBI) scheme was introduced in Nouna health district, Burkina Faso, with the objective of improving financial access to high quality health services. We investigate the role of CBI enrollment in the quality of care provided at primary-care facilities in Nouna district, and measure differences in objective and perceived quality of care and patient satisfaction between enrolled and non-enrolled populations who visit the facilities.MethodsWe interviewed a systematic random sample of 398 patients after their visit to one of the thirteen primary-care facilities contracted with the scheme; 34% (n = 135) of the patients were currently enrolled in the CBI scheme. We assessed objective quality of care as consultation, diagnostic and counselling tasks performed by providers during outpatient visits, perceived quality of care as patient evaluations of the structures and processes of service delivery, and overall patient satisfaction. Two-sample t-tests were performed for group comparison and ordinal logistic regression (OLR) analysis was used to estimate the association between CBI enrollment and overall patient satisfaction.ResultsObjective quality of care evaluations show that CBI enrollees received substantially less comprehensive care for outpatient services than non-enrollees. In contrast, CBI enrollment was positively associated with overall patient satisfaction (aOR = 1.51, p = 0.014), controlling for potential confounders such as patient socio-economic status, illness symptoms, history of illness and characteristics of care received.ConclusionsCBI patients perceived better quality of care, while objectively receiving worse quality of care, compared to patients who were not enrolled in CBI. Systematic differences in quality of care expectations between CBI enrollees and non-enrollees may explain this finding. One factor influencing quality of care may be the type of provider payment used by the CBI scheme, which has been identified as a leading factor in reducing provider motivation to deliver high quality care to CBI enrollees in previous studies. Based on this study, it is unlikely that perceived quality of care and patient satisfaction explain the low CBI enrollment rates in this community.

Provider payment methods and health worker motivation in community-based health insurance: a mixed-methods study.

In a community-based health insurance (CBHI) introduced in 2004 in Nouna health district, Burkina Faso, poor perceived quality of care by CBHI enrollees has been a key factor in observed high drop-out rates. The poor quality perceptions have been previously attributed to health worker dissatisfaction with the provider payment method used by the scheme and the resulting financial risk of health centers. This study applied a mixed-methods approach to investigate how health workers working in facilities contracted by the CBHI view the methods of provider payment used by the CBHI. In order to analyze these relationships, we conducted 23 in-depth interviews and a quantitative survey with 98 health workers working in the CBHI intervention zone. The qualitative in-depth interviews identified that insufficient levels of capitation payments, the infrequent schedule of capitation payment, and lack of a payment mechanism for reimbursing service fees were perceived as significant sources of health worker dissatisfaction and loss of work-related motivation. Combining qualitative interview and quantitative survey data in a mixed-methods analysis, this study identified that the declining quality of care due to the CBHI provider payment method was a source of significant professional stress and role strain for health workers. Health workers felt that the following five changes due to the provider payment methods introduced by the CBHI impeded their ability to fulfill professional roles and responsibilities: (i) increased financial volatility of health facilities, (ii) dissatisfaction with eligible costs to be covered by capitation; (iii) increased pharmacy stock-outs; (iv) limited financial and material support from the CBHI; and (v) the lack of mechanisms to increase provider motivation to support the CBHI. To address these challenges and improve CBHI uptake and health outcomes in the targeted populations, the health care financing and delivery model in the study zone should be reformed. We discuss concrete options for reform based on the study findings.

Street-level workers’ inadequate knowledge and application of exemption policies in Burkina Faso jeopardize the achievement of universal health coverage: evidence from a cross-sectional survey

BackgroundStreet-level workers play a key role in public health policies in Africa, as they are often the ones to ensure their implementation. In Burkina Faso, the State formulated two different user-fee exemption policies for indigents, one for deliveries (2007), and one for primary healthcare (2009). The objective of this study was to measure and understand the determinants of street-level workers’ knowledge and application of these exemption measures.MethodsWe used cross-sectional data collected between October 2013 and March 2014. The survey targeted 1521 health workers distributed in 498 first-line centres, 18 district hospitals, 5 regional hospitals, and 11 private or other facilities across 24 districts. We used four different random effects models to identify factors associated with knowledge and application of each of the above-mentioned exemption policies.ResultsOnly 9.2% of workers surveyed knew of the directive exempting the worst-off, and only 5% implemented it. Knowledge and application of the delivery exemption were higher, with 27% of all health workers being aware of the delivery exemption directive and 24.2% applying it. Mobile health workers were found to be consistently more likely to apply both exemptions. Health workers who were facility heads were significantly more likely to know about the indigent exemption for primary health care and to apply it. Health workers in districts with higher proportions of very poor people were significantly more likely to know about and apply the delivery exemption. Nearly 60% of respondents indicated either 5% or 10% as the percentage of people they would deem adequate to target for exemption.ConclusionThis quantitative study confirmed earlier qualitative results on the importance of training and informing health workers and monitoring the measures targeting equity, to ensure compliance with government directives. The local context (e.g., hierarchy, health system, interventions) and the ideas that street-level workers have about the policy instruments can influence their effective implementation. Methods for remunerating health workers and health centres also need to be adapted to ensure equity measures are applied to achieve universal healthcare.