Sean Orzol

Propensity Score Matching: A Note of Caution for Evaluators of Social Programs

Over the past 25 years, evaluators of social programs have searched for nonexperimental methods that can substitute effectively for experimental ones. Recently, the spotlight has focused on one method, propensity score matching (PSM), as the suggested approach for evaluating employment and education programs. We present a case study of our experience using PSM, under seemingly ideal circumstances, for the evaluation of the State Partnership Initiative employment promotion program. Despite ideal conditions and the passing of statistical tests suggesting that the matching procedure had worked, we find that PSM produced incorrect impact estimates when compared with a randomized design. Based on this experience, we caution practitioners about the risks of implementing PSM-based designs.

Comparing continuous hemofiltration with hemodialysis in patients with severe acute renal failure.

Continuous venovenous hemofiltration (CVVH) or CVVH with additional diffusive dialysis (CVVH-D) has theoretical advantages in treating severe acute renal failure (ARF), but no prospective clinical trials or restrospective comparison studies have clearly shown its superiority over intermittent hemodialysis (HD). To evaluate this question, all 349 adult patients with ARF receiving renal replacement therapy (RRT) at our medical center during 1995 and 1996 were analyzed using multivariate Cox proportional hazards methods. Initial univariate analysis showed the odds of death when receiving initial CVVH to be more than twice those when receiving initial HD (risk for death, 2.03; P < 0.01). Progressive exclusion of patients in whom the RRT modality might not be open to choice and the risk for death was very high (systolic blood pressure < 90 mm Hg; total bilirubin level > 15 mg/dL; or total RRT < 48 hours) for total RRT left 227 patients in whom the risk for death was 1.09 (95% confidence interval [CI], 0.67 to 1.80; P = 0.72) for initial CVVH, virtually equivalent to the risk for initial HD. Comorbid indicators significantly associated with death or failure to recover renal function included: older age; medical rather than surgical diagnosis; preexisting infection or trauma and liver disease as primary diagnoses; and abnormal bilirubin level or vital signs at initiation of RRT. These results show that the high crude mortality rate of patients undergoing CVVH was related to severity of illness and not the treatment choice itself. With the addition of more inclusive comorbidity data and a broader spectrum of interim outcomes, this type of analysis is a practical alternative to what would almost assuredly be a cumbersome and costly prospective, controlled trial comparing traditional HD with CVVH.

Trends in treatment and survival for hemodialysis patients in the United States

Details regarding dialysis therapy have been studied by the US Renal Data System (USRDS) in four random samples of US hemodialysis patients during the years 1986 to 1997. During this decade, the delivered dose of hemodialysis therapy has increased by at least 0.2 Kt/V. The frequency of twice weekly dialysis prescription decreased, whereas the duration of each treatment showed only minor changes. A large shift to more biocompatible membranes, particularly to synthetic membranes, was observed. The use of acetate dialysate almost disappeared. Outcomes research by the USRDS showed significantly lower mortality risk associated independently with higher delivered Kt/V, substituted cellulose or synthetic membranes, and bicarbonate dialysate. The projected reduction in mortality risk from these changes in hemodialysis therapy was of a similar magnitude to the observed 14% to 17% reduction in mortality rate during the years 1990 to 1996. National observational studies of dialysis patients may influence the practice of dialysis and lead to improved survival.

Chronic Illness, Treatment Choice and Workforce Participation

Choices with respect to labor force participation and medical treatment are increasingly intertwined. Technological advances present patients with new choices and may facilitate continued employment for the growing number of chronically ill individuals. We examine joint work/treatment decisions of end stage renal disease patients, a group for whom these tradeoffs are particularly salient. Using a simultaneous equations probit model, we find that treatment choice is a significant predictor of employment status. However, the effect size is considerably smaller than in models that do not consider the joint nature of these choices.

Health insurance and access to care among welfare leavers.

This analysis explores the effects of the 1996 welfare reform on health insurance coverage and access to care among former recipients of cash aid. Using panel data from the Womens Employment Study, which conducted five interviews between 1997 and 2003 in one Michigan county, we find that 25% of welfare leavers lacked health insurance coverage in fall 2003. Uninsured adults were significantly more likely than others to report that they could not afford a medical or dental visit during the year prior to the 2003 interview. Fixed-effect logistic regression analysis indicates that women who had been off the welfare rolls for at least 12 months (the duration of transitional Medicaid) were significantly more likely to be uninsured than women who had made more recent welfare exits, and were significantly more likely to report financial obstacles to the receipt of medical and dental care.