Lorenzo Moreno

Propensity Score Matching: A Note of Caution for Evaluators of Social Programs

Over the past 25 years, evaluators of social programs have searched for nonexperimental methods that can substitute effectively for experimental ones. Recently, the spotlight has focused on one method, propensity score matching (PSM), as the suggested approach for evaluating employment and education programs. We present a case study of our experience using PSM, under seemingly ideal circumstances, for the evaluation of the State Partnership Initiative employment promotion program. Despite ideal conditions and the passing of statistical tests suggesting that the matching procedure had worked, we find that PSM produced incorrect impact estimates when compared with a randomized design. Based on this experience, we caution practitioners about the risks of implementing PSM-based designs.

The Interrelation of Fertility and Geographic Mobility in Peru: A Hazards Model Analysis

The effect of place of residence and migration on fertility and the effect of fertility on migration have been long-standing concerns in population studies. This article makes use of novel longitudinal data from the Peru Demographic and Health Survey and associated statistical techniques to estimate these interrelationships for women of childbearing age. In the case of fertility outcomes, our results are consistent with the findings of others that age and education are negatively associated with fertility. We also find that residence in a city (versus countryside) predicts lower fertility, but residence in an intermediate size town has no effect. Our results are somewhat less robust for migration outcomes, but they do indicate that education and having fewer children are positively associated with geographical mobility. Town and city residents are more likely to be movers; yet, among movers, these more urbanized individuals exhibit slightly lower rates of subsequent mobility. When compared to previous research, our results demonstrate the value of detailed temporal measurement of the outcome event and its associated covariates.

Costs to Medicare of the Informatics for Diabetes Education and Telemedicine (IDEATel) Home Telemedicine Demonstration Findings from an independent evaluation

OBJECTIVE To estimate the impacts on Medicare costs of providing a particular type of home telemedicine to eligible Medicare beneficiaries with type 2 diabetes. RESEARCH DESIGN AND METHODS Two cohorts of beneficiaries (n = 1,665 and 504, respectively) living in two medically underserved areas of New York between 2000 and 2007 were randomized to intensive nurse case management via televisits or usual care. Medicare service use and costs covering a 6-year follow-up period were drawn from claims data. Impacts were estimated using regression analyses. RESULTS Informatics for Diabetes Education and Telemedicine (IDEATel) did not reduce Medicare costs in either site. Total costs were between 71 and 116% higher for the treatment group than for the control group. CONCLUSIONS Although IDEATel had modest effects on clinical outcomes (reported elsewhere), it did not reduce Medicare use or costs for health services. The interventions costs were excessive (over

Costs to Medicare of the IDEATel Home Telemedicine Demonstration: Findings from and Independent Evaluation

8,000 per person per year) compared with programs with similar-sized clinical impacts.

The Impact on Clients of a Community-Based Infant Mortality Reduction Program: The National Healthy Start Program Survey of Postpartum Women

OBJECTIVE To estimate the impacts on Medicare costs of providing a particular type of home telemedicine to eligible Medicare beneficiaries with type 2 diabetes. RESEARCH DESIGN AND METHODS Two cohorts of beneficiaries (n = 1,665 and 504, respectively) living in two medically underserved areas of New York between 2000 and 2007 were randomized to intensive nurse case management via televisits or usual care. Medicare service use and costs covering a 6-year follow-up period were drawn from claims data. Impacts were estimated using regression analyses. RESULTS Informatics for Diabetes Education and Telemedicine (IDEATel) did not reduce Medicare costs in either site. Total costs were between 71 and 116% higher for the treatment group than for the control group. CONCLUSIONS Although IDEATel had modest effects on clinical outcomes (reported elsewhere), it did not reduce Medicare use or costs for health services. The interventions costs were excessive (over

Aftershocks of Chile’s Earthquake for an Ongoing, Large-Scale Experimental Evaluation

8,000 per person per year) compared with programs with similar-sized clinical impacts.

Association Between Extending CareFirst’s Medical Home Program to Medicare Patients and Quality of Care, Utilization, and Spending

OBJECTIVES This study assessed the effect of the national Healthy Start Program on its clients. METHODS We used a cross-sectional survey of a sample from Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) rosters of women less than 6 months postpartum who were residents of Healthy Start Program areas. RESULTS Healthy Start clients revealed higher sociodemographic risk, but not behavioral risk, for adverse pregnancy outcome than other area residents. They did not differ from other residents in receipt of services except for a greater likelihood of receiving case management, using birth control at the time of the interview, and rating their prenatal care more highly. CONCLUSIONS The Healthy Start Program succeeded in enrolling women at high risk. It had little effect on the immediately concluded pregnancy, but it might influence future outcomes.

Could There Be a Role for Home Telemedicine in the U.S. Medicare Program

Evaluation designs for social programs are developed assuming minimal or no disruption from external shocks, such as natural disasters. This is because extremely rare shocks may not make it worthwhile to account for them in the design. Among extreme shocks is the 2010 Chile earthquake. Un Buen Comienzo (UBC), an ongoing early childhood program in Chile, was directly affected by the earthquake. This article discusses (a) the factors the UBC team considered for deciding whether to put on hold or continue implementation and data collection for this experimental study; and (b) how the team reached consensus on those decisions. A lesson learned is that the use of an experimental design for UBC insured that the evaluation’s internal validity was not compromised by the earthquake’s consequences, although cohort comparisons were compromised. Other lessons can be transferred to other contexts where external shocks affect an ongoing experimental or quasi-experimental impact evaluation.

Food Stamp Participants' Access to Food Retailers

Importance CareFirst, the largest commercial insurer in the mid-Atlantic Region of the United States, runs a medical home program focusing on financial incentives for primary care practices and care coordination for high-risk patients. From 2013 to 2015, CareFirst extended the program to Medicare fee-for-service (FFS) beneficiaries in participating practices. If the model extension improved quality while reducing spending, the Centers for Medicare and Medicaid Services could expand the program to Medicare beneficiaries broadly. Objective To test whether extending CareFirst’s program to Medicare FFS patients improves care processes and reduces hospitalizations, emergency department visits, and spending. Design, Setting, and Participants This difference-in-differences analysis compared outcomes for roughly 35 000 Medicare FFS patients attributed to 52 intervention practices (grouped by CareFirst into 14 “medical panels”) to outcomes for 69 000 Medicare patients attributed to 42 matched comparison panels during a 1-year baseline period and 2.5-year intervention at Maryland primary care practices. Main Outcomes and Measures Hospitalizations (all-cause and ambulatory-care sensitive), emergency department visits, Medicare Part A and B spending, and 3 quality-of-care process measures: ambulatory care within 14 days of a hospital stay, cholesterol testing for those with ischemic vascular disease, and a composite measure for those with diabetes. Interventions CareFirst hired nurses who worked with patients’ usual primary care practitioners to coordinate care for 3656 high-risk Medicare patients. CareFirst paid panels rewards for meeting cost and quality targets for their Medicare patients and advised panels on how to meet these targets based on analyses of claims data. Results On average, each of the 14 intervention panels had 9.3 primary care practitioners and was attributed 2202 Medicare FFS patients in the baseline period. The panels’ attributed Medicare patients were, on average, 73.8 years old, 59.2% female, and 85.1% white. The extension of CareFirst’s program to Medicare patients was not statistically associated with improvements in any outcomes, either for the full Medicare population or for a high-risk subgroup in which impacts were expected to be largest. For the full population, the difference-in-differences estimates were 1.4 hospitalizations per 1000 patients per quarter (P = .54; 90% CI, −2.1 to 5.0), −2.5 outpatient ED visits per 1000 patients per quarter (P = .26; 90% CI, −6.2 to 1.1), and −

Evaluation of the Costs to Medicare of Covering Therapeutic Shoes for Diabetic Patients

1 per patient per month in Medicare Part A and B spending (P = .98; 90% CI, −