Fabrizia Mealli

Likelihood-Based Analysis of Causal Effects of Job-Training Programs Using Principal Stratification

Government-sponsored job-training programs must be subject to evaluation to assess whether their effectiveness justifies their cost to the public. The evaluation usually focuses on employment and total earnings, although the effect on wages is also of interest, because this effect reflects the increase in human capital due to the training program, whereas the effect on total earnings may be simply reflecting the increased likelihood of employment without any effect on wage rates. Estimating the effects of training programs on wages is complicated by the fact that, even in a randomized experiment, wages are “truncated” (or less accurately “censored”) by nonemployment, that is, they are only observed and well-defined for individuals who are employed. In this article, we develop a likelihood-based approach to estimate the wage effect of the US federally-funded Job Corps training program using “Principal Stratification”. Our estimands are formulated in terms of: (1) the effect of the training program on wages for those who would be employed whether they were trained or not, also called the survivor average causal effect (SACE), and the proportion of people in this category; (2) the wages when trained for those who would be employed only when trained, and the proportion of people in this category; (3) the wages when not trained for those who would be employed only when not trained, and the proportion of people in this category; (4) the proportion of people who would be not employed whether trained or not. We conduct likelihood-based analysis using the EM algorithm, and investigate the plausibility of important submodels with scaled log-likelihood ratio statistics. We also conduct a sensitivity analysis with respect to specific parametric assumptions. Our results suggest that all four types of people [(1)–(4) previously] exist, which is impossible under the usual monotonicity assumptions made in traditional econometric evaluation methods.

Evaluating the effects of job training programs on wages through principal stratification

In an evaluation of a job training program, the causal effects of the program on wages are often of more interest to economists than the programs effects on employment or on income. The reason is that the effects on wages reflect the increase in human capital due to the training program, whereas the effects on total earnings or income may be simply reflecting the increased likelihood of employment without any effect on wage rates. Estimating the effects of training programs on wages is complicated by the fact that, even in a randomized experiment, wages are truncated by nonemployment, i.e., are only observed and well-defined for individuals who are employed. We present a principal stratification approach applied to a randomized social experiment that classifies participants into four latent groups according to whether they would be employed or not under treatment and control, and argue that the average treatment effect on wages is only clearly defined for those who would be employed whether they were trained or not. We summarize large sample bounds for this average treatment effect, and propose and derive a Bayesian analysis and the associated Bayesian Markov Chain Monte Carlo computational algorithm. Moreover, we illustrate the application of new code checking tools to our Bayesian analysis to detect possible coding errors. Finally, we demonstrate our Bayesian analysis using simulated data.

Occupational Pensions and Job Mobility in Britain: Estimation of a Random-Effects Competing Risks Model

We analyse transitions between pensionable jobs, non-pensionable jobs, and other labour market states, using the 1988/9 UK Retirement Survey. We focus on the positive association between length of job tenure and pensionable status, allowing for the possibility that pension scheme members are less mobile than other workers because they have persistent unobserved characteristics that predispose them towards a high degree of security in both employment and retirement. We use a competing risks model with state-specific random effects. The model is estimated by simulated maximum likelihood. Copyright 1996 by John Wiley & Sons, Ltd.

Evaluating the Effect of Training on Wages in the Presence of Noncompliance, Nonemployment, and Missing Outcome Data

The effects of a job training program, Job Corps, on both employment and wages are evaluated using data from a randomized study. Principal stratification is used to address, simultaneously, the complications of noncompliance, wages that are only partially defined because of nonemployment, and unintended missing outcomes. The first two complications are of substantive interest, whereas the third is a nuisance. The objective is to find a parsimonious model that can be used to inform public policy. We conduct a likelihood-based analysis using finite mixture models estimated by the expectation-maximization (EM) algorithm. We maintain an exclusion restriction assumption for the effect of assignment on employment and wages for noncompliers, but not on missingness. We provide estimates under the “missing at random” assumption, and assess the robustness of our results to deviations from it. The plausibility of meaningful restrictions is investigated by means of scaled log-likelihood ratio statistics. Substantive conclusions include the following. For compliers, the effect on employment is negative in the short term; it becomes positive in the long term, but these effects are small at best. For always employed compliers, that is, compliers who are employed whether trained or not trained, positive effects on wages are found at all time periods. Our analysis reveals that background characteristics of individuals differ markedly across the principal strata. We found evidence that the program should have been better targeted, in the sense of being designed differently for different groups of people, and specific suggestions are offered. Previous analyses of this dataset, which did not address all complications in a principled manner, led to less nuanced conclusions about Job Corps.

Using Secondary Outcomes to Sharpen Inference in Randomized Experiments With Noncompliance

We develop new methods for analyzing randomized experiments with noncompliance and, by extension, instrumental variable settings, when the often controversial, but key, exclusion restriction assumption is violated. We show how existing large-sample bounds on intention-to-treat effects for the subpopulations of compliers, never-takers, and always-takers can be tightened by exploiting the joint distribution of the outcome of interest and a secondary outcome, for which the exclusion restriction is satisfied. The derived bounds can be used to detect violations of the exclusion restriction and the magnitude of these violations in instrumental variables settings. It is shown that the reduced width of the bounds depends on the strength of the association of the auxiliary variable with the primary outcome and the compliance status. We also show how the setup we consider offers new identifying assumptions of intention-to-treat effects. The role of the auxiliary information is shown in two examples of a real social job training experiment and a simulated medical randomized encouragement study. We also discuss issues of inference in finite samples and show how to conduct Bayesian analysis in our partial and point identified settings. Supplementary materials for this article are available online.

A Bayesian Semiparametric Approach to Intermediate Variables in Causal Inference

In causal inference studies, treatment comparisons often need to be adjusted for confounded post-treatment variables. Principal stratification (PS) is a framework to deal with such variables within the potential outcome approach to causal inference. Continuous intermediate variables introduce inferential challenges to PS analysis. Existing methods either dichotomize the intermediate variable, or assume a fully parametric model for the joint distribution of the potential intermediate variables. However, the former is subject to information loss and arbitrary choice of the cutoff point and the latter is often inadequate to represent complex distributional and clustering features. We propose a Bayesian semiparametric approach that consists of a flexible parametric model for the potential outcomes and a Bayesian nonparametric model for the potential intermediate outcomes using a Dirichlet process mixture (DPM) model. The DPM approach provides flexibility in modeling the possibly complex joint distribution of the potential intermediate outcomes and offers better interpretability of results through its clustering feature. Gibbs sampling based posterior inference is developed. We illustrate the method by two applications: one concerning partial compliance in a randomized clinical trial, and one concerning the causal mechanism between physical activity, body mass index, and cardiovascular disease in the observational Swedish National March Cohort study.

Nonparametric Bounds on the Causal Effect of University Studies on Job Opportunities Using Principal Stratification

The authors propose a methodology based on the principal strata approach to causal inference for assessing the relative effectiveness of two degree programs with respect to the employment status of their graduates. An innovative use of nonparametric bounds in the principal strata framework is shown, examining the role of some assumptions in reducing uncertainty about the causal effects and proposing a strategy to use the covariates in the construction of the bounds. In the application, the nonparametric bounds turn out to be quite informative on the average causal effect for the latent group of students who are potentially able to graduate from both degree programs. There is some evidence that the effect is positive for economics with respect to political science, at least for some values of the covariates.

Training Duration and Post-training Outcomes: A Duration-Limited Competing Risks Model

In some practical applications of transition models there is a natural limit on the duration of some state. An important example is the Youth Training Scheme (YTS), which is normally limited to two years. The authors modify the usual competing risks model for this case and derive a diagnostic test for heterogeneity. They apply the techniques to a model of the duration of YTS and find results that suggest an important qualitative effect of completed two-year YTS spells compared to spells completed under two years. Full-term YTS spells are found to be associated with significantly higher employment probabilities. Copyright 1996 by Royal Economic Society.

Assumptions when Analyzing Randomized Experiments with Noncompliance and Missing Outcomes

Randomized trials often suffer from a number of complications, notably noncompliance with assigned treatment and missing outcomes. In this paper, basic complications and associated assumptions are catalogued and discussed. Both noncompliance and missing outcomes are posttreatment variables, and therefore “adjusting” for noncompliance or missing outcomes requires careful analysis. The approach we follow differs from that based on standard “structural” econometrics models where non intuitive distributional and statistical assumptions are usually introduced to identify parameters of interest from observed data. The assumptions we discuss are instead scientific and foster understanding of how identification from observed data is achieved, regardless of the approach used for inference. We illustrate such assumptions in the case of compliance behavior, assumed to be dichotomous (all or none), and response behavior, also assumed to be dichotomous (respondent or nonrespondent). Starting from simple examples, we review and propose different sets of “exclusion restrictions”, which limit the number of potential outcomes, and discuss which assumptions seem to be more appropriate for different settings. An important lesson is that there are no universally appropriate assumptions. Different scientific settings support different assumptions as appropriate.

Estimating binary multilevel models through indirect inference

Abstract Non-normal multilevel models usually lead to intractable likelihood functions, as they involve integrals without closed-form solution. A flexible approach for their estimation consists in replacing the initial model with an approximated one which is easier to handle, as the quasi-likelihood method with a linearising transformation proposed by Goldstein (1991) or the approximated likelihood developed by Longford (1988) . Simulation studies of Rodriguez and Goldman (1995) have shown the occurrence of large biases when such approximated methods are applied; recent works propose second-order corrections ( Goldstein and Rasbash, 1996 ) and iterative bootstrap bias correction ( Goldstein, 1996 ) to improve the estimates. In order to correct for the asymptotic bias of the quasi-likelihood estimator, we propose the use of indirect inference Gourieroux et al., 1993 ; Gallant and Tauchen, 1994 ), which uses simulations performed under the initial model to correct the estimates derived from the auxiliary (approximated) model. We show asymptotic equivalence between indirect inference and iterative bootstrap Kuk, 1995 ) estimators in the just identified case. Some Monte Carlo experiments show the performance of the indirect inference estimator, comparing its correction with the bootstrap one.