Leonardo Grilli

A multilevel multinomial logit model for the analysis of graduates’ skills

The main goal of the paper is to specify a suitable multivariate multilevel model for polytomous responses with a non-ignorable missing data mechanism in order to determine the factors which influence the way of acquisition of the skills of the graduates and to evaluate the degree programmes on the basis of the adequacy of the skills they give to their graduates. The application is based on data gathered by a telephone survey conducted, about two years after the degree, on the graduates of year 2000 of the University of Florence. A multilevel multinomial logit model for the response of interest is fitted simultaneously with a multilevel logit model for the selection mechanism by means of maximum likelihood with adaptive Gaussian quadrature. In the application the multilevel structure has a crucial role, while selection bias results negligible. The analysis of the empirical Bayes residuals allows to detect some extreme degree programmes to be further inspected.

Alternative Specifications of Multivariate Multilevel Probit Ordinal Response Models

Multivariate multilevel models for ordinal variables are quite complex with respect to both interpretation and estimation. The specification in terms of a multivariate latent distribution and a set of thresholds helps in the interpretation of the variance-covariance parameters. However, most existing estimation algorithms for multilevel models can be used only if the model is reparameterized as a univariate model with an additional dummy bottom level. Moreover, the univariate formulation allows the model to be cast in the framework of Generalized Linear Latent and Mixed Models (Rabe-Hesketh, Pickles, & Skrondal, 2001a), a rather general class that includes, as special cases, structural equations and factor models. This article outlines the multivariate latent distribution specification and the corresponding interpretation issues; it then shows the univariate formulation, along with some alternative parameterizations that are useful in the estimation phase. An application to student ratings data illustrates the interpretation of the parameters and the estimation procedures, with a discussion of some computational issues.

Nonparametric Bounds on the Causal Effect of University Studies on Job Opportunities Using Principal Stratification

The authors propose a methodology based on the principal strata approach to causal inference for assessing the relative effectiveness of two degree programs with respect to the employment status of their graduates. An innovative use of nonparametric bounds in the principal strata framework is shown, examining the role of some assumptions in reducing uncertainty about the causal effects and proposing a strategy to use the covariates in the construction of the bounds. In the application, the nonparametric bounds turn out to be quite informative on the average causal effect for the latent group of students who are potentially able to graduate from both degree programs. There is some evidence that the effect is positive for economics with respect to political science, at least for some values of the covariates.

Group versus single handed primary care: A performance evaluation of the care delivered to chronic patients by Italian GPs

OBJECTIVES In family medicine contrasting evidence exists on the effectiveness of team practice compared with solo practice on chronic disease management. In Italy, several experiences of team practice have been introduced since the late 1990s but few studies detail their impact on the quality of care. The aim of this paper is to evaluate the impact of team practice in family medicine in six Italian regions using chronic disease management process indicators as a measure of outcome. METHODS Cross-sectional studies were performed to assess impact on quality of care for diabetes, congestive heart failure and ischaemic heart disease. The impact of team vs. solo practice was approximated through performance comparison of general practitioners (GPs) adhering to a team with respect to GPs working in a solo practice. Among the 2082 practitioners working in the 6 regions those assisting 300+ patients were selected. Quality of care towards 164,267 patients having at least one of three chronic conditions was estimated for the year 2008 using administrative databases. Quality indicators (% of patients receiving appropriate care) were selected (4 for diabetes, 4 for congestive heart failure, 3 for ischaemic heart disease) and a total score was computed for each patient. For each disease the response variable associated to each physician was the average score of the patients on his/her list. A multilevel model was estimated assessing the impact of team vs. solo practice. RESULTS No impact was found for diabetes and heart failure. For ischaemic heart disease a slightly significant impact was observed (0.040; 95% CI: 0.015, 0.065). CONCLUSIONS No significant difference was found between team practice and solo practice on chronic disease management in six Italian regions.

Multilevel models for the evaluation of educational institutions: a review

The methodology for the evaluation of educational systems is being developed in different fields, such as educational statistics, psychometrics, sociology and econometrics. Each discipline has developed approaches suitable for the analysis of particular aspects of the evaluation process. For example, educational statistics focuses on learning curves using standardized scores, while econometrics mainly deals with private returns (e.g. in terms of wages) or social returns (e.g. in terms of productivity). Anyway, there is a considerable overlap among the fields, for example peer effects are studied both in educational statistics, as a major topic, and econometrics, as a minor topic.

Bayesian estimation with integrated nested Laplace approximation for binary logit mixed models

In multilevel models for binary responses, estimation is computationally challenging due to the need to evaluate intractable integrals. In this paper, we investigate the performance of integrated nested Laplace approximation (INLA), a fast deterministic method for Bayesian inference. In particular, we conduct an extensive simulation study to compare the results obtained with INLA to the results obtained with a traditional stochastic method for Bayesian inference (MCMC Gibbs sampling), and with maximum likelihood through adaptive quadrature. Particular attention is devoted to the case of small number of clusters. The specification of the prior distribution for the cluster variance plays a crucial role and it turns out to be more relevant than the choice of the estimation method. The simulations show that INLA has an excellent performance as it achieves good accuracy (similar to MCMC) with reduced computational times (similar to adaptive quadrature).

Modeling Partial Compliance Through Copulas in a Principal Stratification Framework

Within the principal stratification framework for causal inference, modeling partial compliance is challenging because the continuous nature of the principal strata raises subtle specification issues. In this context, we propose an approach based on the assumption that the joint distribution of the degree of compliance to the treatment and the degree of compliance to the control follows a Plackett copula, so that their association is modeled in a flexible way through a single parameter. Moreover, given the two compliances, the distribution of the outcomes is parameterized in a flexible way through a regression model which may include interaction and quadratic terms and may also be heteroscedastic. In order to estimate the parameters of the resulting model, and then the causal effect of the treatment, we adopt a maximum likelihood approach via the EM algorithm. In applying this approach, the marginal distributions of the two compliances are estimated by their empirical distribution functions, so that no constraints are posed on these distributions. Since the two compliances cannot be jointly observed, there is not direct empirical support for the association parameter. We describe a strategy for studying this parameter by a profile likelihood method and discuss an analysis of the sensitivity of the causal inference results to its value. We apply the proposed approach to data from a study about a drug for lowering cholesterol levels previously analyzed by Efron and Feldman and by Jin and Rubin. Estimated causal effects are in line with those of previous analyses, but the pattern of association between the compliances is qualitatively different, apparently due to the flexibility of the copula and to allowing regression equations in the proposed method to include interactions and heteroscedasticity.

Specification issues in stratified variance component ordinal response models

The paper presents some criteria for the specification of ordinal variance component models when the units are grouped in a limited number of strata. The base model is specified using a latent variable approach, allowing the first level variance, the second level variance, and the thresholds to vary according to the strata. However this model is not identifiable. The paper discusses some alternative assumptions that overcome the identification problem and illustrates a general strategy for model selection. The proposed methodology is applied to the analysis of course programme evaluations based on student ratings, referring to three different schools of the University of Florence. The adopted model takes into account both the ordinal scale of the ratings and the hierarchical nature of the phenomenon. In this framework, the specification of the latent variable distributions is crucial, since a different first level variance among the schools would substantially change the interpretation of model parameters, as confirmed by the limited simulation study presented in the paper.

Statistical modelling of gained university credits to evaluate the role of pre-enrolment assessment tests: An approach based on quantile regression for counts

Considering the case of the School of Economics of the University of Florence, the paper investigates whether the pre-enrolment assessment test is an effective tool to predict student performance. The analysis is tailored to evaluate the additional information yielded by the test beyond the background characteristics of the candidates already available from administrative records, such as the high school type and final grade. The student performance is measured by the number of gained credits after one year, which is a count variable with an irregular distribution and a peak in zero. These features pose a challenge in statistical modelling, which is solved by a two-part model with a logit specification for the zeros, while positive values are analyzed by quantile regression for counts. To disentangle direct and indirect effects of background variables, the result of the pre-enrolment assessment test is treated as an intermediate variable in a regression chain graph. The results show that the pre-enrolment test adds some information to predict student performance, which can be exploited for tutoring.

Selection bias in linear mixed models

SummaryThe paper investigates the consequences of sample selection in multilevel or mixed models, focusing on the random intercept two-level linear model under a selection mechanism acting at both hierarchical levels. The behavior of sample selection and the resulting biases on the regression coefficients and on the variance components are studied both theoretically and through a simulation study. Most theoretical results exploit the properties of Normal and Skew-Normal distributions. The analysis allows to outline a taxonomy of sample selection in the multilevel framework that can support the qualitative assessment of the problem in specific applications and the development of suitable techniques for diagnosis and correction.