Mariagiulia Matteucci

Prior Distributions for Item Parameters in IRT Models

The focus of this article is on the choice of suitable prior distributions for item parameters within item response theory (IRT) models. In particular, the use of empirical prior distributions for item parameters is proposed. Firstly, regression trees are implemented in order to build informative empirical prior distributions. Secondly, model estimation is conducted within a fully Bayesian approach through the Gibbs sampler, which makes estimation feasible also with increasingly complex models. The main results show that item parameter recovery is improved with the introduction of empirical prior information about item parameters, also when only a small sample is available.

Uncertainties in the item parameter estimates and robust automated test assembly

Item response theory parameters have to be estimated, and because of the estimation process, they do have uncertainty in them. In most large-scale testing programs, the parameters are stored in item banks, and automated test assembly algorithms are applied to assemble operational test forms. These algorithms treat item parameters as fixed values, and uncertainty is not taken into account. As a consequence, resulting tests might be off target or less informative than expected. In this article, the process of parameter estimation is described to provide insight into the causes of uncertainty in the item parameters. The consequences of uncertainty are studied. Besides, an alternative automated test assembly algorithm is presented that is robust against uncertainties in the data. Several numerical examples demonstrate the performance of the robust test assembly algorithm, and illustrate the consequences of not taking this uncertainty into account. Finally, some recommendations about the use of robust test assembly and some directions for further research are given.

Bayesian computerized adaptive testing

Computerized adaptive testing (CAT) comes with many advantages. Unfortunately, it still is quite expensive to develop and maintain an operational CAT. In this paper, various steps involved in developing an operational CAT are described and literature on these topics is reviewed. Bayesian CAT is introduced as an alternative, and the use of empirical priors is proposed for estimating item and person parameters to reduce the costs of CAT. Methods to elicit empirical priors are presented and a two small examples are presented that illustrate the advantages of Bayesian CAT. Implications of the use of empirical priors are discussed, limitations are mentioned and some suggestions for further research are formulated.

The use of predicted values for item parameters in item response theory models: An application in intelligence tests

In testing, item response theory models are widely used in order to estimate item parameters and individual abilities. However, even unidimensional models require a considerable sample size so that all parameters can be estimated precisely. The introduction of empirical prior information about candidates and items might reduce the number of candidates needed for parameter estimation. Using data for IQ measurement, this work shows how empirical information about items can be used effectively for item calibration and in adaptive testing. First, we propose multivariate regression trees to predict the item parameters based on a set of covariates related to the item-solving process. Afterwards, we compare the item parameter estimation when tree-fitted values are included in the estimation or when they are ignored. Model estimation is fully Bayesian, and is conducted via Markov chain Monte Carlo methods. The results are two-fold: (a) in item calibration, it is shown that the introduction of prior information is effective with short test lengths and small sample sizes and (b) in adaptive testing, it is demonstrated that the use of the tree-fitted values instead of the estimated parameters leads to a moderate increase in the test length, but provides a considerable saving of resources.

An Investigation of Parameter Recovery in MCMC Estimation for the Additive IRT Model

The article aims at evaluating the parameter recovery for the multidimensional additive IRT model (Sheng, 2005; Sheng and Wikle, 2009). By estimating the model parameters via Gibbs sampler, a simulation study is conducted under different testing conditions, e.g., dimensionality, test and subtest lengths, correlation matrices, and different values of discrimination parameters. The results show that, especially when the test length is short and the abilities are highly correlated, the accuracy of the parameter estimates is reduced and more iterations are required to convergence. An application in educational testing is also described to show the effectiveness of the model in use.

On the use of MCMC computerized adaptive testing with empirical prior information to improve efficiency

The paper deals with the introduction of empirical prior information in the estimation of candidate’s ability within computerized adaptive testing (CAT). CAT is generally applied to improve efficiency of test administration. In this paper, it is shown how the inclusion of background variables both in the initialization and the ability estimation is able to improve the accuracy of ability estimates. In particular, a Gibbs sampler scheme is proposed in the phases of interim and final ability estimation. By using both simulated and real data, it is proved that the method produces more accurate ability estimates, especially for short tests and when reproducing boundary abilities. This implies that operational problems of CAT related to weak measurement precision under particular conditions, can be reduced as well. In the empirical examples, the methods were applied to CAT for intelligence testing in the area of personnel selection and to educational measurement. Other promising applications would be in the medical world, where testing efficiency is of paramount importance as well.

Computerized adaptive testing in computer assisted learning

A major goal in computerized learning systems is to optimize learning, while in computerized adaptive tests (CAT) efficient measurement of the proficiency of students is the main focus. There seems to be a common interest to integrate computerized adaptive item selection in learning systems and testing. Item selection is a well founded building block of CAT. However, there are a number of problems that prevent the application of a standard approach, based on item response theory, of computerized adaptive item selection to learning systems. In this work attention will be paid to three unresolved points: item banking, item selection, and choice of IRT model. All problems will be discussed, and an approach to automated item bank generation is presented. Finally some recommendations are given.

Multidimensional IRT Models to Analyze Learning Outcomes of Italian Students at the End of Lower Secondary School

In this paper, different multidimensional IRT models are compared in order to choose the best approach to explain response data on Italian student assessment at the end of lower secondary school. The results show that the additive model with three specific dimensions (reading comprehension, grammar, and mathematics abilities) and an overall ability is able to recover the test structure meaningfully. In this model, the overall ability compensates for the specific ability (or vice versa) in order to determine the probability of a correct response. Given the item characteristics, the overall ability is interpreted as a reasoning and thinking capability. Model estimation is conducted via Gibbs sampler within a Bayesian approach, which allows the use of Bayesian model comparison techniques such as posterior predictive model checking for model comparison and fit.

Exploring Regional Differences in the Reading Competencies of Italian Students

Background: Recently, the study of territorial differences in educational outcomes has assumed a particular importance for the policy strategies related to the socioeconomic conditions of different geographical areas. In Italy, international surveys for student assessments have introduced a regional stratification only recently, and regular national student assessments started only in 2008. Method: In this article, the reading performances of Italian students based on OECD-PISA 2009 are investigated, taking into account regional and macro-area partition. Student outcomes are explored by using a multilevel analysis, where school membership, socioeconomic and cultural background of students, and regional gross domestic product are introduced. Results: The results show that, despite the existence of a unified educational system in Italy, regional and macro-area differences in student reading achievements are consolidated and variability in performances among schools is especially noticeable. Comparisons based on national assessments by INVALSI at the end of compulsory school confirm these findings. Conclusion: Italian policy makers are advised to take into account these results to improve learning opportunities and to reduce educational gaps. In particular, targeted regional policies are needed to improve the mean performance especially in the Southern regions of Calabria, Campania, and Sicilia, and to strengthen the system equity in several regions, such as Emilia-Romagna. To decrease the school differences, possible suggestions are to postpone the choice of the school type (currently, at age 14) and to motivate good teachers to work in schools located in the worst socioeconomic and cultural environments.

Including Empirical Prior Information in Test Administration

In this work, the issue of using prior information in test administration is taken into account. The focus is on the development of procedures to include background variables which are strongly related to the latent ability, adopting a Bayesian approach. Because the desirability of prior information for the ability estimation in item response modelling depends on the goals of the test, only some kinds of educational tests might profit of this approach. The procedures will be evaluated in an empirical context and some recommendations about the use of prior information will be given.