Javier Revuelta

An Investigation of Self-Adapted Testing in a Spanish High School Population.

This study, using Spanish high school students, compared four types of a computer-based English vocabulary test: (a) a self-adapted test (SAT), (b) a computerized adaptive test (CAT), (c) a conventional test of randomly selected items, and (d) a test that combined SAT and CAT (SCAT). No statistically significant differences were found among the test types for either estimated ability or posttest anxiety. Statistically significant differences were found for the number of correct responses (co = .091) and testing time (Co2 = .023). The results suggest caution in generalizations made by researchers and practitioners regarding the effects of SAT on examinees.

Optimization of sample size in controlled experiments: The CLAST rule

Sequential rules are explored in the context of null hypothesis significance testing. Several studies have demonstrated that the fixed-sample stopping rule, in which the sample size used by researchers is determined in advance, is less practical and less efficient than sequential stopping rules. It is proposed that a sequential stopping rule called CLAST (composite limited adaptive sequential test) is a superior variant of COAST (composite open adaptive sequential test), a sequential rule proposed by Frick (1998). Simulation studies are conducted to test the efficiency of the proposed rule in terms of sample size and power. Two statistical tests are used: the one-tailed t test of mean differences with two matched samples, and the chi-square independence test for twofold contingency tables. The results show that the CLAST rule is more efficient than the COAST rule and reflects more realistically the practice of experimental psychology researchers.

Adtest: A Computer-Adaptive Test Based on the Maximum Information Principle

This article describes an easy-to-use program for computer-adaptive testing. Two files have to be provided to use the program: the bank of items and the file containing their parameters. The program has been checked by simulation. Data on the accuracy of ability estimation are offered. Details on how to run the program and the format of both required files are also provided.

Extending the CLAST sequential rule to one-way ANOVA under group sampling.

Several studies have demonstrated that the fixed-sample stopping rule (FSR), in which the sample size is determined in advance, is less practical and efficient than are sequential-stopping rules. The composite limited adaptive sequential test (CLAST) is one such sequential-stopping rule. Previous research has shown that CLAST is more efficient in terms of sample size and power than are the FSR and other sequential rules and that it reflects more realistically the practice of experimental psychology researchers. The CLAST rule has been applied only to thet test of mean differences with two matched samples and to the chi-square independence test for twofold contingency tables. The present work extends previous research on the efficiency of CLAST to multiple group statistical tests. Simulation studies were conducted to test the efficiency of the CLAST rule for the one-way ANOVA for fixed effects models. The ANOVA general test and two linear contrasts of multiple comparisons among treatment means are considered. The article also introduces four rules for allocatingN observations toJ groups under the general null hypothesis and three allocation rules for the linear contrasts. Results show that the CLAST rule is generally more efficient than the FSR in terms of sample size and power for one-way ANOVA tests. However, the allocation rules vary in their optimality and have a differential impact on sample size and power. Thus, selecting an allocation rule depends on the cost of sampling and the intended precision.

A Procedure for Estimating Intrasubject Behavior Consistency.

Trait psychology implicitly assumes consistency of the personal traits. Mischel, however, argued against the idea of a general consistency of human beings. The present article aims to design a statistical procedure based on an adaptation of the π* statistic to measure the degree of intraindividual consistency independently of the measure used. Three studies were carried out for testing the suitability of the π* statistic and the proportion of subjects who act consistently. Results have shown the appropriateness of the statistic proposed and that the percentage of consistent individuals depends on whether test items can be assumed as equivalents and the number of response alternatives they contained. The results suggest that the percentage of consistent subjects is far from 100%, and this percentage decreases when items are equivalent. Moreover, the greater the number of response options, the lesser the percentage of consistent individuals.

Analysis of Distractor Difficulty in Multiple-Choice Items.

Two psychometric models are presented for evaluating the difficulty of the distractors in multiple-choice items. They are based on the criterion of rising distractor selection ratios, which facilitates interpretation of the subject and item parameters. Statistical inferential tools are developed in a Bayesian framework: modal a posteriori estimation by application of an EM algorithm and model evaluation by monitoring posterior predictive replications of the data matrix. An educational example with real data is included to exemplify the application of the models and compare them with the nominal categories model.

Factorial Invariance in a Repeated Measures Design: An Application to the Study of Person-Organization Fit

An important methodological concern of any research based on a person-environment (P-E) fit approach is the operationalization of the fit, which imposes some measurement requirements that are rarely empirically tested with statistical methods. Among them, the assessment of the P and E components along commensurate dimensions is possibly the most cited one. This paper proposes to test the equivalence across the P and E measures by analyzing the measurement invariance of a multi-group confirmatory factor analysis model. From a methodological point of view, the distinct aspect of this approach within the context of P-E fit research is that measurement invariance is assessed in a repeated measures design. An example illustrating the procedure in a person-organization (P-O) fit dataset is provided. Measurement invariance was tested at five different hierarchical levels: (1) configural, (2) first-order factor loadings, (3) second-order factor loadings, (4) residual variances of observed variables, and (5) disturbances of first-order factors. The results supported the measurement invariance across the P and O measures at the third level. The implications of these findings for P-E fit studies are discussed.

Assisted Self-Adapted Testing: A Comparative Study

Abstract: A new type of self-adapted test (S-AT), called Assisted Self-Adapted Test (AS-AT), is presented. It differs from an ordinary S-AT in that prior to selecting the difficulty category, the computer advises examinees on their best difficulty category choice, based on their previous performance. Three tests (computerized adaptive test, AS-AT, and S-AT) were compared regarding both their psychometric (precision and efficiency) and psychological (anxiety) characteristics. Tests were applied in an actual assessment situation, in which test scores determined 20% of term grades. A sample of 173 high school students participated. Neither differences in posttest anxiety nor ability were obtained. Concerning precision, AS-AT was as precise as CAT, and both revealed more precision than S-AT. It was concluded that AS-AT acted as a CAT concerning precision. Some hints, but not conclusive support, of the psychological similarity between AS-AT and S-AT was also found.

Propiedades psicométricas de un test adaptativo informatizado de vocabulario inglés

ResumenSe aplico el modelo logistico de tres parametros de la TRI a un banco de items de vocabulario ingles. Se compararon los niveles de habilidad estimados a partir del banco completo con los estimados a partir de una TAI basado en el principio de maxima informacion y con el rendimiento en el Oxford Placement Test (Allan, 1992). La principales conclusiones fueron: a) El banco de items resulta especialmente informativo para niveles medios de habilidad, b) Las distribuciones de los tres parametros resultan adecuadas, c) Como resultaba previsible, existe una relacion lineal elevada (r = 0.9) entre las estimaciones realizadas a partir de las respuestas al banco completo y al TAI, d) Ambas estimaciones se predicen significativamente a partir de las dos puntuaciones que proporciona el Oxford Placement Test.

ADTEST: A Maximum Information Computerized Adaptive Test

ADTEST implements the computerized adaptive testing (CAT) algorithm described by Wainer (1990, chap. 5) with some minor changes. A normal [N(0,1)] random value from -1 to + 1 is used as the initial trait level. Then the information provided for each item is calculated based on the three-parameter logistic model (see Hambleton & Swaminathan, 1985). The item that provides the highest amount of information is selected. The examinee responds and then a new trait estimate is estimated by maximum likelihood, using the Newton-Raphson method. The information of the unused items is again obtained for the new trait level, the item with the highest information is selected, and so on. The procedure terminates when a target measurement precision has been attained or a preselected number of items has been reached. When the test is complete, data on each selected item and the corresponding trait levels are provided. ADTEST was evaluated using a 225-item vocabulary item bank. The three parameters were obtained for each item. The preassigned termination criterion for the standard error was fixed at .30, and the preselected number of items was 34. A total of 3,750 simulees participated in the evaluation of the program (250 for each of the 15 trait levels between -3.5 and 3.5 at .5 intervals). From each sample of the 250 estimates provided by ADTEST, the mean and the difference between the mean and the corresponding true trait level were computed. The maximum mean difference was .08, and the mean standard error of trait level estimation was .36. The mean number of items administered