Julio Olea

Exploratory factor analysis in validation studies: uses and recommendations

BACKGROUND The Exploratory Factor Analysis (EFA) procedure is one of the most commonly used in social and behavioral sciences. However, it is also one of the most criticized due to the poor management researchers usually display. The main goal is to examine the relationship between practices usually considered more appropriate and actual decisions made by researchers. METHOD The use of exploratory factor analysis is examined in 117 papers published between 2011 and 2012 in 3 Spanish psychological journals with the highest impact within the previous five years. RESULTS RESULTS show significant rates of questionable decisions in conducting EFA, based on unjustified or mistaken decisions regarding the method of extraction, retention, and rotation of factors. CONCLUSIONS Overall, the current review provides support for some improvement guidelines regarding how to apply and report an EFA.

Incorporating randomness in the Fisher information for improving item-exposure control in CATs.

The most commonly employed item selection rule in a computerized adaptive test (CAT) is that of selecting the item with the maximum Fisher information for the estimated trait level. This means a highly unbalanced distribution of item-exposure rates, a high overlap rate among examinees and, for item bank management, strong pressure to replace items with a high discrimination parameter in the bank. An alternative for mitigating these problems involves, at the beginning of the test, basing item selection mainly on randomness. As the test progresses, the weight of information in the selection increases. In the present work we study, for two selection rules, the progressive methods (Revuelta & Ponsoda, 1998) and the proportional method (Segall, 2004a), different functions that define the weight of the random component according to the position in the test of the item to be administered. The functions were tested in simulated item banks and in an operative bank. We found that both the progressive and the proportional methods tolerate a high weight of the random component with minimal or zero loss of accuracy, while bank security and maintenance are improved.

An Investigation of Self-Adapted Testing in a Spanish High School Population.

This study, using Spanish high school students, compared four types of a computer-based English vocabulary test: (a) a self-adapted test (SAT), (b) a computerized adaptive test (CAT), (c) a conventional test of randomly selected items, and (d) a test that combined SAT and CAT (SCAT). No statistically significant differences were found among the test types for either estimated ability or posttest anxiety. Statistically significant differences were found for the number of correct responses (co = .091) and testing time (Co2 = .023). The results suggest caution in generalizations made by researchers and practitioners regarding the effects of SAT on examinees.

Methods for Restricting Maximum Exposure Rate in Computerized Adaptative Testing

The Sympson-Hetter (1985) method provides a means of controlling maximum exposure rate of items in Computerized Adaptive Testing. Through a series of simulations, control parameters are set that mark the probability of administration of an item on being selected. This method presents two main problems: it requires a long computation time for calculating the parameters and the maximum exposure rate is slightly above the fixed limit. Van der Linden (2003) presented two alternatives which appear to solve both of the problems. The impact of these methods in the measurement accuracy has not been tested yet. We show how these methods over-restrict the exposure of some highly discriminating items and, thus, the accuracy is decreased. It also shown that, when the desired maximum exposure rate is near the minimum possible value, these methods offer an empirical maximum exposure rate clearly above the goal. A new method, based on the initial estimation of the probability of administration and the probability of selection of the items with the restricted method (Revuelta & Ponsoda, 1998), is presented in this paper. It can be used with the Sympson-Hetter method and with the two van der Lindens methods. This option, when used with Sympson-Hetter, speeds the convergence of the control parameters without decreasing the accuracy. One of the objectives of administering tests is accurate as- sessment of the examinees trait level. In order to achieve adequate measurement, it is necessary that the probability of responding correctly to the items is marked solely by their psychometric characteristics and by the examinees trait lev- el. In the case of an examinee receiving the test with prior knowledge of the items to which he will have to respond, this would no longer hold and there would be an over-estimation of his trait level that would reduce the tests validity. This risk is especially present when the test is applied by means of a computerized adaptive test (CAT). In this kind of test, the items in the item bank remain operative for a reasonably long period of time. This means that a future examinee can obtain knowledge of part of the item bank if he receives information from an examinee already tested who remembers the items he faced. The risk will be higher the higher the overlap rate between examinees, this being understood as the proportion of items shared, on average,

Psychometric Properties of a New Family Life Satisfaction Scale

Summary: A scale of bipolar adjectives, the Family Satisfaction by Adjectives Scale (F.S.A.S.), is presented, consisting of 27 items designed to measure family satisfaction, mainly related to the affective connotation derived from family interaction. After applying the scale to a sample of 274 subjects and 16 patients in family therapy, we obtained (a) acceptable indicators of internal consistency (α = .976) and temporal stability (rxx = 0.758), (b) clear evidence of unidimensionality, (c) significant linear correlations with other measures of family satisfaction (Family Satisfaction, Olson & Wilson, 1982; Family Satisfaction Scale, Carver, & Jones, 1992), and (d) significant differences between a normal sample and a clinical one.

A Method for the Comparison of Item Selection Rules in Computerized Adaptive Testing

In a typical study comparing the relative efficiency of two item selection rules in computerized adaptive testing, the common result is that they simultaneously differ in accuracy and security, making it difficult to reach a conclusion on which is the more appropriate rule. This study proposes a strategy to conduct a global comparison of two or more selection rules. A plot showing the performance of each selection rule for several maximum exposure rates is obtained and the whole plot is compared with other rule plots. The strategy was applied in a simulation study with fixed-length CATs for the comparison of six item selection rules: the point Fisher information, Fisher information weighted by likelihood, Kullback-Leibler weighted by likelihood, maximum information stratification with blocking, progressive and proportional methods. Our results show that there is no optimal rule for any overlap value or root mean square error (RMSE). The fact that a rule, for a given level of overlap, has lower RMSE than another does not imply that this pattern holds for another overlap rate. A fair comparison of the rules requires extensive manipulation of the maximum exposure rates. The best methods were the Kullback-Leibler weighted by likelihood, the proportional method, and the maximum information stratification method with blocking.

Adtest: A Computer-Adaptive Test Based on the Maximum Information Principle

This article describes an easy-to-use program for computer-adaptive testing. Two files have to be provided to use the program: the bank of items and the file containing their parameters. The program has been checked by simulation. Data on the accuracy of ability estimation are offered. Details on how to run the program and the format of both required files are also provided.

Validity and reliability of situational judgement test scores : a new approach based on cognitive diagnosis models

Conventional methods for assessing the validity and reliability of situational judgment test (SJT) scores have proven to be inadequate. For example, factor analysis techniques typically lead to nonsensical solutions, and assumptions underlying Cronbach’s alpha coefficient are violated due to the multidimensional nature of SJTs. In the current article, we describe how cognitive diagnosis models (CDMs) provide a new approach that not only overcomes these limitations but that also offers extra advantages for scoring and better understanding SJTs. The analysis of the Q-matrix specification, model fit, and model parameter estimates provide a greater wealth of information than traditional procedures do. Our proposal is illustrated using data taken from a 23-item SJT that presents situations about student-related issues. Results show that CDMs are useful tools for scoring tests, like SJTs, in which multiple knowledge, skills, abilities, and other characteristics are required to correctly answer the items. SJT classifications were reliable and significantly related to theoretically relevant variables. We conclude that CDM might help toward the exploration of the nature of the constructs underlying SJT, one of the principal challenges in SJT research.

The Choice of Item Difficulty in Self-Adapted Testing

Summary: The difficulty level choices made by examinees during a self-adapted test were studied. A positive correlation between estimate ability and difficulty choice was found. The mean difficulty level selected by the examinees increased nonlinearly as the testing session progressed. Regression analyses showed that the best predictors of difficulty choice were examinee ability, difficulty of the previous item, and score on the previous item. Four strategies for selecting difficulty levels were examined, and examinees were classified into subgroups based on the best-fitting strategy. The subgroups differed with regard to ability, pretest anxiety, number of items passed, and mean difficulty level chosen. The self-adapted test was found to reduce state anxiety for only some of the strategy groups.

Item Selection Rules in Computerized Adaptive Testing: Accuracy and Security

The item selection rule (ISR) most commonly used in computerized adaptive testing (CAT) is to select the item with maximum Fisher information for the current trait estimation (PFI). Several alternative ISRs have been proposed. Among them, Fisher information considered in an interval (FI*I), Fisher information weighted with the likelihood function (FI*L), Kullback-Leibler information considered in an interval (KL*I) and Kullback-Leibler weighted with the likelihood function (KL*L) have shown a greater precision of trait estimation at the early stages of CAT. A new ISR is proposed, Fisher information by interval with geometric mean (FI*IG), which tries to rectify some detected problems in FI*I. We evaluate accuracy and item bank security for these six ISRs. FI*IG is the only ISR which simultaneously outperforms PFI in both variables. For the other ISRs, there seems to be a trade-off between accuracy and security, PFI being the one with worse accuracy and greater security, and the ISRs using the likelihood function the reverse.