Ulrich Schroeders

Equivalence of Reading and Listening Comprehension across Test Media.

Whether an ability test delivered on either paper or computer provides the same information is an important question in applied psychometrics. Besides the validity, it is also the fairness of a measure that is at stake if the test medium affects performance. This study provides a comprehensive review of existing equivalence research in the field of reading and listening comprehension in English as a foreign language and specifies factors that are likely to have an impact on equivalence. Taking into account these factors, comprehension measures were developed and tested with N = 442 high school students. Using multigroup confirmatory factor analysis, it is shown that reading and listening comprehension both were measurement invariant across test media. Nevertheless, it is argued that equivalence of data gathered on paper and computer depends on the specific measure or construct, the participants or the recruitment mechanisms, and the software and hardware realizations. Therefore, equivalence research is required for specific instantiations unless generalizable knowledge about factors affecting equivalence is available. Multigroup confirmatory factor analysis is an appropriate and effective tool for the assessment of the comparability of test scores across test media.

Computer usage questionnaire: Structure, correlates, and gender differences

Computer usage, computer experience, computer familiarity, and computer anxiety are often discussed as constructs potentially compromising computer-based ability assessment. After presenting and discussing these constructs and associated measures we introduce a brief new questionnaire assessing computer usage. The self-report measure consists of 18 questions asking for the frequency of different computer activities and software usage. Participants were N=976 high school students who completed the questionnaire and several covariates. Based on theoretical considerations and data driven adjustments a model with a general computer usage factor and three nested content factors (Office, Internet, and Games) is established for a subsample (n=379) and cross-validated with the remaining sample (n=597). Weak measurement invariance across gender groups could be established using multi-group confirmatory factor analysis. Differential relations between the questionnaire factors and self-report scales of computer usage, self-concept, and evaluation are reported separately for females and males. It is concluded that computer usage is distinct from other behavior oriented measurement approaches and that it shows a diverging, gender-specific pattern of relations with fluid and crystallized intelligence.

The development of emotional and behavioral self-regulation and their effects on academic achievement in childhood:

Self-regulation is an essential ability of children to cope with various developmental challenges. This study examines the developmental interplay between emotional and behavioral self-regulation during childhood and the relationship with academic achievement using data from the longitudinal Millennium Cohort Study (UK). Using cross-lagged panel analyses, we found that emotional and behavioral self-regulation were separate and stable constructs. In addition, both emotional and behavioral self-regulation had positive cross-lagged effects from ages 3 to 7. At an early developmental stage (ages 3 to 5), emotional regulation affected behavioral regulation more strongly than later developmental stages. However, the difference between the reciprocal effects was small from ages 5 to 7. Moreover, behavioral regulation during the third year of primary education (age 7) had a substantial and positive effect on teachers’ evaluations of educational achievement during the last year of primary school (age 11). In contrast, emotional self-regulation only had a small indirect and positive effect via behavioral self-regulation. The current study suggests the structure of self-regulation was multidimensional and its facets are mutually dependent in the child’s development. In order to gain a complete picture of the development of self-regulation and its effect on educational achievement, the facets emotional and behavioral regulation should both be studied in concert.

Cognitive Abilities Explain Wording Effects in the Rosenberg Self-Esteem Scale:

There is consensus that the 10 items of the Rosenberg Self-Esteem Scale (RSES) reflect wording effects resulting from positively and negatively keyed items. The present study examined the effects of cognitive abilities on the factor structure of the RSES with a novel, nonparametric latent variable technique called local structural equation models. In a nationally representative German large-scale assessment including 12,437 students competing measurement models for the RSES were compared: a bifactor model with a common factor and a specific factor for all negatively worded items had an optimal fit. Local structural equation models showed that the unidimensionality of the scale increased with higher levels of reading competence and reasoning, while the proportion of variance attributed to the negatively keyed items declined. Wording effects on the factor structure of the RSES seem to represent a response style artifact associated with cognitive abilities.

Meta-Heuristics in Short Scale Construction: Ant Colony Optimization and Genetic Algorithm

The advent of large-scale assessment, but also the more frequent use of longitudinal and multivariate approaches to measurement in psychological, educational, and sociological research, caused an increased demand for psychometrically sound short scales. Shortening scales economizes on valuable administration time, but might result in inadequate measures because reducing an item set could: a) change the internal structure of the measure, b) result in poorer reliability and measurement precision, c) deliver measures that cannot effectively discriminate between persons on the intended ability spectrum, and d) reduce test-criterion relations. Different approaches to abbreviate measures fare differently with respect to the above-mentioned problems. Therefore, we compare the quality and efficiency of three item selection strategies to derive short scales from an existing long version: a Stepwise COnfirmatory Factor Analytical approach (SCOFA) that maximizes factor loadings and two metaheuristics, specifically an Ant Colony Optimization (ACO) with a tailored user-defined optimization function and a Genetic Algorithm (GA) with an unspecific cost-reduction function. SCOFA compiled short versions were highly reliable, but had poor validity. In contrast, both metaheuristics outperformed SCOFA and produced efficient and psychometrically sound short versions (unidimensional, reliable, sensitive, and valid). We discuss under which circumstances ACO and GA produce equivalent results and provide recommendations for conditions in which it is advisable to use a metaheuristic with an unspecific out-of-the-box optimization function.

Equivalence of Screen Versus Print Reading Comprehension Depends on Task Complexity and Proficiency

ABSTRACT As reading and reading assessment become increasingly implemented on electronic devices, the question arises whether reading on screen is comparable with reading on paper. To examine potential differences, we studied reading processes on different proficiency and complexity levels. Specifically, we used data from the standardization sample of the German reading comprehension test ELFE II (n = 2,807), which assesses reading at word, sentence, and text level with separate speeded subtests. Children from grades 1 to 6 completed either a test version on paper or via computer under time constraints. In general, children in the screen condition worked faster but at the expense of accuracy. This difference was more pronounced for younger children and at the word level. Based on our results, we suggest that remedial education and interventions for younger children using computer-based approaches should likewise foster speed and accuracy in a balanced way.

The Structure of the Rosenberg Self-Esteem Scale: A Cross-Cultural Meta-Analysis

The Rosenberg Self-Esteem Scale (RSES; Rosenberg, 1965) intends to measure a single dominant factor representing global self-esteem. However, several studies have identified some form of multidimensionality for the RSES. Therefore, we examined the factor structure of the RSES with a fixed-effects meta-analytic structural equation modeling approach including 113 independent samples (N = 140,671). A confirmatory bifactor model with specific factors for positively and negatively worded items and a general self-esteem factor fitted best. However, the general factor captured most of the explained common variance in the RSES, whereas the specific factors accounted for less than 15%. The general factor loadings were invariant across samples from the United States and other highly individualistic countries, but lower for less individualistic countries. Thus, although the RSES essentially represents a unidimensional scale, cross-cultural comparisons might not be justified because the cultural background of the respondents affects the interpretation of the items.

Knowledge Is Power for Medical Assistants: Crystallized and Fluid Intelligence As Predictors of Vocational Knowledge

Medical education research has focused almost entirely on the education of future physicians. In comparison, findings on other health-related occupations, such as medical assistants, are scarce. With the current study, we wanted to examine the knowledge-is-power hypothesis in a real life educational setting and add to the sparse literature on medical assistants. Acquisition of vocational knowledge in vocational education and training (VET) was examined for medical assistant students (n = 448). Differences in domain-specific vocational knowledge were predicted by crystallized and fluid intelligence in the course of VET. A multiple matrix design with 3 year-specific booklets was used for the vocational knowledge tests of the medical assistants. The unique and joint contributions of the predictors were investigated with structural equation modeling. Crystallized intelligence emerged as the strongest predictor of vocational knowledge at every stage of VET, while fluid intelligence only showed weak effects. The present results support the knowledge-is-power hypothesis, even in a broad and more naturalistic setting. This emphasizes the relevance of general knowledge for occupations, such as medical assistants, which are more focused on learning hands-on skills than the acquisition of academic knowledge.

A confirmatory examination of age-associated personality differences: Deriving age-related measurement-invariant solutions using ant colony optimization

OBJECTIVE The goal of this study was to examine age-associated personality differences using a measurement-invariant representation of the higher-order structure of the Five-Factor Model. METHOD We reanalyzed the German NEO-PI-R norm sample (N = 11,724) and applied ant colony optimization in a multigroup confirmatory factor analysis setting in order to select three items per first-order factor that would optimize model fit and measurement invariance across 18 age groups ranging from 16 to 65 years of age. RESULTS Ant colony optimization substantially improved absolute and relative model fit under measurement invariance constraints. However, the results showed that even when selecting items, measurement invariance across a large age span could not be guaranteed. Strong measurement invariance for Extraversion and Agreeableness could not be established. The age-associated mean-level differences of the first-order factors of Neuroticism and Conscientiousness supported the maturity hypothesis. The mean levels of the first-order factors of Openness varied substantially from each other across age. CONCLUSIONS Findings on age differences in personality can be particularly distorted in older age groups. Testing for and ensuring measurement invariance with item selection procedures can help solve this problem. The higher-order structure of personality should be accounted for when personality development is examined.