Joanna S. Gorin

Evaluating Collaborative Learning and Community

The goal of this study was to validate measures and assess the effects of collaborative group-learning methods in real classrooms on 3 specific dependent variables: feelings of campus connectedness, academic classroom community, and effective group processing (2 factors). Confirmatory factor analyses were conducted to evaluate a 4-factor model. Using hierarchical linear modeling techniques, results indicated that campus connectedness and collaborative learning (compared with no collaborative learning) predicted positive academic classroom community. For classes using more formal cooperative group work, campus connectedness and group processingndash;evaluation predicted positive academic classroom community. Suggestions for further applications of the measures are discussed.

Item Difficulty Modeling of Paragraph Comprehension Items.

Recent assessment research joining cognitive psychology and psychometric theory has introduced a new technology, item generation. In algorithmic item generation, items are systematically created based on specific combinations of features that underlie the processing required to correctly solve a problem. Reading comprehension items have been more difficult to model than other item types due to the complexities of quantifying text. However, recent developments in artificial intelligence for text analysis permit quantitative indices to represent cognitive sources of difficulty. The current study attempts to identify generative components for the Graduate Record Examination paragraph comprehension items through the cognitive decomposition of item difficulty. Text comprehension and decision processes accounted for a significant amount of the variance in item difficulties. The decision model variables contributed significantly to variance in item difficulties, whereas the text representation variables did not. Implications for score interpretation and future possibilities for item generation are discussed. Index terms: difficulty modeling, construct validity, comprehension tests, item generation

Reconsidering Issues in Validity Theory

Lissitz and Samuelsen (2007) propose a new framework for validity theory and terminology, emphasizing a shift in theory and practice toward issues of test content rather than constructs. The author of this article argues that several of Lissitz and Samuelsen’s critiques of validity theory focus on previously considered, but subsequently discarded, validity conceptualizations. In addition, she suggests that Lissitz and Samuelsen’s conceptualization returns to methods shown historically to be problematic for score use and interpretation. In doing so, she highlights developments in validity theory and practice centering on cognitively based examinations of test scores that have contributed to increased understanding of score meaning and stronger validity arguments.

Receptive Vocabulary and Cross-Language Transfer of Phonemic Awareness in Kindergarten Children

The authors investigated the influence of language proficiency on the cross-language transfer (CLT) of phonemic awareness in Spanish-speaking kindergarten students and assessed Spanish and English receptive vocabulary and phonemic awareness abilities. Correlation results indicated positive correlations between phonemic awareness across languages; CLT occurred. To investigate the role of proficiency in native language (L1) in CLT, the authors disaggregated the sample into two groups by L1 receptive vocabulary. No evidence for CLT of phonemic awareness emerged among children with below-average L1 skills. Regression results indicated that L1 receptive vocabulary predicted phonemic awareness performance of childrens language of instruction. The authors suggest that prereading skills may transfer from L1 to L2 following a different pattern in children lacking L1 proficiency. Further investigation of CLT among children with below-average L1 skills is needed.

Student feelings of connection to the campus and openness to diversity and challenge at a large research university: Evidence of progress?

Previous research conducted by Pascarella and his colleagues (1996) has shown that undergraduate students tend to change toward greater openness and tolerance to diversity from their freshman to their sophomore year. Although the study by Pascarella includes many different types of universities in the United States, the average size of the entering freshman class in their research was reported to be approximately 4,000 students. While these findings are extremely valued in a general sense, Pascarella believed that they might not be found at very large universities. To our surprise, our findings indicated that large universities may have higher levels of openness to diversity and campus connectedness than what was originally explained by Pascarella.

English-Language Learners: Implications of Limited Vocabulary for Cross-Language Transfer of Phonemic Awareness With Kindergartners

Research examined the influence of native vocabulary development on cross-language transfer of phonemic awareness. Participants were Spanish-speaking kindergartners learning English in immersion classrooms. Results indicated that limited Spanish vocabulary development negatively influenced cross-language transfer of phonemic awareness to English. The results have clear and profound implications for Spanish-speaking children. Without foundational Spanish vocabulary skills needed to facilitate cross-language transfer of phonemic awareness to English, literacy acquisition difficulties will likely arise. Esta investigación examina la influencia en el desarrollo del vocabulario nativo en la transferencia de lenguaje cruzado con atención fonética. Los participantes fueron niños de preprimaria de habla hispana que estaban aprendiendo inglés en salones de inmersión total. Los resultados indicaron que el desarrollo limitado de vocabulario en español influenció negativamente la transferencia de lenguaje cruzado en atención fonética de inglés. Los resultados tienen implicaciones claras y profundas pare niños de habla hispana. Sin las habilidades fundamentales de vocabulario en español necesarias para facilitar la transferencia del lenguaje cruzado de atención fonética en inglés es posible que se formen dificultades de adquisición literaria.

Computerized Adaptive Testing with the Partial Credit Model: Estimation Procedures, Population Distributions, and Item Pool Characteristics

The primary purpose of this research is to examine the impact of estimation methods, actual latent trait distributions, and item pool characteristics on the performance of a simulated computerized adaptive testing (CAT) system. In this study, three estimation procedures are compared for accuracy of estimation: maximum likelihood estimation (MLE), expected a priori (EAP), and Warms weighted likelihood estimation (WLE). Some research has shown that MLE and EAP perform equally well under certain conditions in polytomous CAT systems, such that they match the actual latent trait distribution. However, little research has compared these methods when prior estimates of. distributions are extremely poor. In general, it appears that MLE, EAP, and WLE procedures perform equally well when using an optimal item pool. However, the use of EAP procedures may be advantageous under nonoptimal testing conditions when the item pool is not appropriately matched to the examinees.

Defining and Comparing the Reading Comprehension Construct: A Cognitive-Psychometric Modeling Approach.

As a construct definition, the current study develops a cognitive model describing the knowledge, skills, and abilities measured by critical reading test items on a high-stakes assessment used for selection decisions in the United States. Additionally, in order to establish generalizability of the construct meaning to other similarly structured tests designed for international populations and distinct uses, the skills invoked during a reading comprehension test for English learners from a previous study are contrasted to those in the present study. The results obtained using rule-space methodology suggest that the most difficult skills on reading comprehension items pertain to complex cognitive processes (e.g., understanding implicit ideas), while skills tapping into basic cognitive processes (e.g., word meaning) are mastered with ease by both populations. However, variations across tests in the impact of various cognitive skills on test scores suggest that the differences in construct meaning be considered when interpreting and comparing test scores. Cognitive-psychometric modeling approaches such as those applied in this study prove to be useful in substantively examining score interpretation and construct generalizability.

The Validity of Receptive and Expressive Vocabulary Measures with Spanish-Speaking Kindergarteners Learning English

Research examined construct meaning and validity for two new measures of receptive and expressive vocabulary for Spanish-speaking kindergarteners learning English. Substantive validity and generalizability of score meaning was assessed by examining correlations between an established measure (Peabody Picture Vocabulary Test–PPVT) and new measures (Stanford English Language Proficiency Test–SELP; Dynamic Indicators of Basic Early Literacy Skills, Word Use Fluency, DIBELS–WUF). Positive correlations between the SELP and PPVT supported validity of the SELP as a measure of receptive and expressive vocabulary for these children. DIBELS–WUF results were less clear with lower correlations between the PPVT raising questions about appropriateness.

Cross-Cultural Validity of the TIMSS-1999 Mathematics Test: Verification of a Cognitive Model

As with any test administered across linguistically and culturally diverse groups, evidence suggesting the equivalence of score meaning across countries is needed for valid comparisons. The current study examines the cross-cultural equivalence of score interpretations from the Trends in International Mathematics and Science Study (TIMSS)-1999 from a cognitive-psychometric perspective. A cognitive model describing the knowledge, strategies, and processing skills measured by the TIMSS-R mathematics test was previously validated in several countries. In order to establish the cross-cultural equivalence of TIMSS scores for the Taiwanese student population, the fit of the cognitive model to the Taiwanese item responses was examined. High student-mastery classification rates and good prediction of scores based on attribute mastery probabilities supported the fit of the cognitive model in the current study. Further, we suggest that cognitive-psychometric modeling approaches like those applied in the current study could be useful for examining more substantive issues of score validity and equivalence in test translations and adaptations.