Kikumi Tatsuoka

Patterns of Diagnosed Mathematical Content and Process Skills in TIMSS-R Across a Sample of 20 Countries

This study used a diagnostic testing approach to compare the mathematics achievement of eighth-grade students across a sample of 20 countries, analyzing data from the Third International Math and Science Study–Revised (TIMSS-R, 1999). Using the rule-space method, student mastery was measured on 23 specific content knowledge and processing subskills (“attributes”) underlying students’ item scores, using 23 attributes previously defined and validated. Mean mastery levels for each attribute were compared for the 20 selected countries. Clear differences among the countries were found in patterns of subskill achievement. U.S. students were strong in some content and quantitative reading skills, but weak in others, notably geometry. Interestingly, success in geometry was found to be highly associated with logical reasoning and other important mathematical thinking skills across the sampled countries.

Defining and Comparing the Reading Comprehension Construct: A Cognitive-Psychometric Modeling Approach.

As a construct definition, the current study develops a cognitive model describing the knowledge, skills, and abilities measured by critical reading test items on a high-stakes assessment used for selection decisions in the United States. Additionally, in order to establish generalizability of the construct meaning to other similarly structured tests designed for international populations and distinct uses, the skills invoked during a reading comprehension test for English learners from a previous study are contrasted to those in the present study. The results obtained using rule-space methodology suggest that the most difficult skills on reading comprehension items pertain to complex cognitive processes (e.g., understanding implicit ideas), while skills tapping into basic cognitive processes (e.g., word meaning) are mastered with ease by both populations. However, variations across tests in the impact of various cognitive skills on test scores suggest that the differences in construct meaning be considered when interpreting and comparing test scores. Cognitive-psychometric modeling approaches such as those applied in this study prove to be useful in substantively examining score interpretation and construct generalizability.

Cross-Cultural Validity of the TIMSS-1999 Mathematics Test: Verification of a Cognitive Model

As with any test administered across linguistically and culturally diverse groups, evidence suggesting the equivalence of score meaning across countries is needed for valid comparisons. The current study examines the cross-cultural equivalence of score interpretations from the Trends in International Mathematics and Science Study (TIMSS)-1999 from a cognitive-psychometric perspective. A cognitive model describing the knowledge, strategies, and processing skills measured by the TIMSS-R mathematics test was previously validated in several countries. In order to establish the cross-cultural equivalence of TIMSS scores for the Taiwanese student population, the fit of the cognitive model to the Taiwanese item responses was examined. High student-mastery classification rates and good prediction of scores based on attribute mastery probabilities supported the fit of the cognitive model in the current study. Further, we suggest that cognitive-psychometric modeling approaches like those applied in the current study could be useful for examining more substantive issues of score validity and equivalence in test translations and adaptations.

Group Comparisons of Mathematics Performance from a Cognitive Diagnostic Perspective.

Traditional comparisons of test score means identify group differences in broad academic areas, but fail to provide substantive description of how the groups differ on the specific cognitive attributes required for success in the academic area. The rule space method (RSM) allows for group comparisons at the cognitive attribute level, which consists of the cognitive knowledge, skills, and abilities an individual can employ to solve a problem. In the current study, we extend RSM group comparison methods to include comparisons of the attribute characteristic curves (ACCs) and provide a method for estimating and plotting the ACCs using SAS. We further investigated Taiwanese mathematics performance on TIMSS-1999 by comparing cognitive attributes between students of different achievement levels as well as between male and female students. The results indicated the highest and lowest achieving students differed most on mastery probabilities for Algebra (C3), Open-ended items (S10), and Rule application in algebra (P4). Gender differences in mathematical skills were quite minimal for Taiwanese students. Detailed discussion of these findings is provided in the paper.