Irene Neumann

Evaluating Instrument Quality in Science Education: Rasch‐based analyses of a Nature of Science test

Given the central importance of the Nature of Science (NOS) and Scientific Inquiry (SI) in national and international science standards and science learning, empirical support for the theoretical delineation of these constructs is of considerable significance. Furthermore, tests of the effects of varying magnitudes of NOS knowledge on domain‐specific science understanding and belief require the application of instruments validated in accordance with AERA, APA, and NCME assessment standards. Our study explores three interrelated aspects of a recently developed NOS instrument: (1) validity and reliability; (2) instrument dimensionality; and (3) item scales, properties, and qualities within the context of Classical Test Theory and Item Response Theory (Rasch modeling). A construct analysis revealed that the instrument did not match published operationalizations of NOS concepts. Rasch analysis of the original instrument—as well as a reduced item set—indicated that a two‐dimensional Rasch model fit significantly better than a one‐dimensional model in both cases. Thus, our study revealed that NOS and SI are supported as two separate dimensions, corroborating theoretical distinctions in the literature. To identify items with unacceptable fit values, item quality analyses were used. A Wright Map revealed that few items sufficiently distinguished high performers in the sample and excessive numbers of items were present at the low end of the performance scale. Overall, our study outlines an approach for how Rasch modeling may be used to evaluate and improve Likert‐type instruments in science education.

Analyzing the FCI based on a Force and Motion Learning Progression

In this paper, we bring together a well-established and often-studied instrument assessing students’ understanding about force and motion, the Force Concept Inventory (FCI), and a proposed description of how students develop in understanding of the force concept, the Force and Motion Learning Progression (FM-LP). We report on two phases of content analysis of the FCI and the FM-LP. In the first phase, findings indicate that 17 FCI items address aspects consistent with the FM-LP. In the second phase, our findings show that these 17 items have responses that can be coded to fit the levels of the proposed FM-LP. Implications for future research on both the FCI and the FM-LP are described.