Colin S. Wallace

Do Concept Inventories Actually Measure Anything

Although concept inventories are among the most frequently used tools in the physics and astronomy education communities, they are rarely evaluated using item response theory IRT . When IRT models fit the data, they offer sample-independent estimates of item and person parameters. IRT may also provide a way to measure students’ learning gains that circumvents some known issues with Hake’s normalized gain. In this paper, we review the essentials of IRT while simultaneously applying it to the Star Properties Concept Inventory. We also use IRT to explore an important psychometrics debate that has received too little attention from physics and astronomy education researchers: What do we mean when we say we “measure” a mental process? This question leads us to use IRT to address the provocative question that constitutes the title of this paper: Do concept inventories actually measure anything?

A Study of General Education Astronomy Students’ Understandings of Cosmology. Part IV. Common Difficulties Students Experience with Cosmology

This is our fourth paper in our five paper series describing our national study of general education astronomy students’ conceptual and reasoning difficulties with cosmology. While previous papers in this series focused on the processes by which we collected and quantitatively analyzed our data, this paper presents the most common pre-instruction conceptual and reasoning difficulties identified from our qualitative analysis of students’ written responses. We discuss students’ naı̈ve ideas about the expansion and evolution of the universe, the Big Bang, interpreting Hubble plots, and the evidence for dark matter in spiral galaxies.

A Study of General Education Astronomy Students’ Understandings of Cosmology. Part V. The Effects of a New Suite of Cosmology Lecture-Tutorials on Students’ Conceptual Knowledge

This is the final paper in a five-paper series describing our national study of the teaching and learning of cosmology in general education astronomy college-level courses. A significant portion of this work was dedicated to the development of five new Lecture-Tutorials that focus on addressing the conceptual and reasoning difficulties that our research shows students have with frequently taught cosmology topics, such as the expansion of the universe, the Big Bang, and dark matter. We conducted a systematic investigation of the implementation of these new Lecture-Tutorials and resulting learning gains in order to test the efficacy of these new Lecture-Tutorials. Our investigation included classroom observations, results from pre–post testing using four conceptual cosmology surveys, and comparisons between classes in terms of the class time spent on cosmology topics and other instructional strategies used to teach cosmology. We used this combination of qualitative and quantitative research results to evaluate the conceptual understandings of students who used the new cosmology Lecture-Tutorials compared to those students who did not. The analysis of our data shows that, in many cases, classrooms that used the cosmology Lecture-Tutorials saw a greater increase in their students’ conceptual cosmology knowledge compared to classrooms that did not use the cosmology Lecture-Tutorials. However, our results also indicate how instructors implement the Lecture-Tutorials into their classrooms strongly influences their students’ learning gains.

A Classical Test Theory Analysis of the Light and Spectroscopy Concept Inventory National Study Data Set.

This paper is the first in a series of investigations into the data from the recent national study using the Light and Spectroscopy Concept Inventory (LSCI). In this paper, we use classical test theory to form a framework of results that will be used to evaluate individual item difficulties, item discriminations, and the overall reliability of the LSCI. We perform an analysis of individual students’ normalized gains, providing further insight into the prior results from this data set. This investigation allows us to better understand the efficacy of measuring student achievement using the LSCI. Future papers will discuss our investigation of the data from the recent national study using item response theory (IRT).

A Study of General Education Astronomy Students' Understandings of Cosmology. Part I. Development and Validation of Four Conceptual Cosmology Surveys

This is the first in a series of five articles describing a national study of general education astronomy students’ conceptual and reasoning difficulties with cosmology. In this paper, we describe the process by which we designed four new surveys to assess general education astronomy students’ conceptual cosmology knowledge. These surveys focused on the expansion and evolution of the universe, the Big Bang, and the evidence for dark matter in spiral galaxies. We also present qualitative evidence for the validity of these surveys.

Measuring the conceptual understandings of citizen scientists participating in zooniverse projects: A first approach

The Zooniverse projects turn everyday people into “citizen scientists” who work online with real data to assist scientists in conducting research on a variety of topics related to galaxies, exoplanets, lunar craters, and solar flares, among others. This paper describes our initial study to assess the conceptual knowledge and reasoning abilities of citizen scientists participating in two Zooniverse projects: Galaxy Zoo and Moon Zoo. In order to measure their knowledge and abilities, we developed two new assessment instruments, the Zooniverse Astronomical Concept Survey (ZACS) and the Lunar Cratering Concept Inventory (LCCI). We found that citizen scientists with the highest level of participation in the Galaxy Zoo and Moon Zoo projects also have the highest average correct scores on the items of the ZACS and LCCI. However, the limited nature of the data provided by Zooniverse participants prevents us from being able to evaluate the statistical significance of this finding, and we make no claim about whether there is a causal relationship between one’s participation in Galaxy Zoo or Moon Zoo and one’s level of conceptual understanding or reasoning ability on the astrophysical topics assessed by the ZACS or the LCCI. Overall, both the ZACS and the LCCI provide Zooniverse’s citizen scientists with items that offer a wide range of difficulties. Using the data from the small subset of participants who responded to all items of the ZACS, we found evidence suggesting the ZACS is a reliable instrument (a 1⁄4 0.78), although twenty-one of its forty items appear to have point biserials less than 0.3. The work reported here provides significant insight into the strengths and limitations of various methods for administering assessments to citizen scientists. Researchers who wish to study the knowledge and abilities of citizen scientists in the future should be sure to design their research methods to avoid the pitfalls identified by our initial findings.

A Study of General Education Astronomy Students' Understandings of Cosmology. Part III. Evaluating Four Conceptual Cosmology Surveys: An Item Response Theory Approach.

This is the third of five papers detailing our national study of general education astronomy students’ conceptual and reasoning difficulties with cosmology. In this paper, we use item response theory to analyze students’ responses to three out of the four conceptual cosmology surveys we developed. The specific item response theory model we use is known as the partial credit model. Since readers may be unfamiliar with the partial credit model, we provide a pedagogical introduction to this model. We use the partial credit model to assess the reliabilities of the four survey forms and to determine the probabilities of students achieving different scores on survey items.

Transforming the content, pedagogy and structure of an introductory physics course for life sciences majors

In this paper, we describe how we transformed our large-enrollment introductory physics sequence for life-science students to a Lecture/Studio format and aligned the physics concepts with authentic biological applications. We have reformed the pedagogy to include research-validated practices in interactive engagement, and accomplished our goals of enhanced learning gains, sustainability, and adoptability of our course reforms. The active engagement at the heart of the Lecture/Studio format results in comparable or enhanced learning gains (as measured by validated concept surveys) when compared to traditional instruction. When coupled with appropriate instructor preparation the format is sustainable, requiring no greater financial or human resources than does the traditional mode of teaching such courses. We have developed a complete suite of active-engagement instructional materials and made them available to the physics education community for adoption outside our institution.

A New Lecture-Tutorial for Teaching about Molecular Excitations and Synchrotron Radiation

Light and spectroscopy are among the most important and frequently taught topics in introductory college-level general education astronomy courses (hereafter Astro 101). This is due to the fact that the vast majority of observational data studied by astronomers arrives at Earth in the form of light. While there are many processes by which matter can emit and absorb light, Astro 101 courses typically limit their instruction to the Bohr model of the atom and electron energy level transitions. In this paper, we report on the development of a new Lecture-Tutorial to help students learn about other processes that are responsible for the emission and absorption of light, namely molecular rotations, molecular vibrations, and the acceleration of charged particles by magnetic fields. Note that this paper primarily focuses on describing the variety of representations and reasoning tasks designed for this Lecture-Tutorial; while the end of this paper highlights some data that are suggestive of the Lecture-Tutorials...

Teaching physics with Hubble’s law and dark matter

Physics instructors can enrich, enliven, and enhance their courses with conceptually rich cosmology content. In this paper, we specifically discuss how instructors can integrate lessons on Hubble’s law (as it relates to the expansion of the universe and dark energy) and spiral galaxies’ rotation curves (as they relate to the presence of dark matter) into an introductory, college-level course on mechanics. These cosmology topics intersect with the content of introductory physics in a number of areas, such as students’ abilities to read and interpret graphs and their conceptual understandings of both kinematics and dynamics. Throughout this paper, we draw upon the results from, and research-validated curricula informed by, physics and astronomy education research. In particular, we feature the results from a national study we recently completed with introductory college-level general education astronomy students on the teaching and learning of cosmology.