Kristin L. K. Koskey

Teaching for Transformative Experiences and Conceptual Change: A Case Study and Evaluation of a High School Biology Teacher's Experience

The Teaching for Transformative Experiences in Science (TTES) model is designed to foster transformative experiences (e.g., experiences with science content involving the application of that content in ways that expand perception and value in everyday experience). This study presents a case study of a high school biology teacher learning to implement the TTES model in conjunction with a conceptual change model and evaluates the impact of these models on students’ learning and engagement. In collaboration with the authors, the teacher created three instructional conditions: (a) control (2 classes, 40 students), (b) cognitive conflict (CC, 2 classes, 42 students), and (c) combined TTES and cognitive conflict (TE, 2 classes, 44 students). Descriptive data illustrate how the teacher implemented the models, particular challenges that arose, how the teachers prior beliefs and goals shaped the implementation, and how experience with the models altered the teachers beliefs and engagement with the content. Comparative analyses across conditions illustrate that students in the TE condition displayed more transformative engagement in class but did not report a greater level of independent transformative experiences. Regular-level students in the intervention conditions (CC, TE) displayed greater change in basic knowledge and regular-level students in the TE condition displayed more enduring transfer. Honors-level students in the intervention conditions (CC, TE) displayed greater conceptual change, but not greater transfer.

Developing Conceptual Understanding of Natural Selection: The Role of Interest, Efficacy, and Basic Prior Knowledge.

Changes in high school students’ (n = 94) conceptions of natural selection were examined as a function of motivational beliefs (individual interest, academic self-efficacy), basic prior knowledge, and gender across three assessments (pre, post, follow-up). Results from variable-centered analyses suggested that these variables had relatively little effect on enduring conceptual change; however, academic self-efficacy supported short-term conceptual change for girls. Results from person-centered analyses provided a different picture. Four profiles of motivational beliefs and basic prior knowledge were created using hierarchical cluster analysis: (a) low interest/efficacy, low knowledge; (b) moderate interest/efficacy, low knowledge; (c) moderate-low interest, moderate efficacy, high knowledge; and (d) high interest/efficacy, moderate knowledge. For girls, high interest and efficacy paired with moderate basic prior knowledge (Cluster 4) resulted in the greatest conceptual change. For boys, either moderate interest and efficacy paired with high knowledge (Cluster 3) or high interest and efficacy paired with moderate knowledge (Cluster 4) resulted in the greatest enduring conceptual change.

Comparing Construct Definition in the Angoff and Objective Standard Setting Models Playing in a House of Cards Without a Full Deck

Typical validation studies on standard setting models, most notably the Angoff and modified Angoff models, have ignored construct development, a critical aspect associated with all conceptualizations of measurement processes. Stone compared the Angoff and objective standard setting (OSS) models and found that Angoff failed to define a legitimate and stable construct. The present study replicates and expands this work by presenting results from a 5-year investigation of both models, using two different approaches (equating and annual standard setting) within two testing settings (health care and education). The results support the original conclusion that although the OSS model demonstrates effective construct development, the Angoff approach appears random and lacking in clarity. Implications for creating meaningful and valid standards are discussed.

High school biology students’ transfer of the concept of natural selection: a mixed-methods approach

The concept of natural selection serves as a foundation for understanding diverse biological concepts and has broad applicability to other domains. However, we know little about students’ abilities to transfer (i.e. apply to a new context or use generatively) this concept and the relation between students’ conceptual understanding and transfer ability. Consequently, the purposes of this study were to describe the patterns of transfer displayed by high school biology students learning about natural selection over time, evaluate their overall level of success at transferring the concept across knowledge domains and examine the relation between conceptual understanding and level of transfer of the concept. Transfer ability and conceptual understanding were assessed using open-response items administered to 138 students in the United States. Based on the responses to these items, we identified particular patterns of surface and deep-level transfer and found that deep-level transfer was uncommon. Further, we found that deep-level transfer and conceptual understanding shared a small but significant relation; surface-level transfer was unrelated. Based on these results, we recommend that teachers explicitly focus on fostering transfer of the concept of natural selection and use specific teaching for transfer strategies, in addition to teaching for conceptual understanding.

Applying the Mixed Methods Instrument Development and Construct Validation Process The Transformative Experience Questionnaire

Onwuegbuzie and colleagues proposed the Instrument Development and Construct Validation (IDCV) process as a mixed methods framework for creating and validating measures. Examples applying IDCV are lacking. We provide an illustrative case integrating the Rasch model and cognitive interviews applied to the development of the Transformative Experience Questionnaire (TEQ). TEQ measures student engagement with the content they are learning. IDCV process yielded a high degree of utility for informing the psychometric properties of the TEQ. The quantitative data yielded supportive content, construct, and concurrent-related validity evidence using a sample of middle (n = 115) and high school students (n = 81). The qualitative data revealed item rewording was needed. Moderate to strong test–retest reliability was found for both forms.

A Concurrent Mixed Methods Approach to Examining the Quantitative and Qualitative Meaningfulness of Absolute Magnitude Estimation Scales in Survey Research.

This small n observational study used a concurrent mixed methods approach to address a void in the literature with regard to the qualitative meaningfulness of the data yielded by absolute magnitude estimation scaling (MES) used to rate subjective stimuli. We investigated whether respondents’ scales progressed from less to more and provided for meaningful discriminations, similarities and differences in how they defined their scales, and whether their definitions were congruent with how they used their scales. The respondents completed a think-aloud task while using MES and a follow-up interview regarding how they used MES and defined their scales. The respondents’ scales did not yield meaningful data. Implications for survey research and the use of mixed methods in rating scale construction are discussed.

Using the Cognitive Pretesting Method to Gain Insight Into Participants’ Experiences An Illustration and Methodological Reflection

Quantitative researchers traditionally use cognitive pretesting methods such as think-alouds to determine whether participants are interpreting close-ended survey items as intended. I applied this method to evaluate the score validity of quantitative survey instruments with a middle school and high school sample of students. Through these applications, the cognitive pretesting method also revealed to be a useful tool for cueing participants’ reflection on their experiences related to the phenomenon of interest. Qualitative researchers traditionally use more open-ended interviews guided by general “how” and “why” questions, which might not serve as sufficient prompts to elicit participants’ reflection on their experiences, especially younger participants. To date, there are no illustrative examples focused on how having participants think-aloud as they complete a quantitative survey can inform qualitative-focused or mixed research questions. Therefore, the purpose of this article was to illustrate how the cognitive pretesting method, where participants think-aloud as they complete a survey, can be used as an additional technique to gain insight into their experiences of a phenomenon. The phenomenon in this illustration was defined as the participants’ experiences with the science content that they were learning in school. Limitations and methodological challenges and recommendations for overcoming these barriers are also discussed.