Jody Clarke-Midura

Assessment, Technology, and Change

Abstract Despite three decades of advances in information and communications technology (ICT) and a generation of research on cognition and new pedagogical strategies, the field of assessment has not progressed much beyond paper-and-pencil item-based tests. Research has shown these instruments are not valid measures of sophisticated intellectual performances. Simply using technology to deliver automated versions of item-based tests does not realize the full power of ICT to innovate in assessment via providing rich experiences that enable observing and analyzing student performances. To illustrate this approach, we describe our early research on using immersive technologies to develop virtual performance assessments.

Emerging Technologies for Collaborative, Mediated, Immersive Learning

Three complementary technological interfaces are now shaping how people learn, with multiple implications for K-12 education (Dede, 2002). • The familiar “worldtothedesktop” interface provides access to distributed knowledge and expertise across space and time through networked media. Sitting at their laptop or workstation, students can access distant experts and archives, communicate with peers, and participate in mentoring relationships and virtual communities-of practice. This interface provides the models for learning that now underlie most tools, applications, and media in K-12 education. • Emerging multi-user virtual environment (MUVE) interfaces offer students an engaging “Alice in Wonderland” experience in which their digital emissaries in a graphical virtual context actively engage in experiences with the avatars of other participants and with computerized agents. MUVEs provide rich environments in which participants interact with digital objects and tools, such as historical photographs or virtual microscopes. Moreover, this interface facilitates novel forms of communication among avatars, using media such as text chat and virtual gestures. This type of “mediated immersion” (pervasive experiences within a digitally enhanced context), intermediate in complexity between the real world and paint-by-numbers exercises in K-12 classrooms, allows instructional designers to construct shared simulated experiences otherwise impossible in school settings. Researchers are exploring the affordances of such models for learning in K-12 education (Clarke et al, 2006; Barab et al, 2004). • Augmented reality (AR) interfaces enable “ubiquitous computing” models. Students carrying mobile wireless devices through real world contexts engage with virtual information superimposed on physical landscapes (such as a tree describing its botanical characteristics or an historic photograph offering a contrast with the present scene). This type of mediated immersion infuses digital resources throughout the real world, augmenting students’ experiences and interactions. Researchers are starting to study how these models for learning aid students’ engagement and understanding (Klopfer et al, 2004; Klopfer & Squire, in press).

Robust Designs for Scalability

One-size-fits-all educational innovations do not work because they ignore contextual factors that determine an intervention’s efficacy in a particular local situation. Identifying variables within the intervention’s setting that represent important conditions for success and summarizing the extent to which the impact of the intervention is attenuated by variation in them can provide prospective adopters of the innovation a better sense of what level of effectiveness they are likely to enjoy in their own particular circumstances. This study presents a research framework on how to conduct such an analysis and how to design educational innovations for scalability through enhancing their adaptability for effective usage in a wide variety of settings. The River City MUVE, a technology-based curriculum designed to enhance engagement and learning in middle school science, is presented as a case study.

Predicting Successful Inquiry Learning in a Virtual Performance Assessment for Science

In recent years, models of student inquiry skill have been developed for relatively tightly-scaffolded science simulations. However, there is an increased interest in researching how video games and virtual environments can be used for both learning and assessment of science inquiry skills and practices. Such environments allow students to explore scientific content in a more open-ended context that is designed around actions and choices. In such an environment, students move an avatar around a world, speak to in-game characters, obtain objects, and take those objects to laboratories to run specific tests. While these environments allow for more autonomy and choice, assessing skills in these environments is a more difficult challenge than in closed environments or simulations. In this paper, we present models that can infer two aspects of middle-school students’ inquiry skill, from their interactive behaviors within an assessment in a virtual environment called a “virtual performance assessment” or VPA: 1) whether the student successfully demonstrates the skill of designing controlled experiments within the VPA, and 2) whether a middle-school student can successfully use their inquiry skill to determine the answer to a scientific question with a non-intuitive in-game answer.

Towards general models of effective science inquiry in virtual performance assessments

Recent interest in online assessment of scientific inquiry has led to several new online systems that attempt to assess these skills, but producing models that detect when students are successfully practising these skills can be challenging. In this paper, we study models that assess student inquiry in an immersive virtual environment, where a student navigates an avatar around a world, speaking to in-game characters, collecting samples and conducting scientific tests with those samples in the virtual laboratory. To this goal, we leverage log file data from nearly 2000 middle school students using virtual performance assessment VPA, a software system where students practice inquiry skills in different virtual scenarios. We develop models of student interaction within VPA that predict whether a student will successfully conduct scientific inquiry. Specifically, we identify behaviours that lead to distinguishing causal from non-causal factors to identify a correct final conclusion and to design a causal explanation about these conclusions. We then demonstrate that these models can be adapted with minimal effort between VPA scenarios. We conclude with discussions of how these models serve as a tool for better understanding scientific inquiry in virtual environments and as a platform for the future design of evidence-based interventions.

Some Psychometric and Design Implications of Game-Based Learning Analytics.

The rise of digital game and simulation-based learning applications has led to new approaches in educational measurement that take account of patterns in time, high resolution paths of action, and clusters of virtual performance artifacts. The new exploratory data analysis approaches, which depart from traditional statistical analyses, include data mining, machine learning, and symbolic regression. This article briefly describes the context, methods and broad findings from two game-based exploratory analyses and describes key explanatory constructs used to make claims about the players as well as the implications of the findings for design of digital game-based learning and assessment applications.

Towards Identifying Students' Causal Reasoning Using Machine Learning

Causal reasoning is difficult for middle school students to grasp. In this research, we wanted to test the possibility of using machine learning for modeling students’ causal reasoning in a virtual environment designed to assess this skill. Our findings suggest it is possible to use machine learning to emulate student pathways that are able to predict their causal understanding.

Note-taking and science inquiry in an open-ended learning environment

Highlights • Usage of digital notepad is related to performance in science inquiry tasks in OELE.• Both taking and reaccessing notes facilitate science inquiry performance.• Elaborative and reproductive notes’ relationship with success is content dependent.

Drawing a computer scientist: stereotypical representations or lack of awareness?

ABSTRACT Stereotypes people hold about computer scientists contribute to underrepresentation in computer science. Perceptions of computer scientists have historically been linked to males and a “nerd” culture, which can lead to lack of interest, particularly for girls. This article presents two studies conducted with two groups of middle schoolers: those who attended our programming camp (Study One) and those who did not (Study Two). After analyzing participants’ drawings and two survey questions we found that perceptions youth holds about computer scientists may be improving. Additionally, we found that males (versus females) and those youth who did not attend our camp (versus those who did) tended to have more stereotypical perceptions of computer scientists. With this article, we contribute to the literature on CS stereotypes by examining both positive and negative representations youth had about computer scientists. We also cast light on the lack of awareness about this profession among the youth of this age.

Tipping the Scales: Classroom Feasibility of the Radix Endeavor Game

The Radix Endeavor is an inquiry-based educational game designed to encourage exploration and experimentation. Its design deeply integrates STEM practices as core game mechanics, enabling students to learn by doing in authentic contexts that are distributed across people, places, and time. Radix is a large-scale game that is not primarily designed to directly teach students math and science facts, but rather to serve as a foundational experience that teachers can integrate into their teaching in personalized and meaningful ways. Teachers have a difficult choice each time they consider implementing a unit of game-based curriculum. They must weigh the educational and pedagogical benefits on the one hand, and the barriers to implementation on the other, in order to decide whether the game is worth implementing in their classroom. In this chapter we discuss the feasibility of implementing the game in the classroom, teacher perceptions of student learning with the game, and the interaction between these two factors.