Philip Sheridan Buffum

A Practical Guide to Developing and Validating Computer Science Knowledge Assessments with Application to Middle School

Knowledge assessment instruments, or tests, are commonly created by faculty in classroom settings to measure student knowledge and skill. Another crucial role for assessment instruments is in gauging student learning in response to a computer science education research project, or intervention. In an increasingly interdisciplinary landscape, it is crucial to validate knowledge assessment instruments, yet developing and validating these tests for computer science poses substantial challenges. This paper presents a seven-step approach to designing, iteratively refining, and validating knowledge assessment instruments designed not to assign grades but to measure the efficacy or promise of novel interventions. We also detail how this seven-step process is being instantiated within a three-year project to implement a game-based learning environment for middle school computer science. This paper serves as a practical guide for adapting widely accepted psychometric practices to the development and validation of computer science knowledge assessments to support research.

Mind the Gap: Improving Gender Equity in Game-Based Learning Environments with Learning Companions

Game-based learning environments hold great promise for engaging learners. Yet game mechanics can initially pose barriers for students with less prior gaming experience. This paper examines game-based learning for a population of middle school learners in the US, where female students tend to have less gaming experience than male students. In a pilot study with an early version of Engage, a game-based learning environment for middle school computer science education, female students reported higher initial frustration. To address this critical issue, we developed a prototype learning companion designed specifically to reduce frustration through the telling of autobiographical stories. In a pilot study of two 7th grade classrooms, female students responded especially positively to the learning companion, eliminating the gender gap in reported frustration. The results suggest that introducing learning companions can directly contribute to making the benefits of game-based learning equitable for all learners.

Empowering All Students: Closing the CS Confidence Gap with an In-School Initiative for Middle School Students

The important goal of broadening participation in computing has inspired many successful outreach initiatives. Yet many of these initiatives, such as out-of-school activities or innovative new computer science courses for secondary school students, may disproportionately attract students who already have prior interest and experience in computing. How, then, do we engage the silent majority of students who do not self-select computer science? This paper examines this question in the context of ENGAGE, an in-school outreach initiative for middle school students. ENGAGEs learning activities center on a game-based learning environment for computer science. Results reveal that the initiative improved the computer science attitudes of students who were not already predisposed to study computer science, in a way that a corresponding after-school program could not. The results illustrate how an in-school initiative can empower young students who might not otherwise consider studying computer science.

Leveraging collaboration to improve gender equity in a game-based learning environment for middle school computer science

Game-based learning environments can deliver robust learning gains and also have a unique capacity to engage students. Yet they can unintentionally disadvantage students with less prior gaming experience. This is especially concerning in computer science education, as certain underrepresented groups (such as female students) may on average have less prior experience with games. This paper presents evidence that a collaborative gameplay approach can successfully address this problem at the middle school level. In an iterative, designed-based research study, we first used an experimental pilot study to investigate the nature of collaboration in the Engage game-based learning environment, and then deployed Engage in a full classroom study to measure its effectiveness at serving all students. In earlier phases of the intervention, male students outpaced their female peers in learning gains. However, female students caught up during a multi-week classroom implementation. These findings provide evidence that a collaborative gameplay approach may, over time, compensate for gender differences in experience and lead to equitable learning experiences within game-based learning environments for computer science education.

Collaboration and Gender Equity in Game-Based Learning for Middle School Computer Science

Game-based learning environments can deliver robust learning gains and have a significant capacity to engage students. Yet, they can unintentionally disadvantage students with less prior game experience. This article presents evidence that a collaborative gameplay approach can effectively address this problem at the middle school level. In an iterative, designed-based research study, the authors first used an experimental pilot study to investigate the nature of collaboration in the Engage game-based learning environment and then deployed Engage in a full classroom study to measure its effectiveness at supporting all students during computer science learning. In early phases of the intervention, male students outpaced female peers in learning gains. However, female students caught up during a multiweek classroom implementation. These findings provide evidence that gender differences could dissipate over time within collaborative game-based learning experiences in computer science.

Leveraging Narrative Interactivity to Foster Computer Science Identities

There is growing recognition in the computer science education research community that supporting students in identity formation for computer science is an essential step toward broadening participation in computing. Because students begin establishing their career aspiration identity at an early age, research into computing identity formation in K-8 education is particularly important. Research suggests that narrative-centered learning environments hold great promise for fostering identity formation. My proposed dissertation research focuses on bringing narrative-centered learning environment technology to bear on supporting computer science learners in middle school. I am particularly interested in how we can make these learning environments more equitable for all learners, especially for underrepresented students.

Developing a game-based learning curriculum for "Big Data" in middle school (abstract only)

Exposing students early to computer science may influence their choice of career, and there is increasing recognition that even for students who do not pursue computer science careers, computational literacy is important. This poster reports on a project targeting the development of a new middle school computer science curriculum. This research aims to highlight the role of computation in Big Data in the context of middle school computer science education, which serves as a catalyst to keep students engaged in computer science through middle school via the ENGAGE narrative game-based learning environment. This poster discusses steps taken to validate one activity meant to highlight the role of computation in the context of Big Data: skip list manipulation. While we found that most of the middle school students performed poorly in assessments after the skip list activities, several students showed they were capable of completing the activity successfully, implying that a repetition of the revised skip list study and additional pilot studies for other Big Data activities are needed to pave the way for the development of this Big Data curriculum. This activity will be just one part of a broader curriculum designed to showcase the social relevance and power of Big Data.