Eleanor O'Rourke

The impact of tutorials on games of varying complexity

One of the key challenges of video game design is teaching new players how to play. Although game developers frequently use tutorials to teach game mechanics, little is known about how tutorials affect game learnability and player engagement. Seeking to estimate this value, we implemented eight tutorial designs in three video games of varying complexity and evaluated their effects on player engagement and retention. The results of our multivariate study of over 45,000 players show that the usefulness of tutorials depends greatly on game complexity. Although tutorials increased play time by as much as 29% in the most complex game, they did not significantly improve player engagement in the two simpler games. Our results suggest that investment in tutorials may not be justified for games with mechanics that can be discovered through experimentation.

A Framework for Automatically Generating Interactive Instructional Scaffolding

Interactive learning environments such as intelligent tutoring systems and software tutorials often teach procedures with step-by-step demonstrations. This instructional scaffolding is typically authored by hand, and little can be reused across problem domains. In this work, we present a framework for generating interactive tutorials from an algorithmic representation of the problem-solving thought process. Given a set of mappings between programming language constructs and user interface elements, we step through this algorithm line-by-line to trigger visual explanations of each step. This approach allows us to automatically generate tutorials for any example problem that can be solved with this algorithm. We describe two prototype implementations in the domains of K-12 mathematics and educational games, and present results from two user studies showing that educational technologists can author thought-process procedures and that generated tutorials can effectively teach a new procedure to students.

Smart Connect: last mile data connectivity for rural health facilities

In this paper we address the problem of providing data connection to the periphery of the health system by presenting Smart Connect. Smart Connect is a custom device that uses the cell phone network to provide limited data connectivity between a rural health facility and a server connected to the Internet. Through an analysis of the health systems in Nicaragua and Vietnam, we have identified a variety of processes that could be improved with this device. These include filing incidence reports on diseases, receiving results of diagnostic test, and providing automatic monitoring of vaccine refrigeration equipment. The main contributions of this work include a presentation of the Smart Connect requirements derived from the health care systems of Nicaragua and Vietnam, an argument in favor of facility based communication devices, and a brief discussion of the design for the Smart Connect device.

Brain Points: A Deeper Look at a Growth Mindset Incentive Structure for an Educational Game

Student retention is a central challenge in systems for learning at scale. It has been argued that educational video games could improve student retention by providing engaging experiences and informing the design of other online learning environments. However, educational games are not uniformly effective. Our recent research shows that player retention can be increased by using a brain points incentive structure that rewards behaviors associated with growth mindset, or the belief that intelligence can grow. In this paper, we expand on our prior work by providing new insights into how growth mindset behaviors can be effectively promoted in the educational game Refraction. We present results from an online study of 25,000 children who were exposed to five different versions of the brain points intervention. We find that growth mindset animations cause a large number of players to quit, while brain points encourage persistence. Most importantly, we find that awarding brain points randomly is ineffective; the incentive structure is successful specifically because it rewards desirable growth mindset behaviors. These findings have important implications that can support the future generalization of the brain points intervention to new educational contexts.

Large-Scale Educational Campaigns

Educational technology requires a delivery mechanism to scale. One method that has not yet seen widespread use is the educational campaign: large-scale, short-term events focused on a specific educational topic, such as the Hour of Code campaign. These are designed to generate media coverage and lend themselves nicely to collaborative or competitive goals, thus potentially leveraging social effects and community excitement to increase engagement and reach students who otherwise would not participate. In this article, we present a case study of three such campaigns that we ran to encourage students to play an algebra game—DragonBox Adaptive: the Washington, Norway, and Minnesota Algebra Challenges. We provide several design recommendations for future campaigns based on our experience, including the effects of different incentive schemes, the insertion of “tests” to fast-forward students to levels of appropriate difficulty, and the strengths and weaknesses of campaigns as a method of collecting experimental data.

A Quantitative Assessment of the Impact on Spatial Understanding of Exploring a Complex Immersive Virtual Environment using Augmented Real Walking versus Flying

When an immersive virtual environment spans an area that is larger than the available physical space for real walking, one may use an ‘augmented walking’ method such as Seven League Boots to enable participants to explore the space while gaining proprioceptive feedback that is similar to what they would experience with normal walking. In this paper, we present the results of a preliminary experiment in which we seek to quantitatively assess the extent to which participants are able to make more accurate spatial judgments about the locations of previously-seen targets in a complicated virtual city environment, experienced using a head-mounted display, after traveling to them using augmented real walking (‘boots’) versus virtual walking enabled by a button press on a hand-held wand. In a series of trials, we ask participants to follow paths of increasing complexity from a home base to different hidden targets in the environment and back. At each endpoint, with the path markings turned off, we ask participants to point, through the intervening alleyway walls, to the location they believe they started from. Participants are able to make real turns with their bodies in both locomotion conditions, however they are able to make real forward movement only under the augmented walking condition. Each participant completes eight trials under each locomotion condition, with the target locations and the order of experiencing each method counterbalanced between participants. In data collected from six participants so far, we are finding that the median angle error is significantly greater, overall, in the wand locomotion condition than in the ‘boots’ locomotion condition, and that the errors tend to increase, overall, as the path complexity increases (from two segments to four segments) in the wand locomotion condition but not in the ‘boots’ locomotion condition.

Hint systems may negatively impact performance in educational games

Video games are increasingly recognized as a compelling platform for instruction that could be leveraged to teach students at scale. Hint systems that provide personalized feedback to students in real time are a central component of many effective interactive learning environments, however little is known about how hints impact player behavior and motivation in educational games. In this work, we study the effectiveness of hints by comparing four designs based on successful hint systems in intelligent tutoring systems and commercial games. We present results from a study of 50,000 students showing that all four hint systems negatively impacted performance compared to a baseline condition with no hints. These results suggest that traditional hint systems may not translate well into the educational game environment, highlighting the importance of studying student behavior to understand the impact of new interactive learning technologies.