Thomas W. Price

iSnap: Towards Intelligent Tutoring in Novice Programming Environments

Programming environments intentionally designed to support novices have become increasingly popular, and growing research supports their efficacy. While these environments offer features to engage students and reduce the burden of syntax errors, they currently offer little support to students who get stuck and need expert assistance. Intelligent Tutoring Systems (ITSs) are computer systems designed to play this role, helping and guiding students to achieve better learning outcomes. We present iSnap, an extension to the Snap programming environment which adds some key features of ITSs, including detailed logging and automatically generated hints. We share results from a pilot study of iSnap, indicating that students are generally willing to use hints and that hints can create positive outcomes. We also highlight some key challenges encountered in the pilot study and discuss their implications for future work.

Hint Generation Under Uncertainty: The Effect of Hint Quality on Help-Seeking Behavior

Much research in Intelligent Tutoring Systems has explored how to provide on-demand hints, how they should be used, and what effect they have on student learning and performance. Most of this work relies on hints created by experts and assumes that all help provided by the tutor is correct and of high quality. However, hints may not all be of equal value, especially in open-ended problem solving domains, where context is important. This work argues that hint quality, especially when using data-driven hint generation techniques, is inherently uncertain. We investigate the impact of hint quality on students’ help-seeking behavior in an open-ended programming environment with on-demand hints. Our results suggest that the quality of the first few hints on an assignment is positively associated with future hint use on the same assignment. Initial hint quality also correlates with possible help abuse. These results have important implications for hint design and generation.

BJC in action: Comparison of student perceptions of a computer science principles course

The Beauty and Joy of Computing (BJC) is a curriculum for the new AP Computer Science Principles course. Over the past 2 years, we have collected post-course surveys from 399 students participating in the BJC course. This paper investigates how the responses of females and students from underrepresented racial minority groups (URMs) differed from those of their counterparts. We found that female students had taken fewer CS courses prior to BJC but that students from URMs had taken more prior CS courses. Both groups were nearly equally likely to recommend the course to a friend, with about 80% recommending. We found no evidence to suggest that female students showed more or less interest in specific CS topics, such as learning how computing has changed the world or making mobile apps/games. Despite having taken more CS courses prior to BJC, we found that students from URMs were overall less likely to intend to take additional CS courses. Overall, our findings are fairly consistent with the literature, and suggest that BJC makes some progress towards broadening participation in computing.

Creating Data-Driven Feedback for Novices in Goal-Driven Programming Projects

Programming environments that afford the creation of media-rich, goal-driven projects, such as games, stories and simulations, are effective at engaging novice users. However, the open-ended nature of these projects makes it difficult to generate ITS-style guidance for students in need of help. In domains where students produce similar, overlapping solutions, data-driven techniques can leverage the work of previous students to provide feedback. However, our data suggest that solutions to these projects have insufficient overlap to apply current data-driven methods. We propose a novel subtree-based state matching technique that will find partially overlapping solutions to generate feedback across diverse student programs. We will build a system to generate this feedback, test the technique on historical data, and evaluate the generated feedback in a study of goal-driven programming projects. If successful, this approach will provide insight into how to leverage structural similarities across complex, creative problem solutions to provide data-driven feedback for intelligent tutoring.

iSnap: Automatic Hints and Feedback for Block-based Programming (Abstract Only)

iSnap is a block-based programming environment that supports struggling students with on-demand hints and error-checking feedback. iSnap is an extension of Snap!, a creative and novice-friendly programming environment, used in the Beauty and Joy of Computing (BJC) AP CS Principles curriculum. iSnap is designed to support the open-ended, exploratory programming problems of BJC, while adapting to many possible student solutions. When students ask iSnap for help, it highlights possible errors in their code and suggests next steps they can make. Hints are presented visually, right alongside students/ code, making them easy to interpret and implement. iSnap/s hints are generated automatically from student data, so no teacher input is required to create them, making iSnap appropriate for both new and experienced instructors. The demonstration will showcase iSnap/s hints on a variety of assignments and explain how the algorithm is working behind the scenes to generate data-driven hints. It will also include an overview of the results from two years of research with iSnap on how students seek and use programming help. A key objective of this demonstration is to solicit feedback from SIGCSE attendees on the design of iSnap as we work to make the system ready for deployment in classrooms. More information on iSnap can be found at http://go.ncsu.edu/isnap.

Exploring Instructional Support Design in an Educational Game for K-12 Computing Education

Instructional supports (Supports) help students learn more effectively in intelligent tutoring systems and gamified educational environments. However, the implementation and success of Supports vary by environment. We explored Support design in an educational programming game, BOTS, implementing three different strategies: instructional text (Text), worked examples (Examples) and buggy code (Bugs). These strategies are adapted from promising Supports in other domains and motivated by established educational theory. We evaluated our Supports through a pilot study with middle school students. Our results suggest Bugs may be a promising strategy, as demonstrated by the lower completion time and solution code length in assessment puzzles. We end reflecting on our design decisions providing recommendations for future iterations. Our motivations, design process, and studys results provide insight into the design of Supports for programming games.

The Impact of Data Quantity and Source on the Quality of Data-Driven Hints for Programming

In the domain of programming, intelligent tutoring systems increasingly employ data-driven methods to automate hint generation. Evaluations of these systems have largely focused on whether they can reliably provide hints for most students, and how much data is needed to do so, rather than how useful the resulting hints are to students. We present a method for evaluating the quality of data-driven hints and how their quality is impacted by the data used to generate them. Using two datasets, we investigate how the quantity of data and the source of data (whether it comes from students or experts) impact one hint generation algorithm. We find that with student training data, hint quality stops improving after 15–20 training solutions and can decrease with additional data. We also find that student data outperforms a single expert solution but that a comprehensive set of expert solutions generally performs best.

Sharing and Using Programming Log Data (Abstract Only)

As more programming environments add logging features and programming data becomes more accessible, it is important to have a conversation about how we share and use this data. Uses of programming log data range from big-picture analyses to dashboards for instant teacher feedback, to intelligent, data-driven learning environments. The goal of this BOF is to talk about what data is important to collect, where it can be gathered and shared, what general data formats make sense, how to handle privacy and anonymization, and what ultimately we want to see the data used for. The BOF welcomes both producers of programming log data and current or potential consumers, interested in how it could be applied in their classrooms or research. One hopeful outcome of this BOF is a commitment to documenting and sharing existing programming data in an accessible location and format.

Showpiece: ISnap demonstration

This showpiece will present iSnap, an extension of the block-based, novice programming environment Snap!, which supports struggling students by providing on-demand hints and feedback that help them complete programming assignments. iSnap extends the existing syntactic scaffolding offered by block-based programming to additionally support the implementation of programming tasks. Research on iSnap has explored questions of how visual programming environments can better support learners, the impact of this support, and how learners seek and use computer-based help. The showpiece will consist of an interactive demonstration of iSnap, including the user interface experienced by students and the data-driven algorithm used to automatically generate the programming feedback.

Position paper: Block-based programming should offer intelligent support for learners

Block-based programming environments make learning to program easier by allowing learners to focus on concepts rather than syntax. However, these environments offer little support when learners encounter difficulty with programming concepts themselves, especially in the absence of instructors. Textual programming environments increasingly use AI and data mining to provide intelligent, adaptive support for students, similar to human tutoring, which has been shown to improve performance and learning outcomes. In this position paper, we argue that block-based programming environments should also include these features. This paper gives an overview of promising research in intelligent support for programming and highlights the challenges and opportunities for applying this work to block-based programming.