Kelly Rivers

Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor

To provide personalized help to students who are working on code-writing problems, we introduce a data-driven tutoring system, ITAP (Intelligent Teaching Assistant for Programming). ITAP uses state abstraction, path construction, and state reification to automatically generate personalized hints for students, even when given states that have not occurred in the data before. We provide a detailed description of the system’s implementation and perform a technical evaluation on a small set of data to determine the effectiveness of the component algorithms and ITAP’s potential for self-improvement. The results show that ITAP is capable of producing hints for almost any given state after being given only a single reference solution, and that it can improve its performance by collecting data over time.

Automating Hint Generation with Solution Space Path Construction

Developing intelligent tutoring systems from student solution data is a promising approach to facilitating more widespread application of tutors. In principle, tutor feedback can be generated by matching student solution attempts to stored intermediate solution states, and next-step hints can be generated by finding a path from a students current state to a correct solution state. However, exact matching of states and paths does not work for many domains, like programming, where the number of solution states and paths is too large to cover with data. It has previously been demonstrated that the state space can be substantially reduced using canonicalizing operations that abstract states. In this paper, we show how solution paths can be constructed from these abstract states that go beyond the paths directly observed in the data. We describe a domain-independent algorithm that can automate hint generation through use of these paths. Through path construction, less data is needed for more complete hint generation. We provide examples of hints generated by this algorithm in the domain of programming.

Towards improving programming habits to create better computer science course outcomes

We examine a large dataset collected by the Marmoset system in a CS2 course. The dataset gives us a richly detailed portrait of student behavior because it combines automatically collected program snapshots with unit tests that can evaluate the correctness of all snapshots. We find that students who start earlier tend to earn better scores, which is consistent with the findings of other researchers. We also detail the overall work habits exhibited by students. Finally, we evaluate how students use release tokens, a novel mechanism that provides feedback to students without giving away the code for the test cases used for grading, and gives students an incentive to start coding earlier. We find that students seem to use their tokens quite effectively to acquire feedback and improve their project score, though we do not find much evidence suggesting that students start coding particularly early.

A canonicalizing model for building programming tutors

It is difficult to build intelligent tutoring systems in the domain of programming due to the complexity and variety of possible answers. To simplify this process, we have constructed a language-independent canonicalized model for programming solutions. This model allows for much greater overlap across different students than a basic text model, which enables more self-sustaining hint generation methods in programming tutors.

Learning Curve Analysis for Programming: Which Concepts do Students Struggle With?

The recent surge in interest in using educational data mining on student written programs has led to discoveries about which compiler errors students encounter while they are learning how to program. However, less attention has been paid to the actual code that students produce. In this paper, we investigate programming data by using learning curve analysis to determine which programming elements students struggle with the most when learning in Python. Our analysis extends the traditional use of learning curve analysis to include less structured data, and also reveals new possibilities for when to teach students new programming concepts. One particular discovery is that while we find evidence of student learning in some cases (for example, in function definitions and comparisons), there are other programming elements which do not demonstrate typical learning. In those cases, we discuss how further changes to the model could affect both demonstrated learning and our understanding of the different concepts that students learn.

CloudCoder: building a community for creating, assigning, evaluating and sharing programming exercises (abstract only)

Automatically-tested online programming exercises can be useful in introductory programming courses as self-tests to accompany readings, for in-class assessment, for skills development, and to provide additional practice for students who need it. CloudCoder (http://cloudcoder.org) is an effort to build a community based on an open-source programming exercise system (currently supporting C, Java, and Python) tightly integrated with a repository of freely-redistributable programming exercises written and used by members of the community. The goal of the project is to make programming exercises easy and free to incorporate into any programming course.

Sharing and Using Programming Log Data (Abstract Only)

As more programming environments add logging features and programming data becomes more accessible, it is important to have a conversation about how we share and use this data. Uses of programming log data range from big-picture analyses to dashboards for instant teacher feedback, to intelligent, data-driven learning environments. The goal of this BOF is to talk about what data is important to collect, where it can be gathered and shared, what general data formats make sense, how to handle privacy and anonymization, and what ultimately we want to see the data used for. The BOF welcomes both producers of programming log data and current or potential consumers, interested in how it could be applied in their classrooms or research. One hopeful outcome of this BOF is a commitment to documenting and sharing existing programming data in an accessible location and format.

Designing a Data-Driven Tutor Authoring Tool for CS Educators

Intelligent Tutoring Systems are highly effective at helping students learn, but have required intensive amounts of development time in the past, keeping teachers from making their own. Data-driven tutoring has made it possible to build these tutors more efficiently. For my thesis work, I intend to build an authoring tool for data-driven tutors that is designed to be used by computer science teachers. I plan to design this system based on data gathered in interviews with CS educators and evaluate it on its usability for new users.