Elena L. Glassman

A wavelet-like filter based on neuron action potentials for analysis of human scalp electroencephalographs

This paper describes the development and testing of a wavelet-like filter, named the SNAP, created from a neural activity simulation and used, in place of a wavelet, in a wavelet transform for improving EEG wavelet analysis, intended for brain-computer interfaces. The hypothesis is that an optimal wavelet can be approximated by deriving it from underlying components of the EEG. The SNAP was compared to standard wavelets by measuring Support Vector Machine-based EEG classification accuracy when using different wavelets/filters for EEG analysis. When classifying P300 evoked potentials, the error, as a function of the wavelet/filter used, ranged from 6.92% to 11.99%, almost twofold. Classification using the SNAP was more accurate than that with any of the six standard wavelets tested. Similarly, when differentiating between preparation for left- or right-hand movements, classification using the SNAP was more accurate (10.03% error) than for four out of five of the standard wavelets (9.54% to 12.00% error) and internationally competitive (7% error) on the 2001 NIPS competition test set. Phenomena shown only in maps of discriminatory EEG activity may explain why the SNAP appears to have promise for improving EEG wavelet analysis. It represents the initial exploration of a potential family of EEG-specific wavelets.

Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis

In large introductory programming classes, teacher feedback on individual incorrect student submissions is often infeasible. Program synthesis techniques are capable of fixing student bugs and generating hints automatically, but they lack the deep domain knowledge of a teacher and can generate functionally correct but stylistically poor fixes. We introduce a mixed-initiative approach which combines teacher expertise with data-driven program synthesis techniques. We demonstrate our novel approach in two systems that use different interaction mechanisms. Our systems use program synthesis to learn bug-fixing code transformations and then cluster incorrect submissions by the transformations that correct them. The MistakeBrowser system learns transformations from examples of students fixing bugs in their own submissions. The FixPropagator system learns transformations from teachers fixing bugs in incorrect student submissions. Teachers can write feedback about a single submission or a cluster of submissions and propagate the feedback to all other submissions that can be fixed by the same transformation. Two studies suggest this approach helps teachers better understand student bugs and write reusable feedback that scales to a massive introductory programming classroom.

Feature engineering for clustering student solutions

Open-ended homework problems such as coding assignments give students a broad range of freedom for the design of solutions. We aim to use the diversity in correct solutions to enhance student learning by automatically suggesting alternate solutions. Our approach is to perform a two-level hierarchical clustering of student solutions to first partition them based on the choice of algorithm and then partition solutions implementing the same algorithm based on low-level implementation details. Our initial investigations in domains of introductory programming and computer architecture demonstrate that we need two different classes of features to perform effective clustering at the two levels, namely abstract features and concrete features.

Toward facilitating assistance to students attempting engineering design problems

In engineering design courses, many problems have a specification that the students implementation must meet, but give the student a large range of freedom for the internal design of that implementation. There may be several distinct, correct strategies for solving them, some of which may be unknown to the teaching staff or intelligent tutor designer. When a student is pursuing an unrecognized strategy and begins to struggle, staff may redirect them, costing unnecessary work, and automated hint generators may offer unhelpful feedback. We have taken a first step toward discovering these alternate correct strategies by visualizing many student solutions together, using dynamic and static features of these solutions, so that the teaching staff can understand the space of correct strategies. This approach has been applied to two domains: an online Matlab programming challenge and an undergraduate computer architecture course. We discuss these initial investigations and pose discussion questions to the community about potential enhancement and application of this analysis.

Reducing the number of channels for an ambulatory patient-specific EEG-based epileptic seizure detector by applying recursive feature elimination.

We are building an ambulatory version of a patient-specific epileptic seizure detector based on scalp EEG signals. Since patients have to wear the electrodes all the time, it is desirable to use the minimum number of electrodes needed to achieve good performance. In this paper, we describe a method that uses recursive feature elimination (RFE) to design detectors that use small numbers of electrodes. We also present results that indicate that the appropriate number of electrodes varies across patients. It is frequently the case that a surprisingly small number of electrodes, sometimes as few as two, suffices to construct a detector with expected performance comparable to that of detectors that use a full twenty-one-channel montage

Foobaz: Variable Name Feedback for Student Code at Scale

Current traditional feedback methods, such as hand-grading student code for substance and style, are labor intensive and do not scale. We created a user interface that addresses feedback at scale for a particular and important aspect of code quality: variable names. We built this user interface on top of an existing back-end that distinguishes variables by their behavior in the program. Therefore our interface not only allows teachers to comment on poor variable names, they can comment on names that mislead the reader about the variables role in the program. We ran two user studies in which 10 teachers and 6 students created and received feedback, respectively. The interface helped teachers give personalized variable name feedback on thousands of student solutions from an edX introductory programming MOOC. In the second study, students composed solutions to the same programming assignments and immediately received personalized quizzes composed by teachers in the previous user study.

Exploring the Design Space of Automatically Synthesized Hints for Introductory Programming Assignments

For massive programming classrooms, recent advances in program synthesis offer means to automatically grade and debug student submissions, and generate feedback at scale. A key challenge for synthesis-based autograders is how to design personalized feedback for students that is as effective as manual feedback given by teachers today. To understand the state of hint-giving practice, we analyzed 132 online Q&A posts and conducted a semi-structured interview with a teacher from a local massive programming class. We identified five types of teacher hints that can also be generated by program synthesis. These hints describe transformations, locations, data, behavior, and examples. We describe our implementation of three of these hint types. This work paves the way for future deployments of automatic, pedagogically-useful programming hints driven by program synthesis.

Using and Designing Platforms for In Vivo Educational Experiments

In contrast to typical laboratory experiments, the everyday use of online educational resources by large populations and the prevalence of software infrastructure for A/B testing leads us to consider how platforms can embed in vivo experiments that do not merely support research, but ensure practical improvements to their educational components. Examples are presented of randomized experimental comparisons conducted by subsets of the authors in three widely used online educational platforms -- Khan Academy, edX, and ASSISTments. We suggest design principles for platform technology to support randomized experiments that lead to practical improvements -- enabling Iterative Improvement and Collaborative Work -- and explain the benefit of their implementation by WPI co-authors in the ASSISTments platform.

Visualizing API Usage Examples at Scale

Using existing APIs properly is a key challenge in programming, given that libraries and APIs are increasing in number and complexity. Programmers often search for online code examples in Q&A forums and read tutorials and blog posts to learn how to use a given API. However, there are often a massive number of related code examples and it is difficult for a user to understand the commonalities and variances among them, while being able to drill down to concrete details. We introduce an interactive visualization for exploring a large collection of code examples mined from open-source repositories at scale. This visualization summarizes hundreds of code examples in one synthetic code skeleton with statistical distributions for canonicalized statements and structures enclosing an API call. We implemented this interactive visualization for a set of Java APIs and found that, in a lab study, it helped users (1) answer significantly more API usage questions correctly and comprehensively and (2) explore how other programmers have used an unfamiliar API.

Teamscope: Scalable Team Evaluation via Automated Metric Mining for Communication, Organization, Execution, and Evolution

Teaching software development teams can be difficult to scale. Based on various cloud-based software development tools, Teamscope provides automated or semi-automated metrics to improve the scalability of a course with team projects. Metrics developed in Teamscope provide a synthesized view of a student team. Our preliminary results have shown the validity of these metrics. We also present a case study of applying metrics to teaching software development course in this paper.