Arto Hellas

Preventing Keystroke Based Identification in Open Data Sets

Large-scale courses such as Massive Online Open Courses (MOOCs) can be a great data source for researchers. Ideally, the data gathered on such courses should be openly available to all researchers. Studies could be easily replicated and novel studies on existing data could be conducted. However, very fine-grained data such as source code snapshots can contain hidden identifiers. For example, distinct typing patterns that identify individuals can be extracted from such data. Hence, simply removing explicit identifiers such as names and student numbers is not sufficient to protect the privacy of the users who have supplied the data. At the same time, removing all keystroke information would decrease the value of the shared data significantly. In this work, we study how keystroke data from a programming context could be modified to prevent keystroke latency based identification whilst still retaining information that can be used to e.g. infer programming experience. We investigate the degree of anonymization required to render identification of students based on their typing patterns unreliable. Then, we study whether the modified keystroke data can still be used to infer the programming experience of the students as a case study of whether the anonymized typing patterns have retained at least some informative value. We show that it is possible to modify data so that keystroke latency based identification is no longer accurate, but the programming experience of the students can still be inferred, i.e. the data still has value to researchers. In a broader context, our results indicate that information and anonymity are not necessarily mutually exclusive.

Biosignals reflect pair-dynamics in collaborative work: EDA and ECG study of pair-programming in a classroom environment

Collaboration is a complex phenomenon, where intersubjective dynamics can greatly affect the productive outcome. Evaluation of collaboration is thus of great interest, and can potentially help achieve better outcomes and performance. However, quantitative measurement of collaboration is difficult, because much of the interaction occurs in the intersubjective space between collaborators. Manual observation and/or self-reports are subjective, laborious, and have a poor temporal resolution. The problem is compounded in natural settings where task-activity and response-compliance cannot be controlled. Physiological signals provide an objective mean to quantify intersubjective rapport (as synchrony), but require novel methods to support broad deployment outside the lab. We studied 28 student dyads during a self-directed classroom pair-programming exercise. Sympathetic and parasympathetic nervous system activation was measured during task performance using electrodermal activity and electrocardiography. Results suggest that (a) we can isolate cognitive processes (mental workload) from confounding environmental effects, and (b) electrodermal signals show role-specific but correlated affective response profiles. We demonstrate the potential for social physiological compliance to quantify pair-work in natural settings, with no experimental manipulation of participants required. Our objective approach has a high temporal resolution, is scalable, non-intrusive, and robust.

Social Help-seeking Strategies in a Programming MOOC

Being able to seek help is a crucial part of any learning process. This includes both collaborative models such as asking for help from others as well as independent models such as using course materials and the vast resources provided by the Web. Currently, MOOC research has addressed social help-seeking within the MOOC course, either using MOOC platform tools (forum, chat) or arranging activities using external platforms (Google Hangout, Facebook groups). However, MOOC learning activities take place in a larger social ecology, including friends and teachers, general online communities and alumni communities. Using survey data from a programming MOOC, we show a typology of social learning strategies: non-use of social help-seeking, seeking help from friends and seeking help from alumni and teacher communities. We further show that students using social help-seeking strategies orient themselves more with a surface approach but are also less likely to drop the course. We conclude this work by addressing the various design possibilities identified by this work.

Patterns for Designing and Implementing an Environment for Software Start-Up Education

Todays students are prospective entrepreneurs and employees in modern, start-up like environments within established companies. In these settings, software development projects face extreme requirements in terms of innovation and attractiveness of the end-product. They also suffer severe consequences of failure such as termination of the development effort and bankruptcy. As the abilities needed in start-ups are not among those traditionally taught in universities, new knowledge and skills are required to prepare students for the volatile environment that new market entrants face. This paper reports experiences gained during seven years of teaching start-up knowledge and skills in a higher-education institution. We offer a collection of patterns that help educational institutions to design, implement and operate physical environments, curricula and teaching materials, and to plan interventions that may be required for project-based start-up education.

Stereotype Modeling for Problem-Solving Performance Predictions in MOOCs and Traditional Courses

Stereotypes are frequently used in real life to classify students according to their performance in class. In literature, we can find many references to weaker students, fast learners, struggling students, etc. Given the lack of detailed data about students, these or other kinds of stereotypes could be potentially used for user modeling and personalization in the educational context. Recent research in MOOC context demonstrated that data-driven learner stereotypes could work well for detecting and preventing student dropouts. In this paper, we are exploring the application of stereotype-based modeling to a more challenging task -- predicting student problem-solving and learning in two programming courses and two MOOCs. We explore traditional stereotypes based on readily available factors like gender or education level as well as some advanced data-driven approaches to group students based on their problem-solving behavior. Each of the approaches to form student stereotype cohorts is validated by comparing models of student learning: do students in different groups learn differently? In the search for the stereotypes that could be used for adaptation, the paper examines ten approaches. We compare the performance of these approaches and draw conclusions for future research.

A Contingency Table Derived Method for Analyzing Course Data

We describe a method for analyzing student data from online programming exercises. Our approach uses contingency tables that combine whether or not a student answered an online exercise correctly with the number of attempts that the student made on that exercise. We use this method to explore the relationship between student performance on online exercises done during semester with subsequent performance on questions in a paper-based exam at the end of semester. We found that it is useful to include data about the number of attempts a student makes on an online exercise.

Crowdsourcing programming assignments with CrowdSorcerer

Small automatically assessed programming assignments are an often used resource for learning programming. Creating sufficiently large amounts of such assignments is, however, time consuming. As a consequence, offering large quantities of practice assignments to students is not always possible. CrowdSorcerer is an embeddable open-source system that students and teachers alike can use for creating and evaluating small automatically assessed programming assignments. While creating programming assignments, the students also write simple input-output -tests, and are gently introduced to the basics of testing. Students can also evaluate the assignments of others and provide feedback on them, which exposes them to code written by others early in their education. In this article we both describe the CrowdSorcerer system and our experiences in using the system in a large undergraduate programming course. Moreover, we discuss the motivation for crowdsourcing course assignments and present some usage statistics.

Taxonomizing features and methods for identifying at-risk students in computing courses

Since computing education began, we have sought to learn why students struggle in computer science and how to identify these at-risk students as early as possible. Due to the increasing availability of instrumented coding tools in introductory CS courses, the amount of direct observational data of student working patterns has increased significantly in the past decade, leading to a flurry of attempts to identify at-risk students using data mining techniques on code artifacts. The goal of this work is to produce a systematic literature review to describe the breadth of work being done on the identification of at-risk students in computing courses. In addition to the review itself, which will summarize key areas of work being completed in the field, we will present a taxonomy (based on data sources, methods, and contexts) to classify work in the area.

Early Developmental Activities and Computing Proficiency

As countries adopt computing education for all pupils from primary school upwards, there are challenging indicators: significant proportions of students who choose to study computing at universities fail the introductory courses, and the evidence for links between formal education outcomes and success in CS is limited. Yet, as we know, some students succeed without prior computing experience. Why is this? Some argue for an innate ability, some for motivation, some for the discrepancies between the expectations of instructors and students, and some -- simply -- for how programming is being taught. All agree that becoming proficient in computing is not easy. Our research takes a novel view on the problem and argues that some of that success is influenced by early childhood experiences outside formal education. In this study, we analyzed over 1300 responses to a multi-institutional and multi-national survey that we developed. The survey captures enjoyment of early developmental activities such as childhood toys, games and pastimes between the ages 0 --- 8 as well as later life experiences with computing. We identify unifying features of the computing experiences in later life, and attempt to link these computing experiences to the childhood activities. The analysis indicates that computing proficiency should be seen from multiple viewpoints, including both skill-level and confidence. Our analysis is the first to show, we believe, that particular early childhood experiences are linked to parts of computing proficiency, namely those related to confidence with problem solving using computing technology. These are essential building blocks for more complex use. We recognize issues in the experimental design that may prevent our data showing a link between early activities and more complex computing skills, and suggest adjustments for future studies. Ultimately, we expect that this line of research will feed in to early years and primary education, and thereby improve computing education for all.

Designing and Implementing an Environment for Software Start-up Education: Patterns and Anti-Patterns

Abstract Today’s students are prospective entrepreneurs, as well as potential employees in modern, start-up-like intrapreneurship environments within established companies. In these settings, software development projects face extreme requirements in terms of innovation and attractiveness of the end-product. They also suffer severe consequences of failure such as termination of the development effort and bankruptcy. As the abilities needed in start-ups are not among those traditionally taught in universities, new knowledge and skills are required to prepare students for the volatile environment that new market entrants face. This article reports experiences gained during seven years of teaching start-up knowledge and skills in a higher-education institution. Using a design-based research approach, we have developed the Software Factory, an educational environment for experiential, project-based learning. We offer a collection of patterns and anti-patterns that help educational institutions to design, implement and operate physical environments, curricula and teaching materials, and to plan interventions that may be required for project-based start-up education.