Ari Korhonen

Empirical Study on the Effect of Achievement Badges in TRAKLA2 Online Learning Environment

Achievement badges are a form of gamification that can be used to motivate users and to encourage desired actions. In this study, we describe and evaluate the use of achievement badges in the TRAKLA2 online learning environment where students complete interactive, automatically assessed exercises about data structures and algorithms. The students activity in TRAKLA2 was logged in order to find out whether the achievement badges had an effect on their behavior. We used a between-subject experimental design where the students (N=281) were randomly divided into a treatment and a control group, with and without achievement badges. Students in the treatment group were awarded achievement badges, for example, for solving exercises with only one attempt, returning exercises early, or completing an exercise round with full points. Course grading was similar for both groups, i.e. collecting badges did not affect the final grade. Our results show that achievement badges can be used to affect the behavior of students even when the badges have no impact on the grading. Statistically significant differences in students behavior were observed with some badge types, while some badges did not seem to have such an effect. We also found that students in the two studied courses responded differently to the badges. Based on our findings, achievement badges seem like a promising method to motivate students and to encourage desired study practices.

Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies

Educational data mining and learning analytics promise better understanding of student behavior and knowledge, as well as new information on the tacit factors that contribute to student actions. This knowledge can be used to inform decisions related to course and tool design and pedagogy, and to further engage students and guide those at risk of failure. This working group report provides an overview of the body of knowledge regarding the use of educational data mining and learning analytics focused on the teaching and learning of programming. In a literature survey on mining students programming processes for 2005-2015, we observe a significant increase in work related to the field. However, the majority of the studies focus on simplistic metric analysis and are conducted within a single institution and a single course. This indicates the existence of further avenues of research and a critical need for validation and replication to better understand the various contributing factors and the reasons why certain results occur. We introduce a novel taxonomy to analyse replicating studies and discuss the importance of replicating and reproducing previous work. We describe what is the state of the art in collecting and sharing programming data. To better understand the challenges involved in replicating or reproducing existing studies, we report our experiences from three case studies using programming data. Finally, we present a discussion of future directions for the education and research community.

Characterizing research in computing education: a preliminary analysis of the literature

This paper presents a preliminary analysis of research papers in computing education. While previous analysis has explored what research is being done in computing education, this project explores how that research is being done. We present our classification system, then the results of applying it to the papers from all five years of ICER. We find that this subset of computing education research has more in common with research in information systems than with that in computer science or software engineering; and that the papers published at ICER generally appear to conform to the specified ICER requirements.

How (not) to introduce badges to online exercises

Achievement badges are increasingly used to enhance educational systems and they have been shown to affect student behavior in different ways. However, details on best practices and effective concepts to implement badges from a non-technical point of view are scarce. We implemented badges to our learning management system, used them on a large course and collected feedback from students. Based on our experiences, we present recommendations to other educators that plan on using badges.

Theoretical underpinnings of computing education research: what is the evidence?

We analyze the Computing Education Research (CER) literature to discover what theories, conceptual models and frameworks recent CER builds on. This gives rise to a broad understanding of the theoretical basis of CER that is useful for researchers working in that area, and has the potential to help CER develop its own identity as an independent field of study.n Our analysis takes in seven years of publications (2005-2011, 308 papers) in three venues that publish long research papers in computing education: the journals ACM Transactions of Computing Education (TOCE) and Computer Science Education (CSEd), and the conference International Computing Education Research Workshop (ICER). We looked at the theoretical background works that are used or extended in the papers, not just referred to when describing related work. These background works include theories, conceptual models and frameworks. For each background work we tried to identify the discipline from which it originates, to gain an understanding of how CER relates to its neighboring fields. We also identified theoretical works originating within CER itself, showing that the field is building on its own theoretical works.n Our main findings are that there is a great richness of work on which recent CER papers build; there are no prevailing theoretical or technical works that are broadly applied across CER; about half the analyzed papers build on no previous theoretical work, but a considerable share of these are building their own theoretical constructions. We discuss the significance of these findings for the whole field and conclude with some recommendations.

OpenDSA: beginning a community active-eBook project

In this paper, we present our vision for OpenDSA, an open-source, community-based effort to create a complete active-eBook for Data Structures and Algorithms courses at the undergraduate level. We define active-eBooks as going beyond classic hyper textbooks, being a close integration of text and images with interactive visualizations/simulations and assessment activities. The OpenDSA project is meant to proceed with broad participation from the CS Education community, with maximum flexibility on reuse of materials, and with the ability for a given instructor to pick and choose material from the collection and modify as desired. We discuss the goals of the project, our initial cominunity organization efforts, and the technical infrastructure that we envision for the project. Initial progress is described.

Requirements and design strategies for open source interactive computer science eBooks

Online education supported by digital courseware will radically alter higher education in ways that we cannot predict. New technologies such as MOOCs and Khan Academy have generated interest in new models for knowledge delivery. The nature of Computer Science content provides special opportunities for computer-supported delivery in both traditional and online classes. Traditional CS textbooks are likely to be replaced by online materials that tightly integrate content with visualizations and automatically assessed exercises. We refer to these new textbook-like artifacts as icseBooks (pronounced ice books), for interactive computer science electronic books. IcseBook technology will in turn impact the pedagogy used in CS courses. This report surveys the state of the field, addresses new use cases for CS pedagogy with icseBooks, and lays out a series of research questions for future study.

Automatic recognition of students' sorting algorithm implementations in a data structures and algorithms course

Computing educators often rely on black-box analysis to assess students work automatically and give feedback. This approach does not allow analyzing the quality of programs and checking if they implement the required algorithm. We introduce an instrument for recognizing and classifying algorithms (Aari) in terms of white-box testing to identify authentic students sorting algorithm implementations in a data structures and algorithms course. Aari uses machine learning techniques to classify new instances. The students were asked to submit a program to sort an array of integers in two rounds: at the beginning of the course before sorting algorithms were introduced, and after taking a lecture on sorting algorithms. We evaluated the performance of Aari with the implementations of each round separately. The results show that the sorting algorithms, which Aari has been trained to recognize, are recognized with an average accuracy of about 90%. When considering all the submitted sorting algorithm implementations (including the variations of the standard algorithms), Aari achieved an overall accuracy of 71% and 81% for the first and second round, respectively.n In addition, we analyzed the students implementations manually to gain a better understanding of the reasons of failure in the recognition process. This analysis revealed that students have many misconceptions related to sorting algorithms, which results in problematic implementations that are more inefficient compared with those of standard algorithms. We discuss these variations along with the application of the tool in an educational context, its limitations and some directions for future work.

Misconceptions in Visual Algorithm Simulation Revisited: On UI's Effect on Student Performance, Attitudes and Misconceptions

Mobile devices affect the way we access interactive learning material and exercises in the internet. There are changes both in the technologies used to implement software and in the possibilities and restrictions imposed by this platform. A set of visual algorithm simulation exercises - implemented using the JSAV library allowing them to work on both mobile and desktop machines - were tested on a CS majors data structures and algorithms course. As the problem formulation was not changed from previous years, we were able to study how changes in the UI might affect student performance and if there are any differences in student attitudes and mistakes or misconceptions detected. For the set of exercises studied, the results were in line with previous findings.

Software Architectures for Implementing Achievement Badges - Practical Experiences

There are multiple commercial and non-commercial products available to integrate gamification aspects to existing services. Some of these are platform dependent whilst others are more general purpose. Commercial systems come with some problems - for example, lack of control and privacy issues. To avoid these problems, we created two iterations of badge systems and tested both of them on large courses (ca. 300 students each). In this paper, we present these systems and evaluate their merits and flaws. Based on our experiences, we present design principles on how to implement badge systems to existing online learning environments.