Marcelo Worsley

Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming

New high-frequency, automated data collection and analysis algorithms could offer new insights into complex learning processes, especially for tasks in which students have opportunities to generate unique open-ended artifacts such as computer programs. These approaches should be particularly useful because the need for scalable project-based and student-centered learning is growing considerably. In this article, we present studies focused on how students learn computer programming, based on data drawn from 154,000 code snapshots of computer programs under development by approximately 370 students enrolled in an introductory undergraduate programming course. We use methods from machine learning to discover patterns in the data and try to predict final exam grades. We begin with a set of exploratory experiments that use fully automated techniques to investigate how much students change their programming behavior throughout all assignments in the course. The results show that students’ change in programming patterns is only weakly predictive of course performance. We subsequently hone in on 1 single assignment, trying to map students’ learning process and trajectories and automatically identify productive and unproductive (sink) states within these trajectories. Results show that our process-based metric has better predictive power for final exams than the midterm grades. We conclude with recommendations about the use of such methods for assessment, real-time feedback, and course improvement.

Towards the development of multimodal action based assessment

In this paper, we describe multimodal learning analytics techniques for understanding and identifying expertise as students engage in a hands-on building activity. Our techniques leverage process-oriented data, and demonstrate how this temporal data can be used to study student learning. The proposed techniques introduce useful insights in how to segment and analyze gesture- and action-based generally, and may also be useful for other sources of process rich data. Using this approach we uncover new ideas about how experts engage in building activities. Finally, a primary objective of this work is to motivate additional research and development in the area of authentic, automated, process-oriented assessments.

Multimodal learning analytics: enabling the future of learning through multimodal data analysis and interfaces

Project-based learning has found its way into a range of formal and informal learning environments. However, systematically assessing these environments remains a significant challenge. Traditional assessments, which focus on learning outcomes, seem incongruent with the process-oriented goals of project-based learning. Multimodal interfaces and multimodal learning analytics hold significant promise for assessing learning in open-ended learning environments. With its rich integration of a multitude of data streams and naturalistic interfaces, this area of research may help usher in a new wave of education reform by supporting alternative modes of learning.

Leveraging multimodal learning analytics to differentiate student learning strategies

Multimodal analysis has had demonstrated effectiveness in studying and modeling several human-human and human-computer interactions. In this paper, we explore the role of multimodal analysis in the service of studying complex learning environments. We compare uni-modal and multimodal; manual and semi-automated methods for examining how students learn in a hands-on, engineering design context. Specifically, we compare human annotations, speech, gesture and electro-dermal activation data from a study (N=20) where student participating in two different experimental conditions. The experimental conditions have already been shown to be associated with differences in learning gains and design quality. Hence, one objective of this paper is to identify the behavioral practices that differed between the two experimental conditions, as this may help us better understand how the learning interventions work. An additional objective is to provide examples of how to conduct learning analytics research in complex environments and compare how the same algorithm, when used with different forms of data can provide complementary results.

Multimodal Learning Analytics as a Tool for Bridging Learning Theory and Complex Learning Behaviors

The recent emergence of several low-cost, high resolution, multimodal sensors has greatly facilitated the ability for researchers to capture a wealth of data across a variety of contexts. Over the past few years, this multimodal technology has begun to receive greater attention within the learning community. Specifically, the Multimodal Learning Analytics community has been capitalizing on new sensor technology, as well as the expansion of tools for supporting computational analysis, in order to better understand and improve student learning in complex learning environments. However, even as the data collection and analysis tools have greatly eased the process, there remain a number of considerations and challenges in framing research in such a way that it lends to the development of learning theory. Moreover, there are a multitude of approaches that can be used for integrating multimodal data, and each approach has different assumptions and implications. In this paper, I describe three different types of multimodal analyses, and discuss how decisions about data integration and fusion have a significant impact on how the research relates to learning theories.

1st international workshop on multimodal learning analytics: extended abstract

This summary describes the 1st International Workshop on Multimodal Learning Analytics. This area of study brings together the technologies of multimodal analysis with the learning sciences. The intersection of these domains should enable researchers to foster an improved understanding of student learning, lead to the creation of more natural and enriching learning interfaces, and motivate the development of novel techniques for tackling challenges that are specific of education.

Multimodal interactive spaces: MagicTV and magicMAP

Through the growing popularity of voice-enabled search, multimodal applications are finally starting to get into the hands of consumers. However, these applications are principally for mobile platforms and generally involve highly-moded interaction where the user has to click or hold a button in order to speak. Significant technical challenges remain in bringing multimodal interaction to other environments such as smart living rooms and classrooms, where users speech and gesture is directed toward large displays or interactive kiosks and the microphone and other sensors are ‘always on’. In this demonstration, we present a framework combining low cost hardware and open source software that lowers the barrier of entry for exploration of multimodal interaction in smart environments. Specifically, we will demonstrate the combination of infrared tracking, face detection, and open microphone speech recognition for media search (magicTV) and map navigation (magicMap).

Programming pathways: A technique for analyzing novice programmers' learning trajectories

Introductory computer science courses are a valuable resource to students of all disciplines. While we often look at students’ end products to judge their proficiency, little analysis is done on the most integral aspect of learning to programming, the process. We also have a hard time quantifying how students’ programming changes over the course of a semester. In order to address these we show how a process-oriented analysis can identify meaningful trends in how programmers develop proficiency across various assignments.

Multimodal learning analytics data challenges

This is a proposal for organizing a Multimodal Learning Analytics (MLA) data challenge as part of the workshop offering of the Learning Analytics and Knowledge (LAK) conference. It explains the motivation of the event, its objectives, target groups, expected format, organization, dissemination strategy and schedule.

Using learning analytics to study cognitive disequilibrium in a complex learning environment

Cognitive disequilibrium has received significant attention for its role in fostering student learning in intelligent tutoring systems and in complex learning environments. In this paper, we both add to and extend this discussion by analyzing the emergence of four affective states associated with disequilibrium: joy, surprise, neutrality and confusion; in a collaborative hands-on, engineering design task. Specifically, we conduct a comparison between two learning strategies to make salient how the strategies are associated with different affective states. This comparison is grounded in the construction of a probabilistic model of student affective state as defined by the frequency of each state, and the rate of transition between affective states. Through this comparison we confirm prior research that highlights the importance of confusion as a marker of knowledge construction, but put to question the notion that surprise is a significant mediator of cognitive disequilibrium. Overall, we show how modeling learner affect is useful for understanding and improving learning in complex, hands-on learning environments.