Christopher G. Brinton

Learning about Social Learning in MOOCs: From Statistical Analysis to Generative Model

We study user behavior in the courses offered by a major massive online open course (MOOC) provider during the summer of 2013. Since social learning is a key element of scalable education on MOOC and is done via online discussion forums, our main focus is on understanding forum activities. Two salient features of these activities drive our research: (1) high decline rate: for each course studied, the volume of discussion declined continuously throughout the duration of the course; (2) high-volume, noisy discussions: at least 30 percent of the courses produced new threads at rates that are infeasible for students or teaching staff to read through. Further, a substantial portion of these discussions are not directly course-related. In our analysis, we investigate factors that are associated with the decline of activity on MOOC forums, and we find effective strategies to classify threads and rank their relevance. Specifically, we first use linear regression models to analyze the forum activity count data over time, and make a number of observations; for instance, the teaching staffs active participation in the discussions is correlated with an increase in the discussion volume but does not slow down the decline rate. We then propose a unified generative model for the discussion threads, which allows us both to choose efficient thread classifiers and to design an effective algorithm for ranking thread relevance. Further, our algorithm is compared against two baselines using human evaluation from Amazon Mechanical Turk.

MOOC performance prediction via clickstream data and social learning networks

We study student performance prediction in Massive Open Online Courses (MOOCs), where the objective is to predict whether a user will be Correct on First Attempt (CFA) in answering a question. In doing so, we develop novel techniques that leverage behavioral data collected by MOOC platforms. Using video-watching clickstream data from one of our MOOCs, we first extract summary quantities (e.g., fraction played, number of pauses) for each user-video pair, and show how certain intervals/sets of values for these behaviors quantify that a pair is more likely to be CFA or not for the corresponding question. Motivated by these findings, our methods are designed to determine suitable intervals from training data and to use the corresponding success estimates as learning features in prediction algorithms. Tested against a large set of empirical data, we find that our schemes outperform standard algorithms (i.e., without behavioral data) for all datasets and metrics tested. Moreover, the improvement is particularly pronounced when considering the first few course weeks, demonstrating the “early detection” capability of such clickstream data. We also discuss how CFA prediction can be used to depict graphs of the Social Learning Network (SLN) of students, which can help instructors manage courses more effectively.

Individualization for Education at Scale: MIIC Design and Preliminary Evaluation

We present the design, implementation, and preliminary evaluation of our Adaptive Educational System (AES): the Mobile Integrated and Individualized Course (MIIC). MIIC is a platform for personalized course delivery which integrates lecture videos, text, assessments, and social learning into a mobile native app, and collects clickstream-level behavioral measurements about each student as they interact with the material. These measurements can subsequently be used to update the students user model, which can in turn be used to determine the content adaptation. Recruiting students from one of our Massive Open Online Courses (MOOCs), we have conducted two preliminary trials with MIIC, in which we found (i) that the majority of students (70 percent) preferred MIIC overall to a one-size-fits-all (OSFA) presentation of the same material, (ii) that the mean level of engagement, when quantified as the number of pages viewed, was statistically higher (by 72 percent) among students using MIIC than among OSFA, and (iii) that the integrated multimedia learning features were generally favorable among the students (e.g., 87 percent found the videos helpful).

Social learning networks: A brief survey

Social Learning Network (SLN) is a type of social network among students, instructors, and modules of learning. It consists of the dynamics of learning behavior over a variety of graphs representing the relationships among the people and processes involved in learning. Recent innovations in online education, including open online courses at various scales, in flipped classroom instruction, and in professional and corporate training have presented interesting questions about SLN. Collecting, analyzing, and leveraging data about SLN lead to potential answers to these questions, with help from a convergence of modeling languages and design methods, such as social network theory, science of learning, and education information technology. This survey article overviews some of these topics, including prediction, recommendation, and personalization, in this emergent research area.

Mining MOOC Clickstreams: Video-Watching Behavior vs. In-Video Quiz Performance

Student video-watching behavior and quiz performance are studied in two Massive Open Online Courses (MOOCs). In doing so, two frameworks are presented by which video-watching clickstreams can be represented: one based on the sequence of events created, and another on the sequence of positions visited. With the event-based framework, recurring subsequences of student behavior are extracted, which contain fundamental characteristics such as reflecting (i.e., repeatedly playing and pausing) and revising (i.e., plays and skip backs). It is found that some of these behaviors are significantly correlated with changes in the likelihood that a student will be Correct on First Attempt (CFA) or not in answering quiz questions, and in ways that are not necessarily intuitive. Then, with the position-based framework, models of quiz performance are devised based on positions visited in a video. In evaluating these models through CFA prediction, it is found that three of them can substantially improve prediction quality, which underlines the ability to relate this type of behavior to quiz scores. Since this prediction considers videos individually, these benefits also suggest that these models are useful in situations where there is limited training data, e.g., for early detection or in short courses.

Social learning networks: Efficiency optimization for MOOC forums

A Social Learning Network (SLN) emerges when users exchange information on educational topics with structured interactions. The recent proliferation of massively scaled online (human) learning, such as Massive Open Online Courses (MOOCs), has presented a plethora of research challenges surrounding SLN. In this paper, we ask: How efficient are these networks? We propose a framework in which SLN efficiency is determined by comparing user benefit in the observed network to a benchmark of maximum utility achievable through optimization. Our framework defines the optimal SLN through utility maximization subject to a set of constraints that can be inferred from the network. Through evaluation on four MOOC discussion forum datasets and optimizing over millions of variables, we find that SLN efficiency can be rather low (from 68% to 82% depending on the specific parameters and dataset), which indicates that much can be gained through optimization. We find that the gains in global utility (i.e., average across users) can be obtained without making the distribution of local utilities (i.e., utility of individual users) less fair. We also discuss ways of realizing the optimal network in practice, through curated news feeds in online SLN.

An Intelligent Satellite Multicast and Caching Overlay for CDNs to Improve Performance in Video Applications

Over the past decade, video has become the dominant form of traffic consumed over content delivery networks (CDNs). This trend, coupled with the ever-increasing subscriber base, has caused an explosion of data demands in a wide variety of scenarios. Such trends have resulted in heightened levels of congestion within today’s terrestrial networks and are expected to become more acute in the coming years. To combat network congestion, we propose a satellite-based overlay for existing terrestrial CDNs. Satellite networking has distinct advantages over terrestrial networks in being able to distribute delay-tolerant high bandwidth content across a wide geographic area simultaneously, with few limitations to the distance between requestor and source, nor the number of locations being served. Additionally, our solution calls for cache storage at local proxy servers one-hop from the end users, which in most instances will improve the response time of current network architectures. The proposed cache algorithm leverages the homogeneous coverage area provided by satellite to allow each proxy server to compare its local network view to the global picture, learn the popularity distributions quickly, and make its own caching decisions. Through simulations of two CDN case studies Cellular and Video on Demand we find that multicasting can provide significant reductions in required network bandwidth as compared to terrestrial-based unicast, for situations dominated by video traffic. Further, by leveraging advantages offered by our caching algorithm, we show that the multicast solution scales well, both with increasing cache storage and coverage area. Our solution appears robust as relevant traffic parameters, such as heavy-tail characteristics and global file popularity, are varied. The work presented in this paper is the result of an ongoing collaboration between Princeton University and SES. We believe that our solution incorporates the technologies best suited for the networking challenges being faced today and is forward looking in its ability to scale with demand, content type and size, which enables new market opportunities for the satellite industry.

Behavior in social learning networks: Early detection for online short-courses

We study learning outcome prediction for online courses. Whereas prior work has focused on semester-long courses with frequent student assessments, we focus on short-courses that have single outcomes assigned by instructors at the end. The lack of performance data makes the behavior of learners, captured as they interact with course content and with one another in Social Learning Networks (SLN), essential for prediction. Our method defines several (machine) learning features based on behaviors collected on the modes of (human) learning in a course, and uses them in appropriate classifiers. Through evaluation on data captured from three two-week courses hosted through our delivery platforms, we make three key observations: (i) behavioral data is predictive of learning outcomes in short-courses (our classifiers achieving AUCs ≥ 0.8 after the two weeks), (ii) it has an early detection capability (AUCs ≥ 0.7 with the first week of data), and (iii) the content features have an “earliest” detection capability (with higher AUC in the first few days), while the SLN features become the more predictive set over time, as the network matures. We also discuss how our method can generate behavioral analytics for instructors.

Behavior-Based Grade Prediction for MOOCs Via Time Series Neural Networks

We present a novel method for predicting the evolution of a students grade in massive open online courses (MOOCs). Performance prediction is particularly challenging in MOOC settings due to per-student assessment response sparsity and the need for personalized models. Our method overcomes these challenges by incorporating another, richer form of data collected from each student—lecture video-watching clickstreams—into the machine learning feature set, and using that to train a time series neural network that learns from both prior performance and clickstream data. Through evaluation on two MOOC datasets, we find that our algorithm outperforms a baseline of average past performance by more than 60% on average, and a lasso regression baseline by more than 15%. Moreover, the gains are higher when the student has answered fewer questions, underscoring their ability to provide instructors with early detection of struggling and/or advanced students. We also show that despite these gains, when taken alone, none of the behavioral features are particularly correlated with performance, emphasizing the need to consider their combined effect and nonlinear predictors. Finally, we discuss how course instructors can use these predictive learning analytics to stage student interventions.

Learner Behavioral Feature Refinement and Augmentation Using GANs

Learner behavioral data (e.g., clickstream activity logs) collected by online education platforms contains rich information about learners and content, but is often highly redundant. In this paper, we study the problem of learning low-dimensional, interpretable features from this type of raw, high-dimensional behavioral data. Based on the premise of generative adversarial networks (GANs), our method refines a small set of human-crafted features while also generating a set of additional, complementary features that better summarize the raw data. Through experimental validation on a real-world dataset that we collected from an online course, we demonstrate that our method leads to features that are both predictive of learner quiz scores and closely related to human-crafted features.