Justin Reich

Rebooting MOOC Research

Improve assessment, data sharing, and experimental design The chief executive officer of edX, Anant Agarwal, declared that Massive Open Online Courses (MOOCs) should serve as “particle accelerator for learning” (1). MOOCs provide new sources of data and opportunities for large-scale experiments that can advance the science of learning. In the years since MOOCs first attracted widespread attention, new lines of research have begun, but findings from these efforts have had few implications for teaching and learning. Big data sets do not, by virtue of their size, inherently possess answers to interesting questions. For MOOC research to advance the science of learning, researchers, course developers, and other stakeholders must advance the field along three trajectories: from studies of engagement to research about learning, from investigations of individual courses to comparisons across contexts, and from a reliance on post hoc analyses to greater use of multidisciplinary, experimental design.

The State of Wiki Usage in U.S. K–12 Schools Leveraging Web 2.0 Data Warehouses to Assess Quality and Equity in Online Learning Environments

To document wiki usage in U.S. K–12 settings, this study examined a representative sample drawn from a population of nearly 180,000 wikis. The authors measured the opportunities wikis provide for students to develop 21st-century skills such as expert thinking, complex communication, and new media literacy. The authors found four types of wiki usage: (a) trial wikis and teacher resource-sharing sites (40%), (b) teacher content-delivery sites (34%), (c) individual student assignments and portfolios (25%), and (d) collaborative student presentations and workspaces (1%). Wikis created in schools serving low-income students have fewer opportunities for 21st-century skill development and shorter lifetimes than wikis from schools serving affluent students. This study illustrates the exciting potential that Web 2.0 data warehouses offer for educational research.

Privacy, anonymity, and big data in the social sciences

Open data has tremendous potential for science, but, in human subjects research, there is a tension between privacy and releasing high-quality open data. Federal law governing student privacy and the release of student records suggests that anonymizing student data protects student privacy. Guided by this standard, we de-identified and released a data set from 16 MOOCs (massive open online courses) from MITx and HarvardX on the edX platform. In this article, we show that these and other de-identification procedures necessitate changes to data sets that threaten replication and extension of baseline analyses. To balance student privacy and the benefits of open data, we suggest focusing on protecting privacy without anonymizing data by instead expanding policies that compel researchers to uphold the privacy of the subjects in open data sets. If we want to have high-quality social science research and also protect the privacy of human subjects, we must eventually have trust in researchers. Otherwise, we’ll always have the strict tradeoff between anonymity and science illustrated here.

Democratizing Education? Examining Access and Usage Patterns in Massive Open Online Courses

Toward a level playing field? Do free learning resources benefit the disadvantaged and decrease gaps between rich and poor? Hansen and Reich studied the relationships between socioeconomic status (SES) and enrollment in and completion of free Massive Open Online Courses (MOOCs) offered by Harvard and MIT. Students from low-SES backgrounds were less likely to enroll in MOOCs and earn a certificate than their high-SES peers. Thus, although there are many free online learning opportunities, it is not safe to assume that they will “level the playing field.” Science, this issue p. 1245 The availability of online courses may not reduce the effects of socioeconomic disparities for adolescents and young adults. Massive open online courses (MOOCs) are often characterized as remedies to educational disparities related to social class. Using data from 68 MOOCs offered by Harvard and MIT between 2012 and 2014, we found that course participants from the United States tended to live in more-affluent and better-educated neighborhoods than the average U.S. resident. Among those who did register for courses, students with greater socioeconomic resources were more likely to earn a certificate. Furthermore, these differences in MOOC access and completion were larger for adolescents and young adults, the traditional ages where people find on-ramps into science, technology, engineering, and mathematics (STEM) coursework and careers. Our findings raise concerns that MOOCs and similar approaches to online learning can exacerbate rather than reduce disparities in educational outcomes related to socioeconomic status.

Closing global achievement gaps in MOOCs

Brief interventions address social identity threat at scale Advocates for free massive open online courses (MOOCs) have heralded them as vehicles for democratizing education and bridging divides within and across countries (1). More than 25 million people enrolled in MOOCs between 2012 and 2015, including 39% from less-developed countries (LDCs) (2). But the educated and affluent in all countries enroll in and complete MOOCs at relatively higher rates (3, 4). Judged by completion rates, MOOCs do not spread benefits equitably across global regions. Rather, they reflect prevailing educational disparities between nations (see the first chart) (5). Although the global achievement gap could be caused by barriers in LDCs, such as less broadband Internet access, formal education, and English proficiency, we explore another potential but underappreciated cause. Members of LDCs may suffer from the cognitive burden of wrestling with feeling unwelcome while trying to learn and, therefore, underperform. This can be exacerbated by social identity threat, which is the fear of being seen as less capable because of ones group (6). We discuss field experiments with interventions that targeted social identity threat and caused substantial improvements in MOOC persistence and completion rates among learners in LDCs, eliminating the global achievement gap.

Forecasting student achievement in MOOCs with natural language processing

Student intention and motivation are among the strongest predictors of persistence and completion in Massive Open Online Courses (MOOCs), but these factors are typically measured through fixed-response items that constrain student expression. We use natural language processing techniques to evaluate whether text analysis of open responses questions about motivation and utility value can offer additional capacity to predict persistence and completion over and above information obtained from fixed-response items. Compared to simple benchmarks based on demographics, we find that a machine learning prediction model can learn from unstructured text to predict which students will complete an online course. We show that the model performs well out-of-sample, compared to a standard array of demographics. These results demonstrate the potential for natural language processing to contribute to predicting student success in MOOCs and other forms of open online learning.

The Civic Mission of MOOCs: Measuring Engagement across Political Differences in Forums

In this study, we develop methods for computationally measuring the degree to which students engage in MOOC forums with other students holding different political beliefs. We examine a case study of a single MOOC about education policy, Saving Schools, where we obtain measures of student education policy preferences that correlate with political ideology. Contrary to assertions that online spaces often become echo chambers or ideological silos, we find that students in this case hold diverse political beliefs, participate equitably in forum discussions, directly engage (through replies and upvotes) with students holding opposing beliefs, and converge on a shared language rather than talking past one another. Research that focuses on the civic mission of MOOCs helps ensure that open online learning engages the same breadth of purposes that higher education aspires to serve.

The Life Between Big Data Log Events: Learners’ Strategies to Overcome Challenges in MOOCs

Big data from massive open online courses (MOOCs) have enabled researchers to examine learning processes at almost infinite levels of granularity. Yet, such data sets do not track every important element in the learning process. Many strategies that MOOC learners use to overcome learning challenges are not captured in clickstream and log data. In this study, we interviewed 92 MOOC learners to better understand their worlds, investigate possible mechanisms of student attrition, and extend conversations about the use of big data in education. Findings reveal three important domains of the experience of MOOC students that are absent from MOOC tracking logs: the practices at learners’ workstations, learners’ activities online but off-platform, and the wider social context of their lives beyond the MOOC. These findings enrich our understanding of learner agency in MOOCs, clarify the spaces in-between recorded tracking log events, and challenge the view that MOOC learners are disembodied autodidacts.

Due dates in MOOCs: does stricter mean better?

Massive Open Online Courses (MOOCs) employ a variety of components to engage students in learning (eg. videos, forums, quizzes). Some components are graded, which means that they play a key role in a students final grade and certificate attainment. It is not yet clear how the due date structure of graded components affects student outcomes including academic performance and alternative modes of learning of students. Using data from HarvardX and MITx, Harvards and MITs divisions for online learning, we study the structure of due dates on graded components for 10 completed MOOCs. We find that stricter due dates are associated with higher certificate attainment rates but fewer students who join late being able to earn a certificate. Our findings motivate further studies of how the use of graded components and deadlines affects academic and alternative learning of MOOC students, and can help inform the design of online courses.

Using and Designing Platforms for In Vivo Educational Experiments

In contrast to typical laboratory experiments, the everyday use of online educational resources by large populations and the prevalence of software infrastructure for A/B testing leads us to consider how platforms can embed in vivo experiments that do not merely support research, but ensure practical improvements to their educational components. Examples are presented of randomized experimental comparisons conducted by subsets of the authors in three widely used online educational platforms -- Khan Academy, edX, and ASSISTments. We suggest design principles for platform technology to support randomized experiments that lead to practical improvements -- enabling Iterative Improvement and Collaborative Work -- and explain the benefit of their implementation by WPI co-authors in the ASSISTments platform.