René F. Kizilcec

Deconstructing disengagement: analyzing learner subpopulations in massive open online courses

As MOOCs grow in popularity, the relatively low completion rates of learners has been a central criticism. This focus on completion rates, however, reflects a monolithic view of disengagement that does not allow MOOC designers to target interventions or develop adaptive course features for particular subpopulations of learners. To address this, we present a simple, scalable, and informative classification method that identifies a small number of longitudinal engagement trajectories in MOOCs. Learners are classified based on their patterns of interaction with video lectures and assessments, the primary features of most MOOCs to date. In an analysis of three computer science MOOCs, the classifier consistently identifies four prototypical trajectories of engagement. The most notable of these is the learners who stay engaged through the course without taking assessments. These trajectories are also a useful framework for the comparison of learner engagement between different course structures or instructional approaches. We compare learners in each trajectory and course across demographics, forum participation, video access, and reports of overall experience. These results inform a discussion of future interventions, research, and design directions for MOOCs. Potential improvements to the classification mechanism are also discussed, including the introduction of more fine-grained analytics.

Attrition and Achievement Gaps in Online Learning

Attrition in online learning is generally higher than in traditional settings, especially in large-scale online learning environments. A systematic analysis of individual differences in attrition and performance in 20 massive open online courses (N > 67,000) revealed a geographic achievement gap and a gender achievement gap. Online learners in Africa, Asia, and Latin America scored substantially lower grades and were only half as likely to persist than those in Europe, Oceania, and Northern America. Women also exhibited lower persistence and performance than men. Yet more persistent learners were only marginally more satisfied with their achievement. The primary obstacle for most learners was finding time for the course, which was partly related to low levels of volitional control. Self-ascribed successful learners reported higher levels of goal striving, growth mindset, and feelings of social belonging than unsuccessful ones. Insights into why learners leave online courses inform models of attrition and targeted interventions to support learners achieve their goals.

The Relationship between Virtual Self Similarity and Social Anxiety.

In virtual reality (VR), it is possible to embody avatars that are dissimilar to the physical self. We examined whether embodying a dissimilar self in VR would decrease anxiety in a public speaking situation. We report the results of an observational pilot study and two laboratory experiments. In the pilot study (N = 252), participants chose an avatar to use in a public speaking task. Trait public speaking anxiety correlated with avatar preference, such that anxious individuals preferred dissimilar self-representations. In Study 1 (N = 82), differences in anxiety during a speech in front of a virtual audience were compared among participants embodying an assigned avatar whose face was identical to their real self, an assigned avatar whose face was other than their real face, or embodied an avatar of their choice. Anxiety differences were not significant, but there was a trend for lower anxiety with the assigned dissimilar avatar compared to the avatar looking like the real self. Study 2 (N = 105) was designed to explicate that trend, and further investigated anxiety differences with an assigned self or dissimilar avatar. The assigned dissimilar avatar reduced anxiety relative to the assigned self avatar for one measure of anxiety. We discuss implications for theories of self-representation as well as for applied uses of VR to treat social anxiety.

Recommending Self-Regulated Learning Strategies Does Not Improve Performance in a MOOC

Many committed learners struggle to achieve their goal of completing a Massive Open Online Course (MOOC). This work investigates self-regulated learning (SRL) in MOOCs and tests if encouraging the use of SRL strategies can improve course performance. We asked a group of 17 highly successful learners about their own strategies for how to succeed in a MOOC. Their responses were coded based on a SRL framework and synthesized into seven recommendations. In a randomized experiment, we evaluated the effect of providing those recommendations to learners in the same course (N = 653). Although most learners rated the study tips as very helpful, the intervention did not improve course persistence or achievement. Results suggest that a single SRL prompt at the beginning of the course provides insufficient support. Instead, embedding technological aids that adaptively support SRL throughout the course could better support learners in MOOCs.

How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface

The rising prevalence of algorithmic interfaces, such as curated feeds in online news, raises new questions for designers, scholars, and critics of media. This work focuses on how transparent design of algorithmic interfaces can promote awareness and foster trust. A two-stage process of how transparency affects trust was hypothesized drawing on theories of information processing and procedural justice. In an online field experiment, three levels of system transparency were tested in the high-stakes context of peer assessment. Individuals whose expectations were violated (by receiving a lower grade than expected) trusted the system less, unless the grading algorithm was made more transparent through explanation. However, providing too much information eroded this trust. Attitudes of individuals whose expectations were met did not vary with transparency. Results are discussed in terms of a dual process model of attitude change and the depth of justification of perceived inconsistency. Designing for trust requires balanced interface transparency - not too little and not too much.

Closing global achievement gaps in MOOCs

Brief interventions address social identity threat at scale Advocates for free massive open online courses (MOOCs) have heralded them as vehicles for democratizing education and bridging divides within and across countries (1). More than 25 million people enrolled in MOOCs between 2012 and 2015, including 39% from less-developed countries (LDCs) (2). But the educated and affluent in all countries enroll in and complete MOOCs at relatively higher rates (3, 4). Judged by completion rates, MOOCs do not spread benefits equitably across global regions. Rather, they reflect prevailing educational disparities between nations (see the first chart) (5). Although the global achievement gap could be caused by barriers in LDCs, such as less broadband Internet access, formal education, and English proficiency, we explore another potential but underappreciated cause. Members of LDCs may suffer from the cognitive burden of wrestling with feeling unwelcome while trying to learn and, therefore, underperform. This can be exacerbated by social identity threat, which is the fear of being seen as less capable because of ones group (6). We discuss field experiments with interventions that targeted social identity threat and caused substantial improvements in MOOC persistence and completion rates among learners in LDCs, eliminating the global achievement gap.

Estimating peer effects in networks with peer encouragement designs

Peer effects, in which the behavior of an individual is affected by the behavior of their peers, are central to social science. Because peer effects are often confounded with homophily and common external causes, recent work has used randomized experiments to estimate effects of specific peer behaviors. These experiments have often relied on the experimenter being able to randomly modulate mechanisms by which peer behavior is transmitted to a focal individual. We describe experimental designs that instead randomly assign individuals’ peers to encouragements to behaviors that directly affect those individuals. We illustrate this method with a large peer encouragement design on Facebook for estimating the effects of receiving feedback from peers on posts shared by focal individuals. We find evidence for substantial effects of receiving marginal feedback on multiple behaviors, including giving feedback to others and continued posting. These findings provide experimental evidence for the role of behaviors directed at specific individuals in the adoption and continued use of communication technologies. In comparison, observational estimates differ substantially, both underestimating and overestimating effects, suggesting that researchers and policy makers should be cautious in relying on them.

Eight-minute self-regulation intervention raises educational attainment at scale in individualist but not collectivist cultures.

Significance High attrition from educational programs is a major obstacle to social mobility and a persistent source of economic inefficiency. Over two-thirds of students entering a 2-y institution fail to earn a credential in the United States. In online courses, attrition rates are even higher. In two large field experiments, we tested the conditions under which a writing activity that facilitates goal commitment and goal-directed behavior reduces attrition in online courses. The activity raised completion rates by up to 78% for members of individualist cultures and primarily for those who contended with predictable and surmountable obstacles in the form of everyday obligations, but it was ineffective in collectivist cultures and for people contending with other types of obstacles. Academic credentials open up a wealth of opportunities. However, many people drop out of educational programs, such as community college and online courses. Prior research found that a brief self-regulation strategy can improve self-discipline and academic outcomes. Could this strategy support learners at large scale? Mental contrasting with implementation intentions (MCII) involves writing about positive outcomes associated with a goal, the obstacles to achieving it, and concrete if–then plans to overcome them. The strategy was developed in Western countries (United States, Germany) and appeals to individualist tendencies, which may reduce its efficacy in collectivist cultures such as India or China. We tested this hypothesis in two randomized controlled experiments in online courses (n = 17,963). Learners in individualist cultures were 32% (first experiment) and 15% (second experiment) more likely to complete the course following the MCII intervention than a control activity. In contrast, learners in collectivist cultures were unaffected by MCII. Natural language processing of written responses revealed that MCII was effective when a learner’s primary obstacle was predictable and surmountable, such as everyday work or family obligations but not a practical constraint (e.g., Internet access) or a lack of time. By revealing heterogeneity in MCII’s effectiveness, this research advances theory on self-regulation and illuminates how even highly efficacious interventions may be culturally bounded in their effects.

Learning innovation at scale

The rapid developments in online education raise new issues for the future of learning and universities, practical questions about what counts as good design, and new opportunities for research. This workshop brings together practitioners, learning platform innovators, and researchers who draw on a multi-disciplinary range of theory and methodology. We will share insights about the current state and next directions for research and practice in online learning and technology.

Towards Equal Opportunities in MOOCs: Affirmation Reduces Gender & Social-Class Achievement Gaps in China

The presence of achievement gaps in Massive Open Online Courses (MOOCs) implies that not everyone who can gain access to a course shares the same opportunities to succeed. This study advances research on a social psychological barrier to achievement that exists alongside important structural barriers (e.g., Internet access, insufficient prior knowledge). Learners who experience social identity threat (SIT) - a fear of being judged negatively in light of a social group they identify with - are at risk of underperforming. An initial survey identified lower-class men as an at-risk group in an English language learning MOOC for Chinese learners (N = 1,664). In a subsequent randomized experiment, an interdependent value relevance affirmation intervention raised grades, persistence, and completion rates exclusively among lower-class men - the lowest performing group in the course (N = 1,990). Efforts to establish equal opportunities in online learning should go beyond initiatives that increase access through technology to incorporate strategies that lower psychological barriers to create safe and inclusive learning environments.