Carly D. Robinson

Forecasting student achievement in MOOCs with natural language processing

Student intention and motivation are among the strongest predictors of persistence and completion in Massive Open Online Courses (MOOCs), but these factors are typically measured through fixed-response items that constrain student expression. We use natural language processing techniques to evaluate whether text analysis of open responses questions about motivation and utility value can offer additional capacity to predict persistence and completion over and above information obtained from fixed-response items. Compared to simple benchmarks based on demographics, we find that a machine learning prediction model can learn from unstructured text to predict which students will complete an online course. We show that the model performs well out-of-sample, compared to a standard array of demographics. These results demonstrate the potential for natural language processing to contribute to predicting student success in MOOCs and other forms of open online learning.

Questionnaires as interventions: can taking a survey increase teachers’ openness to student feedback surveys?

Abstract Administrators often struggle in getting teachers to trust their school’s evaluation practices – a necessity if teachers are to learn from the feedback they receive. We attempted to bolster teachers’ support for receiving evaluative feedback from a particularly controversial source: student-perception surveys. For our intervention, we took one of two approaches to asking 309 teachers how they felt about students evaluating their teaching practice. Control participants responded only to core questions regarding their attitudes towards student-perception surveys. Meanwhile, treatment participants were first asked whether teachers should evaluate administrators in performance reviews and were then asked the core items about student-perception surveys. Congruent with cognitive dissonance theory, this juxtaposition of questions bolstered treatment teachers’ support for using student surveys in teacher evaluations relative to the control group. We discuss the implications of these findings with respect to increasing teacher openness to alternative evaluation approaches, and consider whether surveys show promise as a vehicle for delivering interventions.

Reducing Student Absenteeism in the Early Grades by Targeting Parental Beliefs

Attendance in kindergarten and elementary school robustly predicts student outcomes. Despite this well-documented association, there is little experimental research on how to reduce absenteeism in the early grades. This paper presents results from a randomized field experiment in ten school districts evaluating the impact of a low-cost, parent-focused intervention on student attendance in grades K-5. The intervention targeted commonly held parental misbeliefs undervaluing the importance of regular K-5 attendance as well as the number of school days their child had missed. The intervention decreased chronic absenteeism by 15%. This study presents the first experimental evidence on how to improve student attendance in grades K-5 at scale, and has implications for increasing parental involvement in education.

Some Middle School Students Want Behavior Commitment Devices (but Take-Up Does Not Affect Their Behavior)

Commitment devices impose costs on ones future self for failing to follow through on ones intentions, offer no additional benefit to ones future self for following through on the intention, and people voluntarily enroll in them. Enrollment in commitment devices reflects self-awareness that one may lack sufficient self-control to fulfill ones intentions. There is little experimental research on whether school-age children possess the self-awareness necessary to enroll in a commitment device, despite evidence that children and young adolescents have many positive intentions that they fail to live up to, such as demonstrating improved school conduct or eating healthier. We report the first field experiment examining the demand for, and impact of, commitment devices among middle school students. We offered students a commitment device that imposed future costs for failing to improve in-school conduct. When presented with the opportunity to actively opt-in (default not enrolled), over one-third of students elected to enroll. When presented with the opportunity to actively opt-out (default enrolled), more than half elected to remain enrolled, showing that changing default options can increase commitment device enrollment. Despite demand for the self-control strategy, taking-up the commitment device did not affect student behavior. These findings have implications for youth-based behavioral interventions broadly, as well as those focused on eating behaviors.

The Demotivating Effect (and Unintended Message) of Retrospective Awards

It is common for organizations to offer awards to motivate individual behavior, yet few empirical studies evaluate their effectiveness in the field. We report a randomized field experiment (N = 15,329) that tests the impact of two types of symbolic awards on student attendance: pre-announced awards (prospective) and surprise awards (retrospective). Contrary to our pre-registered hypotheses, prospective awards had no impact while the retrospective awards decreased subsequent attendance. Survey studies provide evidence suggesting that receiving retrospective awards may demotivate the behavior being awarded by inadvertently signaling (a) that recipients have performed the behavior more than their peers have; and (b) that recipients have performed the behavior to a greater degree than was organizationally expected. A school leaders survey shows that awards for attendance are common, and that the organizational leaders who offer these awards are unaware of their potential demotivating impact.

Mitigating Illusory Results through Preregistration in Education.

ABSTRACT Like performance-enhancing drugs inflating apparent athletic achievements, several common social science practices contribute to the production of illusory results. In this article, we examine the processes that lead to illusory findings and describe their consequences. We borrow from an approach used increasingly by other disciplines—the norm of preregistering studies. Specifically, we examine how this practice of publicly posting documentation of ones prespecified hypotheses and other key decisions of a study prior to study implementation or data analysis could improve scientific integrity within education. In an attempt to develop initial guidelines to facilitate preregistrations in education, we discuss the types of studies that ought to be preregistered and the logistics of how educational researchers might execute preregistrations. We conclude with ideas for how researchers, reviewers, and the field of education more broadly might speed the adoption of this new norm.

Mitigating Illusory Results Through Pre-Registration in Education

Like performance enhancing drugs inflating apparent athletic achievements, several common social science practices contribute to the production of illusory results. In this article, we examine the processes that lead to illusory findings and describe their consequences. Borrowing from an approach used increasingly by other disciplines—the norm of pre-registering studies—we examine how this practice could improve scientific integrity within education. As an initial attempt to develop guidelines that facilitate pre-registrations in education, we discuss the types of studies that ought to be pre-registered and the logistics of how educational researchers might execute pre-registrations. We conclude with ideas for how researchers, reviewers, and the field of education more broadly might speed the adoption of this new norm.