Characterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites
1111Characterizing Student Engagement Moods for DropoutPrediction in Question Pool Websites
REZA HADI MOGAVI,
Hong Kong University of Science and Technology, Hong Kong SAR
XIAOJUAN MA,
Hong Kong University of Science and Technology, Hong Kong SAR
PAN HUI,
Hong Kong University of Science and Technology & University of Helsinki, Hong Kong SAR &FinlandProblem-Based Learning (PBL) is a popular approach to instruction that supports students to get hands-ontraining by solving problems. Question Pool websites (QPs) such as LeetCode, Code Chef, and Math Playgroundhelp PBL by supplying authentic, diverse, and contextualized questions to students. Nonetheless, empiricalfindings suggest that 40% to 80% of students registered in QPs drop out in less than two months. This researchis the first attempt to understand and predict student dropouts from QPs via exploiting students’ engagementmoods. Adopting a data-driven approach, we identify five different engagement moods for QP students, whichare namely challenge-seeker , subject-seeker , interest-seeker , joy-seeker , and non-seeker . We find that studentshave collective preferences for answering questions in each engagement mood, and deviation from thosepreferences increases their probability of dropping out significantly. Last but not least, this paper contributesby introducing a new hybrid machine learning model (we call Dropout-Plus) for predicting student dropoutsin QPs. The test results on a popular QP in China, with nearly 10K students, show that Dropout-Plus canexceed the rival algorithms’ dropout prediction performance in terms of accuracy, F1-measure, and AUC. Wewrap up our work by giving some design suggestions to QP managers and online learning professionals toreduce their student dropouts.CCS Concepts: • Human-centered computing → HCI theory, concepts and models ; •
Applied comput-ing → Interactive learning environments ; •
Computing methodologies → Machine learning approaches .Additional Key Words and Phrases: Question Pool website (QP), online judge, Problem-Based Learning (PBL),online learning, engagement mood, dropout prediction.
ACM Reference Format:
Reza Hadi Mogavi, Xiaojuan Ma, and Pan Hui. 2021. Characterizing Student Engagement Moods for DropoutPrediction in Question Pool Websites.
J. ACM
37, 4, Article 111 (August 2021), 22 pages. https://doi.org/10.1145/1122445.1122456
Problem-Based Learning (PBL) is a student-centered approach to instruction where students learnthrough solving problems [82]. Question Pool websites such as LeetCode, Code Chef, Timus, Jutge,and Math Playground support PBL by supplying students with a variety of questions, quizzes, andcompetitions in different subjects [85, 118, 120]. However, empirical statistics from these websites
Authors’ addresses: Reza Hadi Mogavi, [email protected], Hong Kong University of Science and Technology, HongKong SAR; Xiaojuan Ma, [email protected], Hong Kong University of Science and Technology, Hong Kong SAR; Pan Hui,[email protected], Hong Kong University of Science and Technology & University of Helsinki, Hong Kong SAR & Finland.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without feeprovided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requiresprior specific permission and/or a fee. Request permissions from [email protected].© 2021 Association for Computing Machinery.0004-5411/2021/8-ART111 $15.00https://doi.org/10.1145/1122445.1122456 J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. a r X i v : . [ c s . H C ] F e b show that 40% to 80% of the registered students in QPs tend to drop out before completing theirsecond-month of membership [22, 54, 64, 87, 106]. Having said this, practical insights into thisphenomenon can help educators and online learning professionals to improve their QP designsand reduce dropouts.Nevertheless, the majority of empirical studies to date in the area of computer-mediated edu-cation are only focused on studying dropouts from Massive Open Online Courses (MOOCs) andCommunity Question Answering (CQA) websites [25, 72, 76, 102]. Therefore, there is a researchgap in the literature for studying student dropouts in comparatively new platforms like QPs. Ourresearch aims to fill this gap and inspect the problem of student dropouts in QPs through thelens of student engagement moods. By doing so, we draw Human-Computer Interaction (HCI)and Computer Supported Cooperative Work (CSCW) researchers’ attention to the importance ofpersonalization in QPs. More formally, this work answers three research questions as follows: • RQ1:
What are student engagement moods in QPs? • RQ2:
How are student engagement moods and dropout rates correlated? • RQ3:
Can student engagement moods help to predict student dropouts more precisely?We utilize a probabilistic graphical model, known as Hidden Markov Model (HMM), to extractand visually distinguish different student engagement moods in QPs. We identify five dominantstudent engagement moods, which are (E1) challenge-seeker , (E2) subject-seeker , (E3) interest-seeker , (E4) joy-seeker , and (E5) non-seeker . We distinguish each mood according to students’ data-drivenbehavioral patterns that emerge in the process of interacting with QPs. We are inspired by theHexad user types of Tondello et al. (see [108]) for naming the extracted engagement moods, butthe context and concepts we introduce are genuine and specialized for QPs.To the best of our knowledge, this work is the first research that casts a typology for studentbehaviors in QPs. By adopting a data-driven approach, we identify some distinctive behavioralpatterns for different QP students. For example, when students are in the challenge-seeker mood,they search for challenging types of questions that are commensurate with their high-level skills.Students who are in the subject-seeker mood are described best as mission or task-oriented individ-uals. Interestingly, they do not search much to find their questions and often restrict themselvesto a predefined study plan around specific subject matter and contexts. Students in this mood aremore in need of a mentor, guide, or a study plan to keep them focused and help them find theirquestions easily. When students are in an interest-seeker mood, they rummage around for a varietyof topics to find their questions of interest. However, they do not chase the challenging questionsas challenge-seekers do.Furthermore, we notice that the students in joy-seeking and non-seeking moods are not ascommitted as students in the other moods to study and exercise their knowledge. Joy-seekerstudents have a high tendency to game the platform or misuse it for purposes other than education[7, 9, 31]. Technically speaking, students who exploit the platform’s properties rather than theirknowledge or skills to become successful in an educational platform are considered to be gamingthe platform [8]. Finally, students in a non-seeker mood tend to leave the platform earlier thanstudents in other moods. These students are seldom determined to answer any questions. They onlycheck the platform to see what is new and if any questions can attract their attention by chance.These findings underline the familiar point that a one-size design QP does not fit all students[28, 65, 104].We also find that students have collective preferences for answering questions in each engagementmood, and deviation from those preferences increases their probability of dropping out significantly. Depending on the context of research, the terms attrition , churning , and dropping out can be used interchangeably to implysimilar concepts.J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:3 Finally, inspired by the HMM findings and insights about student dropouts, we feed HMM results toa Long Short-Term Memory (LSTM) recurrent neural network to find if it can improve the accuracyof dropout predictions compared with a plain LSTM model and five other baselines, includingan XGBoost model, Random Forest, Decision Tree (DT), Logistic Regression, and Support VectorMachine (SVM) [76, 110]. We notice improvements in the accuracy of dropout predictions whenHMM and LSTM recurrent neural networks are combined. More precisely, we reach an accuracy of78.22%, F1-measure of 81.28%, and AUC measure of 89.10%, which until now set the bar for dropoutprediction models in QPs.
Contributions.
This work is important to HCI and CSCW because it presents the first typologyand dropout prediction model for QP students. Furthermore, it reinforces the need for personalizingQP websites by revealing that different students have different preferences for selecting andanswering QP questions. Such knowledge could help QPs to make more informed design decisions.We also provide some design suggestions for QP managers and online learning professionals toreduce student dropouts in QPs.
Question Pool websites (QP) provide students with a collection of questions to learn and practicetheir knowledge online [120]. The QPs such as Timus, Jutge.org, Optil.io, Code Chef, and HDUvirtual judge are among the most popular web-based platforms for code education [86, 117, 118, 120].These platforms usually include a large repository of programming questions from which studentschoose to answer. The students submit their solutions to the QP and wait for the feedback to findif their code is correct. Dropout prediction is a challenging but necessary study for ensuring thesustainability and service continuation of these platforms.In this paper, we concentrate on a popular and publicly accessible QP in China that is knownas HDU virtual judge platform (henceforth HDU). The website originally belongs to HangzhouDianzi University’s ACM team and is designed to provide students hands-on exercises to hone theirprogramming and coding skills [120]. HDU, on average, hosts more than one hundred studentsevery day and receives more than 300K programming code submissions every month. It is also afamiliar platform for researchers who work in the field of HCI [120]. Figure 1 shows snapshots ofHDU website.Similar to the literature works [34, 93, 124], we formulate the dropout prediction problem as abinary classification task. Our definition of dropout is similar to [72], which temporally splits adataset into observation and inspection periods. The students whose number of solution submissionsin the inspection period drops to less than 20% of the observation period are considered to bedropped students. Since QP platforms usually do not have fixed start and end time points likein MOOCs [76], we regulate the observation and inspection periods similar to CQA platforms[34, 72, 88]. We use equally long periods of time for observation and inspection time windows. The HCI and CSCW literature is abundant with various conceptualizations of engagement [4, 20, 32,36, 55, 105]. However, there is a lack of consensus on the definition of engagement [10]. The threemost widely used conceptualizations of engagement moods are behavioral , emotional , and cognitive [2, 39]. In the context of students’ learning research, behavioral engagement refers to attention,participation, and effort a student puts in doing his or her academic activities [2, 19, 78]. For example,students can be in off-task or on-task moods when doing their learning tasks [2]. The emotional http://code.hdu.edu.cn/ J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. (a) Pool of Questions (b) Realtime Status of the QPFig. 1. Snapshots from HDU, the QP we study in this work. Figure (a) shows the pool of questions and eachquestion’s acceptance rate. Figure (b) shows the QP’s realtime status that helps students become aware ofthe evaluation of their own and their friends’ performances after each submission. engagement refers to students’ affective responses to learning activities and the individuals involvedin those activities [78, 83]. For instance, students might be concerned about how their instructorsperceive their performances. Finally, cognitive engagement is about how intrinsically invested andmotivated students are in their learning process [78]. For example, students might make mentalefforts to debate in an online forum [49].The complexity of discovering student engagement moods has resulted in the appearance ofa diversity of data mining techniques [4, 17, 74, 112]. K-means, the clustering algorithm, is oneof the most popular techniques many researchers use to extract naturally occurring typology ofstudents’ engagement moods [44, 74, 94]. Saenz et al. use K-means and exploratory cluster analysisto extract different engagement moods for college students [94]. By comparison of similarities anddissimilarities between different features, they characterize 15 different clusters. Furtado et al. use acombination of hierarchical and non-hierarchical clustering algorithms to identify the contributors’profiles in the context of CQA platforms [40]. They categorize CQA contributors’ behavior into10 types based on how much and how well they contribute to the platform over time. However,K-Means is more useful when features show Euclidean distances properties [47].Latent variable models like Hidden Markov Models (HMMs) are also quite popular. Faucon etal. use a semi-Markov Model for simulating and capturing the behavioral engagement of MOOCstudents [35]. They provide a graphical representation of the dynamics of the transitions betweendifferent states, such as forum participation or video watching. Mogavi et al. combine HMM and aReinforcement Learning Model (RLM) to capture users’ flow experience on a famous CQA platform[72]. Flow experience is a positive mental state in psychology that occurs when the challengelevel of an activity (e.g., answering a question, doing a task, or solving a homework problem)is commensurate with the user’s skill level [30, 72, 81]. Our paper is the first to use HMM tocharacterize students’ engagement moods in QPs. Understanding students’ engagement moods canhelp educators to manage students behaviors better and set more customized curricula [97]. Educational Data Mining (EDM) is a relatively new discipline that has recently caught the attentionof HCI and CSCW communities [29, 53, 68]. One of EDM’s primary interests is to know whetherstudents would drop out soon or continue their studies until the end of their courses or at least
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:5
Table 1. The summary of the benchmark dataset
Measure Number Min Max Median Mean SD
Answer Submission 1.2 M 1 62 5.23 7.14 18.53Accepted Answers 227 K 0 39 4.16 5.01 12.89Endurance (in minute) N/A 0 209.65 15.91 21.38 67.43Attendance Gap (in hour) N/A 1.18 673.38 134.22 192.48 125.68Number of Students = 9,941 Number of Dropped Students = 5,261 for a long time [12, 57, 92]. Qiu et al. introduce a latent variable model called LadFG based onstudents’ demographics, forum activities, and learning behaviors to predict students’ success incompleting the courses they start in XuetangX, one of the largest MOOCs in China [89]. They showthat having friends on MOOCs can increase students’ chances of success in receiving the finalcertificates of any course dramatically by three-fold, but surprisingly, being more active on theprogram does not guarantee the student will take the final certificate. Wang et al. propose a hybriddeep learning-based dropout prediction model that combines two architectures of ConvolutionalNeural Network (CNN) and Recurrent Neural Network (RNN) to advance the accuracy of dropoutprediction models in MOOCs [115]. They show that their model can achieve a high accuracycomparable to feature-engineered data mining methods. As another example, Xing et al. proposea simple deep learning-based model with features such as students’ access times to the platformand their history of active days to predict dropouts [121]. They suggest finding students’ dropoutprobability on a weekly basis to take better measures in preventing students’ dropouts. Studentengagement is one key factor among all of these studies. In fact, engagement can be considered abasis for students’ retention, and a lack of it confronts and cancels any positive learning outcomes[125]. Therefore, we use this rationale to utilize students’ engagement moods to predict dropoutsin QPs.
After receiving the approval of our local university’s Institutional Review Board (IRB), we followthe ethical guidelines of AoIR for the study of student behavior on Hangzhou Dianzi University’sQP platform (known as HDU). The dataset we study includes near 10K student records in the rangefrom January 25th to July 15th, 2019 (172 days). We utilize the student data before April 21st forthe observation period feed of the prediction models, and the data after that for the inspection ofthe dropouts (similar to [77, 88]). More than half of the students drop out of the platform in theinspection period. We exclude the students with no submissions in the observation period fromour study to avoid the inclusion of the students who have already dropped out and to alleviate theproblem of imbalanced class labels between dropped and continuing students (see [61, 72]). Table 1summarizes the main statistics of our dataset. We use an unsupervised HMM for decoding student engagement moods, and a supervised LSTMnetwork for predicting dropouts. Both components are trained with the student data features duringthe observation period. The HMM inputs include simple observable features that imply student performance , challenge , endurance , and attendance gap states after each submission to the QP. Theparameters of the HMM are estimated by iterations of a standard Expectation Maximization (EM) https://aoir.org/ethics/ J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. algorithm known as Baum-Welch [13]. Five dominant student engagement moods are identified,which are (E1) challenge-seeker, (E2) subject-seeker, (E3) interest-seeker, (E4) joy-seeker, and (E5)non-seeker. We run a user study with 26 local students to evaluate our HMM findings.After decoding the student engagement moods with the HMM, we associate the platform ques-tions with the engagement moods that are more likely to submit solutions for those questions. Wenotice that students have collective preferences for answering questions in each engagement mood,and deviation from those preferences increases their probability of dropping out significantly.Finally, we feed the generated engagement moods and questions’ associativity features alongwith other common features to an LSTM network to predict student dropout in the inspectionperiod. All the dropout predictions are reported based on the 10-fold cross-validation. Hidden Markov Models (HMMs) are statistical tools that help to make inferences about the latent(unobservable) variables through analyzing the manifest (observable) features [26, 58, 98]. In thecontext of education systems, HMMs are used for the detection of various phenomenons such as student social loafing [127], flow zone engagement [72], and academic pathway choices [42]. Thewidespread use of HMMs in the literature and the capability of capturing complex data structures[42] inspire our work to apply HMMs to help distinguish between different student engagementmoods in the QP platforms. We use the hmmlearn module in Python to train our model. • Inputs.
In order to build an HMM, we utilize the manifest features for student performance , challenge , endurance , and attendance gap . We pick the manifest features by performing an extensivethematic analysis [84, 116] across the literature about student engagement [43, 50, 73]. The featureswe use are as follows.- Performance: The feedback QP platforms provide each submission, such as if the answer is Wrong or Accepted .- Challenge: The past acceptance rate of a question, which is often shown along with a guidenext to each question, can resemble the challenge.- Endurance: The time students spend on the platform to answer questions and compile codesin one session is student endurance. Similar to [46], we define a “session” in QP platforms asa period of time in which the interval between two consecutive submissions does not exceedone hour. We measure the student endurance in minutes.- Attendance gap: The time interval between two consecutive sessions is a student’s attendancegap. We measure the attendance gap in hours.We should mention here that these features serve only as cues to infer students’ cognitive (per-formance and challenge) and behavioral (endurance and attendance gap) engagement moods[45, 59, 62, 80]. • Parameters.
Hereafter, we use a four-element vector 𝑂 𝑡 to refer to the corresponding studentobservations after each answer submission to the QP in time 𝑡 . The HMM assumes the observations 𝑂 𝑡 are generated by an underlying state space of hidden variables 𝑍 = { 𝑧 𝑖 } , where 𝑖 ≥ . For conve-nience, we use a triplet 𝜆 HMM = ( 𝐴, 𝐵, 𝜋 ) to denote the HMM we train for extraction of the studentengagement moods. The transition matrix 𝐴 shows the probabilities of moving between differentengagement moods over time. The emission matrix 𝐵 shows the conditional probability for anobservation 𝑂 𝑡 to be emitted (generated) from a certain engagement mood 𝑧 . The vector 𝜋 denotesthe initial probabilities of being in each of the engagement moods of 𝑍 . The initial probabilitiesare often assumed to be || 𝑍 || , with || 𝑍 || showing the cardinality of the hidden state space (i.e., thenumber of engagement moods) [72]. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:7
A I C B I C
N u m b e r o f H i d d e n S t a t e s
AIC
BIC
Fig. 2. Estimating the best number of hidden states, where AIC and BIC measures both take the least values. • Model training.
We optimize 𝜆 HMM parameters according to the student behavior in the obser-vation period with a standard EM algorithm known as Baum-Welch [13]. The aim is to optimizethe 𝜆 HMM parameters such that 𝑃𝑟 ( 𝜆 HMM | 𝑂 ) is maximized with 𝑂 = { 𝑂 𝑡 } . To avoid the local maximumproblem with the EM algorithm, we train the HMM with ten random seeds until they converge atthe global maximum.The HMM representation is completed by choosing the best number of hidden states [72]. Whilethis task appears to be simple conceptually, finding the best number of hidden states in a meaningfulway is quite challenging [72, 111]. The main reason is that an HMM with a small number of hiddenstates cannot capture the underlying behavioral kinetics adequately, and an HMM with too manyhidden states is difficult to interpret [79]. However, we need a criterion to compromise, and thuswe use the conventional Akaike (AIC) [96] and Bayes (BIC) [23] measures in our model training(also see [72]). Figure 2 plots the AIC and BIC measures against the number of hidden states in ourtrained HMMs (i.e., from 2 to 10 hidden states are tested). We choose an HMM with || 𝑍 || = hiddenstates since the global values of AICs and BICs measures are the lowest in this representation. Thesmaller AIC and BIC show more descriptive and less complicated 𝜆 HMM [72].
Similar to the literature, we use data distributions within each hidden state to characterize andvisualize the distinction between different engagement moods [5, 40, 72]. The features we inspecthere are the number of incorrect and accepted answers, the average ease of questions, the averagetime spent in the platform, the average time gap in attendance, and the number of repeatedsubmissions. They are inspired by the manifest features we had before, but instead of an individualstudent, they aim to examine all students’ collective behaviors in a specific hidden state. Thecumulative distribution function (CDF) of these features are plotted in Figure 3. We also utilizethe frequency plots of student submissions over different question IDs to demonstrate question-answering patterns in each hidden state (Figure 4). The number of submissions in each hidden stateis a value normalized between 0 to 100. This representation is to facilitate the visualization andcomparison of the patterns in different hidden states. The marked characteristics of each hiddenstate are as follows: • (E1) Challenge-seeker (Hidden State 1) : As shown in Figure 3c, the students in this mood arebest described for their tendency to look for more challenging questions, i.e., the questions withthe least acceptance rate. From Figures 3d and 3e, we find they also spend the longest average J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021.
Percentage (%)
N u m b e r o f I n c o r r e c t A n s w e r s
J o y -s e e k e r In te re s t-s e e k e r C h a lle n g e -s e e k e r S u b je c t-s e e k e r N o n -s e e k e r (a) Incorrect Answers
Percentage (%)
N u m b e r o f A c c e p t e d A n s w e r s
J o y -s e e k e r In te re s t-s e e k e r C h a lle n g e -s e e k e r S u b je c t-s e e k e r N o n -s e e k e r (b) Accepted Answers
Percentage (%)
A v e r a g e Q u e s t i o n E a s e
J o y -s e e k e r In te re s t-s e e k e r C h a lle n g e -s e e k e r S u b je c t-s e e k e r N o n -s e e k e r (c) Question Ease
Percentage (%)
A v e r a g e t i m e S p e n t i n P l a t f o r m ( m i n u t e )
J o y -s e e k e r In te re s t-s e e k e r C h a lle n g e -s e e k e r S u b je c t-s e e k e r N o n -s e e k e r (d) Spent Time
Percentage (%)
A v e r a g e t i m e G a p i n A t t e n d a n c e ( h o u r )
J o y -s e e k e r In te re s t-s e e k e r C h a lle n g e -s e e k e r S u b je c t-s e e k e r N o n -s e e k e r (e) Gap in Attendance
Percentage (%)
N u m b e r o f R e p e a t e d S u b m i s s i o n s
J o y -s e e k e r In te re s t-s e e k e r C h a lle n g e -s e e k e r S u b je c t-s e e k e r N o n -s e e k e r (f) Repeated SubmissionsFig. 3. CDF plots of the student engagement moods resolved time on the platform, and attend the platform more frequently in comparison with the otherengagement moods. Furthermore, Figure 4a shows that challenge-seekers show more interest inthe last questions of the platform. • (E2) Subject-seeker (Hidden State 2) : As shown in Figure 4b, students in this mood tend toanswer specific sets of questions. They usually answer the specific-context questions (e.g., greedyalgorithms) sequentially. Furthermore, from Figure 3d we can notice that subject-seekers comesecond after the students in the challenge-seeker mood for spending the longest time on theplatform. Figures 3a and 3b show that the students in the subject-seeker mood have the largestaverage number of incorrect answers on the platform, whereas their average number of acceptedanswers is quite similar to those of the challenge-seeker students (the distributions are also similar). • (E3) Interest-seeker (Hidden State 3): From Figure 4c, we realize that the students in this mooddo not answer specific-context questions and regularly search for their questions of interest. Thequestion types they answer has the largest variance. Furthermore, the distribution of the easinessof the questions shown in Figure 3c makes no tangible difference in comparison with the othermoods except the challenge-seeker mood. As shown in Figure 3b, the students in this mood hold thehighest average number of accepted answers after the students in the joy-seeker mood. Moreover,according to Figure 3a, interest-seekers’ distribution of producing incorrect answers is close to auniform distribution, which sharply distinguishes them from the other student moods. • (E4) Joy-seeker (Hidden State 4): As shown in Figures 3c and 3f, the students who are inthis mood tend to answer the easiest questions on the platform in a highly repetitive manner.Interestingly, these students choose their questions from a small and selective number of QPquestions (probably those with compilation loopholes) (see Figure 4d). Also, their number ofaccepted answers has the highest value among all the other moods (see Figure 3b). Based on these
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:9
Normalized Number of Submissions
Q u e s t i o n I D s (a)
Challenge-seeker
Normalized Number of Submissions
Q u e s t i o n I D s (b)
Subject-seeker
Normalized Number of Submissions
Q u e s t i o n I D s (c)
Interest-seeker
Normalized Number of Submissions
Q u e s t i o n I D s (d)
Joy-seeker
02 04 06 08 01 0 0
Normalized Number of Submissions
Q u e s t i o n I D s (e)
Non-seeker
Fig. 4. Frequency plots of student submissions over different question IDs. signs, we come to the conclusion that these students are at the highest risk of gaming the platform [7, 9] in comparison with the other mood groups. Finally, we notice that the distribution of incorrectanswers for this mood type resembles a mixture of two Gaussian distributions, which distinguishesitself in sharp contrast to the other mood groups (see Figure 3a). • (E5) Non-seeker (Hidden State 5): According to Figure 3e, we notice that the students in thismood are well-distinguished by holding the largest average time gap among all moods for attendingthe platform, which means that they seldom visit the platform. Furthermore, the least averagetime spent on the platform is also another characteristic of this mood (see Figure 3d). Finally, as isexpected, the lowest number of incorrect answers, accepted answers, and repeated submissions arethe outcomes of this short visit (see Figures 3a, 3b, and 3f).
With the approval of our university’s Institutional Review Board (IRB), we conduct a preliminaryuser study with 26 local students to evaluate the accuracy of the hidden states resolved. A combina-tion of snowball and convenience sampling methods is used to recruit participants from our localuniversities. The participants include 9 females and 17 males, all undergraduate computer-majorstudents in the age range between 18 to 23 (mean = = J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021.
Engagement Mood Options: Responses(1) I am looking for a challenging question. (Challenge-seeker) 46.03%(2) I am looking for a specific question. (Subject-seeker) 19.04%(3) I am looking for an interesting (non-challenging) question. (Interest-seeker) 12.69%(4) I want to use the platform for purposes other than learning. (Joy-seeker) 11.11%(5) I want to solve a random question. (Non-seeker) 11.11%(6) None of the above (explain) N/A
Table 2. Engagement mood options participants could choose from. in our study from their homes’ safety and comfort. As a token of our appreciation, we compensateeach participants’ time with small cash (50HK$) at the end of our study.We use the event-focused version of Experience Sampling Method (ESM) [128], to collect theparticipants’ self-reported engagement moods after each answer submission to HDU. The partici-pants report their engagement moods through a Google form. After training each participant aboutthe definition of each engagement mood, we ask them to identify their moods through one of theoptions shown in Table 2. The study lasts for a period of two weeks long, from September 18th toOctober 2nd, 2020, and 63 responses are recorded. Parallel to each self-reported engagement moodwe receive, an HMM-based label is also generated through students’ observed data from HDU. Wenotice that there is a 76.19% agreement between the engagement mood labels extracted from theHMM and ESM. This is more than 50% better than a random labeler with an accuracy of 20%.Interestingly, none of the participants mention any engagement moods other than the fiveengagement moods we have extracted. However, we predict that there would be more personalizedand detailed engagement moods with an increased number of participants [94].
Next, we use the optimal 𝜆 HMM to find the most probable engagement mood sequence 𝑋 𝑧 = { 𝑥 ∈ 𝑍 } for each student with respect to their observed sequence 𝑂 so as to maximize the 𝑃𝑟 ( 𝑋 𝑧 | 𝑂, 𝜆
HMM ) (inferences at [37]). Furthermore, we associate every question 𝑞 𝑗 on the platform a distribution 𝑄 𝑗 based on its probabilities for receiving answers when students are in different hidden states.Here, the index 𝑗 refers to the identity number of the questions on the platform. We define theaverage question mismatch as the probability that a question does not match with the currentengagement mood of a student. Illustratively, the question mismatch is the complement of thequestion’s associativity measure, as is represented in Figure 5a.In this subsection, we test the null hypothesis that the average question mismatch has nocorrelation with the percentage of student dropouts. The regression analysis shown in Figure 5breveals the positive coefficient factor of 92 .
45, and the Pearson’s r value of 0 .
927 with the significanceof 𝑃 𝑣𝑎𝑙𝑢𝑒 < .
01 between the average question mismatch and the student dropouts. Therefore, thenull hypothesis is rejected, and the alternative hypothesis is accepted. That is to say, as the averagequestion mismatch increases, the percentage of students who drop out also increases. Furthermore,it can be inferred that the students in each engagement mood have a collective preference foranswering questions if from which they deviate, their risk of dropping out also increases.
We suggest using engagement moods extracted from the HMM with an LSTM network to predictstudent dropouts in QPs more precisely. We refer to this hybrid machine learning architecture asDropout-Plus (DP).
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:11
E 1 E 2 E 3 E 4 E 5
E n g a g e m e n t M o o d s
Sample Questions (ID)
P r o b a b i l i t y (a)
S t u d e n t R e g r e s s i o n L i n e
Dropout Percentage (%)
A v e r a g e M i s m a t c h (b)Fig. 5. (a) Finding the question associativity for sample questions, (b) The linear correlation between theaverage question mismatch and the percentage of student dropouts from the platform
We use
Keras in Python to build an attention-based Long Short-Term Memory (LSTM) network todetect the student dropouts [95]. The features we input into the network at every time 𝑡 the studentsubmits an answer to the QP are of two types: 1) We call the set of non-handcrafted features thecommon features that are directly acquired from student behavior. The common features are oftenshared among different QP platforms and include the features of student’s membership period,rank, nationality, acceptance rate, error type distributions, and the average time gap betweensubmissions; and 2) the set of preprocessed features the HMM renders. In order to feed data intothe network more effectively and reduce the effect of correlated features, we use a fully connectedfeed-forward neural network to combine the features and get a distributional feature set to trainour model [123].We compare the performance of DP for predicting student dropouts with five competitivebaselines picked from the literature. Since there are no preceding dropout prediction models forQPs before our work, we pick our baselines from previous MOOC and CQA studies [76, 88]. Tokeep the comparisons relevant and unbiased, we avoid the models where it is not clear how tomatch MOOC or CQA features to QPs. The baselines include: • XGBoost : Extreme Gradient Boosting (XGBoost) algorithm is one of the most dominant machinelearning tools for classification and regression [60, 71, 75, 113]. It comprises a collection of basedecision tree models that are built sequentially, and their final results are summed together toreduce the bias [24]. Each decision tree boosts attributes that led to misclassification of the previousdecision tree. • Random Forest : Random Forest is another decision tree-based ensemble algorithm that has arich literature in HCI and CSCW communities [56, 67, 90, 99]. However, different from XGBoost,Random Forest combines the decision trees all uniformly by using an ensemble algorithm knownas bootstrap aggregation [16, 18]. In other words, every decision tree is independent of the others,and thus the final classification result is resolved based on a majority voting [16]. • Decision Tree (DT) : As a base model of Random Forest and XGBoost algorithms, DT’s perfor-mance sometimes exceeds the two [76]. However, even a small change in data would dramaticallyreshape the model, which is an adverse point. Nevertheless, we add DTs in our analysis to have amore comprehensive set of baselines [34, 76, 88].
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021.
Models with HMM Engagement Features (EF)
Models without HMM Engagement Features (EF)
Name of the Model Accuracy F1-measure AUC Name of the Model Accuracy F1-measure AUC
Dropout-Plus (DP) * Plain LSTM 71.40 74.21 77.21 XGBoost with EF 75.12 76.60 78.41 XGBoost 67.35 68.83 70.56 Random Forest with EF 71.33 78.98 79.27 Random Forest 64.12 69.18 73.90 Decision Tree (DT) with EF 70.15 72.90 74.43 Decision Tree (DT) 62.60 63.85 67.24 Logistic Regression with EF 73.56 77.36 78.85 Logistic Regression 66.93 67.90 70.59 SVM (RBF kernel) with EF 74.39 70.30 75.06 SVM (RBF kernel) 73.20 67.25 70.13 * Best performance
Table 3. Performances of different methods under different evaluation metrics (all reported in %). • Logistic Regression : As a popular statistical tool in HCI and CSCW quantitative analysis[21, 38, 56, 69], Logistic Regression in its basic form uses a logistic (sigmoid function) functionto model a binary dependent variable [69]. In our work, the dependent variable is the dropout’soutcome [69]. • SVM (RBF Kernel) : Support Vector Machine (SVM) is another supervised algorithm that canbe applied for both classification and regression purposes [16]. SVM tries to identify hyperplanes(boundaries) that can separate all data points into groups with high margins [14]. Gaussian RadialBasis Function (RBF) is one of the most common kernel functions researchers apply to train theirmodels [66, 114, 122].We also run an ablation study (see [41, 52]) with each baseline by including or excluding Engage-ment Features (EF) produced by the HMM. More precisely, EF includes students’ engagement moodsand their questions’ associativity features after making each submission during the observationperiod. However, all of the models apply the common features we introduced before for modeltraining. We remind the reader that the common features include student’s membership period,rank, nationality, acceptance rate, error type distributions, and the average time gap betweensubmissions. All of the baselines are implemented through sklearn module in Python.Table 3 summarizes the results of our analyses based on 10-fold cross-validation [16, 91]. Theresults of our analyses show that DP outperforms all the other baselines we have suggested.Moreover, the models with HMM Engagement Features (EF) have all performed better than modelswithout EF. In fact, adding EF has leveled up the performance of all models close to DP’s performance,which can attest that the information about engagement moods can improve dropout prediction.
To the best of our knowledge, this research is the first work that characterizes the students’engagement moods and sets the dropout prediction baseline for the QP platforms. Rendering andexposing student engagement moods is the sweet spot of our work, and the implications can providepractical insights for online learning professionals to manage students’ behaviors better and tailortheir services accordingly.
One of the difficulties of studying human behavior in HCI and CSCW studies is that people changetheir behaviors dynamically over time [1, 33, 63, 100]. Therefore, studying the dynamics betweenstudent engagement moods is an essential part of understanding students’ behavior.
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:13 S t a r t i n g M oo d s Challenge-seeker Subject-seeker Interest-seeker Joy-seeker Non-seeker Challenge-seeker
0 0
Subject-seeker
Interest-seeker
0 0.030 0.240 0.195
Joy-seeker
0 0 0.036
Non-seeker
The values in cells show the probabilities of transitions.
Ending Moods
Table 4. The transition matrix of moving between different engagement moods
Based on the observations from students’ engagement mood changes in our dataset, we havecalculated the transition probabilities between different engagement moods, as shown in Table4. Therefore, these probabilities show a frequentist perspective of engagement mood transitionsfor a restricted time period (i.e., 172 days) [101]. As is shown, the engagement moods are well-distinguished with respect to their probabilistic distributions. The general dynamics from Table4 show that holistically students have the highest probability of getting into the interest-seekermood and the least probability for getting into the challenge-seeker mood.Interestingly, the students in the challenge-seeker mood do not become joy-seekers or non-seekers. We also realize that the joy-seeker students do not become challenge-seekers or subject-seekers. They look for easy questions, and the subjects seem not to interest them. Furthermore,we observe that the challenge-seeker students have the highest probability of becoming interest-seekers. On the other hand, interest-seeker students do not become challenge-seekers at any time.However, this observation may be the result of the inappropriate question assignments to thestudents. According to our findings, subject-seeker students are also not terribly motivated to changetheir moods, which sounds reasonable because of the directed and specific-context nature of thequestions they choose to answer. Interest-seekers are more likely to become non-seekers and vice-versa. Since finding the programming topics of interest is often the main intention for most of thestudent explorations in QPs, a fit question recommender system can be advantageous for students’satisfaction and quality of experience [72, 120]. Finally, among all the resolved engagement moods,students in the joy-seeker mood have the least tendency to change their moods. This implies thatthe platform resources are largely at risk of being wasted unless we find and guide the joy-seekerstudents on time [118].
We define a dominant engagement mood as a mood that has happened most frequently in a student’sengagement mood trajectory. As is shown in Table 5, the majority of the students (30 . . . . . . J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021.
AoS = Abundance of Students; AQM = Average Question Mismatch; DR = Dropout Rate.
Challenge-seeker Subject-seeker Interest-seeker Joy-seeker Non-seeker AoS
AQM DR Table 5. Dropout analysis according to the dominant engagement moods of the students
Based on our findings, we provide some design suggestions for online learning professionals tobetter control student dropouts in QPs. • Recommender systems.
Current recommender systems do not consider when is the right timefor a student to learn something, they just consider what students want to learn [120]. The problemholds for many educators as well who do not know when it is the right time to engage students forstudying a specific topic. Our resolved engagement moods can play a role as an assistant to addressthis issue better. For example, if a topic is difficult and challenging, we would recommend askingstudents to solve it when they are in a challenge-seeker mood. If students are in a joy-seeker mood,we would suggest educators to skip asking questions in that session because it might actually resultin adverse and unwanted educational outcomes in students, such as feeling frustrated or dislikingthe subject matter of that course [6]. Combining learning path recommenders or personal teaching(coaching) styles with engagement mood exposers would probably enhance the quality of educationand deter dropouts as students would feel more satisfied with what they learn and do [103]. • Strategic dropout management.
Before our work, researchers have not paid much attentionto who is dropping out [85, 119], but we think that this is an important question to be answered formanaging student dropouts more strategically. In contrast with the existing practice in dropoutstudies, we believe not all of the dropouts are equally bad. For example, imagine an educatorwho is undecided about which types of interventions she should apply to retain students fromdropping out. She is confused about customizing the system either according to challenge-seekers’or joy-seekers’ tastes. We would recommend prioritizing challenge-seekers as joy-seekers wouldmost probably game the system without learning anything. Joy-seekers might also mischievouslyhave adverse side effects on other students’ experiences and make them feel disappointed [6, 27].For more complex dropout scenarios, educators can also heed to the abundance of students ineach group and the transition probabilities between different engagement moods to make betterdecisions. • Gamification and reinforcement tools.
According to Skinner’s refined Law of Effect, rein-forced behaviors tend to repeat more in the future, and those behaviors which are not reinforcedtend to dissipate over time [15]. Therefore, educators can use gamification incentives such as badges,points, and leaderboards to steer students’ behaviors [3]. We suggest QP designers to considerstudent engagement moods for designing their gamification mechanics [107]. For example, if educa-tors are preparing students for difficult exams like ACM programming contests, they can reinforce
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:15 challenge-seekers by providing bigger gamification prizes [126]. Likewise, negative reinforcementcan be applied to diminish non-productive behaviors like being in a joy-seeker mood. Besidesthat, QP designers can provision special gamification mechanics or side games to satisfy students’playful intentions somewhere else. • Putting students into groups.
At this point, collaboration and social communication are hugelymissing in the context of QPs, and instead, all the website functions only promote competencyamong students. Although competency can provide an initial motive to attract students [11], manyeducators believe that collaboration and social communication among students are also needed tobetter achieve the intended educational outcomes [51, 109]. We suggest QPs to add affordances tosupport student communication and social collaboration as well. Meanwhile, student engagementmoods can help to find initial common attitudes among students and form groups. For example, ifone student is a subject-seeker, there is a good chance that this student would successfully fit intoone group with other subject-seeker students. This rule is known as Homophily in social networks[70]. Nevertheless, more in-depth studies are required to figure out which combination of studentengagement moods work and match better together.
As with any study, there are several limitations and challenges in this study. First, the platformdata we analyzed is cross-sectional and is restricted in its size and time, but not type. We findthis problem to be a commonplace issue in many related studies as well [72, 76]. Also, we wantto emphasize the difficulty of working with student data and the scarceness of datasets about QPplatforms, which are relatively new research subjects in the context of educational data mining.Regarding the model limitations, although HMM can successfully profile the student behavior ina few hidden states, it is often a time-consuming task to characterize the resolved hidden states.For example, in this research, we have spent more than twelve hours to carefully visualize andcompare the probabilistic distributions behind the hidden states considering different aspects andfeatures to finally announce our engagement mood typology. We should also explain that sincehidden states are found by an estimation procedure, different platforms might result in differentnumbers of hidden states. Generally, we expect these states to be semantically close to what wehave introduced in this work. We emphasize that this feature should be viewed more as a positivepoint for future work, which leaves more complex engagement moods to be mined and comparedwith our extracted typology. Hence, it remains as future work to cross-validate our work on otherQP platforms such as Timus, quera.ir, CodeChef, and the like. Moreover, it would be a particularlyinteresting direction to examine the effect of the HMM’s state space granularity on the performanceof the dropout predictions.
10 CONCLUSION
We used the powerful tool Hidden Markov Models (HMMs) to expose underlying student engage-ment moods in QP platforms and point out that the mismatch between students’ engagementmoods and the question types they answer over time can significantly increase the dropout risk.Furthermore, we developed a novel and more accurate computational framework called Dropout-Plus (DP) to predict student dropouts and explain the possible reasons why dropouts happen in QPplatforms. However, we believe there is still a long path in front of HCI and CSCW researchers tofully understand dropouts on different educational platforms. Our future work includes developinga more exact time prediction for student dropouts and enriching the explanations to the questionof “why dropouts happen?” Finally, this study can benefit researchers and practitioners of online
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. education platforms to promote their work by understanding student dropouts more profoundly,building better prediction models, and providing more customized services.
ACKNOWLEDGMENTS
This research has been supported in part by project 16214817 from the Research Grants Council ofHong Kong, and the 5GEAR and FIT projects from Academy of Finland. It is also partially supportedby the Research Grants Council of the Hong Kong Special Administrative Region, China underGeneral Research Fund (GRF) Grant No. 16204420.
REFERENCES [1] Najwa Alghamdi, Nora Alrajebah, and Shiroq Al-Megren. 2019. Crowd Behavior Analysis Using Snap Map: APreliminary Study on the Grand Holy Mosque in Mecca. In
Conference Companion Publication of the 2019 on Com-puter Supported Cooperative Work and Social Computing (Austin, TX, USA) (CSCW ’19) . Association for ComputingMachinery, New York, NY, USA, 137–141. https://doi.org/10.1145/3311957.3359473[2] Nese Alyuz, Eda Okur, Utku Genc, Sinem Aslan, Cagri Tanriover, and Asli Arslan Esme. 2017. An Unobtrusiveand Multimodal Approach for Behavioral Engagement Detection of Students. In
Proceedings of the 1st ACM SIGCHIInternational Workshop on Multimodal Interaction for Education (Glasgow, UK) (MIE 2017) . Association for ComputingMachinery, New York, NY, USA, 26–32. https://doi.org/10.1145/3139513.3139521[3] Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. 2013. Steering User Behavior with Badges.In
Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW ’13) . Associationfor Computing Machinery, New York, NY, USA, 95–106. https://doi.org/10.1145/2488388.2488398[4] Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E. Mete, Eda Okur, Sidney K. D’Mello, and Asli Arslan Esme.2019. Investigating the Impact of a Real-Time, Multimodal Student Engagement Analytics Technology in AuthenticClassrooms. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19) . Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300534[5] R. Babbie. 2016.
The Basics of Social Research . Cengage Learning. https://books.google.com/books?id=0CJTCwAAQBAJ[6] Ryan Baker, Jason Walonoski, Neil Heffernan, Ido Roll, Albert Corbett, and Kenneth Koedinger. 2008. Why studentsengage in “gaming the system” behavior in interactive learning environments.
Journal of Interactive Learning Research
19, 2 (2008), 185–224.[7] Ryan Shaun Baker. 2005.
Designing intelligent tutors that adapt to when students game the system . Ph.D. Dissertation.Carnegie Mellon University Pittsburgh.[8] Ryan S.J.d. Baker. 2007. Modeling and Understanding Students’ off-Task Behavior in Intelligent Tutoring Systems. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07) .Association for Computing Machinery, New York, NY, USA, 1059–1068. https://doi.org/10.1145/1240624.1240785[9] Ryan Shaun Baker, Albert T. Corbett, Kenneth R. Koedinger, and Angela Z. Wagner. 2004. Off-Task Behavior in theCognitive Tutor Classroom: When Students "Game the System". In
Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems (Vienna, Austria) (CHI ’04) . Association for Computing Machinery, New York, NY, USA,383–390. https://doi.org/10.1145/985692.985741[10] E.F. Barkley. 2009.
Student Engagement Techniques: A Handbook for College Faculty . Wiley. https://books.google.com/books?id=muAStyrwyZgC[11] Alexander Bartel, Paula Figas, and Georg Hagel. 2015. Towards a Competency-Based Education with GamificationDesign Elements. In
Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (London,United Kingdom) (CHI PLAY ’15) . Association for Computing Machinery, New York, NY, USA, 457–462. https://doi.org/10.1145/2793107.2810325[12] Jonathan Bassen, Bharathan Balaji, Michael Schaarschmidt, Candace Thille, Jay Painter, Dawn Zimmaro, Alex Games,Ethan Fast, and John C. Mitchell. 2020. Reinforcement Learning for the Adaptive Scheduling of Educational Activities.In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20) .Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376518[13] Leonard E. Baum, Ted Petrie, George Soules, and Norman Weiss. 1970. A Maximization Technique Occurring in theStatistical Analysis of Probabilistic Functions of Markov Chains.
The Annals of Mathematical Statistics
Proceedings of theHalfway to the Future Symposium 2019 (Nottingham, United Kingdom) (HTTF 2019) . Association for ComputingMachinery, New York, NY, USA, Article 8, 11 pages. https://doi.org/10.1145/3363384.3363392J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:17 [15] Muhammad Ilyas Bhutto. 2011. Effects of Social Reinforcers on Students’Learning Outcomes at Secondary SchoolLevel.
International Journal of Academic Research in Business and Social Sciences
1, 2 (2011), 71.[16] Christopher M Bishop. 2006. Machine learning and pattern recognition.
Information science and statistics. Springer,Heidelberg (2006).[17] Nigel Bosch. 2016. Detecting Student Engagement: Human Versus Machine. In
Proceedings of the 2016 Conference onUser Modeling Adaptation and Personalization (Halifax, Nova Scotia, Canada) (UMAP ’16) . Association for ComputingMachinery, New York, NY, USA, 317–320. https://doi.org/10.1145/2930238.2930371[18] Leo Breiman. 2001. Random forests.
Machine Learning
45, 1 (2001), 5–32. https://doi.org/10.1023/a:1010933404324[19] Elise Cappella, Ha Yeon Kim, Jennifer W. Neal, and Daisy R. Jackson. 2013. Classroom Peer Relationships andBehavioral Engagement in Elementary School: The Role of Social Network Equity.
American Journal of CommunityPsychology
52, 3-4 (Oct. 2013), 367–379. https://doi.org/10.1007/s10464-013-9603-5[20] Jonathan Carlton, Andy Brown, Caroline Jay, and John Keane. 2019. Inferring User Engagement from Interaction Data.In
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHIEA ’19) . Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3313009[21] Xunru Che, Danaë Metaxa-Kakavouli, and Jeffrey T. Hancock. 2018. Fake News in the News: An Analysis ofPartisan Coverage of the Fake News Phenomenon. In
Companion of the 2018 ACM Conference on Computer SupportedCooperative Work and Social Computing (Jersey City, NJ, USA) (CSCW ’18) . Association for Computing Machinery,New York, NY, USA, 289–292. https://doi.org/10.1145/3272973.3274079[22] Code Chef. 2019.
Code Chef
Proc. DARPA broadcast news transcription and understanding workshop ,Vol. 8. Virginia, USA, 127–132.[24] Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In
Proceedings of the 22nd ACMSIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD’16) . Association for Computing Machinery, New York, NY, USA, 785–794. https://doi.org/10.1145/2939672.2939785[25] Yuanzhe Chen, Qing Chen, Mingqian Zhao, Sebastien Boyer, Kalyan Veeramachaneni, and Huamin Qu. 2016. Dropout-Seer: Visualizing learning patterns in Massive Open Online Courses for dropout reasoning and prediction. In . IEEE, 111–120.[26] Yinghan Chen, Steven Andrew Culpepper, Shiyu Wang, and Jeffrey Douglas. 2017. A Hidden Markov Model forLearning Trajectories in Cognitive Diagnosis With Application to Spatial Rotation Skills.
Applied PsychologicalMeasurement
42, 1 (Sept. 2017), 5–23. https://doi.org/10.1177/0146621617721250[27] Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017. Anyone Can Become aTroll: Causes of Trolling Behavior in Online Discussions. In
Proceedings of the 2017 ACM Conference on ComputerSupported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17) . Association for ComputingMachinery, New York, NY, USA, 1217–1230. https://doi.org/10.1145/2998181.2998213[28] Chia-Fang Chung, Nanna Gorm, Irina A. Shklovski, and Sean Munson. 2017. Finding the Right Fit: UnderstandingHealth Tracking in Workplace Wellness Programs. In
Proceedings of the 2017 CHI Conference on Human Factors inComputing Systems (Denver, Colorado, USA) (CHI ’17) . Association for Computing Machinery, New York, NY, USA,4875–4886. https://doi.org/10.1145/3025453.3025510[29] Evandro B. Costa, Baldoino Fonseca, Marcelo Almeida Santana, Fabrísia Ferreira de Araújo, and Joilson Rego. 2017.Evaluating the effectiveness of educational data mining techniques for early prediction of students’ academic failurein introductory programming courses.
Computers in Human Behavior
73 (2017), 247 – 256. https://doi.org/10.1016/j.chb.2017.01.047[30] M Csikszentmihalyi. 1975. Beyond boredom and anxiety. San Francisco: JosseyBass.
Well-being: Thefoundations ofhedonic psychology (1975), 134–154.[31] Ryan S. J. d. Baker, Albert T. Corbett, Ido Roll, and Kenneth R. Koedinger. 2008. Developing a generalizable detectorof when students game the system.
User Modeling and User-Adapted Interaction
18, 3 (Jan. 2008), 287–314. https://doi.org/10.1007/s11257-007-9045-6[32] Matt Dixon, Nalin Asanka Gamagedara Arachchilage, and James Nicholson. 2019. Engaging Users with EducationalGames: The Case of Phishing. In
Extended Abstracts of the 2019 CHI Conference on Human Factors in ComputingSystems (Glasgow, Scotland Uk) (CHI EA ’19) . Association for Computing Machinery, New York, NY, USA, 1–6.https://doi.org/10.1145/3290607.3313026[33] Edson B. dos Santos Junior, Carlos Simões, Ana Cristina Bicharra Garcia, and Adriana S. Vivacqua. 2018. WhatDoes a Crowd Routing Behavior Change Reveal?. In
Companion of the 2018 ACM Conference on Computer SupportedCooperative Work and Social Computing (Jersey City, NJ, USA) (CSCW ’18) . Association for Computing Machinery,New York, NY, USA, 297–300. https://doi.org/10.1145/3272973.3274081J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. [34] Gideon Dror, Dan Pelleg, Oleg Rokhlenko, and Idan Szpektor. 2012. Churn Prediction in New Users of Yahoo! Answers.In
Proceedings of the 21st International Conference on World Wide Web (Lyon, France) (WWW ’12 Companion) . ACM,New York, NY, USA, 829–834. https://doi.org/10.1145/2187980.2188207[35] Louis Faucon, Lukasz Kidzinski, and Pierre Dillenbourg. 2016. Semi-Markov Model for Simulating MOOC Students.
International Educational Data Mining Society (2016).[36] Sarah Foley, Nadia Pantidi, and John McCarthy. 2020. Student Engagement in Sensitive Design Contexts: A CaseStudy in Dementia Care. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu,HI, USA) (CHI ’20) . Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376161[37] G David Forney Jr. 2005. The viterbi algorithm: A personal history. arXiv preprint cs/0504020 (2005). http://arxiv.org/abs/cs/0504020[38] Pascal E. Fortin, Elisabeth Sulmont, and Jeremy Cooperstock. 2019. Detecting Perception of Smartphone NotificationsUsing Skin Conductance Responses. In
Proceedings of the 2019 CHI Conference on Human Factors in ComputingSystems (Glasgow, Scotland Uk) (CHI ’19) . Association for Computing Machinery, New York, NY, USA, 1–9. https://doi.org/10.1145/3290605.3300420[39] Jennifer A Fredricks, Phyllis C Blumenfeld, and Alison H Paris. 2004. School Engagement: Potential of the Con-cept, State of the Evidence.
Review of Educational Research
74, 1 (March 2004), 59–109. https://doi.org/10.3102/00346543074001059[40] Adabriand Furtado, Nazareno Andrade, Nigini Oliveira, and Francisco Brasileiro. 2013. Contributor Profiles, TheirDynamics, and Their Importance in Five Q&a Sites. In
Proceedings of the 2013 Conference on Computer SupportedCooperative Work (San Antonio, Texas, USA) (CSCW ’13) . Association for Computing Machinery, New York, NY, USA,1237–1252. https://doi.org/10.1145/2441776.2441916[41] W.L. Gardiner. 1974.
Psychology: a story of a search . Brooks/Cole Pub. Co. https://books.google.fr/books?id=q50bAQAAMAAJ[42] Nate Gruver, Ali Malik, Brahm Capoor, Chris Piech, Mitchell L Stevens, and Andreas Paepcke. 2019. Using LatentVariable Models to Observe Academic Pathways. In
Proceedings of The 12th International Conference on EducationalData Mining . EDM, 294–299. http://educationaldatamining.org/edm2019/proceedings/[43] Juho Hamari, David J. Shernoff, Elizabeth Rowe, Brianno Coller, Jodi Asbell-Clarke, and Teon Edwards. 2016. Chal-lenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning.
Computers in Human Behavior
54 (2016), 170 – 179. https://doi.org/10.1016/j.chb.2015.07.045[44] Siti Suhaila Abdul Hamid, Novia Admodisastro, Noridayu Manshor, Azrina Kamaruddin, and Abdul Azim Abd Ghani.2018. Dyslexia adaptive learning model: student engagement prediction using machine learning approach. In
International Conference on Soft Computing and Data Mining . Springer, 372–384.[45] Mitchell M Handelsman, William L Briggs, Nora Sullivan, and Annette Towler. 2005. A measure of college studentcourse engagement.
The Journal of Educational Research
98, 3 (2005), 184–192.[46] Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018. AData-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk. In
Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18) . ACM, New York, NY, USA, Article 449,14 pages. https://doi.org/10.1145/3173574.3174023[47] Xiaofei He, Xinbo Gao, Yanning Zhang, Zhi-Hua Zhou, Zhi-Yong Liu, Baochuan Fu, Fuyuan Hu, and ZhanchengZhang. 2015.
Intelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques: 5th InternationalConference, IScIDE 2015, Suzhou, China, June 14-16, 2015, Revised Selected Papers . Vol. 9243. Springer.[48] Matthew Hertz. 2010. What Do "CS1" and "CS2" Mean? Investigating Differences in the Early Courses. In
Proceedingsof the 41st ACM Technical Symposium on Computer Science Education (Milwaukee, Wisconsin, USA) (SIGCSE ’10) .Association for Computing Machinery, New York, NY, USA, 199–203. https://doi.org/10.1145/1734263.1734335[49] Hayati Hind, Mohammed Khalidi Idrissi, and Samir Bennani. 2017. Applying Text Mining to Predict Learners’ CognitiveEngagement. In
Proceedings of the Mediterranean Symposium on Smart City Application (Tangier, Morocco) (SCAMS ’17) .Association for Computing Machinery, New York, NY, USA, Article 2, 6 pages. https://doi.org/10.1145/3175628.3175655[50] Beryl Hoffman, Ralph Morelli, and Jennifer Rosato. 2019. Student Engagement is Key to Broadening Participation inCS. In
Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19) . ACM, New York, NY, USA, 1123–1129. https://doi.org/10.1145/3287324.3287438[51] Jeroen Janssen and Paul A Kirschner. 2020. Applying collaborative cognitive load theory to computer-supportedcollaborative learning: towards a research agenda.
Educational Technology Research and Development (2020), 1–23.[52] Kenneth Joseph, Wei Wei, and Kathleen M. Carley. 2017. Girls Rule, Boys Drool: Extracting Semantic and AffectiveStereotypes from Twitter. In
Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work andSocial Computing (Portland, Oregon, USA) (CSCW ’17) . Association for Computing Machinery, New York, NY, USA,1362–1374. https://doi.org/10.1145/2998181.2998187J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:19 [53] Libor Juhaňák, Jiří Zounek, and Lucie Rohlíková. 2019. Using process mining to analyze students’ quiz-takingbehavior patterns in a learning management system.
Computers in Human Behavior
92 (2019), 496 – 506. https://doi.org/10.1016/j.chb.2017.12.015[54] Jutge. 2019. jutge . Retrieved January 24, 2019 from https://jutge.org/[55] Saskia M. Kelders and Hanneke Kip. 2019. Development and Initial Validation of a Scale to Measure Engagementwith EHealth Technologies. In
Extended Abstracts of the 2019 CHI Conference on Human Factors in ComputingSystems (Glasgow, Scotland Uk) (CHI EA ’19) . Association for Computing Machinery, New York, NY, USA, 1–6.https://doi.org/10.1145/3290607.3312917[56] Jina Kim, Kunwoo Bae, Eunil Park, and Angel P. del Pobil. 2019. Who Will Subscribe to My Streaming Channel?The Case of Twitch. In
Conference Companion Publication of the 2019 on Computer Supported Cooperative Work andSocial Computing (Austin, TX, USA) (CSCW ’19) . Association for Computing Machinery, New York, NY, USA, 247–251.https://doi.org/10.1145/3311957.3359470[57] Juho Kim, Philip J. Guo, Daniel T. Seaton, Piotr Mitros, Krzysztof Z. Gajos, and Robert C. Miller. 2014. UnderstandingIn-Video Dropouts and Interaction Peaks Inonline Lecture Videos. In
Proceedings of the First ACM Conference onLearning @ Scale Conference (Atlanta, Georgia, USA) (L@S ’14) . Association for Computing Machinery, New York, NY,USA, 31–40. https://doi.org/10.1145/2556325.2566237[58] Marios Kokkodis. 2019. Reputation Deflation Through Dynamic Expertise Assessment in Online Labor Markets.In
The World Wide Web Conference (San Francisco, CA, USA) (WWW ’19) . ACM, New York, NY, USA, 896–905.https://doi.org/10.1145/3308558.3313479[59] George D Kuh. 2001. Assessing what really matters to student learning inside the national survey of studentengagement.
Change: The Magazine of Higher Learning
33, 3 (2001), 10–17.[60] Il-Youp Kwak, Jun Ho Huh, Seung Taek Han, Iljoo Kim, and Jiwon Yoon. 2019. Voice Presentation Attack Detectionthrough Text-Converted Voice Command Analysis. In
Proceedings of the 2019 CHI Conference on Human Factors inComputing Systems (Glasgow, Scotland Uk) (CHI ’19) . Association for Computing Machinery, New York, NY, USA,1–12. https://doi.org/10.1145/3290605.3300828[61] Young D. Kwon, Dimitris Chatzopoulos, Ehsan ul Haq, Raymond Chi-Wing Wong, and Pan Hui. 2019. GeoLifecycle:User Engagement of Geographical Exploration and Churn Prediction in LBSNs.
Proc. ACM Interact. Mob. WearableUbiquitous Technol.
3, 3, Article 92 (Sept. 2019), 29 pages. https://doi.org/10.1145/3351250[62] D Langley. 2006. The student engagement index: A proposed student rating system based on the national benchmarksof effective educational practice.
University of Minnesota: Center for Teaching and Learning Services (2006).[63] Hyunsoo Lee, Uichin Lee, and Hwajung Hong. 2019. Commitment Devices in Online Behavior Change SupportSystems. In
Proceedings of Asian CHI Symposium 2019: Emerging HCI Research Collection (Glasgow, Scotland, UnitedKingdom) (AsianHCI ’19) . Association for Computing Machinery, New York, NY, USA, 105–113. https://doi.org/10.1145/3309700.3338446[64] LeetCode. 2019.
LeetCode . Retrieved January 24, 2019 from https://leetcode.com/[65] Pascal Lessel, Maximilian Altmeyer, Lea Verena Schmeer, and Antonio Krüger. 2019. "Enable or Disable Gamification?":Analyzing the Impact of Choice in a Gamified Image Tagging Task. In
Proceedings of the 2019 CHI Conference onHuman Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19) . Association for Computing Machinery, NewYork, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300380[66] Zipeng Liu, Zhicheng Liu, and Tamara Munzner. 2020. Data-Driven Multi-Level Segmentation of Image Editing Logs.In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20) .Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376152[67] Suman Kalyan Maity, Aishik Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2017. Detection of Sockpuppetsin Social Media. In
Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and SocialComputing (Portland, Oregon, USA) (CSCW ’17) . Association for Computing Machinery, New York, NY, USA, 243–246.https://doi.org/10.1145/3022198.3026360[68] Jorge Maldonado-Mahauad, Mar Pérez-Sanagustín, René F. Kizilcec, Nicolás Morales, and Jorge Munoz-Gama. 2018.Mining theory-based patterns from Big data: Identifying self-regulated learning strategies in Massive Open OnlineCourses.
Computers in Human Behavior
80 (2018), 179 – 196. https://doi.org/10.1016/j.chb.2017.11.011[69] Hiroaki Masaki, Kengo Shibata, Shui Hoshino, Takahiro Ishihama, Nagayuki Saito, and Koji Yatani. 2020. ExploringNudge Designs to Help Adolescent SNS Users Avoid Privacy and Safety Threats. In
Proceedings of the 2020 CHIConference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20) . Association for ComputingMachinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376666[70] Miller McPherson, Lynn Smith-Lovin, and James M Cook. 2001. Birds of a feather: Homophily in social networks.
Annual review of sociology
27, 1 (2001), 415–444.[71] Pardis Miri, Emily Jusuf, Andero Uusberg, Horia Margarit, Robert Flory, Katherine Isbister, Keith Marzullo, and James J.Gross. 2020. Evaluating a Personalizable, Inconspicuous Vibrotactile(PIV) Breathing Pacer for In-the-Moment AffectJ. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021.
Regulation. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20) . Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376757[72] Reza Hadi Mogavi, Sujit Gujar, Xiaojuan Ma, and Pan Hui. 2019. HRCR: Hidden Markov-Based Reinforcement toReduce Churn in Question Answering Forums. In
PRICAI 2019: Trends in Artificial Intelligence . Springer InternationalPublishing, 364–376. https://doi.org/10.1007/978-3-030-29908-8_29[73] Michael Morgan, Matthew Butler, Neena Thota, and Jane Sinclair. 2018. How CS Academics View Student Engagement.In
Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca,Cyprus) (ITiCSE 2018) . ACM, New York, NY, USA, 284–289. https://doi.org/10.1145/3197091.3197092[74] Abdallah Moubayed, Mohammadnoor Injadat, Abdallah Shami, and Hanan Lutfiyya. 2020. Student Engagement Levelin an e-Learning Environment: Clustering Using K-means.
American Journal of Distance Education
34, 2 (March 2020),137–156. https://doi.org/10.1080/08923647.2020.1696140[75] Benjamin Murauer and Günther Specht. 2018. Detecting Music Genre Using Extreme Gradient Boosting. In
CompanionProceedings of the The Web Conference 2018 (Lyon, France) (WWW ’18) . International World Wide Web ConferencesSteering Committee, Republic and Canton of Geneva, CHE, 1923–1927. https://doi.org/10.1145/3184558.3191822[76] Saurabh Nagrecha, John Z. Dillon, and Nitesh V. Chawla. 2017. MOOC Dropout Prediction: Lessons Learned fromMaking Pipelines Interpretable. In
Proceedings of the 26th International Conference on World Wide Web Companion (Perth, Australia) (WWW ’17) . International World Wide Web Conferences Steering Committee, Republic and Cantonof Geneva, CHE, 351–359. https://doi.org/10.1145/3041021.3054162[77] Junpei Naito, Yukino Baba, Hisashi Kashima, Takenori Takaki, and Takuya Funo. 2018. Predictive modeling of learningcontinuation in preschool education using temporal patterns of development tests. In
Thirty-Second AAAI Conferenceon Artificial Intelligence .[78] Tuan Dinh Nguyen, Marisa Cannata, and Jason Miller. 2016. Understanding student behavioral engagement: Impor-tance of student interaction with peers and teachers.
The Journal of Educational Research
Hidden Markov Models . SpringerNew York, 103–113. https://doi.org/10.1007/978-1-4939-6753-7_7[80] Judith A Ouimet and Robert A Smallwood. 2005. Assessment Measures: CLASSE–The Class-Level Survey of StudentEngagement.
Assessment Update
17, 6 (2005), 13–15.[81] Steven Pace. 2004. A grounded theory of the flow experiences of Web users.
International Journal of Human-ComputerStudies
60, 3 (2004), 327 – 363. https://doi.org/10.1016/j.ijhcs.2003.08.005[82] Zilong Pan, Chenglu Li, and Min Liu. 2020. Learning Analytics Dashboard for Problem-Based Learning. In
Proceedingsof the Seventh ACM Conference on Learning @ Scale (Virtual Event, USA) (L@S ’20) . Association for ComputingMachinery, New York, NY, USA, 393–396. https://doi.org/10.1145/3386527.3406751[83] Sira Park, Susan D. Holloway, Amanda Arendtsz, Janine Bempechat, and Jin Li. 2011. What Makes Students Engagedin Learning? A Time-Use Study of Within- and Between-Individual Predictors of Emotional Engagement in Low-Performing High Schools.
Journal of Youth and Adolescence
41, 3 (Dec. 2011), 390–401. https://doi.org/10.1007/s10964-011-9738-3[84] Sophie Parsons, Peter M. Atkinson, Elena Simperl, and Mark Weal. 2015. Thematically Analysing Social NetworkContent During Disasters Through the Lens of the Disaster Management Lifecycle. In
Proceedings of the 24thInternational Conference on World Wide Web (Florence, Italy) (WWW ’15 Companion) . ACM, New York, NY, USA,1221–1226. https://doi.org/10.1145/2740908.2741721[85] Filipe D. Pereira, Elaine Oliveira, Alexandra Cristea, David Fernandes, Luciano Silva, Gene Aguiar, Ahmed Alamri, andMohammad Alshehri. 2019. Early Dropout Prediction for Programming Courses Supported by Online Judges. In
LectureNotes in Computer Science . Springer International Publishing, 67–72. https://doi.org/10.1007/978-3-030-23207-8_13[86] Jordi Petit, Omer Giménez, and Salvador Roura. 2012. Jutge.Org: An Educational Programming Judge. In
Proceedingsof the 43rd ACM Technical Symposium on Computer Science Education (Raleigh, North Carolina, USA) (SIGCSE ’12) .ACM, New York, NY, USA, 445–450. https://doi.org/10.1145/2157136.2157267[87] Math Playground. 2019.
Math Playground
Proceedings of the 23rd International Conference on World Wide Web (Seoul,Korea) (WWW ’14 Companion) . ACM, New York, NY, USA, 469–474. https://doi.org/10.1145/2567948.2576965[89] Jiezhong Qiu, Jie Tang, Tracy Xiao Liu, Jie Gong, Chenhui Zhang, Qian Zhang, and Yufei Xue. 2016. Modeling andPredicting Learning Behavior in MOOCs. In
Proceedings of the Ninth ACM International Conference on Web Search andData Mining (San Francisco, California, USA) (WSDM ’16) . Association for Computing Machinery, New York, NY,USA, 93–102. https://doi.org/10.1145/2835776.2835842[90] Rezvaneh Rezapour and Jana Diesner. 2017. Classification and Detection of Micro-Level Impact of Issue-FocusedDocumentary Films Based on Reviews. In
Proceedings of the 2017 ACM Conference on Computer Supported Cooperative
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. haracterizing Student Engagement Moods for Dropout Prediction in Question Pool Websites 111:21
Work and Social Computing (Portland, Oregon, USA) (CSCW ’17) . Association for Computing Machinery, New York,NY, USA, 1419–1431. https://doi.org/10.1145/2998181.2998201[91] Juan D Rodriguez, Aritz Perez, and Jose A Lozano. 2009. Sensitivity analysis of k-fold cross validation in predictionerror estimation.
IEEE transactions on pattern analysis and machine intelligence
32, 3 (2009), 569–575.[92] Cristóbal Romero and Sebastián Ventura. 2016. Educational data science in massive open online courses.
WileyInterdisciplinary Reviews: Data Mining and Knowledge Discovery
7, 1 (Sept. 2016), e1187. https://doi.org/10.1002/widm.1187[93] Matthew Rowe. 2013. Mining user lifecycles from online community platforms and their application to churnprediction. In . IEEE, 637–646.[94] Victor B. Saenz, Deryl Hatch, Beth E. Bukoski, Suyun Kim, Kye hyoung Lee, and Patrick Valdez. 2011. CommunityCollege Student Engagement Patterns.
Community College Review
39, 3 (July 2011), 235–267. https://doi.org/10.1177/0091552111416643[95] Koya Sato, Mizuki Oka, and Kazuhiko Kato. 2019. Early Churn User Classification in Social Networking Service UsingAttention-Based Long Short-Term Memory. In
Lecture Notes in Computer Science . Springer International Publishing,45–56. https://doi.org/10.1007/978-3-030-26142-9_5[96] Junfeng Shang and Joseph E. Cavanaugh. 2008. Bootstrap variants of the Akaike information criterion for mixed modelselection.
Computational Statistics & Data Analysis
52, 4 (2008), 2004 – 2021. https://doi.org/10.1016/j.csda.2007.06.019[97] David J Shernoff and Mihaly Csikszentmihalyi. 2009. Cultivating engaged learners and optimal learning environments.
Handbook of positive psychology in schools (2009), 131–145.[98] Hongzhi Shi, Chao Zhang, Quanming Yao, Yong Li, Funing Sun, and Depeng Jin. 2019. State-Sharing Sparse HiddenMarkov Models for Personalized Sequences. In
Proceedings of the 25th ACM SIGKDD International Conference onKnowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD ’19) . Association for Computing Machinery, NewYork, NY, USA, 1549–1559. https://doi.org/10.1145/3292500.3330828[99] Hyunjin Shin, Bugeun Kim, and Gahgene Gweon. 2020. Guessing or Solving? Exploring the Use of Motion Featuresfrom Educational Game Logs. In
Extended Abstracts of the 2020 CHI Conference on Human Factors in ComputingSystems (Honolulu, HI, USA) (CHI EA ’20) . Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3383005[100] Manya Sleeper, Alessandro Acquisti, Lorrie Faith Cranor, Patrick Gage Kelley, Sean A. Munson, and Norman Sadeh.2015. I Would Like To..., I Shouldn’t..., I Wish I...: Exploring Behavior-Change Goals for Social Networking Sites. In
Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC,Canada) (CSCW ’15) . Association for Computing Machinery, New York, NY, USA, 1058–1069. https://doi.org/10.1145/2675133.2675193[101] A. Spanos. 2019.
Probability Theory and Statistical Inference: Empirical Modeling with Observational Data . CambridgeUniversity Press. https://books.google.com/books?id=nm_IDwAAQBAJ[102] Ivan Srba and Maria Bielikova. 2016. A Comprehensive Survey and Classification of Approaches for CommunityQuestion Answering.
ACM Trans. Web
10, 3, Article 18 (Aug. 2016), 63 pages. https://doi.org/10.1145/2934687[103] Cor JM Suhre, Ellen PWA Jansen, and Egbert G Harskamp. 2007. Impact of degree program satisfaction on thepersistence of college students.
Higher Education
54, 2 (2007), 207–226.[104] Stephanie D. Teasley. 2017. Student Facing Dashboards: One Size Fits All?
Technology, Knowledge and Learning
22, 3(01 Oct 2017), 377–384. https://doi.org/10.1007/s10758-017-9314-3[105] Jacob Thebault-Spieker, Anbang Xu, Jilin Chen, Jalal Mahmud, and Jeffrey Nichols. 2016. Exploring Engagement in a’Social Crowd’ on Twitter. In
Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work andSocial Computing Companion (San Francisco, California, USA) (CSCW ’16) . Association for Computing Machinery,New York, NY, USA, 417–420. https://doi.org/10.1145/2818052.2869112[106] Timus. 2019.
Timus Online Judge . Retrieved January 24, 2019 from https://acm.timus.ru/[107] Gustavo F. Tondello, Dennis L. Kappen, Marim Ganaba, and Lennart E. Nacke. 2019. Gameful Design Heuristics:A Gamification Inspection Tool. In
Human-Computer Interaction. Perspectives on Design . Springer InternationalPublishing, 224–244. https://doi.org/10.1007/978-3-030-22646-6_16[108] Gustavo F. Tondello, Rina R. Wehbe, Lisa Diamond, Marc Busch, Andrzej Marczewski, and Lennart E. Nacke. 2016. TheGamification User Types Hexad Scale. In
Proceedings of the 2016 Annual Symposium on Computer-Human Interactionin Play (Austin, Texas, USA) (CHI PLAY ’16) . Association for Computing Machinery, New York, NY, USA, 229–243.https://doi.org/10.1145/2967934.2968082[109] Richard Tucker. 2016.
Collaboration and Student Engagement in Design Education . IGI Global.[110] T. Vafeiadis, K.I. Diamantaras, G. Sarigiannidis, and K.Ch. Chatzisavvas. 2015. A comparison of machine learningtechniques for customer churn prediction.
Simulation Modelling Practice and Theory
55 (2015), 1 – 9. https://doi.org/10.1016/j.simpat.2015.03.003[111] Wil Van Der Aalst. 2011.
Process mining: discovery, conformance and enhancement of business processes . Vol. 2. Springer.J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2021. [112] Devin Waldrop, Amy L. Reschly, Kathleen Fraysier, and James J. Appleton. 2018. Measuring the Engagement of CollegeStudents: Administration Format, Structure, and Validity of the Student Engagement Instrument–College.
Measurementand Evaluation in Counseling and Development
52, 2 (Nov. 2018), 90–107. https://doi.org/10.1080/07481756.2018.1497429[113] Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019. Designing Theory-Driven User-Centric ExplainableAI. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI’19) . Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300831[114] Qianwen Wang, Yao Ming, Zhihua Jin, Qiaomu Shen, Dongyu Liu, Micah J. Smith, Kalyan Veeramachaneni, andHuamin Qu. 2019. ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19) .Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300911[115] Wei Wang, Han Yu, and Chunyan Miao. 2017. Deep Model for Dropout Prediction in MOOCs. In
Proceedings of the2nd International Conference on Crowd Science and Engineering (Beijing, China) (ICCSE’17) . Association for ComputingMachinery, New York, NY, USA, 26–32. https://doi.org/10.1145/3126973.3126990[116] Vicky Ward, Allan House, and Susan Hamer. 2009. Developing a framework for transferring knowledge into action: athematic analysis of the literature.
Journal of health services research & policy
14, 3 (2009), 156–164.[117] Szymon Wasik, Maciej Antczak, Jan Badura, Artur Laskowski, and Tomasz Sternal. 2016. Optil.Io: Cloud Based PlatformFor Solving Optimization Problems Using Crowdsourcing Approach. In
Proceedings of the 19th ACM Conference onComputer Supported Cooperative Work and Social Computing Companion (San Francisco, California, USA) (CSCW ’16) .Association for Computing Machinery, New York, NY, USA, 433–436. https://doi.org/10.1145/2818052.2869098[118] Szymon Wasik, Maciej Antczak, Jan Badura, Artur Laskowski, and Tomasz Sternal. 2018. A Survey on Online JudgeSystems and Their Applications.
ACM Comput. Surv.
51, 1, Article 3 (Jan. 2018), 34 pages. https://doi.org/10.1145/3143560[119] Jacob Whitehill, Kiran Mohan, Daniel Seaton, Yigal Rosen, and Dustin Tingley. 2017. MOOC Dropout Prediction:How to Measure Accuracy?. In
Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (Cambridge,Massachusetts, USA) (L@S ’17) . Association for Computing Machinery, New York, NY, USA, 161–164. https://doi.org/10.1145/3051457.3053974[120] Meng Xia, Mingfei Sun, Huan Wei, Qing Chen, Yong Wang, Lei Shi, Huamin Qu, and Xiaojuan Ma. 2019. PeerLens:Peer-inspired Interactive Learning Path Planning in Online Question Pool. In
Proceedings of the 2019 CHI Conferenceon Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19) . ACM, New York, NY, USA, Article 634,12 pages. https://doi.org/10.1145/3290605.3300864[121] Wanli Xing and Dongping Du. 2018. Dropout Prediction in MOOCs: Using Deep Learning for Personalized Intervention.
Journal of Educational Computing Research
57, 3 (March 2018), 547–570. https://doi.org/10.1177/0735633118757015[122] Yuan Xuan, Yang Chen, Huiying Li, Pan Hui, and Lei Shi. 2016. LBSNShield: Malicious Account Detection in Location-Based Social Networks. In
Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and SocialComputing Companion (San Francisco, California, USA) (CSCW ’16) . Association for Computing Machinery, NewYork, NY, USA, 437–440. https://doi.org/10.1145/2818052.2869094[123] Carl Yang, Xiaolin Shi, Luo Jie, and Jiawei Han. 2018. I Know You’ll Be Back: Interpretable New User Clustering andChurn Prediction on a Mobile Social Application. In
Proceedings of the 24th ACM SIGKDD (London, United Kingdom) (KDD ’18) . ACM, New York, NY, USA, 914–922. https://doi.org/10.1145/3219819.3219821[124] Yang Yang, Zongtao Liu, Chenhao Tan, Fei Wu, Yueting Zhuang, and Yafeng Li. 2018. To Stay or to Leave: ChurnPrediction for Urban Migrants in the Initial Period. In
Proceedings of the 2018 World Wide Web Conference (Lyon,France) (WWW ’18) . International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva,CHE, 967–976. https://doi.org/10.1145/3178876.3186144[125] Zamzami Zainuddin, Samuel Kai Wah Chu, Muhammad Shujahat, and Corinne Jacqueline Perera. 2020. The impactof gamification on learning and instruction: A systematic review of empirical evidence.
Educational Research Review (2020), 100326.[126] Jiawei Zhang, Xiangnan Kong, and S Yu Philip. 2016. Social badge system analysis. In . IEEE, 453–460.[127] Xi Zhang, Shan Jiang, and Yihang Cheng. 2017. Inferring the Student Social Loafing State in Collaborative Learningwith a Hidden Markov Model: A Case on Slack. In
Proceedings of the 26th International Conference on World Wide WebCompanion (Perth, Australia) (WWW ’17) . International World Wide Web Conferences Steering Committee, Republicand Canton of Geneva, CHE, 149–152. https://doi.org/10.1145/3041021.3054145[128] Sabrina Zirkel, Julie A Garcia, and Mary C Murphy. 2015. Experience-sampling research methods and their potentialfor education research.