Asli Arslan Esme

Behavioral Engagement Detection of Students in the Wild

This paper aims to investigate students’ behavioral engagement (On-Task vs. Off-Task) in authentic classrooms. We propose a two-phased approach for automatic engagement detection: In Phase 1, contextual logs are utilized to assess active usage of the content platform. If there is active use, the appearance information is utilized in Phase 2 to infer behavioral engagement. Through authentic classroom pilots, we collected around 170 hours of in-the-wild data from 28 students in two different classrooms using two different content platforms (one for Math and one for English as a Second Language (ESL)). Our data collection application captured appearance data from a 3D camera and context data from uniform resource locator (URL) logs. We experimented with two test cases: (1) Cross-classroom, where trained models were tested on a different classroom’s data; (2) Cross-platform, where the data collected in different subject areas (Math or ESL) were utilized in training and testing, respectively. For the first case, the behavioral engagement was detected with an F1-score of 77%, using only appearance. Incorporating the contextual information improved the overall performance to 82%. For the second case, even though the subject areas and content platforms changed, the proposed appearance classifier still achieved 72% accuracy (compared to 77%). Our experiments proved that the accuracy of the proposed model is not adversely impacted considering different set of students or different subject areas.

Learner Engagement Measurement and Classification in 1:1 Learning

We explore the feasibility of measuring learner engagement and classifying the engagement level based on machine learning applied on data from 2D/3D camera sensors and eye trackers in a 1:1 learning setting. Our results are based on nine pilot sessions held in a local high school where we recorded features related to student engagement while consuming educational content. We label the collected data as Engaged or NotEngaged while observing videos of the students and their screens. Based on the collected data, perceptual user features (e.g., body posture, facial points, and gaze) are extracted. We use feature selection and classification methods to produce classifiers that can detect whether a student is engaged or not. Accuracies of up to 85-95% are achieved on the collected dataset. We believe our work pioneers in the successful classification of student engagement based on perceptual user features in a 1:1 authentic learning setting.

An unobtrusive and multimodal approach for behavioral engagement detection of students

In this paper, we investigate detection of students’ behavioral engagement states (On-Task vs. Off-Task) in authentic classroom settings. We propose a multimodal detection approach, based on three unobtrusive modalities readily available in a 1:1 learning scenario where learning technologies are incorporated. These modalities are: (1)Appearance: upper-body video captured using a camera; (2) Context-Performance: students’ interaction and performance data related to learning content; and (3) Mouse: data related to mouse movements during learning process. For each modality, separate unimodal classifiers were trained, and decision-level fusion was applied to obtain final behavioral engagement states. We also analyzed each modality based on Instructional and Assessment sections separately (i.e., Instructional where a student is reading an article or watching an instructional video vs. Assessment where a student is solving exercises on the digital learning platform). We carried out various experiments on a dataset collected in an authentic classroom, where students used laptops equipped with cameras and they consumed learning content for Math on a digital learning platform. The dataset included multimodal data of 17 students who attended a Math course for 13 sessions (40 minutes each). The results indicate that it is beneficial to have separate classification pipelines for Instructional and Assessment sections: For Instructional, using only Appearance modality yields an F1-measure of 0.74, compared to fused performance of 0.70. For Assessment, fusing all three modalities (F1-measure of 0.89) provide a prominent improvement over the best performing unimodality (i.e., 0.81 for Appearance).

Towards Human Affect Modeling: A Comparative Analysis of Discrete Affect and Valence-Arousal Labeling

There is still considerable disagreement on key aspects of affective computing - including even how affect itself is conceptualized. Using a multi-modal student dataset collected while students were watching instructional videos and answering questions on a learning platform, we investigated the two key paradigms of how affect is represented through a comparative approach: (1) Affect as a set of discrete states and (2) Affect as a combination of a two-dimensional space of attributes. We specifically examined a set of discrete learning-related affects (Satisfied, Confused, and Bored) that are hypothesized to map to specific locations within the Valence-Arousal dimensions of Circumplex Model of Emotion. For each of the key paradigms, we had five human experts label student affect on the dataset. We investigated two major research questions using their labels: (1) Whether the hypothesized mappings between discrete affects and Valence-Arousal are valid and (2) whether affect labeling is more reliable with discrete affect or Valence-Arousal. Contrary to the expected, the results show that discrete labels did not directly map to Valence-Arousal quadrants in Circumplex Model of Emotion. This indicates that the experts perceived and labeled these two relatively differently. On the other side, the inter-rater agreement results show that the experts moderately agreed with each other within both paradigms. These results imply that researchers and practitioners should consider how affect information would operationally be used in an intelligent system when choosing from the two key paradigms of affect.

Role of Socio-cultural Differences in Labeling Students’ Affective States

The development of real-time affect detection models often depends upon obtaining annotated data for supervised learning by employing human experts to label the student data. One open question in labeling affective data for affect detection is whether the labelers (i.e., human experts) need to be socio-culturally similar to the students being labeled, as this impacts the cost and feasibility of obtaining the labels. In this study, we investigate the following research questions: For affective state labeling, how does the socio-cultural background of human expert labelers, compared to the subjects (i.e., students), impact the degree of consensus and distribution of affective states obtained? Secondly, how do differences in labeler background impact the performance of affect detection models that are trained using these labels? To address these questions, we employed experts from Turkey and the United States to label the same data collected through authentic classroom pilots with students in Turkey. We analyzed within-country and cross-country inter-rater agreements, finding that experts from Turkey obtained moderately better inter-rater agreement than the experts from the U.S., and the two groups did not agree with each other. In addition, we observed differences between the distributions of affective states provided by experts in the U.S. versus Turkey, and between the performances of the resulting affect detectors. These results suggest that there are indeed implications to using human experts who do not belong to the same population as the research subjects.

Human Expert Labeling Process: Valence-Arousal Labeling for Students’ Affective States

Affect has emerged as an important part of the interaction between learners and computers, with important implications for learning outcomes. As a result, it has emerged as an important area of research within learning analytics. Reliable and valid data labeling is a key tenet for training machine learning models providing such analytics. In this study, using Human Expert Labeling Process (HELP) as a baseline labeling protocol, we investigated an optimized method through several experiments for labeling student affect based on Circumplex Model of Emotion (Valence-Arousal). Using the optimized method, we then had the human experts label a larger quantity of student data so that we could test and validate this method on a relatively larger and different dataset. The results showed that using the optimized method, the experts were able to achieve an acceptable consensus in labeling outcomes as aligned with affect labeling literature.