Zacharoula K. Papamitsiou

Temporal learning analytics for computer based testing

Predicting students performance is a challenging, yet complicated task for institutions, instructors and learners. Accurate predictions of performance could lead to improved learning outcomes and increased goal achievement. In this paper we explore the predictive capabilities of students time-spent on answering (in-)correctly each question of a multiple-choice assessment quiz, along with students final quiz-score, in the context of computer-based testing. We also explore the correlation between the time-spent factor (as defined here) and goal-expectancy. We present a case study and investigate the value of using this parameter as a learning analytics factor for improving prediction of performance during computer-based testing. Our initial results are encouraging and indicate that the temporal dimension of learning analytics should be further explored.

On the subjective difficulty of Joystick-based robot arm teleoperation with auditory feedback

Joystick-based teleoperation is a dominant method for remotely controlling various types of robots, such as excavators, cranes, and space telerobotics. Our ultimate goal is to create effective methods for training and assessing human operators of joystick-controlled robots. Towards that goal, in this paper we present an extensive study consisting of 18 experimental subjects controlling a simulated robot, using either no feedback or auditory feedback. Multiple observables were recorded, including not only joystick and robot angles and timings, but also subjective measures of difficulty, personality and usability data, and automated analysis of facial expressions and blink rate of the subjects. Our initial results indicate that: First, that the subjective difficulty of teleoperation with auditory feedback has smaller variance as compared to teleoperation without feedback, and second, that the subjective difficulty of a task is linearly related with the logarithm of task completion time. We conclude with a forward-looking discussion including future steps.

Learning Analytics for Smart Learning Environments: A Meta-Analysis of Empirical Research Results from 2009 to 2015

Although several qualitative analyses appeared in the domain of Learning Analytics (LA), a systematic quantitative analysis of the effects of the empirical research findings toward the development of more reliable Smart Learning Environments (SLE) is still missing. This chapter aims at preserving and enhancing the chronicles of recent LA developments as well as covering the Z. Papamitsiou (*) • A.A. Economides Interdepartmental Programme of Postgraduate Studies in Information Systems, University of Macedonia, Thessaloniki, Greece e-mail: papamits@uom.edu.gr; economid@uom.gr # Springer International Publishing Switzerland 2016 J.M. Spector et al. (eds.), Learning, Design, and Technology, DOI 10.1007/978-3-319-17727-4_15-1 1 abovementioned gap. The core question is where these two research areas intersect and how the significant LA research findings could be beneficial for guiding the construction of SLEs. This meta-analysis study synthesizes research on the effectiveness of LA and targets at determining the influence of its dimensions on learning outcomes so far. Sixty-six experimental and quasi-experimental papers published from 2009 through September 2015 in the domain of LA were coded and analyzed. Overall, the weighted random effects mean effect size (g) was 0.433 (p = 0.001). The collection was heterogeneous (Qt(66) = 78.47). Here, the results of the statistical and classification processes applied during the meta-analysis process are presented and the most important issues raised are discussed.

Process Mining of Interactions During Computer-Based Testing for Detecting and Modelling Guessing Behavior

Detecting, recognizing and modelling patterns of observed examinee behaviors during assessment is a topic of great interest for the educational research community. In this paper we investigate the perspectives of process-centric inference of guessing behavior patterns. The underlying idea is to extract knowledge from real processes (i.e., not assumed nor truncated), logged automatically by the assessment environment. We applied a three-step process mining methodology on logged interaction traces from a case study with 259 undergraduate university students. The analysis revealed sequences of interactions in which low goal-orientation students answered quickly and correctly on difficult items, without reviewing them, while they submitted wrong answers on easier items. We assumed that this implies guessing behavior. From the conformance checking and performance analysis we found that the fitness of our process model is almost 85 %. Hence, initial results are encouraging towards modelling guessing behavior. Potential implications and future work plans are also discussed.

An Assessment Analytics Framework (AAF) for Enhancing Students’ Progress

e-Assessment has been acknowledged as one of the most accurate methods to track and measure students’ progress. Assessment analytics are about revealing the intelligence held in e-assessment systems. However, a framework that would organize empirical assessment analytics results is missing. In this chapter, we suggest a theoretical framework aiming at: (a) developing a conceptual representation that will act as a reference point for discussing the literature, (b) developing a theory that could be used to move beyond descriptions of “what” to explanations of “why” and “how,” and (c) providing a structure that could act as a useful guide to understand more deeply, evaluate and design analytics for assessment. We followed an inductive and deductive inquiry methodology for conceptual mapping for sensemaking during construction of the framework. In this chapter, we discuss about the main concepts involved in the proposed theory, explaining how former research papers fit in the suggested framework.

Temporal learning analytics visualizations for increasing awareness during assessment

Visual representations of student-generated trace data during learning activities help both students and instructors interpret them intuitively and perceive hidden aspects of these data quickly. In this paper, we elaborate on the visualization of temporal trace data during assessment. The goals of the study were twofold: a) to depict students’ engagement in the assessment procedure in terms of time spent and temporal factors associated with learning-specific characteristics, and b) to explore the factors that influence the teachers’ Behavioural Intention to use the proposed system as an information system and their perceptions of the effectiveness and acceptance of our approach. The proposed visualizations have been explored in a study with 32 Secondary Education teachers. We adopted a design-based research methodology and employed a survey instrument — based on the Learning Analytics Acceptance Model (LAAM) — in order to measure the expected impact of the proposed visualizations. The analysis of the findings indicates that a) temporal factors can be used for visualizing students’ behaviour during assessment, and b) the visualization of the temporal dimension of students’ behaviour increases teachers’ awareness of students’ progress, possible misconceptions (e.g., guessing the correct answer) and task difficulty.ResumenLas representaciones visuales de datos de trazas generados por el alumnado durante las actividades de aprendizaje ayudan tanto a los estudiantes como a los profesores a interpretarlos intuitivamente y a percibir con rapidez aspectos ocultos. En este trabajo, describimos la visualización de datos de trazas temporales durante la evaluación. El estudio tenía un doble objetivo: a) describir la implicación de los estudiantes en el proceso de evaluación en cuanto a tiempo invertido y factores temporales asociados con características concretas del aprendizaje, y b) explorar los factores que influyen en la intención comportamental del profesorado en cuanto a emplear el sistema propuesto como sistema de información y sus percepciones de la efectividad y la aceptación de nuestro enfoque. Las visualizaciones propuestas se han examinado en un estudio con 32 profesores de educación secundaria. Adoptamos una metodología de investigación basada en el diseño y utilizamos un instrumento de encuesta -basada en el modelo de aceptación del análisis del aprendizaje- para medir el impacto esperado de las visualizaciones propuestas. El análisis de los hallazgos indica que a) los factores temporales se pueden utilizar para visualizar el comportamiento de los estudiantes durante la evaluación, y b) la visualización de la dimensión temporal del comportamiento de los estudiantes aumenta el conocimiento del profesor respecto al progreso de los alumnos, posibles conceptos erróneos (por ejemplo, adivinar la respuesta correcta) y dificultad de la tarea.

Exhibiting achievement behavior during computer-based testing: What temporal trace data and personality traits tell us?

Abstract Personalizing computer-based testing services to examinees can be improved by considering their behavioral models. This study aims to contribute towards deeper understanding the examinee’s time-spent and achievement behavior during testing according to the five personality traits by exploiting assessment analytics. Further, it aims to investigate assessment analytics appropriateness for classifying students and generating enhanced student models to guide personalization of testing services. In this study, the LAERS assessment environment and the Big Five Inventory were used to track the response times of 112 undergraduate students and to extract their personality traits respectively. Partial Least Squares was used to detect fundamental relationships between the collected data, and Supervised Learning Algorithms were used to classify students. Results indicate a positive effect of extraversion and agreeableness on goal-expectancy, a positive effect of conscientiousness on both goal-expectancy and level of certainty, and a negative effect of neuroticism and openness on level of certainty. Further, extraversion, agreeableness and conscientiousness have statistically significant indirect impact on students’ response-times and level of achievement. Moreover, the ensemble RandomForest method provides accurate classification results, indicating that a time-spent driven description of students’ behavior could have added value towards dynamically reshaping the respective models. Further implications of these findings are also discussed.

Visualitzacions d’anàlisi temporal de l’aprenentatge per augmentar el coneixement durant l’avaluació

Visual representations of student-generated trace data during learning activities help both students and instructors interpret them intuitively and perceive hidden aspects of these data quickly. In this paper, we elaborate on the visualization of temporal trace data during assessment. The goals of the study were twofold: a) to depict students’ engagement in the assessment procedure in terms of time spent and temporal factors associated with learning-specific characteristics, and b) to explore the factors that influence the teachers’ Behavioural Intention to use the proposed system as an information system and their perceptions of the effectiveness and acceptance of our approach. The proposed visualizations have been explored in a study with 32 Secondary Education teachers. We adopted a design-based research methodology and employed a survey instrument — based on the Learning Analytics Acceptance Model (LAAM) — in order to measure the expected impact of the proposed visualizations. The analysis of the findings indicates that a) temporal factors can be used for visualizing students’ behaviour during assessment, and b) the visualization of the temporal dimension of students’ behaviour increases teachers’ awareness of students’ progress, possible misconceptions (e.g., guessing the correct answer) and task difficulty.ResumenLas representaciones visuales de datos de trazas generados por el alumnado durante las actividades de aprendizaje ayudan tanto a los estudiantes como a los profesores a interpretarlos intuitivamente y a percibir con rapidez aspectos ocultos. En este trabajo, describimos la visualización de datos de trazas temporales durante la evaluación. El estudio tenía un doble objetivo: a) describir la implicación de los estudiantes en el proceso de evaluación en cuanto a tiempo invertido y factores temporales asociados con características concretas del aprendizaje, y b) explorar los factores que influyen en la intención comportamental del profesorado en cuanto a emplear el sistema propuesto como sistema de información y sus percepciones de la efectividad y la aceptación de nuestro enfoque. Las visualizaciones propuestas se han examinado en un estudio con 32 profesores de educación secundaria. Adoptamos una metodología de investigación basada en el diseño y utilizamos un instrumento de encuesta -basada en el modelo de aceptación del análisis del aprendizaje- para medir el impacto esperado de las visualizaciones propuestas. El análisis de los hallazgos indica que a) los factores temporales se pueden utilizar para visualizar el comportamiento de los estudiantes durante la evaluación, y b) la visualización de la dimensión temporal del comportamiento de los estudiantes aumenta el conocimiento del profesor respecto al progreso de los alumnos, posibles conceptos erróneos (por ejemplo, adivinar la respuesta correcta) y dificultad de la tarea.

Visualizaciones del análisis temporal del aprendizaje para aumentar el conocimiento durante la evaluación

Visual representations of student-generated trace data during learning activities help both students and instructors interpret them intuitively and perceive hidden aspects of these data quickly. In this paper, we elaborate on the visualization of temporal trace data during assessment. The goals of the study were twofold: a) to depict students’ engagement in the assessment procedure in terms of time spent and temporal factors associated with learning-specific characteristics, and b) to explore the factors that influence the teachers’ Behavioural Intention to use the proposed system as an information system and their perceptions of the effectiveness and acceptance of our approach. The proposed visualizations have been explored in a study with 32 Secondary Education teachers. We adopted a design-based research methodology and employed a survey instrument — based on the Learning Analytics Acceptance Model (LAAM) — in order to measure the expected impact of the proposed visualizations. The analysis of the findings indicates that a) temporal factors can be used for visualizing students’ behaviour during assessment, and b) the visualization of the temporal dimension of students’ behaviour increases teachers’ awareness of students’ progress, possible misconceptions (e.g., guessing the correct answer) and task difficulty.ResumenLas representaciones visuales de datos de trazas generados por el alumnado durante las actividades de aprendizaje ayudan tanto a los estudiantes como a los profesores a interpretarlos intuitivamente y a percibir con rapidez aspectos ocultos. En este trabajo, describimos la visualización de datos de trazas temporales durante la evaluación. El estudio tenía un doble objetivo: a) describir la implicación de los estudiantes en el proceso de evaluación en cuanto a tiempo invertido y factores temporales asociados con características concretas del aprendizaje, y b) explorar los factores que influyen en la intención comportamental del profesorado en cuanto a emplear el sistema propuesto como sistema de información y sus percepciones de la efectividad y la aceptación de nuestro enfoque. Las visualizaciones propuestas se han examinado en un estudio con 32 profesores de educación secundaria. Adoptamos una metodología de investigación basada en el diseño y utilizamos un instrumento de encuesta -basada en el modelo de aceptación del análisis del aprendizaje- para medir el impacto esperado de las visualizaciones propuestas. El análisis de los hallazgos indica que a) los factores temporales se pueden utilizar para visualizar el comportamiento de los estudiantes durante la evaluación, y b) la visualización de la dimensión temporal del comportamiento de los estudiantes aumenta el conocimiento del profesor respecto al progreso de los alumnos, posibles conceptos erróneos (por ejemplo, adivinar la respuesta correcta) y dificultad de la tarea.