Gert Janssenswillen

Measuring the Quality of Models with Respect to the Underlying System: An Empirical Study

Fitness and precision are two widely studied criteria to determine the quality of a discovered process model. These metrics measure how well a model represents the log from which it is learned. However, often the goal of discovery is not to represent the log, but the underlying system. This paper discusses the need to explicitly distinguish between a log and system perspective when interpreting the fitness and precision of a model. An empirical analysis was conducted to investigate whether the existing log-based fitness and precision measures are good estimators for system-based metrics. The analysis reveals that incompleteness and noisiness of event logs significantly impact fitness and precision measures. This makes them biased estimators of a model’s ability to represent the true underlying process.

A comparative study of existing quality measures for process discovery

Abstract Evaluating the quality of discovered process models is an important task in many process mining analyses. Currently, several metrics measuring the fitness, precision and generalization of a discovered model are implemented. However, there is little empirical evidence how these metrics relate to each other, both within and across these different quality dimensions. In order to better understand these relationships, a large-scale comparative experiment was conducted. The statistical analysis of the results shows that, although fitness and precision metrics behave very similar within their dimension, some are more pessimistic while others are more optimistic. Furthermore, it was found that there is no agreement between generalization metrics. The results of the study can be used to inform decisions on which quality metrics to use in practice. Moreover, they highlight issues which give rise to new directions for future research in the area of quality measurement.

Affordances and limitations of learning analytics for computer-assisted language learning: a case study of the VITAL project

ABSTRACT Learning analytics (LA) has emerged as a field that offers promising new ways to prevent drop-out and aid retention. However, other research suggests that large datasets of learner activity can be used to understand online learning behaviour and improve pedagogy. While the use of LA in language learning has received little attention to date, available research suggests that LA could provide valuable insights into task design for instructors and materials designers, as well as help students with effective learning strategies and personalised learning pathways. This paper first discusses previous CALL research based on learner tracking and specific affordances of LA for CALL, as well as its inherent limitations and challenges. The second part of the paper analyses data arising from the VITAL project that implemented LA in different blended or distance learning settings. Statistical and process-mining techniques were applied to data from 285 undergraduate students on a Business French course. Results suggested that most students planned their self-study sessions in accordance with the flipped classroom design. Other metrics measuring active online engagement indicated significant differences between successful and non-successful students’ learner patterns. The research implied that valuable insights can be acquired through LA and the use of visualisation and process-mining tools.

Detecting train reroutings with process mining: A Belgian application

One of the objectives of railway infrastructure managers is to improve the punctuality of their operations while satisfying safety requirements and coping with limited capacity. To fulfil this objective, capacity planning and monitoring have become an absolute necessity. Railway infrastructure managers possess tremendous amounts of data about the railway operations, which are recorded in so-called train describer systems. In this paper, a set of methods is proposed to guide the analysis of capacity usage based on these data. In particular, train connections are categorized according to the severity of train reroutings as well as the diversity of these reroutings. The applied method is able to highlight areas in the railway network, where trains have a higher tendency to diverge from their allocated route. The method is independent from the underlying infrastructure, and can, therefore, be reused effortlessly on new cases. The analysis provides a starting point to improve the planning of capacity usage and can be used to facilitate the communication between capacity planning at one hand and operations on the other hand. At the same time, it presents an illustration on how process mining can be used for analysis of train describer data.