Toon Jouck

Measuring the Quality of Models with Respect to the Underlying System: An Empirical Study

Fitness and precision are two widely studied criteria to determine the quality of a discovered process model. These metrics measure how well a model represents the log from which it is learned. However, often the goal of discovery is not to represent the log, but the underlying system. This paper discusses the need to explicitly distinguish between a log and system perspective when interpreting the fitness and precision of a model. An empirical analysis was conducted to investigate whether the existing log-based fitness and precision measures are good estimators for system-based metrics. The analysis reveals that incompleteness and noisiness of event logs significantly impact fitness and precision measures. This makes them biased estimators of a model’s ability to represent the true underlying process.

A comparative study of existing quality measures for process discovery

Abstract Evaluating the quality of discovered process models is an important task in many process mining analyses. Currently, several metrics measuring the fitness, precision and generalization of a discovered model are implemented. However, there is little empirical evidence how these metrics relate to each other, both within and across these different quality dimensions. In order to better understand these relationships, a large-scale comparative experiment was conducted. The statistical analysis of the results shows that, although fitness and precision metrics behave very similar within their dimension, some are more pessimistic while others are more optimistic. Furthermore, it was found that there is no agreement between generalization metrics. The results of the study can be used to inform decisions on which quality metrics to use in practice. Moreover, they highlight issues which give rise to new directions for future research in the area of quality measurement.

An Improved Way for Measuring Simplicity During Process Discovery

In the domain of process discovery, there are four quality dimensions for evaluating process models of which simplicity is one. Simplicity is often measured using the size of a process model, the structuredness and the entropy. It is closely related to the process model understandability. Researchers from the domain of business process management (BPM) proposed several metrics for measuring the process model understandability. A part of these understandability metrics focus on the control-flow perspective, which is important for evaluating models from process discovery algorithms. It is remarkable that there are more of these metrics defined in the BPM literature compared to the number of proposed simplicity metrics. To research whether the understandability metrics capture more understandability dimensions than the simplicity metrics, an exploratory factor analysis was conducted on 18 understandability metrics. A sample of 4450 BPMN models, both manually modelled and artificially generated, is used. Four dimensions are discovered: token behaviour complexity, node IO complexity, path complexity and degree of connectedness. The conclusion of this analysis is that process analysts should be aware that the measurement of simplicity does not capture all dimensions of the understandability of process models.