Gayane Sedrakyan

Process-mining enabled feedback

Fast advancement of technology has led to an increased interest for using information technology to provide feedback based on learning behavior observations. This work outlines a novel approach for analyzing behavioral learner data through the application of process mining techniques specifically targeting a complex problem solving process. We realize this in the context of one particular learning case, namely, domain modeling. This work extends our previous research on process-mining analysis of domain modeling behavior of novices by elaborating with new insights from a replication study enhanced with an extra observation on how novices verify/validate models. The findings include a set of typical modeling and validation patterns that can be used to improve teaching guidance for domain modeling courses. From a scientific viewpoint, the results contribute to improving our knowledge on the cognitive aspects of problem-solving behavior of novices in the area of domain modeling, specifically regarding process-oriented feedback as opposed to traditional outcome feedback (is a solution correct? Why (not)?) usually applied in this type of courses. Ultimately, the outcomes of the work can be inspirational outside of the area of domain modeling as learning event data is becoming readily available through virtual learning environments and other information systems. This work extends our previous research on process mining of learner data.Complex task solving process for conceptual modeling by novices is analized.The extension includes an observation of extra activities for model validation.Modeling/validation patterns indicative for learning outcomes are found.The findings provide a basis for process-oriented guidance and feedback.

Technology-Enhanced Support for Learning Conceptual Modeling

This paper describes an optimized didactic environment to support and improve learning achievements for conceptual modeling. In particular, it describes computer-aided techniques to address various learning challenges observed in the teaching process such as: hybrid background of students, enrollment of a large number of students, the complexity of industrial tools and difficulties in abstract thinking. The didactic environment has been developed and subsequently optimized in the context of the course Architecture and Modeling of Management Information Systems. It includes 1) diagnostic testing with automated feedback 2) an adapted modeling tool 3) an MDA based simulation feature. The didactic tools were evaluated positively by the students and a positive impact was observed on the student’s capabilities to construct object-oriented conceptual models.

Feedback-enabled MDA-prototyping effects on modeling knowledge

This paper describes the effects of a feedback-enabled MDA prototyping tool on the validation cycle for conceptual models. We observe the effects of such prototyping method on learning outcomes of novice modelers. The impact is assessed based on the quality dimensions introduced by Conceptual Modeling Quality Framework (CMQF), more specifically with respect to semantic quality being affected by modeling knowledge. The current work proposes an extension to the techniques introduced in previous work in particular, experimenting with the prototyping tool by novice modelers. A positive impact has been observed on the learning achievements of novice modelers improving both modeling and language knowledge.

Evaluating Student-Facing Learning Dashboards of Affective States

Detection and visualizations of affective states of students in computer based learning environments have been proposed to support student awareness and improve learning. However, the evaluation of such visualizations with students in real life settings is an open issue. This research reports on our experiences from the use of four different types of dashboard visualizations in two user studies (n = 115). Students who participated in the studies were bachelor and master level students from two different study programs at two universities. The results indicate that usability, measured by interpretability, perceived usefulness and insight, is overall acceptable. However, the findings also suggest that interpretability of some visualizations, in terms of the capability to support emotion awareness, still needs to be improved. The level of students awareness about their emotions during learning activities based on the visualization interpretation varied depending on previous knowledge on visualization techniques. Furthermore, simpler visualizations resulted in better outcomes than more complex techniques.

Effects of simulation on novices' understanding of the concept of inheritance in conceptual modeling

In this paper we present our experience in the experimental development and use of simulation instrument for learning object-oriented conceptual modeling in a master level course on analysis and design of information systems. The focus of our research is on the teaching of one particular topic in object-oriented conceptual modeling - inheritance. The results from the pilot experimental study (with a student sample N = 32), demonstrate a positive effect of simulation-based learning method on the understanding by novice business analysts of the concept of inheritance when applied in a conceptual model.

Lightweight Semantic Prototyper for Conceptual Modeling

While much research work was devoted to conceptual model quality validation techniques, most of the existing tools in this domain focus on syntactic quality. Tool support for checking semantic quality (correspondence between the conceptual model and requirements of a domain to be engineered) is largely lacking. This work introduces a lightweight model-driven semantic prototyper to test/validate conceptual models. The goal of the tool is twofold: (1) to assist business analysts in validating semantic quality of conceptual business specifications using a fast prototyper to communicate with domain experts; (2) to support the learning perspective of conceptual modeling for less experienced modelers (such as students or novice analysts in their early career) to facilitate their progression to advanced level of expertise. The learning perspective is supported by providing automated feedback that visually links the test results to their causes in the model’s design. The effectiveness of the tool has been confirmed by means of empirical experimental studies.

Enriching Model Execution with Feedback to Support Testing of Semantic Conformance between Models and Requirements - Design and Evaluation of Feedback Automation Architecture

Model Driven Development (MDD) has traditionally been used to support model transformations and code generation. While plenty of techniques and tools are available to support modeling and transformations, tool support for checking the model quality in terms of semantic conformance with respect to the domain requirements is largely absent. In this work we present a model verification and validation approach based on model-driven feedback generation in a model-to-code transformation. The transformation is achieved using a single click. The generated output of the transformation is a compiled code which is achieved by a single click. This also serves as a rapid prototyping instrument that allows simulating a model (the terms prototyping and simulation are thus used interchangeably in the paper). The proposed feedback incorporation method in the generated prototype allows linking event execution in the generated code to its causes in the model used as input for the generation. The goal of the feedback is twofold: (1) to assist a modeler in validating semantic conformance of a model with respect to a domain to be engineered; (2) to support the learning perspective of less experienced modelers (such as students or junior analysts in their early career) by allowing them to detect modeling errors that result from the misinterpreted use of modeling language constructs. Within this work we focus on conceptual and platform independent models (PIM) that make use of two prominent UML diagrams – a class diagram (for modeling the structure of a system) and multiple interacting statecharts (for modeling a system’s dynamic behavior). The tool has been used in the context of teaching a requirements analysis and modeling course at KU Leuven. The proposed feedback generation technique has been constantly validated by means of “usability” evaluations, and demonstrates a high level of self-reported utility of the feedback. Additionally, the findings of our experimental studies also show a significant positive impact of feedback-enabled rapid prototyping method on semantic validation capabilities of novices. Despite our focus on specific diagramming techniques, the principles of the approach presented in this work can be used to support educational feedback automation for a broader spectrum of diagram types in the context of MDD and simulation.

Automating immediate and personalized feedback taking conceptual modelling education to a next level

Providing individual and immediate feedback to students is a critical factor for improving knowledge and skills acquisition in higher education. However, a growing number of students with very different heterogeneous profiles and unequal problem solving skills, as well as the lack of teaching resources, make it very challenging and sometimes even impossible to give immediate feedback at individual level. This paper presents a synthesis and progress of a long-term project that addresses this challenge in the context of conceptual modelling by developing SAiLE@CoMo, a smart and adaptive learning environment. By crafting innovative process analytics techniques and expert knowledge on feedback automation, SAiLE@CoMo can automatically provide personalized and immediate feedback to leaners.

Assessing the influence of feedback-inclusive rapid prototyping on understanding the semantics of parallel UML statecharts by novice modellers

Abstract Context UML diagrams are the de facto standard for analysing, communicating and designing software systems, as well as automated code generation. However there is a certain degree of difficulty in understanding a system represented by means of UML diagrams. Object Our previous research demonstrates a significant improvement in understanding the structural aspects of a system represented as a UML class diagram when using a feedback-inclusive prototype of a model. This paper extends our previous work with an empirical validation study for the effectiveness of the feedback-inclusive rapid prototyping (FIRP) method, on the comprehension of system dynamics represented as multiple interacting UML statecharts . Because models often combine structural and behavioural views that are highly intertwined, we additionally evaluate the effectiveness of the proposed method with respect to comprehension of the between-view consistency. Method The FIRP environment was built following the principles of Design Science Research in Information Systems. This study targets the empirical validation of the effectiveness of the proposed technique using an experimental study method. Two experiments were conducted with the participation of 65 final-year master students in the context of different modelling courses from different study programs at KU Leuven using two two-group factorial experimental designs. The effectiveness of the FIRP method was measured by comparing students’ performance between the cycles with and without the use of the method, using the understandability (comprehension test results) as the dependent variable and the use of FIRP as the independent variable. Effects from unknown variables were neutralized by means of randomized group compositions. The effectiveness of FIRP was additionally assessed with respect to personal characteristics (age, gender, previous knowledge, self-efficacy) and user acceptance (perceived ease of use, perceived utility, preference, satisfaction). Results The findings reveal a significant positive impact of the use of the prototyping technique on students’ comprehension of system dynamics represented as multiple interacting statecharts. Conclusions The findings provide empirical support for the advantage of the use of FIRP over manual inspection of interacting statecharts. The findings also suggest that the method is suitable for training systems analysis and modelling skills when UML statecharts are involved.

Cognitive Feedback and Behavioral Feedforward Automation Perspectives for Modeling and Validation in a Learning Context

State-of-the-art technologies have made it possible to provide a learner with immediate computer-assisted feedback by delivering a feedback targeting cognitive aspects of learning, (e.g. reflecting on a result, explaining a concept, i.e. improving understanding). Fast advancement of technology has recently generated increased interest for previously non-feasible approaches for providing feedback based on learning behavioral observations by exploiting different traces of learning processes stored in information systems. Such learner behavior data makes it possible to observe different aspects of learning processes in which feedback needs of learners (e.g. difficulties, engagement issues, inefficient learning processes, etc.) based on individual learning trajectories can be traced. By identifying problems earlier in a learning process it is possible to deliver individualized feedback helping learners to take control of their own learning, i.e. to become self-regulated learners, and teachers to understand individual feedback needs and/or adapt their teaching strategies. In this work we (i) propose cognitive computer-assisted feedback mechanisms using a combination of MDE based simulation augmented with automated feedback, and (ii) discuss perspectives for behavioral feedback, i.e. feedforward, that can be based on learning process analytics in the context of learning conceptual modeling. Aggregated results of our previous studies assessing the effectiveness of the proposed cognitive feedback method with respect to improved understanding on different dimensions of knowledge, as well as feasibility of behavioral feedforward automation based on learners behavior patterns, are presented. Despite our focus on conceptual modeling and specific diagrams, the principles of the approach presented in this work can be used to support educational feedback automation for a broader spectrum of diagram types beyond the scope of conceptual modeling.