Michel C. Desmarais

A review of recent advances in learner and skill modeling in intelligent learning environments

In recent years, learner models have emerged from the research laboratory and research classrooms into the wider world. Learner models are now embedded in real world applications which can claim to have thousands, or even hundreds of thousands, of users. Probabilistic models for skill assessment are playing a key role in these advanced learning environments. In this paper, we review the learner models that have played the largest roles in the success of these learning environments, and also the latest advances in the modeling and assessment of learner skills. We conclude by discussing related advancements in modeling other key constructs such as learner motivation, emotional and attentional state, meta-cognition and self-regulated learning, group learning, and the recent movement towards open and shared learner models.

Human-Centered Software Engineering - Integrating Usability in the Software Development Lifecycle

The fields of HCI and Software Engineering have evolved almost independently of each other until the last decade, when it became apparent that an integrated and combined perspective would benefit the development of interactive software applications. The chapters in this book are written by prominent researchers who bring to light the major integration issues and challenges, and offer a variety of solutions to bridging the HCI and SE gap, including: Extending software engineering artifacts for UI specification, such as annotating use cases with task descriptions, Enhancing object-oriented software engineering notations and models Possible extensions of HCI methods for requirements gathering through field observations and interviews, deriving a conceptual design model from scenario, task models and use cases and using personae as a way to understand and model end-users, New methodologies for interactive systems design, as well as approaches complementing existing methodologies.

Human-Centered Software Engineering

This book aims at establishing a meaningful dialog between the Human-Computer Interaction (HCI) community and Software Engineering (SE) practitioners and researchers on the results (both good and bad), obstacles, and lessons learned associated with applying software development practices in the field of user interface. Human-Centered Software Engineering provides accounts of the application of software engineering practices (which may be principles, techniques, tools, methods, processes, etc.) to a specific domain or to the development of a significant interactive system. The book gathers experiences gained by various companies and research centers working in the field of user interface engineering over a significant amount of time.

Mapping question items to skills with non-negative matrix factorization

Intelligent learning environments need to assess the student skills to tailor course material, provide helpful hints, and in general provide some kind of personalized interaction. To perform this assessment, question items, exercises, and tasks are presented to the student. This assessment relies on a mapping of tasks to skills. However, the process of deciding which skills are involved in a given task is tedious and challenging. Means to automate it are highly desirable, even if only partial automation that provides supportive tools can be achieved. A recent technique based on Non-negative Matrix Factorization (NMF) was shown to offer valuable results, especially due to the fact that the resulting factorization allows a straightforward interpretation in terms of a Q-matrix. We investigate the factors and assumptions under which NMF can effectively derive the underlying high level skills behind assessment results. We demonstrate the use of different techniques to analyze and interpret the output of NMF. We propose a simple model to generate simulated data and to provide lower and upper bounds for quantifying skill effect. Using the simulated data, we show that, under the assumption of independent skills, the NMF technique is highly effective in deriving the Q-matrix. However, the NMF performance degrades under different ratios of variance between subject performance, item difficulty, and skill mastery. The results corroborates conclusions from previous work in that high level skills, corresponding to general topics like World History and Biology, seem to have no substantial effect on test performance, whereas other topics like Mathematics and French do. The analysis and visualization techniques of the NMF output, along with the simulation approach presented in this paper, should be useful for future investigations using NMF for Q-matrix induction from data.

Learned student models with item to item knowledge structures

Probabilistic and learned approaches to student modeling are attractive because of the uncertainty surrounding the student skills assessment and because of the need to automatize the process. Item to item structures readily lend themselves to probabilistic and fully learned models because they are solely composed of observable nodes, like answers to test questions. Their structure is also well grounded in the cognitive theory of knowledge spaces. We study the effectiveness of two Bayesian frameworks to learn item to item structures and to use the induced structures to predict item outcome from a subset of evidence. One approach, Partial Order Knowledge Structures (POKS), relies on a naive Bayes framework whereas the other is based on the Bayesian network (BN) learning and inference framework. Both approaches are assessed over their predictive ability and their computational efficiency in different experimental simulations. The results from simulations over three data sets show that they both can effectively perform accurate predictions, but POKS generally displays higher predictive power than the BN. Moreover, the simplicity of POKS translates to a time efficiency between one to three orders of magnitude greater than the BN runs. We further explore the use of the item to item approach for handling concepts mastery assessment. The approach investigated consist in augmenting an initial set of observations, based on inferences with the item to item structure, and feed the augmented set to a BN containing a number of concepts. The results show that augmented set can effectively improve predictive power of a BN for item outcome, but that improvement does not transfer to the concept assessment in this particular experiment. We discuss different explanations for the results and outline future research avenues.

Intelligent Guide: Combining User Knowledge Assessment with Pedagogical Guidance

Despite their many successes, Intelligent Tutoring Systems (ITS) are not yet practical enough to be employed in the real world educational/training environments. We argue that this undesirable scenario can be changed by focusing on developing an ITS development methodology that transforms current ITS research to consider practical issues that are part of the main causes of underemployment of ITSs. Here we describe an ambitious research project to develop an ITS that has recently completed its first phase of development at the Computer Research Institute of Montreal. This project aims to address issues, such as, making ITS handle multiple domains, developing cost-effective knowledge assessment methodologies, organizing and structuring domains around curriculum views and addressing the needs of users by considering their immediate goals and educational/training settings. This paper concentrates on the outcomes of the first phase of our project that includes the architecture and functionality (specially user knowledge assessment and pedagogical guidance) of the Intelligent Guide.

A Matrix Factorization Method for Mapping Items to Skills and for Enhancing Expert-Based Q-Matrices

Uncovering the right skills behind question items is a difficult task. It requires a thorough understanding of the subject matter and of the cognitive factors that determine student performance. The skills definition, and the mapping of item to skills, require the involvement of experts. We investigate means to assist experts for this task by using a data driven, matrix factorization approach. The two mappings of items to skills, the expert on one side and the matrix factorization on the other, are compared in terms of discrepancies, and in terms of their performance when used in a linear model of skills assessment and item outcome prediction. Visual analysis shows a relatively similar pattern between the expert and the factorized mappings, although differences arise. The prediction comparison shows the factorization approach performs slightly better than the original expert Q-matrix, giving supporting evidence to the belief that the factorization mapping is valid. Implications for the use of the factorization to design better item to skills mapping are discussed.

Fast Markov blanket discovery algorithm via local learning within single pass

Learning of Markov blanket (MB) can be regarded as an optimal solution to the feature selection problem. In this paper, an efficient and effective framework is suggested for learning MB. Firstly, we propose a novel algorithm, called Iterative Parent-Child based search of MB (IPC-MB), to induce MB without having to learn a whole Bayesian network first. It is proved correct, and is demonstrated to be more efficient than the current state of the art, PCMB, by requiring much fewer conditional independence (CI) tests. We show how to construct an AD-tree into the implementation so that computational efficiency is further increased through collecting full statistics within a single data pass. We conclude that IPC-MB plus AD-tree appears a very attractive solution in very large applications.

Item to skills mapping: deriving a conjunctive q-matrix from data

Uncovering which skills are determining the success to questions and exercises is a fundamental task in ITS. This task is notoriously difficult because most exercise and question items involve multiple skills, and because skills modeling may involve subtle concepts and abilities. Means to derive this mapping from test results data are highly desirable. They would provide objective and reproductible evidence of item to skills mapping that can either help validate predefine skills models, or give guidance to define such models. However, the progress towards this end has been relatively elusive, in particular for a conjunctive skills model, where all required skills of an item must be mastered to obtain a success. We extend a technique based on Non-negative Matrix Factorization, that was previously shown successful for single skill items, to construct a conjunctive item to skills mapping from test data with multiple skills per item. Using simulated student test data, the technique is shown to yield reliable mapping for items involving one or two skills from a set of six skills.

HCI, Usability and Software Engineering Integration: Present and Future

In the last five years, several studies and workshops have highlighted the gap between software design approaches in HCI (Human Computer Interaction) and software engineering. Although the fields are complementary, these studies emphasize that they are not well integrated with each other. Several frameworks have been proposed for integrating HCI and usability techniques into the software development lifecycle. This chapter reviews some of the most relevant frameworks. It assesses their strengths and weaknesses as well as how far the objective of integrating HCI methods and principles within different software engineering methods has been reached. Finally, it draws conclusions about research directions towards the development of a generic framework that can: (1) facilitate the integration of usability engineering methods in software development practices and, (2) foster the cross-pollination of the HCI and software engineering disciplines.