Christopher J. MacLellan

Authoring Tutors with SimStudent: An Evaluation of Efficiency and Model Quality

Authoring Intelligent Tutoring Systems is expensive and time consuming. To reduce costs, the Cognitive Tutor Authoring Tools and the Example-Tracing Tutor paradigm were developed to make the tutor authoring process more efficient. Under this paradigm, tutors are constructed by demonstrating behavior directly in a tutor interface, reducing the need for programming expertise. This paper evaluates the efficiency of authoring a tutor with SimStudent, an extension to the Example-Tracing paradigm that is designed to produce greater generality in less time by induction from past demonstrations and feedback. We found that authoring an algebra tutor in SimStudent is faster than Example-Tracing while maintaining equivalent final model quality. Furthermore, we found that the SimStudent model generalizes beyond the problems that were used to author it.

Using extracted features to inform alignment-driven design ideas in an educational game

As educational games have become a larger field of study, there has been a growing need for analytic methods that can be used to assess game design and inform iteration. While much previous work has focused on the measurement of student engagement or learning at a gross level, we argue that new methods are necessary for measuring the alignment of a game to its target learning goals at an appropriate level of detail to inform design decisions. We present a novel technique that we have employed to examine alignment in an open-ended educational game. The approach is based on examining how the game reacts to representative student solutions that do and do not obey target principles. We demonstrate this method using real student data and discuss how redesign might be informed by these techniques.

Problem Map: An Ontological Framework for a Computational Study of Problem Formulation in Engineering Design

Studies of design cognition often face two challenges. One is a lack of formal cognitive models of design processes that have the appropriate granularity: fine enough to distinguish differences among individuals and coarse enough to detect patterns of similar actions. The other is the inadequacies in automating the recourse-intensive analyses of data collected from large samples of designers. To overcome these barriers, we have developed the problem map (P-maps) ontological framework. It can be used to explain design thinking through changes in state models that are represented in terms of requirements, functions, artifacts, behaviors, and issues. The different ways these entities can be combined, in addition to disjunctive relations and hierarchies, support detailed modeling and analysis of design problem formulation. A node‐link representation of P-maps enables one to visualize how a designer formulates a problem or to compare how different designers formulate the same problem. Descriptive statistics and time series of entities provide more detailed comparisons. Answer set programming (ASP), a predicate logic formalism, is used to formalize and trace strategies that designers adopt. Data mining techniques (association rule and sequence mining) are used to search for patterns among large number of designers. Potential uses of P-maps are computer-assisted collection of large data sets for design research, development of a test for the problem formulation skill, and a tutoring system. [DOI: 10.1115/1.4030076]

A Computational Aid for Problem Formulation in Early Conceptual Design

Conceptual design is a high-level cognitive activity that draws upon distinctive human mental abilities. An early and fundamental part of the design process is problem formulation, in which designers determine the structure of the problem space they will later search. Although many tools have been developed to aid the later stages of design, few tools exist that aid designers in the early stages. In this paper, we describe Problem Formulator, an interactive environment that focuses on this stage of the design process. This tool has representations and operations that let designers create, visualize, explore, and reflect on their formulations. Although this process remains entirely under the user’s control, these capabilities make the system well positioned to aid the early stages of conceptual design. [DOI: 10.1115/1.4024714]

The Structure of Creative Design: What Problem Maps Can Tell Us About Problem Formulation and Creative Designers

Problem formulation is an important part of the design process that has been largely underexplored. Similarly, the relationship between how designers formulate problems and creative outcome is not well understood. To shed light on what the process of problem formulation can tell us about creativity in design, we use the problem map model ‐ a flexible, domainindependent ontology for modeling the design formulation process ‐ to analyze protocols from eight expert designers. In this paper, we discuss the effectiveness of using problem maps for coding design protocols and what the problem map model can tell us about the protocols of designers. In this exploratory study, we use the problem map model to code and analyze the problem formulation stage of the design process.

Replay Analysis in Open-Ended Educational Games

Designers of serious games have an interest in understanding if their games are well-aligned, i.e., whether in-game rewards incentivize behaviors that will lead to learning. Few existing serious games analytics solutions exist to serve this need. Open-ended games in particular run into issues of alignment due to their affordances for wide player freedom. In this chapter, we first define open-ended games as games that have a complex functional solution spaces. Next, we describe our method for exploring alignment issues in an open-ended educational game using replay analysis. The method uses multiple data mining techniques to extract features from replays of player behavior. Focusing on replays rather than logging play-time metrics allows designers and researchers to run additional metric calculations and data transformations in a post hoc manner. We describe how we have applied this replay analysis methodology to explore and evaluate the design of the open-ended educational game RumbleBlocks. Using our approach, we were able to map out the solution space of the game and highlight some potential issues that the game’s designers might consider in iteration. Finally, we discuss some of the limitations of the replay approach.

Developmental changes in semantic knowledge organization

Semantic knowledge is a crucial aspect of higher cognition. Theoretical accounts of semantic knowledge posit that relations between concepts provide organizational structure that converts information known about individual entities into an interconnected network in which concepts can be linked by many types of relations (e.g., taxonomic, thematic). The goal of the current research was to address several methodological shortcomings of prior studies on the development of semantic organization, by using a variant of the spatial arrangement method (SpAM) to collect graded judgments of relatedness for a set of entities that can be cross-classified into either taxonomic or thematic groups. In Experiment 1, we used the cross-classify SpAM (CC-SpAM) to obtain graded relatedness judgments and derive a representation of developmental changes in the organization of semantic knowledge. In Experiment 2, we validated the findings of Experiment 1 by using a more traditional pairwise similarity judgment paradigm. Across both experiments, we found that an early recognition of links between entities that are both taxonomically and thematically related preceded an increasing recognition of links based on a single type of relation. The utility of CC-SpAM for evaluating theoretical accounts of semantic development is discussed.

Modeling Strategy Use in an Intelligent Tutoring System: Implications for Strategic Flexibility

Education research has identified strategic flexibility as an important aspect of math proficiency and learning. This aspect of student learning has been largely ignored by Intelligent Tutoring Systems (ITSs). In the current study, we demonstrate how Hidden Markov Modeling can be used to identify groups of students who use similar strategies during tutoring and relate these findings to a measure of strategic flexibility. We use these results to explore how strategy use is expressed in an ITS and consider how tutoring systems could integrate a measure of strategy use to improve learning.

Learning Cognitive Models Using Neural Networks

A cognitive model of human learning provides information about skills a learner must acquire to perform accurately in a task domain. Cognitive models of learning are not only of scientific interest, but are also valuable in adaptive online tutoring systems. A more accurate model yields more effective tutoring through better instructional decisions. Prior methods of automated cognitive model discovery have typically focused on well-structured domains, relied on student performance data or involved substantial human knowledge engineering. In this paper, we propose Cognitive Representation Learner (CogRL), a novel framework to learn accurate cognitive models in ill-structured domains with no data and little to no human knowledge engineering. Our contribution is two-fold: firstly, we show that representations learnt using CogRL can be used for accurate automatic cognitive model discovery without using any student performance data in several ill-structured domains: Rumble Blocks, Chinese Character, and Article Selection. This is especially effective and useful in domains where an accurate human-authored cognitive model is unavailable or authoring a cognitive model is difficult. Secondly, for domains where a cognitive model is available, we show that representations learned through CogRL can be used to get accurate estimates of skill difficulty and learning rate parameters without using any student performance data. These estimates are shown to highly correlate with estimates using student performance data on an Article Selection dataset.

Assessing the Creativity of Designs at Scale

How best to assess the creativity of a large number of designed artifacts remains an open problem. The typical approach is to have a panel of experts answer likert questions about individual artifacts. This process typically requires a substantial amount of training to ensure the judges achieve an acceptable level of agreement. Consequently, the approach does not scale well as it is infeasible to have a panel of experts regularly evaluate the creativity of a large number of designs. The current work explores an alternative approach that uses both individual and pairwise judgements from novice crowd workers to support reliable and scalable assessment of creative designs. This approach, which we call TrueCreativity, can operate over a set of evaluations from a large number of judges and appropriately weights their evaluations based on their past reliability and agreement with other judges. We show that this approach produces results that strongly correlate with another measure of creativity.