Mark K. Singley

Integrating theory development with design evaluation

Abstract In this paper, we recruit the construct of psychological design rationale as a framework for integrating theory development with design evaluation in HCI. We propose that, in some cases, part of an artefacts psychological design rationale can be regarded as inherited from second-order artefacts (prescriptive design models, architectures and genres, tools and environments, interface styles). We show how evaluation data pertaining to an artefact can be used to test and develop the second-order artefact from which it inherits.

Comparative usability evaluation: critical incidents and critical threads

Em irical usability evaluations (particularly forrna~ive J ev uations [13]) hinge on observing and interpreting critical incidents [8] of use. We proposed [3,5] augmenting critical incident methods by analysis of what we called critical threads: sets of causally related user episodes that, taken together, define major usability themes. This paper extents this work to the comparative usability analysis of a related artifact. We discuss how our earlier claims analysis was used to orient and simplify our current evaluation efforts.

The Reification of Goal Structures in a Calculus Tutor: Effects on Problem‐Solving Performance

Abstract In an initial experiment with a minimal version of a calculus tutor, it was determined through analyses of verbal protocol data that students were attempting to execute a fairly standard working‐backwards, means‐ends strategy to solve systems of equations, but were having difficulty maintaining the requisite goal stack. To remedy this problem, an enhancement to the interface of the tutor was designed which allowed students to post and display the subgoals required by the means‐ends strategy. As students progressed through problems, individual subgoals were boxed and shaded to indicate which subgoals were active and which had been satisfied, respectively. An experiment testing the effects of this type of goal posting showed that student problem‐solving performance improved in terms of both speed and accuracy while the goal blackboard was present. Furthermore, many of the positive effects persisted after the goal blackboard was taken away. Two explanations for the beneficial effects of goal posting...

Algebra jam: supporting teamwork and managing roles in a collaborative learning environment

We are building a collaborative learning environement that supports teams of students as they collaborate synchronously and remotely to solve situated, multi-step problems involving algebraic modeling. Our system, named Algebra Jam, provides a set of tools to help students overcome two of the most serious impediments to successful collaboration: establishing common ground and maintaining group focus. These tools include tethered and untethered modes of operation including discrepancy notification, a goal-oriented team blackboard, object-oriented chat with collabicons, reification of problem solving roles, and the optional inclusion of a tutor agent as a virtual team participant. The tutor agent not only offers help and feedback on problem solving actions but also accumulates evidence about individual and group problem solving performance in a Bayesian inference network. The system is envisioned as a testbed for developing theories of teaming.

Team tutoring systems: reifying roles in problem solving

We are pursuing the notion of a Team Tutoring System, an intelligent tutoring system that monitors and manages teams of students as they collaborate synchronously and remotely to solve extended, distributed, multi-step problems. Our system, named Algebra Jam, provides opportunities for each team member to take on a variety of roles in order to come to know the domain from a variety of perspectives. To do this, we are defining a typology of collaborative problem solving roles, reifying these roles at the problem solving interface, and accumulating evidence about individual behavior and group problem solving performance in a Bayesian inference network.

Scaffolding group learning in a collaborative networked environment

Scaffolding students in a collaborative networked learning environment requires different instructional methods than in a traditional home or classroom setting. The goal of this research is to understand computer-mediated collaboration in an instructional setting in order to create an effective computer-mediated collaboration tool. We identify ways to support collaboration by examining the interaction and strategies employed by a peer tutor and teacher and between peers working in our collaborative learning environment. We found that supporting collaboration in an electronic setting requires diagnosing impasses, facilitating problem-solving interaction, and suggesting ways to divide the problem into sub-tasks.

Open-Ended Approaches to Science Assessment Using Computers

We discuss the potential role of technology in evaluating learning outcomes in large-scale, widespread science assessments of the kind typically done at ETS, such as the GRE, or the College Board SAT II Subject Tests. We describe the current state-of-the-art in this area, as well as briefly outline the history of technology in large-scale science assessment and ponder possibilities for the future. We present examples from our own work in the domain of chemistry, in which we are designing problem solving interfaces and scoring programs for stoichiometric and other kinds of quantitative problem solving. We also present a new scientific reasoning item type that we are prototyping on the computer. It is our view that the technological infrastructure for large-scale constructed response science assessment is well on its way to being available, although many technical and practical hurdles remain.

Incidental Reification of Goals in an Intelligent Tutor for Smalltalk

We describe the design of an instructional system for Smalltalk that attempts to provide feedback on learners’ goals and plans through a technique called incidental reification. We contrast our approach with other current attempts at reifying learners’ goals and plans and sketch out problems and prospects for future work.

PASTEL: pattern-driven adaptive simulations

We propose a new kind of learning environment called an adaptive simulation that more deeply explores and exploits the potential of simulations as pedagogical and explanatory tools. In an adaptive simulation, the simulation configuration is not fixed but rather can be modified by an instructional agent for optimal pedagogical effect. Types of adaptations include manipulations of simulation time and state, changes in representation to facilitate explanations and/or task performance, and adjustments in simulation complexity by the addition and/or removal of components. We briefly describe a system we are developing called PASTEL that is designed to enable these kinds of adaptations. Open research issues include precisely how to perform these adaptations and when to employ them for optimal effect.

MOLEHILL: an instructional system for Smalltalk programming

Smalltalk, although rccognizcci as a good platform for rapid protolypirrg and software reuse, is widely regarded as ciifEcult to learn. Programming in Smalltalk invo~ves adopting the object-orielltcd, message-pmsing model of program design and execution and becoming familiar with a large set of existing object classes, methods, and their inheritance relationships. [Jnlike learning a procedural language like Pascal or C, learning Smalltaik is dominated by browsing and code comprehension, I,earners of Smalltalk typically experience a long, slow startup phase in which they becc)me familiar with the hierarchy and object-oriented model but do little meaningful work (this has been dubbed climbing the Smalltaik mountain [1]). In an industrial environment, this is particularly troublesome because of the real or perhaps perceived need to get started quickly.