John Rieman

Cognitive walkthroughs: a method for theory-based evaluation of user interfaces

This paper presents a new methodology for performing theory-based evaluations of user interface designs early in the design cycle. The methodology is an adaptation of the design walkthrough techniques that have been used for many years in the software engineering community. Traditional walkthroughs involve hand simulation of sections of code to ensure that they implement specified functionality. The method we present involves hand simulation of the cognitive activities of a user, to ensure that the user can easily learn to perform tasks that the system is intended to support. The cognitive walkthrough methodology, described in detail, is based on a theory of learning by exploration presented in this paper. There is a summary of preliminary results of effectiveness and comparisons with other design methods.

Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces

The value of theoretical analyses in user interface design has been hotly debated. All sides agree that it is difficult to apply current theoretical models within the constraints of real-world development projects. We attack this problem in the context of bringing the theoretical ideas within a model of exploratory learning [19] to bear on the evaluation of alternative interfaces for walk-up-and-use systems. We derived a “cognitive walkthrough” procedure for systematically evaluating features of an interface in the context of the theory. Four people independently applied this procedure to four alternative interfaces for which we have empirical usability data. Consideration of the walkthrough sheds light on the consistency with which such a procedure can be applied as well as the accuracy of the results.

A field study of exploratory learning strategies

It has suggested that interactive computer users find “exploratory learning” to be an effective and attractive strategy for learning a new system or investigating unknown features of familiar software. In exploratory learning, instead of working through precisely sequenced training materials, the user investigates a system on his or her own initiative, often in pursuit of a real or artificial task. The value of exploratory learning has been studied in controlled settings, with special attention newly acquired systems, be there has been little investigation of its occurrence in natural situations or in support of ongoing learning. To address this question, a field study of the behavior and attitudes of computer users in everyday working situations was performed, using diaries and structured interviews that focused on learning events. The study showed that task-oriented exploration was a widely accepted method for learning, but that it often required support from manuals and from other users or system support personnel. Exploration not related to a current or pending task was infrequent, and most users believed it to be inefficient. These findings have implications for the design of systems, documentation, and training.

The diary study: a workplace-oriented research tool to guide laboratory efforts

Methods for studying user behavior in HCI can be informally divided into two approaches: experimental psychology in the laboratory and observations in the workplace. The first approach has been faulted for providing results that have little effect on system usability, while the second can often be accused of yielding primarily anecdotal data that do not support general conclusions. This paper describes two similar approaches in another field, the study of animal behavior, and shows how they produce complementary results. To support similar complementary interactions between research approaches in the HCI field, the paper describes the diary study technique, a tool for research in the workplace that achieves a relatively high standard of objectivity. A diary study is reported that focuses on exploratory learning.

A dual-space model of iteratively deepening exploratory learning

Abstract When users of interactive computers must work with new software without formal training, they rely on strategies for “exploratory learning”. These include trial and error, asking for help from other users, and looking for information in printed and on-line documentation. This paper describes a cognitive model of exploratory learning, which covers both trial-and-error and instruction-taking activities. The model, implemented in Soar, is grounded in empirical data of subjects in a task-oriented, trial-and-error exploratory learning situation. A key empirical finding reflected in the model is the repeated scanning of a subset of the available menu items, with increased attention to items on each successive scan. This is explained in terms of dual search spaces, the external interface and the users internal knowledge, both of which must be tentatively explored with attention to changing costs and benefits. The model implements this dual-space search by alternating between external scanning and internal comprehension, a strategy that gradually shifts its focus to a potentially productive route through an interface. Ways in which interfaces might be designed to capitalize on this behaviour are suggested. The research demonstrates how cognitive modelling can describe behaviour of the kind discussed by theories of “situated cognition”.

An automated cognitive walkthrough

The cognitive walkthrough is a new method for assessing the usability of a system and assigning causes to usability problems early in the design process. Early versions of the walkthrough suffered from tedious and repetitious paperwork. To address this problem we developed an automated prompting system, which reduces the method’s overhead and focuses the analyst’s attention on critical issues. The methodology can be introduced in about an hour, and the automated system allows a user action in a proposed interface to be analyzed in as little as 5 to 10 minutes.

Minimalist explanations in knowledge-based systems

Research in discourse comprehension and human-computer interaction indicates that good explanations are usually brief. A system that provides brief explanations, however, must plan for the case where brevity comes at the expense of understanding. Human-to-human dialogue is, to a large part, concerned with conversational repair and question-answer episodes; computer systems need to provide similar fallback techniques to their users. The authors have designed such an explanation system in the context of a knowledge-based critiquing system, LISP-CRITIC. The system provides several levels of explanations, specifically tailored to the user. If the initial, brief explanation is insufficient, the system positions the user at an appropriate point within a more complete, hypertext-based documentation system. Rather than attempting to design a system that can generate a perfect one-shot explanation for any given situation, this approach concentrates on matching the communication abilities provided by current computer technology to the cognitive needs of the human user.<<ETX>>

Getting to know users and their tasks

Publisher Summary This chapter discusses the significance of interacting with users and knowing their tasks while designing an interface. The first step is to find some real people who would be potential users of the interface being built. Once there are some people to talk with, the system building team should develop concrete, detailed examples of tasks they perform or want to perform that the system should support. Most large software projects are developed using some version of the waterfall method. The basic waterfall method assumes that a piece of software is produced through a clearly defined series of steps, or phases:(1) Requirements analysis,(2)Specification,(3)Planning,(4) Design, (5) Implementation, (6) Integration, and (7) Maintenance.

Natural Training Wheels: Learning and Transfer Between Two Versions of a Computer Application

Users of personal computers must deal with frequent major upgrades of software packages. Major upgrades typically provide increased functionality without changing the conceptual framework of the program, but they may force the user to learn how to use new menus, dialog boxes, and other controls. We suggest that early versions of a program provide a natural training wheels environment, in which novice users can learn a programs basic operation while avoiding potential confusion caused by advanced features. Experiments with two versions of a graphing program confirm this hypothesis, with some restrictions.

Using the programming walkthrough to aid in programming language design

The programming walkthrough is a method for assessing how easy or hard it will be for users to write programs in a programming language. It is intended to enable language designers to identify problems early in design and to help them choose among alternative designs. We describe the method and present experience in applying it in four language design projects. Results indicate that the method is a useful supplement to existing design approaches.