David Kirsh

Distributed cognition: toward a new foundation for human-computer interaction research

We are quickly passing through the historical moment when people work in front of a single computer, dominated by a small CRT and focused on tasks involving only local information. Networked computers are becoming ubiquitous and are playing increasingly significant roles in our lives and in the basic infrastructures of science, business, and social interaction. For human-computer interaction to advance in the new millennium we need to better understand the emerging dynamic of interaction in which the focus task is no longer confined to the desktop but reaches into a complex networked world of information and computer-mediated interactions. We think the theory of distributed cognition has a special role to play in understanding interactions between people and technologies, for its focus has always been on whole environments: what we really do in them and how we coordinate our activity in them. Distributed cognition provides a radical reorientation of how to think about designing and supporting human-computer interaction. As a theory it is specifically tailored to understanding interactions among people and technologies. In this article we propose distributed cognition as a new foundation for human-computer interaction, sketch an integrated research framework, and use selections from our earlier work to suggest how this framework can provide new opportunities in the design of digital work materials.

On Distinguishing Epistemic from Pragmatic Action

We present data and argument to show that in Tetris-a real-time, interactive video game-certain cognitive and perceptual problems ore more quicktv, easily, and reliably solved by performing actions in the world than by performing computational actions in the head atone. We have found that some of the translations and rotations made by players of this video game are best understood as actions that use the world to improve cognition. These actions are not used to implement a plan, or to implement a reaction; they are used to change the world in order to simplify the problem-solving task. Thus, we distinguish pragmatic octionsactions performed to bring one physically closer to a goal-from epistemic actions -actions performed to uncover informatioan that is hidden or hard to compute mentally. To illustrate the need for epistemic actions, we first develop a standard information-processing model of Tetris cognition and show that it cannot explain performance data from human players of the game-even when we relax the assumption of fully sequential processing. Standard models disregard many actions taken by players because they appear unmotivated or superfluous. However, we show that such actions are actually far from superfluous; they play a valuable role in improving human performance. We argue that traditional accounts are limited because they regard action as having o single function: to change the world. By recognizing a second function of action-an epistemic function-we can explain many of the actions that a traditional model cannot. Although our argument is supported by numerous examples specifically from Tetris, we outline how the new category of epistemic action can be incorporated into theories of action more generally.

The intelligent use of space

Abstract The objective of this essay is to provide the beginning of a principled classification of some of the ways space is intelligently used. Studies of planning have typically focused on the temporal ordering of action, leaving as unaddressed, questions of where to lay down instruments, ingredients, work-in-progress, and the like. But, in having a body, we are spatially located creatures: we must always be facing some direction, have only certain objects in view, be within reach of certain others. How we manage the spatial arrangement of items around us, is not an afterthought; it is an integral part of the way we think, plan and behave. The proposed classification has three main categories: spatial arrangements that simplify choice; spatial arrangements that simplify perception; and spatial dynamics that simplify internal computation. The data for such a classification is drawn from videos of cooking, assembly and packing, everyday observations in supermarkets, workshops and playrooms, and experimental studies of subjects playing Tetris, the computer game. This study, therefore, focusses on interactive processes in the medium and short term: on how agents set up their workplace for particular tasks, and how they continuously manage that workplace.

Thinking with external representations

Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation—external representations save internal memory and computation—is only part of the story. I discuss seven ways external representations enhance cognitive power: they change the cost structure of the inferential landscape; they provide a structure that can serve as a shareable object of thought; they create persistent referents; they facilitate re-representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought—they help coordinate thought. Jointly, these functions allow people to think more powerfully with external representations than without. They allow us to think the previously unthinkable.

Today the earwig, tomorrow man?

Abstract A startling amount of intelligent activity can be controlled without reasoning or thought. By tuning the perceptual system to task relevant properties a creature can cope with relatively sophisticated environments without concepts. There is a limit, however, to how far a creature without concepts can go. Rod Brooks, like many ecologically oriented scientists, argues that the vast majority of intelligent behaviour is concept-free. To evaluate this position I consider what special benefits accrue to concept-using creatures. Concepts are either necessary for certain types of perception, learning, and control, or they make those processes computationally simpler. Once a creature has concepts its capacities are vastly multiplied.

Adapting the environment instead of oneself

This article examines some of the methods used by animals and humans to adapt their environment. Because there are limits on the number of different tasks a creature can be designed to do well in, creatures with the capacity to redesign their environments have an adaptive advantage over those who can adapt only passively to existing environmental structures. To clarify environmental redesign, l rely on the formal notion of a task environment as a directed graph in which the nodes are states and the links are actions. One natural form of redesign is to change the topology of this graph structure so as to increase the likelihood of task success or to reduce its expected cost, measured in physical terms. This may be done by eliminating initial states, hence eliminating choice points; by changing the action repertoire; by changing the consequence function; and, lastly, by adding choice points. Another major method for adapting the environment is to change its cognitive congeniality. Such changes leave the state space formally intact but reduce the number and cost of mental operations needed for task success; they reliably increase the speed, accuracy, or robustness of performance. The last section of the article describes several of these epistemic or complementary actions found in human performance.

The context of work

The question of how to conceive and represent the context of work is explored from the theoretical perspective of distributed cognition. It is argued that to understand the office work context we need to go beyond tracking superficial physical attributes such as who or what is where and when and consider the state of digital resources, peoples concepts, task state, social relations, and the local work culture, to name a few. In analyzing an office more deeply, three concepts are especially helpful: entry points, action landscapes, and coordinating mechanisms. An entry point is a structure or cue that represents an invitation to enter an information space or office task. An activity landscape is part mental construct and part physical; it is the space users interactively construct out of the resources they find when trying to accomplish a task. A coordinating mechanism is an artifact, such as a schedule or clock, or an environmental structure such as the layout of papers to be signed, which helps a user manage the complexity of his task. Using these three concepts we can abstract away from many of the surface attributes of work context and define the deep structure of a setting-the invariant structure that many office settings share. A long-term challenge for context-aware computing is to operationalize these analytic concepts.

Embodied cognition and the magical future of interaction design

The theory of embodied cognition can provide HCI practitioners and theorists with new ideas about interaction and new principles for better designs. I support this claim with four ideas about cognition: (1) interacting with tools changes the way we think and perceive -- tools, when manipulated, are soon absorbed into the body schema, and this absorption leads to fundamental changes in the way we perceive and conceive of our environments; (2) we think with our bodies not just with our brains; (3) we know more by doing than by seeing -- there are times when physically performing an activity is better than watching someone else perform the activity, even though our motor resonance system fires strongly during other person observation; (4) there are times when we literally think with things. These four ideas have major implications for interaction design, especially the design of tangible, physical, context aware, and telepresence systems.

Interactivity and multimedia interfaces

Multimedia technology offers instructional designers an unprecedented opportunity to create richly interactive learning environments. With greater design freedom comes complexity. The standard answer to the problems of too much choice, disorientation, and complex navigation is thought to lie in the way we design the interactivity in a system. Unfortunately, the theory of interactivity is at an early stage of development. After critiquing the decision cycle model of interaction – the received theory in human computer interaction – I present arguments and observational data to show that humans have several ways of interacting with their environments which resist accommodation in the decision cycle model. These additional ways of interacting include: preparing the environment, maintaining the environment, and reshaping the cognitive congeniality of the environment. Understanding how these actions simplify the computational complexity of our mental processes is the first step in designing the right sort of resources and scaffolding necessary for tractable learner controlled learning environments.

Worldlets—3D thumbnails for wayfinding in virtual environments

Virtual environment landmarks are essential in wayfinding: they anchor routes through a region and provide memorable destinations to return to later. Current virtual environment browsers provide user interface menus that characterize available travel destinations via landmark textual descriptions or thumbnail images. Such characterizations lack the depth cues and context needed to reliably recognize 3D landmarks. This paper introduces a new user interface affordance that captures a 3D representation of a virtual environment landmark into a 3D thumbnail, called a worldlet. Each worldlet is a miniature virtual world fragment that may be interactively viewed in 3D, enabling a traveler to gain first-person experience with a travel destination. In a pilot study conducted to compare textual, image, and worldlet landmark representations within a wayfinding task, worldlet use significantly reduced the overall travel time and distance traversed, virtually eliminating unnecessary backtracking. Landmarks are the subject of landmark knowledge, but also play a part in survey and procedural knowledge. In survey knowledge, landmarks provide regional anchors with which to calibrate distances and directions. In procedural knowledge, landmarks mark decision points along a route, helping in the recall of procedures to get to and from destinations of interest. Overall, landmarks help to structure an environment and provide directional cues to facilitate wayfinding.