Morten Hertzum

Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?

Usability comprises the aspects effectiveness, efficiency, and satisfaction. The correlations between these aspects are not well understood for complex tasks. We present data from an experiment where 87 subjects solved 20 information retrieval tasks concerning programming problems. The correlation between efficiency, as indicated by task completion time, and effectiveness, as indicated by quality of solution, was negligible. Generally, the correlations among the usability aspects depend in a complex way on the application domain, the users experience, and the use context. Going through three years of CHI Proceedings, we find that 11 out of 19 experimental studies involving complex tasks account for only one or two aspects of usability. When these studies make claims concerning overall usability, they rely on risky assumptions about correlations between usability aspects. Unless domain specific studies suggest otherwise, effectiveness, efficiency, and satisfaction should be considered independent aspect of usability and all be included in usability testing.

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations.

The information-seeking practices of engineers: searching for documents as well as for people

Abstract Engineers get most of their information from colleagues and internal reports. This study investigates how engineers’ information-seeking practices intertwine looking for informing documents with looking for informed people. Based on case studies in two product-development organisations we find that engineers search for documents to find people, search for people to get documents, and interact socially to get information without engaging in explicit searches. This intricate interplay between document and people sources can be explained by the nature of the design task. Many possible solutions are normally available to the designer and in choosing one over the others the designer must take into account a complex set of issues involving both the product as such and its context. However, design documentation seems to be biased toward technical aspects of the chosen solution, while information about the context of the design process is typically not available. Hence, people become a critical source of information because they can explain and argue about why specific decisions were made and what purpose is served by individual parts of the design. While document retrieval is a well-established field, this study concludes by briefly outlining how computer systems could support searches for people. Given the immense practical importance of searches for people there seems to be a large need for such systems and, consequently, for addressing the open research questions involved in designing them.

Trust in information sources: seeking information from people, documents, and virtual agents

Abstract The notion of trust has been virtually absent from most work on how people assess and choose their information sources. Based on two empirical cases this study shows that software engineers and users of e-commerce websites devote a lot of attention to considerations about the trustworthiness of their sources, which include people, documents, and virtual agents. In the project-based software engineering environment trust tends to be a collaborative issue and the studied software engineers normally know their sources first-hand or have them recommended by colleagues. Outside this network people are cautious and alert to even feeble cues about source trustworthiness. For example, users of e-commerce websites—generally perceived as single-user environments—react rather strongly to the visual appearance of virtual agents, though this is clearly a surface attribute. Across the two cases people need access to their sources in ways that enable them to assess source trustworthiness, access alone is not enough.

Collaborative information seeking: The combined activity of information seeking and collaborative grounding

Since common ground is pivotal to collaboration, this paper proposes to define collaborative information seeking as the combined activity of information seeking and collaborative grounding. While information-seeking activities are necessary for collaborating actors to acquire new information, the activities involved in information seeking are often performed by varying subgroups of actors. Consequently, collaborative grounding is necessary to share information among collaborating actors and, thereby, establish and maintain the common ground necessary for their collaborative work. By focusing on the collaborative level, collaborative information seeking aims to avoid both individual reductionism and group reductionism, while at the same time recognizing that only some information and understanding need be shared.

The evaluator effect in usability tests

Usability tests are applied in industry to evaluate systems and in research as a yardstick for other usability evaluation methods. However, one potential threat to the reliability of usability tests has been letI unaddressed: the evaluator effect. In this study, four evaluators analyzed four videotaped usability test sessions. Only 20% of the 93 unique problems were detected by all four evaluators and 46% were detected by only a single evaluator. Severe problems were detected more often by all four evaluators (41%) and less often by only one evaluator (22%) but a substantial evaluator effect remained.

Browsing and querying in online documentation: a study of user interfaces and the interaction process

A user interface study concerning the usage effectiveness of selected retrieval modes was conducted using an experimental text retrieval system, TeSS, giving access to online documentation of certain programming tools. Four modes of TeSS were compared: (1) browsing, (2) conventional boolean retrieval, (3) boolean retrieval based on Venn diagrams, and (4) these three combined. Further, the modes of TeSS were compared to the use of printed manuals. The subjects observed were 87 computing new to them. In the experiment the use of printed manuals is faster and provides answers of higher quality than any of the electronic modes. Therefore, claims about the effectiveness of computer-based text retrieval have to by vary in situations where printed manuals are manageable to the user. Among the modes of TeSS, browsing is the fastest and the one causing the fewest operational errors. On the same two variables, time and operational errors, the Venn diagram mode performs better than conventional boolean retrieval. The combined mode scores worst on the objective performance measures; nonetheless nearly all subject prefer this mode. Concerning the interaction process, the subjects tend to manage the complexities of the information retrieval tasks by issuing series of simple commands and exploiting the interactive capabilities of TeSS. To characterize the dynamics of the interaction process two concepts are introduced; threads and sequences of tactics. Threads in a query sequence describes the continuity during retrieval. Sequences of tactics concern the combined mode and describe how different retrieval modes succeed each other as the retrieval process evolves.

Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?

Thinking aloud is widely used for usability evaluation. The validity of the method is, however, debatable because it is generally used in a relaxed way that conflicts with the prescriptions of the classic model for obtaining valid verbalisations of thought processes. This study investigates whether participants that think aloud in the classic or relaxed way behave differently compared to performing in silence. Results indicate that whereas classic thinking aloud has little or no effect on behaviour apart from prolonging tasks, relaxed thinking aloud affects behaviour in multiple ways. During relaxed thinking aloud participants took longer to solve tasks, spent a larger part of tasks on general distributed visual behaviour, issued more commands to navigate both within and between the pages of the websites used in the experiment, and experienced higher mental workload. Implications for usability evaluation are discussed.

Images of Usability

The term usability is ubiquitous in human–computer interaction, so much so that it is commonly used without definition. Rather than one established meaning of usability, there are, however, multiple images of usability. Although each image provides a partial view, the partiality remains implicit unless confronted with alternative images. This study delineates six images of usability: universal usability, situational usability, perceived usability, hedonic usability, organizational usability, and cultural usability. The different foci of the images provide opportunities for becoming sensitized to manifold aspects of the use of a system and thereby acquiring a genuine understanding of its usability. The six images differ, for example, in the extent to which they include aspects of the outcome of the process of using a system or merely the process of use, whether they involve collaborative use or merely individual use, and in their view of usability as perceived by individuals or shared by groups. Several challenges result from recognizing that usability is a set of images rather than a coherent concept, including a risk of misunderstandings in discussions of usability because participants may assume different images of usability and a need for supplementary methods addressing the collaborative and long-term aspects of usability. Moreover, the images call for extending the scope of practical usability work to include the effects achieved by users during their use of systems for real work.

The notion of overview in information visualization

Overview is a frequently used notion and design goal in information-visualization research and practice. However, it is difficult to find a consensus on what an overview is and to appreciate its relation to how users understand and navigate an information space. We review papers that use the notion of overview and develop a model. The model highlights the awareness that makes up an overview, the process by which users acquire it, the usefulness of overviews, and the role of user-interface components in developing an overview. We discuss the model in relation to classic readings in information visualization and use it to generate recommendations for future research.