Robert Anthony Virzi

A Comparison of Three Usability Evaluation Methods: Heuristic, Think-Aloud, and Performance Testing:

A high-fidelity prototype of an extended voice mail application was created. We tested it using three distinct usability testing paradigms so that we could compare the quantity and quality of the information obtained using each. The three methods employed were (1) heuristic evaluation, in which usability experts critique the user interface, (2) think-aloud testing, in which naive subjects comment on the system as they use it, and (3) performance testing, in which task completion times and error rates are collected as naive subjects interact with the system. The three testing methodologies were roughly equivalent in their ability to detect a core set of usability problems on a per evaluator basis. However, the heuristic and think-aloud evaluations were generally more sensitive, uncovering a broader array of problems in the user interface. Implications of these findings are discussed in terms of the costs of doing the evaluations and in light of other work on this topic.

Expert Reviews: How Many Experts is Enough?

We asked five usability specialists to review the user interface to a phone-based, interactive voice response system. The experts were instructed to conduct their review independently in three one-hour sessions and to record each usability problem on a Problem Description Sheet along with the elapsed time from the beginning of the hour. Each expert then spent one hour reviewing their problem sheets and making a summary list of problems. Finally, the experts spent two hours together on a conference call discussing their impressions and coming to consensus on a prioritized list of problems and solutions. The results showed that when allocating expert time, it is more effective to have a greater number of experts spend fewer hours than to use fewer experts for more hours. The individual summaries included the majority of the severe problems, but left out many less severe problems and added new problems. The group report did not surface any new problems, but described the problems as being caused by more basic design flaws and proposed solutions that focused on the conceptual model on which the design was based.