Peter Jan Schellens

Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue

Think-aloud protocols are a dominant method in usability testing. There is, however, only little empirical evidence on the actual validity of the method. This paper describes an experiment that compares concurrent and retrospective think-aloud protocols for a usability test of an online library catalogue. There were three points of comparison: usability problems detected, overall task performance, and participant experiences. Results show that concurrent and retrospective think-aloud protocols reveal comparable sets of usability problems, but that these problems come to light in different ways. In retrospective think-aloud protocols, more problems were detected by means of verbalisation, while in concurrent think-aloud protocols, more problems were detected by means of observation. Moreover, in the concurrent think-aloud protocols, the requirement to think aloud while working had a negative effect on the task performance. This raises questions about the reactivity of concurrent think-aloud protocols, especially in the case of high task complexity.Think-aloud protocols are a dominant method in usability testing. There is, however, only little empirical evidence on the actual validity of the method. This paper describes an experiment that compares concurrent and retrospective think-aloud protocols for a usability test of an online library catalogue. There were three points of comparison: usability problems detected, overall task performance, and participant experiences. Results show that concurrent and retrospective think-aloud protocols reveal comparable sets of usability problems, but that these problems come to light in different ways. In retrospective think-aloud protocols, more problems were detected by means of verbalisation, while in concurrent think-aloud protocols, more problems were detected by means of observation. Moreover, in the concurrent think-aloud protocols, the requirement to think aloud while working had a negative effect on the task performance. This raises questions about the reactivity of concurrent think-aloud protocols, especially in the case of high task complexity.

Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison.

This paper describes a comparative study of three usability test approaches: concurrent think-aloud protocols, retrospective think-aloud protocols, and constructive interaction. These three methods were compared by means of an evaluation of an online library catalogue, which involved four points of comparison: number and type of usability problems detected; relevance of the problems detected; overall task performance; and participant experiences. The results of the study showed that there were only few significant differences between the usability test approaches, mainly with respect to manner of problem detecting, task performance and participant experience. For the most part, the usability methods proved very much comparable, revealing similar numbers and types of problems that were equally relevant. Taking some practical aspects into account, a case can be made for preferring the concurrent think-aloud protocols over the other two methods.

Writing profiles: the effect of the writing mode on pausing and revision patterns of experienced writers

We investigated how writing processes are affected by physical aspects of the task environment, specifically the use of a word processor, with respect to patterns of pausing and revision. Consistent with the tradition of cognitive writing research, the writing processes of experienced writers were examined (60 involving the use of a word processor, 20 involving pen&paper). In comparison with writers using pen&paper, those using a word processor (i) spent more time on a first draft and less on finalizing a text, (ii) pursued a more fragmentary writing process, (iii) tended to revise more extensively at the beginning of the writing process, (iv) attended more to lower linguistic levels (letter, word) and formal properties of the text, and (v) did not normally undertake any systematic revision of their work before finishing. By clustering the various processes with respect to twelve relevant parameters, we developed a new typology of writing processes which distinguishes five writing profiles: the initial planner, the fragmentary Stage I writer, the Stage II writer, the non-stop writer, and the average writer. Our quantitative approach to describing the cognitive behavior of the different writers revealed that the profiles they adopt depend greatly on the constraints of the writing environment.

Reader-Focused Text Evaluation An Overview of Goals and Methods

This article presents a review of the literature on reader-focused text evaluation. First, an account is given of the document characteristics that can be evaluated. Then the possible functions of evaluations are considered, a distinction being made between verifying, troubleshooting, and choice-supporting research. Finally, an overview is presented of methods appropriate for the various document characteristics and evaluation functions. Relevant research findings on the methodological strengths and constraints of each method are discussed.

Toward a document evaluation methodology: what does research tell us about the validity and reliability of evaluation methods?

Although the usefulness of evaluating documents has become generally accepted among communication professionals, the supporting research that puts evaluation practices empirically to the test is only beginning to emerge. This article presents an overview of the available research on troubleshooting evaluation methods. Four lines of research are distinguished concerning the validity of evaluation methods, sample composition, sample size, and the implementation of evaluation results during revision.

Evaluating municipal websites: A methodological comparison of three think-aloud variants

Usability methods have received relatively little methodological attention within the field of E-Government. This paper aims to address this gap by reporting on a usability test of the municipal website of Deventer (the Netherlands), carried out by means of three variants of the think-aloud method (concurrent/retrospective think-aloud protocols and constructive interaction). These three methods had proved successful in a previous evaluation of a different municipal website, yet we decided to replicate our study in order to investigate whether the three methods would reveal different results when applied to another municipal website with a different information architecture. The results of our study showed that, as in the previous municipal website evaluation, the three evaluation methods were largely comparable in terms of output. Nevertheless, we did find a number of differences between the present and previous municipal website evaluation regarding the workings of the three methods—differences that could be explained by the different information architectures of the municipal websites tested. This suggests that the three evaluation methods might indeed work differently depending on the nature of the website that is being evaluated, and calls for more research into the effect of task type on the validity of evaluation methods.

Argumentation schemes in persuasive brochures

Many public information documents attempt to persuade the recipients that they should engage in or refrain from specific behaviour. This is based on the assumption that the recipient will decide about his or her behaviour on the basis of the information given and a rational evaluation of the pros and cons. An analysis of 20 public information brochures shows that the argumentation in persuasive brochures is often not marked as such. Argumentation is presented as factual information, and in many instances the task of making argumentational links and drawing conclusions is left to the reader. However, since the information offered does follow familiar argumentational schemes, readers can, in principle, reconstruct the argument. All the brochures make use of pragmatic argumentation (argumentation from consequences),i. e.,they formulate at least certain benefits of the desirable behaviour or disadvantages of the undesirable behaviour. In addition, they make regular use of argumentation from cause to effect and argumentation from example. Argumentation from rules and argumentation from authority are less frequently used. This empirical analysis of the use of argumentation schemes is a solid base for interesting and rich hypotheses about the cognitive processing of persuasive brochures. Central processing requires the reader to be able to reconstruct argumentation from informational texts and to identify and evaluate various types of argumentation.

Arguing about desirable consequences: what constitutes a convincing argument?

Argument quality has consistently been shown to have strong and lasting persuasive effects. The question is what criteria people use to distinguish strong from weak arguments and how these criteria relate to the ones proposed in normative argumentation theory. In an experiment 235 participants without training in argumentation theory rated the acceptance of 30 claims about the desirability of a consequence that were supported by either an argument from analogy, from authority, or from consequences. The supporting arguments were systematically manipulated to violate argument type specific criteria. Participants proved sensitive to the violation of most, but not all, argument type specific criteria. From a normative perspective these findings suggest that people act in a fairly adequate way. These findings also enable a more precise description of what people may do when critically appraising arguments, which has important implications for the use of argument quality as a methodological tool in persuasion research.

Verbal irony: Differences in usage across written genres

According to Gibbs and Colston, one of the biggest challenges for irony research is the uncovering of the various ways in which irony is used in discourse. This article takes up a genre-based approach to deal with this research challenge. In a content analysis of ironic utterances from six written genres (commercial and noncommercial advertisements, columns, cartoons, letters to the editor, book and film reviews), ironic utterances are compared on the usage of irony factors and irony markers. Results indicate that every genre in the corpus differs from the general distribution for at least one irony factor and one category of irony markers. Taken together, the clustering of irony factors and markers in specific genres is a first step toward identifying the various ways in which verbal irony is used differently across various genres.

Revision of Public Information Brochures on the Basis of Reader Feedback

The literature on formative text evaluation pays scant attention to the revision phase following data collection. This article describes a small-scale experiment in which five professional writers were asked to revise brochure fragments on the basis of feedback from readers. The feedback consisted of readers comments, selected from the results of a pretest of the brochures, regarding their acceptance of the information and their appreciation of text elements. Despite the wide variety of solutions that resulted, some interesting tendencies were found: In response to problems with factual acceptance, writers often decided to add information; in response to problems with normative acceptance, they often chose to substitute material; and in response to appreciation problems, they either deleted the problematic passage or substituted a different phrase.