Thea van der Geest

User-centered evaluation of adaptive and adaptable systems: A literature review

Adaptive and adaptable systems provide tailored output to various users in various contexts. While adaptive systems base their output on implicit inferences, adaptable systems use explicitly provided information. Since the presentation or output of these systems is adapted, standard user-centered evaluation methods do not produce results that can be easily generalized. This calls for a reflection on the appropriateness of standard evaluation methods for user-centered evaluations of these systems. We have conducted a literature review to create an overview of the methods that have been used. When reviewing the empirical evaluation studies we have, among other things, focused on the variables measured and the implementation of results in the (re)design process. The goal of our review has been to compose a framework for user-centered evaluation. In the next phase of the project, we intend to test some of the most valid and feasible methods with an adaptive or adaptable system.

Requirements engineering for e-Government services: A citizen-centric approach and case study

Throughout the last decade, user involvement in e-Government service design has been virtually non-existent. Over time, e-Government experts began to realize that these services would benefit from a citizen-centric requirements engineering approach which has led to a demand for such an approach for this particular field. This article presents a citizen-centric approach towards user requirements engineering for e-Government services. It utilizes interviews and citizen walkthroughs of low-fidelity prototypes. A case study of a social support portal illustrates the approach and shows the need for repeated citizen inquiry, as the implementation of user requirements in low-fidelity prototype design is not always accepted by prospective end-users.

LEMtool: measuring emotions in visual interfaces

In this paper the development process and validation of the LEMtool (Layered Emotion Measurement tool) are described. The LEMtool consists of eight images that display a cartoon figure expressing four positive and four negative emotions using facial expressions and body postures. The instrument can be used during interaction with a visual interface, such as a website, and allows participants to select elements of the interface that elicit a certain emotion. The images of the cartoon figure were submitted to a validation study, in which participants rated the recognizability of the images as specific emotions. All images were found to be recognizable above chance level. In another study, the LEMtool was used to assess visual appeal judgements of a number of web pages. The LEMtool ratings were supported by visual appeal ratings of web pages both for very brief (50 milliseconds) and for long (free-viewing) stimulus exposures. Furthermore, the instrument provided insight into the elements of the web pages that elicited the emotional responses.

Measuring the emotional impact of websites: a study on combining a dimensional and discrete emotion approach in measuring visual appeal of university websites

In this paper a combined dimensional and discrete emotional approach is introduced for measuring emotions elicited by the visual appeal of websites. An online experiment was set out to measure the emotional experience of twelve screenshots of university websites. By indicating a position on the dimensions of pleasure and arousal, participants were asked to score corresponding emotion words that would relate to their position on the dimensions, plus random emotion words that were not related to their position. It was expected that scores on the corresponding emotion words would be higher than scores on the random emotion words. As expected, results showed that corresponding emotion words scored significantly higher, indicating that a combined dimensional and discrete emotion approach was feasible in measuring emotional experience elicited by visual appeal of websites.

Engineering User Requirements for e-Government Services: A Dutch Case Study

E-Government services can be made more user-friendly by involving prospective users in the requirements engineering stage. In this paper, we present user requirements engineering activities for e-Government services, demonstrate their effectiveness by means of a case study, and reflect upon their use. We used a combination of interviews with clients and involved service employees, which resulted in a set of requirements covering the different needs of future users. The design based on these requirements was tested with fifteen potential end-users, using rapid prototyping and a citizen walkthrough. These sessions show that it is crucial to test user requirements with potential end-users to create an e-Service that successfully caters to clients, providing such things as personalization and interoperability in an e-Government setting.

Eliciting User Input for Requirements on Personalization: The Case of a Dutch ERP System

For small and medium-sized companies, the fit between their business processes and their Enterprise Resource Planning (ERP) system is a critical success factor. The functions and features for essential tasks must be geared to the demands and skills of the individual users. This article reports on the usefulness of several methods for eliciting user input which served as a basis for requirements for a personalized ERP system. It describes the yield of heuristic evaluations, both by experts and by developers, and a focus group with six users representing the main user types. The focus group consisted of an identification of the most important functions, task demonstrations, and a mini design workshop. As a demonstration of the results of the various user-focused methods, some noteworthy findings on the personalization of ERP systems are presented.

Review as a Method for Improving Professional Texts.

In this article, the review process is described as a method of formative evaluation of texts. The description is based on three empirical studies of professional writing practices. It includes the goals of review, the actors involved in the process, the moments in the text production process that review is taking place, and the procedures followed. The studies make clear that review serves more goals than just improving the text. For improving the text, other methods than review probably produce better and more reliable results, especially when the goal is to improve the usability of the text. But review also has the function of having the information checked by experts and of building consensus and commitment in the organization. Because in most organizations review is taking place anyway, all remarks about the quality and acceptability of the document that are collected in the review process can be considered additional information that writers could use—with caution.In this article, the review process is described as a method of formative evaluation of texts. The description is based on three empirical studies of professional writing practices. It includes the goals of review, the actors involved in the process, the moments in the text production process that review is taking place, and the procedures followed. The studies make clear that review serves more goals than just improving the text. For improving the text, other methods than review probably produce better and more reliable results, especially when the goal is to improve the usability of the text. But review also has the function of having the information checked by experts and of building consensus and commitment in the organization. Because in most organizations review is taking place anyway, all remarks about the quality and acceptability of the document that are collected in the review process can be considered additional information that writers could use—with caution.

Self-assessed and actual Internet skills of people with visual impairments

The Internet can make available to people with a visual impairment information and services that are otherwise inaccessible. But do visually impaired users actually use common Internet applications and do they have the necessary skills? This article reports a two-part study addressing these questions. The first part was an interview study in which 73 young and 69 older Dutch people with a visual impairment were questioned about usage of applications such as e-mail, chat, and web forms, and their self-perceived competence. The young participants reported more frequent use of Internet applications and mentioned multiple goals (i.e., social and educational), compared to the older. Both groups considered themselves reasonably competent, with the young rating themselves higher. The second part was a case study with 20 young and 20 older participants from the first study, who performed common Internet tasks, using websites or applications that complied with accessibility guidelines. Task performance was analyzed in detail for demonstrated skills. Actual performance proved to be unrelated to self-rated competence. Moreover, the competence of both young and older participants fell far short of what active participation in society requires, especially for the more complex information and strategic skills. The success rate on the performance tasks was low. People with a visual impairment should receive extensive support for the acquisition of higher-level skills that are called upon when using Internet information and services in order to participate in society.

Identifying usability issues for personalization during formative evaluations: a comparison of three methods

A personalized system is one that generates unique output for each individual. As a result, personalization has transformed the interaction between the user and the system, and specific new usability issues have arisen. Methods used for evaluating personalized systems should be able to reveal the issues and problems specifically associated with personalization. Therefore this study evaluated three of the most common test methods used to detect usability problems in a personalized search engine. This was done by comparing the comments generated from thinking-aloud, questionnaires, and interviews. Questionnaires and interviews appear to be more useful for assessing specific usability issues for personalization, whereas thinking-aloud generates more comments on the usefulness of the system in the intended context of use and identifies the most critical and serious problems. Interviews, on the other hand, appear to yield a disproportionate number of positive comments. During the formative evaluation of a personalized system it is best to use a combination of thinking-aloud and questionnaires. This article concludes with a summary of implications for practitioners.

What is beautiful is useful - Visual appeal and expected information quality

Would users, when having a first glance on websites, expect that visually appealing websites contain better information than websites that are less appealing? And if they looked longer, would that change their judgment? We created two versions for 12 homepages of websites, one with low visual appeal, the other one with high visual appeal. Through a pre-test, we made sure we entered the main study with validated ‘beautiful’ and ‘ugly’ sites. In the main study, 588 participants asked to judge the expected information quality of the sites for a given task. For 11 of the 12 sites, we demonstrated that a positive judgment of the visual appeal was consistently transferred to a positive expectation of the information quality of the site, after a short exposure time. In a follow-up experiment, a week later, with 355 of the same participants, we proved that also with a longer exposure time the high visual appeal sites were expected to contain better information than the low appeal ones, although the difference between the two versions decreased somewhat. We conclude that visual appeal is an important shortcut for users to determine the information quality of a website.