Erica L. Olmsted-Hawala

Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability

We describe an empirical, between-subjects study on the use of think-aloud protocols in usability testing of a federal data-dissemination Web site. This double-blind study used three different types of think-aloud protocols: a traditional protocol, a speech-communication protocol, and a coaching protocol. A silent condition served as the control. Eighty participants were recruited and randomly pre-assigned to one of four conditions. Accuracy and efficiency measures were collected, and participants rated their subjective satisfaction with the site. Results show that accuracy is significantly higher in the coaching condition than in the other conditions. The traditional protocol and the speech-communication protocol are not statistically different from each other with regard to accuracy. Participants in the coaching condition are more satisfied with the Web site than participants in the traditional or speech-communication condition. In addition, there are no significant differences with respect to efficiency (time-on-task). This paper concludes with recommendations for usability practitioners.

Age-Related Differences in Eye Tracking and Usability Performance: Website Usability for Older Adults

Cognitive decline is inherent with age. Despite known cognitive limitations, older adults are generally not taken into account during website design. Understanding age-related differences in website navigation is instructive for website design, especially considering the growing number of older adults who use the Internet. This article presents usability and eye-tracking data from five independent website usability studies that included younger and older participants. Overall results revealed age-dependent differences in eye movement and performance during website navigation on some of the sites. In particular, older participants had lower accuracy in one study and took longer to complete tasks in two studies compared to younger participants, they looked at the central part of the screen more frequently than younger participants in two studies, and they looked at the peripheral left part of the screen less frequently and took longer to first look at the peripheral top part of the screen than younger participants in one study. These data highlight the potential for age-related differences in performance while navigating websites and provide motivation for further exploration. Implications for website design and for usability practitioners are discussed.

Think-aloud protocols: Analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich Web site

We describe an empirical, between-subjects study on the use of think-aloud protocols in usability testing of an information-rich Web site. This double-blind study used three different types of think-aloud protocols: a traditional protocol, a speech-communication protocol, and a coaching protocol. A silent condition served as the control. Eighty participants were recruited and randomly pre-assigned to one of four conditions. With the goal of keeping unintended bias to a minimum, data analysis did not count the number of identified usability problems by condition, which was considered too subjective. Rather, the study collected the number of verbalized and non-verbalized counts of frustration by condition that users experienced. The study also did a count of the number of verbalized and non-verbalized instances of positive comments by condition that users expressed. Results show that there were no statistical differences in the number of counts by condition with respect to the traditional, speech communication, or coaching condition. The study concludes that simply counting the verbalizations of users by condition does not give enough information to determine whether any of the conditions would lead to a better understanding of the usability problems associated with the Web site.

Age-Related differences in search strategy and performance when using a data-rich web site

In a usability study on a portion of the Census Bureau Web site, we assess how people of different ages search for specific information by evaluating performance and strategy differences between age groups. We collected usability metrics of accuracy and efficiency, including mouse click data. Eye-tracking data were also collected including eye-movement patterns in pre-defined areas of interest. This paper focuses on the number of fixations (whether participants looked at the correct area of the screen) and number of unique visits (whether participants re-checked their answers). Results show that on the hard task only, older adults took longer to make the first click when initially starting the task, and there was a trend for younger adults to have higher accuracy. All age groups re-checked their answers suggesting that users of all ages experience difficulties when reading and comprehending complex data tables.

3 – Usability Testing

An overview of how eye tracking can be incorporated into usability testing. Using real-world examples from recent usability studies, this chapter explains the basics on how to combine usability testing with eye tracking, how eye tracking captures a broad picture of users’ experiences with an interface, how eye tracking aids in communicating results more effectively with clients, and the challenges related to using eye tracking during a usability test.

The use of paper-prototyping in a low-fidelity usability study

The Usability Laboratory at the US Census Bureau participated in the redesign effort for the American FactFinder Web site. The Web site was undergoing a major redesign in anticipation of receiving the data from the 2010 Census and making the data available to the public. To obtain user feedback on the conceptual design, the team used the technique of paper prototyping in a low-fidelity usability study. The low-fidelity study identified user problems, and the team was able to implement design recommendations quickly and easily. This article highlights the benefits of paper prototyping in low-fidelity usability testing.

Card Sorting, Information Architecture And Usability: Adding in Our Users' Perspective to Re-Design the Census Bureau Web Site

Previous research shows that one of the major problems for users of the information rich Census Bureau Web site was in locating or navigating to desired content. This knowledge motivated Bureau staff to begin thinking of how to reorganize the site to attain a more usable information architecture. The team determined that bringing in user-centered design practices into the process could benefit. Thus, in the initial stages of redesigning the Web site, Census Bureau staff in the usability lab conducted two rounds of card sorting. These studies were intended to reveal a better understanding of the content, and the organization of the content that should be on the Census Bureau main page and on the lower-level target pages of the site. This paper looks at how the card sorting studies identified ways that users organize site content and shows how bringing in users to help organize and understand site content gave a better basis for our Web site content development

Experimentation for Developing Evidence-Based UI Standards of Mobile Survey Questionnaires

With the growing use of smartphones, many surveys can now be administered using those phones. Such questionnaires are called mobile survey questionnaires. The designer of a mobile survey questionnaire is challenged with presenting text and controls on a small display, while allowing respondents to correctly understand and answer questions with ease. To address this challenge, we are developing an evidence-based framework of user interface design for mobile survey questionnaires. The framework includes two parts: standards for the basic elements of survey-relevant mobile device operation and guidelines for the building blocks of mobile survey questionnaires. In this presentation, we will describe five behavioral experiments designed to collect evidence for developing the standards. These experiments cover visual perception and motor actions relevant to survey completion. Some preliminary results from ongoing data collection are presented.

Answers for Self and Proxy – Using Eye Tracking to Uncover Respondent Burden and Usability Issues in Online Questionnaires

In a study of the American Community Survey online instrument, we assessed how people answered questions about themselves and other individuals living in their household using eye-tracking data and other qualitative measures. This paper focuses on the number of fixations (whether participants looked at specific areas of the screen), fixation duration (how long participants looked at the questions and answers), and number of unique visits (whether participants rechecked the question and answer options). Results showed that for age, date of birth and employment duty questions participants had more fixations and unique visit counts, and spent more time on the screen when answering about unrelated members of their household than when answering about themselves. Differing eye movements for proxy reporting suggest that answering some survey questions for other unrelated people poses more burden on respondents than answering about oneself. However, not all questions showed this tendency, so eye tracking alone is not enough to detect burden.

Information architecture: Strategies for analysis of card-sorting data for organizing information on the Census Bureau Web site

Previous research showed that one of the major problems for users of the information-rich Census Bureau Web site was in locating or navigating to desired content. Card sorting was chosen as a way to improve the site. Two important aspects of the card-sorting study are reviewed in this paper: preparation of card labels and interpretation of the results. For the preparation of the labels, the focus was on getting and using user-derived terminology during the sorts. For the analysis, the focus was on using additional ways to visualize and analyze the data. Data were analyzed by cluster analysis in the form of a distance matrix in a tree hierarchy as well as a multidimensional scaling view. In addition, spreadsheet analysis was used to identify terms that were overlapping. Results highlight the importance of using the different views/tools to get a better understanding of the data. The results identify clusters of terms that should be grouped together, terms that need to go in more than one location and outlier terminology, which were seen by users as fitting into many different categories. These results will inform the re-organization of site content.