Philip Kortum

An Empirical Evaluation of the System Usability Scale

This article presents nearly 10 years worth of System Usability Scale (SUS) data collected on numerous products in all phases of the development lifecycle. The SUS, developed by Brooke (1996), reflected a strong need in the usability community for a tool that could quickly and easily collect a users subjective rating of a products usability. The data in this study indicate that the SUS fulfills that need. Results from the analysis of this large number of SUS scores show that the SUS is a highly robust and versatile tool for usability professionals. The article presents these results and discusses their implications, describes nontraditional uses of the SUS, explains a proposed modification to the SUS to provide an adjective rating that correlates with a given score, and provides details of what constitutes an acceptable SUS score.

LiveLab: measuring wireless networks and smartphone users in the field

We present LiveLab, a methodology to measure real-world smartphone usage and wireless networks with a reprogrammable indevice logger designed for long-term user studies. We discuss the challenges of privacy protection and power impact in LiveLab and offer our solutions. We present an iPhone 3GS based deployment of LiveLab with 25 users intended for one year. Early results from the data collection so far highlight the unique strengths and potential of LiveLab. We have two objectives in this position paper. First, we demonstrate the feasibility and capability of LiveLab. By sharing our experience, we seek to advocate LiveLab as a network and user measurement methodology. Second, we present our preliminary findings, and seek feedback from the community regarding what data to collect.

Implementation of a foveated image coding system for image bandwidth reduction

We have developed a preliminary version of a foveated imaging system, implemented on a general purpose computer, which greatly reduces the transmission bandwidth of images. The system is based on the fact that the spatial resolution of the human eye is space variant, decreasing with increasing eccentricity from the point of gaze. By taking advantage of this fact, it is possible to create an image that is almost perceptually indistinguishable from a constant resolution image, but requires substantially less information to code it. This is accomplished by degrading the resolution of the image so that it matches the space-variant degradation in the resolution of the human eye. Eye movements are recorded so that the high resolution region of the image can be kept aligned with the high resolution region of the human visual system. This system has demonstrated that significant reductions in bandwidth can be achieved while still maintaining access to high detail at any point in an image. The system has been tested using 256 by 256 8 bit gray scale images with a 20 degree field-of-view and eye-movement update rates of 30 Hz (display refresh was 60 Hz). users of the system have reported minimal perceptual artifacts at bandwidth reductions of up to 94.7% (a factor of 18.8). Bandwidth reduction factors of over 100 are expected once lossless compression techniques are added to the system.

Characterizing web use on smartphones

The current paper establishes empirical patterns associated with mobile internet use on smartphones and explores user differences in these behaviors. We apply a naturalistic and longitudinal logs-based approach to collect real usage data from 24 iPhone users in the wild. These data are used to describe smartphone usage and analyze revisitation patterns of web browsers, native applications, and physical locations where phones are used. Among our findings are that web page revisitation through browsers occurred very infrequently (approximately 25% of URLs are revisited by each user), bookmarks were used sparingly, physical traversing patterns mirrored virtual (internet) traversing patterns and users systematically differed in their web use. We characterize these differences and suggest ways to support users with enhanced design of smartphone technologies and content.

The impact of inaccurate Internet health information in a secondary school learning environment.

Background Patients in the United States commonly use the Internet to acquire health information. While a significant amount of health-related information is available on the Internet, the accuracy of this information is highly variable. Objectives The objective of the study was to determine how effectively students can assess the accuracy of Internet-based material when gathering information on a controversial medical topic using simple keyword searches. Methods A group of 34 students from the science magnet high school in Houston, Texas searched for the terms “vaccine safety” and “vaccine danger” using Google and then answered questions regarding the accuracy of the health information on the returned sites. The students were also asked to describe the lessons they learned in the exercise and to answer questions regarding the strength of evidence for seven statements regarding vaccinations. Because of the surprising revelation that the majority of students left the exercise with inaccurate information concerning the safety and efficacy of vaccines, these same students participated in a follow-up study in which a fact-based vaccine video was shown, after which the assessment of student knowledge was repeated. Results Of the 34 participants, 20 (59%) thought that the Internet sites were accurate on the whole, even though over half of the links (22 out of 40, 55%) that the students viewed were, in fact, inaccurate on the whole. A high percentage of the students left the first exercise with significant misconceptions about vaccines; 18 of the 34 participants (53%) reported inaccurate statements about vaccines in the lessons they learned. Of the 41 verifiable facts about vaccines that were reported by participants in their lessons-learned statement, 24 of those facts (59%) were incorrect. Following presentation of the film, the majority of students left the exercise with correct information about vaccines, based on their lessons-learned statement. In this case, 29 of the 31 participants (94%) reported accurate information about vaccines. Of the 49 verifiable facts about vaccines that were reported by participants, only 2 (4%) were incorrect. Students had higher correct scores in the “strength of evidence” exercise following exposure to the video as well. Conclusions Allowing students to use the Internet to gain information about medical topics should be approached with care since students may take away predominantly incorrect information. It is important to follow up conflicting information with a solid, unambiguous message that communicates those lessons that the instructor deems most important. This final message should be fact based but may need to contain an anecdotal component to counter the strong emotional message that is often delivered by inaccurate Internet sites.

Usability Ratings for Everyday Products Measured With the System Usability Scale

This article characterizes the usability of 14 common, everyday products using the System Usability Scale (SUS). More than 1,000 users were queried about the usability of these products using an online survey methodology. The study employed two novel applications of the SUS. First, participants were not asked to perform specific tasks on these products before rating their usability but were rather asked to assess usability based on their overall integrated experience with a given product. Second, some of the evaluated products were assessed as a class of products (e.g., “microwaves”) rather than a specific make and model, as is typically done. The results show clear distinctions among different products and will provide practitioners and researchers with important known benchmarks as they seek to characterize and describe results from their own usability studies.

Exploring iPhone usage: the influence of socioeconomic differences on smartphone adoption, usage and usability

Previous studies have found that smartphone users differ by orders of magnitude. We explore this variability to understand how users install and use native applications in ecologically-valid environments. A quasi-experimental approach is applied to compare how users in different socio-economic status (SES) groups adopt new smartphone technology along with how applications are installed and used. We present a longitudinal study of 34 iPhone 3GS users. 24 of these participants were chosen from two carefully selected SES groups who were otherwise similar and balanced. Usage data collected through an in-device programmable logger, as well as several structured interviews, identify similarities, differences, and trends, and highlight systematic differences in smartphone usage. A group of 10 lower SES participants were later recruited and confirm the influence of SES diversity on device usage. Among our findings are that a large number of applications were uninstalled, lower SES groups spent more money on applications and installed more applications overall, and the lowest SES group perceived the usability of their iPhones poorly in comparison to the other groups. We further discuss the primary reasons behind this low score, and suggest design implications to better support users across SES brackets.

The Effect of Content Desirability on Subjective Video Quality Ratings

Objective: The purpose of this study was to examine the impact of the desirability of content on viewers’ ratings of subjective video quality. Background: Most subjective video quality studies use short-duration clips that are specially designed to exercise the encoding algorithms and do not consider the desirability of the content as a variable. Method: In four studies, we employed a total of 100 participants and 180 movie clips encoded at nine levels from 550 kbps up to DVD quality. Participants viewed clips that were 2 min in length and then were asked about video quality of the clips and desirability of the movie content. Results: The results of these studies show that there is a strong correlation between the desirability of movie content and subjective ratings of video quality. This strong relationship holds across a wide range of encoding levels and movie content when that content is viewed under longer, more naturalistic viewing conditions. Conclusion: The effects of content should be considered when evaluating the subjective quality of encoded video content, as these effects can be as large as those seen between low- and high-quality encodings. Application: Researchers and practitioners trying to determine acceptable levels of video quality for actual consumption by consumers may find that the results and methods described here allow for a more accurate assessment of levels of video quality that are acceptable in a fielded service.

Measuring the Usability of Mobile Applications for Phones and Tablets

The use of applications on mobile devices has reached historic levels. Using the System Usability Scale (SUS), data were collected on the usability of applications used on two kinds of mobile platforms—phones and tablets—across two general classes of operating systems, iOS and Android. Over 4 experiments, 3,575 users rated the usability of 10 applications that had been selected based on their popularity, as well as 5 additional applications that users had identified as using frequently. The average SUS rating for the top 10 apps across all platforms was 77.7, with a nearly 20-point spread (67.7–87.4) between the highest and lowest rated apps. Overall, applications on phone platforms were judged to be more usable than applications on the tablet platforms. Practitioners can use the information in this article to make better design decisions and benchmark their progress against a known universe of apps for their specific mobile platform.

Adaptation mechanisms in spatial vision. II: Flash thresholds and background adaptation

To examine how the mechanisms of light adaptation affect spatial pattern vision, contrast detection thresholds were measured for sinusoidal (increment-Gabor) probes on flashed backgrounds in the presence of steady adapting backgrounds. The thresholds for all spatial frequencies (1-12 c/deg), flashed-background intensities (dark to 4 log td) and adapting-background intensities (dark to 4 log td) were adequately described by a simple model consisting of a compressive nonlinearity (a modified Naka-Rushton function), a subtractive adaptation factor, and a multiplicative adaptation factor. For all five subjects the compressive nonlinearity was found to vary systematically with spatial frequency; for all but one subject, the subtractive and multiplicative factors were found to be relatively constant.