Simone Borsci

On the dimensionality of the System Usability Scale: a test of alternative measurement models

The System Usability Scale (SUS), developed by Brooke (Usability evaluation in industry, Taylor & Francis, London, pp 189–194, 1996), had a great success among usability practitioners since it is a quick and easy to use measure for collecting users’ usability evaluation of a system. Recently, Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) have proposed a two-factor structure—Usability (8 items) and Learnability (2 items)—suggesting that practitioners might take advantage of these new factors to extract additional information from SUS data. In order to verify the dimensionality in the SUS’ two-component structure, we estimated the parameters and tested with a structural equation model the SUS structure on a sample of 196 university users. Our data indicated that both the unidimensional model and the two-factor model with uncorrelated factors proposed by Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) had a not satisfactory fit to the data. We thus released the hypothesis that Usability and Learnability are independent components of SUS ratings and tested a less restrictive model with correlated factors. This model not only yielded a good fit to the data, but it was also significantly more appropriate to represent the structure of SUS ratings.

Assessing user satisfaction in the era of user experience: comparison of the SUS, UMUX and UMUX-LITE as a function of product experience

Nowadays, practitioners extensively apply quick and reliable scales of user satisfaction as part of their user experience analyses to obtain well-founded measures of user satisfaction within time and budget constraints. However, in the human–computer interaction literature the relationship between the outcomes of standardized satisfaction scales and the amount of product usage has been only marginally explored. The few studies that have investigated this relationship have typically shown that users who have interacted more with a product have higher satisfaction. The purpose of this article was to systematically analyze the variation in outcomes of three standardized user satisfaction scales (SUS, UMUX, UMUX-LITE) when completed by users who had spent different amounts of time with a website. In two studies, the amount of interaction was manipulated to assess its effect on user satisfaction. Measurements of the three scales were strongly correlated and their outcomes were significantly affected by the amount of interaction time. Notably, the SUS acted as a unidimensional scale when administered to people who had less product experience but was bidimensional when administered to users with more experience. Previous findings of similar magnitudes for the SUS and UMUX-LITE (after adjustment) were replicated but did not show the previously reported similarities of magnitude for the SUS and the UMUX. Results strongly encourage further research to analyze the relationships of the three scales with levels of product exposure. Recommendations for practitioners and researchers in the use of the questionnaires are also provided.

Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation

The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach -- which we call the Grounded Procedure -- is introduced and its value argued.

An ideal model of an assistive technology assessment and delivery process

The purpose of the present work is to present some aspects of the Assistive Technology Assessment (ATA) process model compatible with the Position Paper 2012 by AAATE/EASTIN. Three aspects of the ATA process will be discussed in light of three topics of the Position Paper 2012: (i) The dimensions and the measures of the User eXperience (UX) evaluation modelled in the ATA process as a way to verify the efficient and the evidence-based practices of an AT service delivery centre; (ii) The relevance of the presence of the psychologist in the multidisciplinary team of an AT service delivery centre as necessary for a complete person-centred assistive solution empowering users to make their own choices; (iii) The new profession of the psychotechnologist, who explores users needs by seeking a proper assistive solution, leading the multidisciplinary team to observe critical issues and problems. Through the foundation of the Position Paper 2012, the 1995 HEART study, the Matching Person and Technology model, the ICF framework, and the pillarsof the ATAprocess, thispaper sets fortha concept and approach that emphasise the personal factors of the individual consumer and UX as key to positively impacting a successful outcome and AT solution. 1. Background and purpose The model of the Assistive Technology Assess- ment (ATA) process was developed by Federici and Scherer (1) with the contribution of 55 scholars from five continents. It models the functioning process of centres for assistive technology (AT) evaluation and provision independentlyfrom the model of local or na- tional service delivery systems. The aim is to suggest practical guidelines for a quality control of effective processes of matching individual users with the most appropriate technology. The ATA process borrows a user-driven working methodology from the Matching Person and Technology (MPT) model of Scherer (3,4).

Web popularity: an illusory perception of a qualitative order in information

Using a psychotechnological perspective, this study discusses the current model of information ranking by search engines, based on quantitative Web Popularity (WP), which binds users to a cognitive adaptation to the rank-system restrictions. This phenomenon gives rise to a “rich-get-richer” effect on the Web. This paper claims that such an effect could be limited or reversed by the introduction of quality factors in ranking, and addresses the case of accessibility as a fundamental such factor. A study is reported which, through introducing an accessibility factor in a well-known popularity ranking algorithm, demonstrates that this transformation allows a qualitative rearrangement, without modifying or weighing on the properties of the rank. The overall approach is grounded on two development factors: the analysis of accessibility through specific tools and the employment of this analysis within all components used to build up the ranking. The results show that it is important to reconsider WP as including not only on the number of inbound and outbound links of a website, but also on its level of accessibility for all users, and on users’ judgment of the website use as efficient, effective, and satisfactory.

How many testers are needed to assure the usability of medical devices

Before releasing a product, manufacturers have to follow a regulatory framework and meet standards, producing reliable evidence that the device presents low levels of risk in use. There is, though, a gap between the needs of the manufacturers to conduct usability testing while managing their costs, and the requirements of authorities for representative evaluation data. A key issue here is the number of users that should complete this evaluation to provide confidence in a product’s safety. This paper reviews the US FDA’s indication that a sample composed of 15 participants per major group (or a minimum of 25 users) should be enough to identify 90–97% of the usability problems and argues that a more nuanced approach to determining sample size (which would also fit well with the FDA’s own concerns) would be beneficial. The paper will show that there is no a priori cohort size that can guarantee a reliable assessment, a point stressed by the FDA in the appendices to its guidance, but that manufacturers can terminate the assessment when appropriate by using a specific approach – illustrated in this paper through a case study – called the ‘Grounded Procedure’.

The partial concurrent thinking aloud : A new usability evaluation technique for blind users

The aim of this study is to build up a verbal protocol technique for samples of visual impaired users in order to overcome the limits of concurrent and retrospective protocols. Indeed, when blind users surf using a screen reader and talk about the way they interact with the computer, the evaluation is influenced by a structural interference. Users are force to think aloud and listen to the screen reader at the same time. The technique we improved, called Partial Concurrent Thinking Aloud (PCTA), integrates a modified set of concurrent verbalization and retrospective analysis. One group of 6 blind user and another group of 6 sighted users evaluated the usability of a website by PCTA. Estimating the number of users needed with an asymptotic test, we found out that the two groups had an equivalent ability of identifying usability problems, both over 80%. The result suggest that PCTA, even respecting the properties of classic verbal protocols, also allows to overcome the structural interference and the limits of concurrent and retrospective protocols when used with screen-reader users.

Usability evaluation with screen reader users : A video presentation of the PCTA's experimental setting and rules

In the study entitled “Web usability evaluation with screen reader users: Implementation of the Partial Concurrent Thinking Aloud technique” (Federici et al. 2010), we have proposed a modified protocol of usability evaluation technique for blind users, which integrates the features of the concurrent and the retrospective techniques. This new technique, called partial concurrent thinking aloud (PCTA), while respecting the properties of classic verbal protocols, overcomes the structural interference and the limits of concurrent and retrospective protocols when used with screen reader users. In order to facilitate understanding and acquisition of the PCTA’s technique for practitioners and researchers, we have video recorded three different verbal protocols by visualizing five experimental sections. In the first two videos, we have compared a concurrent with a retrospective’s verbal protocol of a sighted user, showing the difference of the verbalizations provided by the user in these two conditions. The third video shows the structural interference of the screen reader, during a blind user concurrent thinking aloud. In the last two videos, we show the difference of a blind user behaviour when PCTA or retrospective protocol is adopted. The videos clearly visualize the advantage of the PCTA use in respect of the two other protocols. In conclusion, the visualization of the PCTA technique confirms that this new verbal protocol promotes and guarantees a more user-driven usability assessment with disabled people, by better involving screen reader users, overcoming the structural interference and the limits of the concurrent and retrospective protocols.

The Bootstrap Discovery Behaviour (BDB): a new outlook on usability evaluation

The value of λ is one of the main issues debated in international usability studies. The debate is centred on the deficiencies of the mathematical return on investment model (ROI model) of Nielsen and Landauer (1993). The ROI model is discussed in order to identify the base of another model that, respecting Nielsen and Landauer’s one, tries to consider a large number of variables for the estimation of the number of evaluators needed for an interface. Using the bootstrap model (Efron 1979), we can take into account: (a) the interface properties, as the properties at zero condition of evaluation and (b) the probability that the population discovery behaviour is represented by all the possible discovery behaviours of a sample. Our alternative model, named Bootstrap Discovery Behaviour (BDB), provides an alternative estimation of the number of experts and users needed for a usability evaluation. Two experimental groups of users and experts are involved in the evaluation of a website (http://www.serviziocivile.it). Applying the BDB model to the problems identified by the two groups, we found that 13 experts and 20 users are needed to identify 80% of usability problems, instead of 6 experts and 7 users required according to the estimation of the discovery likelihood provided by the ROI model. The consequence of the difference between the results of those models is that in following the BDB the costs of usability evaluation increase, although this is justified considering that the results obtained have the best probability of representing the entire population of experts and users.

A visual sonificated web search clustering engine

Information Visualization is a widespread approach of the Information and Communication Technologies, and it facilitates the manipulation of abstract information by representing it by geometric models. It has been shown that spatial representation can be independent by the sensorial way in which it is perceived (Avraamides et al. 2004; De Vega et al. 2001), leading to the hypothesis of an amodal spatial representation (Bryant 1992). In light of these studies, an important alternative to Information Visualization methods appears to be the Sonification approach, an information representation method (Olivetti Belardinelli et al. 2009) that implements non-speech audio information to ‘‘represent data relations into perceived relations in an acoustic signal for the purposes of facilitating communication and interpretation’’ (Kramer et al. 1997). The main focus of the sonification approach is on the interactivity between the user and the data representation, since it allows user to manipulate complex data and their inner relations in a dynamic way. Currently, the information representation techniques of main search engines are based on a top-down hierarchic order, according to quantitative indexes of ranking. In a recent study (Borsci et al. 2008; Federici et al. 2009), it has been shown that high ranking values of web popularity (WP) do not match with high levels of accessibility. The top-down output of search engines leads users to assign a qualitative value to the hierarchy organization based on WP, although it is calculated without considering quality indexes (i.e., accessibility). Moreover, Andronico et al. highlighted the lack of qualitative ratings on WP, showing that the main search engines do not comply with W3C accessibility guidelines (Andronico et al. 2008). In this study, we aim to describe the redesign process of WhatsOnWeb (WoW) (Di Giacomo et al. 2007, 2008), a visual graph based search engine implemented by the Department of Electronic and Information Engineering of the University of Perugia. WoW is an autonomous application able to order indexed Web data using semantic nodes in a single page network diagram. In this way, WoW overcomes the page scrolling normally required by the reports of the traditional search engines (Search Engines Report Page), overpassing the limits of traditional Web information representation methods pointed out by Borsci et al. (2008).