Michael Quade

Automated Usability Evaluation during Model-Based Interactive System Development

In this paper we describe an approach to efficiently evaluate the usability of an interactive application that has been realized to support various platforms and modalities. Therefore we combine our Multi-Access Service Platform (MASP), a model-based runtime environment to offer multimodal user interfaces with the MeMo workbench which is a tool supporting an automated usability analysis. Instead of deriving a system model by reverse-engineering or annotating screenshots for the automated usability analysis, we use the semantics of the runtime models of the MASP. This allows us to reduce the evaluation effort by automating parts of the testing process for various combinations of platforms and user groups that should be addressed by the application. Furthermore, by testing the application at runtime, the usability evaluation can also consider system dynamics and information that are unavailable at design time.

Analysis of a new simulation approach to dialog system evaluation

The evaluation of spoken dialog systems still relies on subjective interaction experiments for quantifying interaction behavior and user-perceived quality. In this paper, we present a simulation approach replacing subjective tests in early system design and evaluation phases. The simulation is based on a model of the system, and a probabilistic model of user behavior. Probabilities for the next user action vary in dependence of system features and user characteristics, as defined by rules. This way, simulations can be conducted before data have been acquired. In order to evaluate the simulation approach, characteristics of simulated interactions are compared to interaction corpora obtained in subjective experiments. As was previously proposed in the literature, we compare interaction parameters for both corpora and calculate recall and precision of user utterances. The results are compared to those from a comparison of real user corpora. While the real corpora are not equal, they are more similar than the simulation is to the real data. However, the simulations can predict differences between system versions and user groups quite well on a relative level. In order to derive further requirements for the model, we conclude with a detailed analysis of utterances missing in the simulated corpus and consider the believability of entire dialogs.

Automated Usability Evaluation of Model-Based Adaptive User Interfaces for Users with Special and Specific Needs by Simulating User Interaction

Adaptive applications have the potential to help users with special and specific needs. However, evaluating the usability of such adaptive applications tends to become very complex. This chapter presents an integrated concept for the automated usability evaluation of model-based adaptive user interfaces. The approach is supposed to be used complementary to custom usability evaluations at an early stage of development. Interaction of a user is simulated and evaluated by combining a user model with user interface models from a model-based development framework, which is capable of providing different adaptation alternatives based on user attributes and the context of use. The main benefit of the approach is that no additional descriptions of the application’s UI and tasks need to be created for the usability evaluation because they are already available from the development process. As a result, different design alternatives and adaptation variants can be compared under equal usability evaluation criteria. Further, the complexity and costs for applying automated usability evaluation to adaptive user interfaces for users with special and specific needs can be reduced.

Predicting task execution times by deriving enhanced cognitive models from user interface development models

Adaptive user interfaces (UI) offer the opportunity to adapt to changes in the context, but this also poses the challenge of evaluating the usability of many different versions of the resulting UI. Consequently, usability evaluations tend to become very complex and time-consuming. We describe an approach that combines model-based usability evaluation with development models of adaptive UIs. In particular, we present how a cognitive user behavior model can be created automatically from UI development models and thus save time and costs when predicting task execution times. With the help of two usability studies, we show that the resulting predictions can be further improved by using information encoded in the UI development models.

Predicting user error for ambient systems by integrating model-based UI development and cognitive modeling

With the move to ubiquitous computing, user interfaces (UI) are no longer bound to specific devices. While this problem can be tackled using the model-based UI development (MBUID) process, the usability of the device-specific interfaces is still an open question. We are presenting a combined system that integrates MBUID with a cognitive modeling framework in order to provide usability predictions at development time. Because of their potential impact, our focus within usability problems lies on user errors. These are captured in a cognitive model that capitalizes on meta-information provided by the MBUID system such as the abstract role of a UI element within a task sequence (e.g., input, output, command). The free parameters of the cognitive model were constrained using data from two previous studies. A validation experiment featuring a new application and UI yielded an unexpected error pattern that was nonetheless consistent with the model predictions.

Rule-Based Approach for Simulating Age-Related Usability Problems

Ambient Assisted Living requires easy to use interfaces, making usability a critical feature. Because usability evaluations are resource and time consuming, several automation efforts have been made, one of which is the simulation of users interacting with UIs. In this article, we present ongoing work of a tool for automated usability simulations that allows simulating agerelated deficits. The tool is specifically intended to be used by IT practitioners, i.e. in difference to cognitive architectures that allow similar simulations, this tool does not require extensive knowledge in cognitive science. A core component of the simulation tool is its rule-based User Model (UM). During a simulated interaction, the UM selects actions causing a model of the UI to change states until a specified task goal is satisfied or the UM “gives up”. Interactions of the UM are calculated from probabilities which are informed by rules drawing on user and UI attributes. Using a Monte Carlo approach, the simulation is iterated, resulting in a set of task solutions where non-optimal solutions may indicate usability problems. By analyzing which rules led the UM to interact non-optimally, our approach can offer hints on how to improve the UI. While our approach cannot render user-based evaluations unnecessary, our aim is to substantially reduce the effort involved in usability testing of UIs as well as to provide an automated tool that can be used early on in the development process.

Designing New Experiences in the Smart Home: Multi-camera Person Localization Framework to Document Predefined Situations

In an interdisciplinary team, we developed personas and use cases addressing higher-level needs of residents in a smart home, such as safety, self reflection or evocation of memories. For one of the use cases, we describe a concrete usage scenario in which photos are taken automatically in predefined situations. We show how we implemented the required technology, a multi-camera person localization framework, and present results from a lab study to evaluate the performance and user experience of the technology.

How Can Cognitive Modeling Benefit from Ontologies? Evidence from the HCI Domain

Cognitive modeling as a method has proven successful at reproducing and explaining human intelligent behavior in specific laboratory situations, but still struggles to produce more general intelligent capabilities. A promising strategy to address this weakness is the addition of large semantic resources to cognitive architectures. We are investigating the usefulness of this approach in the context of human behavior during software use. By adding world knowledge from a Wikipedia-based ontology to a model of human sequential behavior, we achieve quantitatively and qualitatively better fits to human data.The combination of model and ontology yields additional insights that cannot be explained by the model or the ontology alone.

Requirements for Applying Simulation-Based Automated Usability Evaluation to Model-Based Adaptive User Interfaces for Smart Environments

Users in smart environments benefit from context-aware applications that are able to adapt their user interfaces UI to specific situations. In the same way as the development of adaptive applications poses high demands on the designers, the evaluation of their usability also becomes more complex and time consuming because the context of use and different adaptation variants need to be considered. While automated usability evaluations cannot fully replace user tests in this domain, they can be applied to multiple adaptation variants at an early stage of development and thus reduce time and complexity. This paper presents general requirements for applying automated model-based usability evaluations that apply simulated user interaction as an approach to evaluate UIs of adaptive applications based on the underlying development models.

Model-Based Evaluation of Adaptive User Interfaces

This thesis presents an approach for the automated usability evaluation of model-based adaptive user interfaces. The described approach is supposed to be used complementary to custom evaluations at development time. By combining a user model with user interface models from a model-based development framework, providing different adaptation alternatives, interaction can be simulated and evaluated accordingly. This way, no additional descriptions of the application need to be created for the evaluation. As a result the complexity and costs for applying an automated usability evaluation can be reduced significantly.