Cédric Bach

Comparing Inspections and User Testing for the Evaluation of Virtual Environments

This article describes an experiment comparing three Usability Evaluation Methods: User Testing (UT), Document-based Inspection (DI), and Expert Inspection (EI) for evaluating Virtual Environments (VEs). Twenty-nine individuals (10 end-users and 19 junior usability experts) participated during 1 hr each in the evaluation of two VEs (a training VE and a 3D map). Quantitative results of the comparison show that the effectiveness of UT and DI is significantly better than the effectiveness of EI. For each method, results show their problem coverage: DI- and UT-based diagnoses lead to more problem diversity than EI. The overlap of identified problems amounts to 22% between UT and DI, 20% between DI and EI, and 12% between EI and UT for both virtual environments. The identification impact of the whole set of usability problems is 60% for DI, 57% for UT, and only 36% for EI for both virtual environments. Also reliability of UT and DI is significantly better than reliability of EI. In addition, a qualitative analysis identified 35 classes describing the profile of usability problems found with each method. It shows that UT seems particularly efficient for the diagnosis of problems that require a particular state of interaction to be detectable. On the other hand, DI supports the identification of problems directly observable, often related to learnability and basic usability. This study shows that DI could be viewed as a “4-wheel drive SUV evaluation type” (less powerful under certain conditions but able to go everywhere, with any driver), whereas UT could be viewed as a “Formula 1 car evaluation type” (more powerful but requiring adequate road and a very skilled driver). EI is found (considering all metrics) to be not efficient enough to evaluate usability of VEs.

Ergonomic criteria adapted to human virtual environment interaction

This paper introduces a version of Ergonomic Criteria adapted to HVEIs (Human Virtual Environment Interactions) which have been tested for validity. An experiment based on an assignment task (Ergonomic Criteria to usability problems) has been conducted to ensure appropriate understanding of the different criteria. The results of this study identify criteria that are clearly understood and criteria that need further improvement.

MACS: combination of a formal mixed interaction model with an informal creative session

In this paper, we propose a collaborative design method combining the informal power of creative session and the formal generative power of a mixed interaction model called MACS (Model Assisted Creativity Session). By using a formal notation during creative sessions, interdisciplinary teams systematically explore combinations between the physical and digital spaces and remain focused on the design problem to address. In this paper, we introduce the MACS method principles and illustrate its application on two case studies.

Un modèle préliminaire du domaine des systèmes mixtes

This paper describes a multidisciplinary approach to the modelling of the mixed reality systems domain. Such modelling, useful for structuring such a recent interaction domain, and to facilitate design and evaluation from both software engineering and ergonomics perspectives, is based on the combination of two approaches: the ASUR notation and an ergonomic recommendations-based model. Both approaches are presented as well as the results of their combination. The resulting model, an extension of the ASUR notation, is described as well as its limits and perspectives.

Developing Mixed Interactive Systems: A Model-Based Process for Generating and Managing Design Solutions

Taking advantage of the physical objects surrounding the user and the human ability to manipulate them fosters the development of multiple, new and advanced interaction techniques, called mixed interactive systems (MIS). Much work has been done to address specific aspects of the development of MIS. However, there is still no unifying conceptual framework to link these contributions and that presents a global approach for the development of MIS. In this context, this chapter presents a domain-specific development process that goes beyond ad hoc approaches and attempts to overcome barriers between different types of developer expertise, through a set of connections between steps of the MIS development process. Furthermore, to facilitate iteration in the design, these connections are observable, thus allowing a designer to review their decisions. The development process is illustrated via a concrete museum application.

Participatory Design Meets Mixed Reality Design Models

Participatory design and model-based approaches are two major HCI design approaches. Traditionally opposed, the first ones promote user’s creativity while the second ones support a more systematic approach of the design space. In Mixed Reality domain, combining these two aspects is especially crucial in order to support the design and to help the users to take into account the wide variety of emerging technologies. We introduce in this paper a solution to bring these two approaches together. In addition, we illustrate how the outcomes of this combination of formal and informal approaches serve as input for the implementation of the designed solution

Identification of UX dimensions for incident reporting systems with mobile applications in urban contexts: a longitudinal study

Incident reporting systems enable end-users to report problems that they have experienced in their working activities to authorities. Such applications are sought to sense the quality of the environment, thus enabling authorities to promote safety and well-being among citizens. Many governments are now promoting the use of mobile applications allowing citizens to report incidents in their neighbourhood to the administration. Nonetheless, it is not clear which user experience dimensions affect the adoption of incident reporting systems, and to what extent anticipated use of the system (anticipated UX) is a determinant for predicting the user experience with the final application. In order to understand how citizens perceive incident reporting systems and which factors affect the user experience (UX), we have performed empirical studies including interviews in early phases of the development process and empirical user testing of advanced prototypes. In this paper, we present the results of a longitudinal study on the evolution of the perception of UX dimensions along the development process, from interviews to running prototypes. Hereafter, we describe the method that has been used for coding the findings of these empirical studies according to six UX dimensions (including visual and aesthetic experience, emotions, stimulation, identification, meaning & value and social relatedness/co-experience). Moreover, we describe how the findings have been associated with users’ tasks. The findings from interviews and user testing indicate that whilst the perceived importance of some UX dimensions (such as identification and meaning & value) remains similar over time, other dimensions such as stimulation and emotions do evolve. Beyond the practical implications of this study for the design of incident reporting systems, this work presents an approach that allows comparing the results of UX assessments in different phases of the process.

Design and evaluation of mixed interactive museographic exhibits

This paper describes the four steps of a codesign process dedicated to the development of mixed interactive exhibits in Museums. The goal is to propose a well identified set of steps for accompanying the development of such systems, to document them and to suggest the most appropriate tools for each step. Based on a concrete design experience of a mixed interactive system in a museum, each step is illustrated and design decisions are highlighted with regards to the steps of the design process. A comparison of our codesign process with similar approaches applied in the field of interactive software engineering, participatory design or museum codesign is then performed; lessons learnt from the use of the process highlight benefits and limitations. Tightly coupled to this process, we then present different mechanisms and settings for evaluating visitors’ experience. Such mechanism involves lab and field activities requiring different methodologies and technical platforms. These formative evaluation activities are illustrated through some results of document-based usability inspections, user tests, field experiments and field studies. This paper reports lessons learnt and descriptive results of the prototype evaluations. Finally, we propose the ways for improving the use of codesign and evaluation of mixed interactive museographic exhibits.

Co-conception d'expériences interactives augmentées dédiées aux situations muséales

This paper shows the different steps of a co-design process dedicated to build augmented interactive experiments in Museums. This process enables an analytic framework listing the different dimensions to tackle during the design of augmented interactions. These different steps are illustrated by a case of study in the classification of life.

Instrumentation de focus-group dans la co-conception de systèmes mixtes

This paper introduces and illustrates a process that integrates Focus-Group into early phases of a mixed systems design process. This process supports the systematic exploration of mixed interaction design solutions and offers a framework to present data collected in a Focus-Group. We also present a connection between this process and an existing tool for the design of mixed systems.