Marilyn C. Salzman

Damaged merchandise? a review of experiments that compare usability evaluation methods

An interest in the design of interfaces has been a core topic for researchers and practitioners in the field of human-computer interaction (HCI); an interest in the design of experiments has not. To the extent that reliable and valid guidance for the former depends on the results of the latter, it is necessary that researchers and practitioners understand how small features of an experimental design can cast large shadows over the results and conclusions that can be drawn. In this review we examine the design of 5 experiments that compared usability evaluation methods (UEMs). Each has had an important influence on HCI thought and practice. Unfortunately, our examination shows that small problems in the way these experiments were designed and conducted call into serious question what we thought we knew regarding the efficacy of various UEMs. If the influence of these experiments were trivial, then such small problems could be safely ignored. Unfortunately, the outcomes of these experiments have been used to justify advice to practitioners regarding their choice of UEMs. Making such choices based on misleading or erroneous claims can be detrimental--compromising the quality and integrity of the evaluation, incurring unnecessary costs, or undermining the practitioners credibility within the design team. The experimental method is a potent vehicle that can help inform the choice of a UEM as well as help to address other HCI issues. However, to obtain the desired outcomes, close attention must be paid to experimental design.

A Model for Understanding How Virtual Reality Aids Complex Conceptual Learning

Designers and evaluators of immersive virtual reality systems have many ideas concerning how virtual reality can facilitate learning. However, we have little information concerning which of virtual realitys features provide the most leverage for enhancing understanding or how to customize those affordances for different learning environments. In part, this reflects the truly complex nature of learning. Features of a learning environment do not act in isolation; other factors such as the concepts or skills to be learned, individual characteristics, the learning experience, and the interaction experience all play a role in shaping the learning process and its outcomes. Through Project Science Space, we have been trying to identify, use, and evaluate immersive virtual realitys affordances as a means to facilitate the mastery of complex, abstract concepts. In doing so, we are beginning to understand the interplay between virtual realitys features and other important factors in shaping the learning process and learning outcomes for this type of material. In this paper, we present a general model that describes how we think these factors work together and discuss some of the lessons we are learning about virtual realitys affordances in the context of this model for complex conceptual learning.

ScienceSpace: virtual realities for learning complex and abstract scientific concepts

Three virtual worlds have been built to investigate the effect of immersive, multisensory computer-generated experiences on learning topics in science. Currently targeted at high school and beginning college students, these worlds address Newtonian mechanics, electrostatics, and molecular structure and dynamics. Data has been collected on usability and learning through questionnaires, pre- and post-tests, in situ prediction and experiment and post-session interviews. The results are not uniformly conclusive but suggest that students can improve their mastery of abstract concepts through the use of virtual environments that have been designed for learning. Moreover, usability studies have identified many significant problems that have been addressed in successive refinements of these worlds. Future work will include collaborative learning studies (both local and distant), use of intelligent agents, and comparison with two-dimensional microworlds.

Multisensory Immersion as a Modeling Environment for Learning Complex Scientific Concepts

In every aspect of our knowledge-based society, fluency in understanding complex information spaces is an increasingly crucial skill (Dede and Lewis, 1995). In research and industry, many processes depend on peolple utilizing complicated representations of information (Rieber, 1994). Increasingly, workers must navigate complex information spaces to locate data they need, must find patterns in information for problem solving, and must use sophisticated representations of information to communicate their ideas (Kohn, 1994; Studt, 1995). Further, to make informed decisions about public-policy issues such as global warming and environmental contamination, citizens must comprehend the strenghts and limitations of scientific models based on multivariate interactions. In many academic areas, students’ success now depends on their ability to envision and manipulate abstract multidimensional information spaces (Gordin and Pea, 1995). Fields in which students struggle with mastering these types of representations include mathematics, science, engineering, statistics, and finance.

The Development of a Virtual world for Learning Newtonian Mechanics

We are collaboratively designing “ScienceSpace,” a collection of virtual worlds designed to explore the potential utility of physical immersion and multisensory perception to enhance science education. This paper describes the creation and formative evaluation of NewtonWorld, a virtual environment for investigating the kinematics and dynamics of one-dimensional motion. Through this research, we are developing design heuristics, assessment methodologies, and insights about multisensory learning generalizable to a wide range of educational environments. We are also gaining an understanding the strengths and weaknesses of virtual reality as a vehicle for learning.

VR's frames of reference: a visualization technique for mastering abstract multidimensional information

This paper describes a research study that investigated howdesigners can use frames of reference (egocentric, exocentric, anda combination of the two) to support the mastery of abstractmultidimensional information. The primary focus of this study wasthe relationship between FORs and mastery; the secondary focus wason other factors (individual characteristics and interactionexperience) that were likely to influence the relationship betweenFORs and mastery. This studys outcomes (1) clarify how FORs work inconjunction with other factors in shaping mastery, (2) highlightstrengths and weaknesses of different FORs, (3) demonstrate thebenefits of providing multiple FORs, and (4) provide the basis forour recommendations to HCI researchers and designers.

ScienceSpace: lessons for designing immersive virtual realities

ScienceSpace is a collection of immersive virtual realities designed to explore the potential utility of physical immersion and multisensory perception to aid in the learning of science. Through the design and evaluation of ScienceSpace, we are learning lessons about the virtual reality interface and the development of immersive virtual worlds for education. This paper descxibesthese lessons.

Welcome to the wireless world: problems using and understanding mobile telephony

Wireless telephony adoption is on the rise in many parts of the world (Cahners In-Stat Group, 2000), with recent technological innovations continuing to enhance the capabilities of the technology (McGinity, 1999). The market is extremely competitive. Service providers and wireless handset manufacturers, therefore, are faced with the challenge of designing wireless services and handsets to retain customers.

Usability and Learning in Educational Virtual Realities

Designing ScienceSpace, a series of virtual realities for teaching difficult science concepts and skills, has implications for designing sensorily immersive educational virtual realities. Through the design and evaluation of the worlds in ScienceSpace we are gaining insights into the general utility of sensorial immersion, as well as virtual realitys potential and limitations for enhancing learning. This paper focuses on the learner-centered design and evaluation of NewtonWorld, one of the virtual worlds in ScienceSpace. NewtonWorld is a sensorily immersive virtual learning environment in which students can challenge their intuitions about Newtons laws and the conservation of energy and momentum through game-like inquiry activities. We discuss how usability and learning issues have shaped the design and refinement of NewtonWorld. Additionally, we discuss implications of our work for designing sensorily immersive virtual reality interfaces that are usable and facilitate learning.

Assessing Virtual Reality's Potential for Teaching Abstract Science:

Understanding how to leverage the features of immersive, three-dimensional (3-D) multisensory virtual reality to meet user needs presents a challenge for human factors researchers. This paper describes our approach to evaluating this mediums potential as a tool for teaching abstract science. It describes some of our early research outcomes and discusses an evaluation comparing a 3-D VR microworld to an alternative 2-D computer-based microworld. Both are simulations in which students learn about electrostatics. The outcomes of the comparison study suggest: 1) the immersive 3-D VR microworld facilitated conceptual and three-dimensional learning that the 2-D computer microworld did not, and 2) VRs multisensory information aided students who found the electrostatics concepts challenging. As a whole, our research suggests that VRs immersive representational abilities have promise for teaching and for visualization. It also demonstrates that characteristics of the learning experience such as usability, motivation, and simulator sickness are important part of assessing this mediums potential.