Arthur E. Kirkpatrick

Testing pointing device performance and user assessment with the ISO 9241, Part 9 standard

The IS0 9241, Part 9 Draft International Standard for testingcomputer pointing devices proposes an evaluation of performance andcomfort. In this paper we evaluate the scientific validity andpracticality of these dimensions for two pointing devices forlaptop computers, a finger-controlled isometric joystick and atouchpad. Using a between-subjects design, evaluation ofperformance using the measure of throughput was done forone-direction and multi-directional pointing and selecting. Resultsshow a significant difference in throughput for themulti-directional task, with the joystick 27% higher; results forthe one-direction task were non-significant. After the experiment,participants rated the device for comfort, including operation,fatigue, and usability. The questionnaire showed no overalldifference in the responses, and a significant statisticaldifference in only the question concerning force required tooperate the device - the joystick requiring slightly more force.The paper concludes with a discussion of problems in implementingthe IS0 standard and recommendations for improvement.

Visualization task performance with 2D, 3D, and combination displays

We describe a series of experiments that compare 2D displays, 3D displays, and combined 2D/3D displays (orientation icon, ExoVis, and clip planes) for relative position estimation, orientation, and volume of interest tasks. Our results indicate that 3D displays can be very effective for approximate navigation and relative positioning when appropriate cues, such as shadows, are present. However, 3D displays are not effective for precise navigation and positioning except possibly in specific circumstances, for instance, when good viewing angles or measurement tools are available. For precise tasks in other situations, orientation icon and ExoVis displays were better than strict 2D or 3D displays (displays consisting exclusively of 2D or 3D views). The combined displays had as good or better performance, inspired higher confidence, and allowed natural, integrated navigation. Clip plane displays were not effective for 3D orientation because users could not easily view more than one 2D slice at a time and had to frequently change the visibility of individual slices. Major factors contributing to display preference and usability were task characteristics, orientation cues, occlusion, and spatial proximity of views that were used together.

Application-based evaluation of haptic interfaces

A taxonomy of haptic modes is proposed as a basis for evaluating haptic interfaces. Haptic modes are defined as distinct styles of using haptic perception for purposeful activity. An evaluation method that exercises a single haptic mode tests the hardware and software of the interface on a task that has a clear relationship to actual applications. To demonstrate this approach to evaluation, the mode of geometric perception was evaluated for a point force device. Twelve observers performed a shape recognition task using a PHANToM display. The task proved difficult, with a mean recognition time of 22 s. This class of devices does not appear to support adequate performance for applications that require geometric perception. The paper concludes with possible extensions to both the shape recognition protocol and the taxonomy of modes.

Combining 2D and 3D views for orientation and relative position tasks

We compare 2D/3D combination displays to displays with 2D and 3D views alone. Combination displays we consider are: orientation icon (i.e., side-by-side), in-place methods (e.g., clip planes), and a new method called ExoVis. We specifically analyze performance differences (i.e., time and accuracy) for 3D orientation and relative position tasks. Empirical results show that 3D displays are effective for approximate navigation and relative positioning whereas 2D/3D combination displays (orientation icon and ExoVis) are useful for precise orientation and position tasks. Combination 2D/3D displays had as good or better performance as 2D displays. Clip planes were not effective for a 3D orientation task, but may be useful when only one slice is needed.

Model and representation: the effect of visual feedback on human performance in a color picker interface

User interfaces for color selection consist of a visible screen representation, an input method, and the underlying conceptual organization of the color model. We report a two-way factorial, between-subjects variable experiment that tested the effect of high and low visual feedback interfaces on speed and accuracy of color matching for RGB and HSV color models. The only significant effect was improved accuracy due to increased visual feedback. Using color groups as a within-subjects variable, we found differences in performance of both speed and accuracy. We recommend that experimental tests adopt a color test set that does not show bias toward a particular model, but is based instead on a range of colors that would be most likely matched in practice by people using color selection software. We recomment the Macbeth Color Checker naturals, primaries, and grays. As a follow-up study, a qualitative case analysis of the way users navigated through the color space indicates that feedback helps users with limited knowledge of the model, allowing them to refine their match to a higher degree of accuracy. Users with very little or a lot of knowledge of the color model do not appear to be aided by increased feedback. In conclusion, we suggest that visual feedback and design of the interface may be a more important factor in improving the usability of a color selection interface than the particular color model used.

On visual quality of optimal 3D sampling and reconstruction

This paper presents a user study of the visual quality of an imaging pipeline employing the optimal body-centered cubic (BCC) sampling lattice. We provide perceptual evidence supporting the theoretical expectation that sampling and reconstruction on the BCC lattice offer superior imaging quality over the traditionally popular Cartesian cubic (CC) sampling lattice. We asked 12 participants to choose the better of two images: one image rendered from data sampled on the CC lattice and one image that is rendered from data sampled on the BCC lattice. We used both synthetic and CT volumetric data, and confirm that the theoretical advantages of BCC sampling carry over to the perceived quality of rendered images. Using 25% to 35% fewer samples, BCC sampled data result in images that exhibit comparable visual quality to their CC counterparts.

A mixing board interface for graphics and visualization applications

We use a haptically enhanced mixing board with a video projector as an interface to various data visualization tasks. We report results of an expert review with four participants, qualitatively evaluating the board for three different applications: dynamic queries (abstract task), parallel coordinates interface (multi-dimensional combinatorial search), and ExoVis (3D spatial navigation). Our investigation sought to determine the strengths of this physical input given its capability to facilitate bimanual interaction, constraint maintenance, tight coupling of input and output, and other features. Participants generally had little difficulty with the mappings of parameters to sliders. The graspable sliders apparently reduced the mental exertion needed to acquire control, allowing participants to attend more directly to understanding the visualization. Participants often designated specific roles for each hand, but only rarely moved both hands simultaneously.

A new approach to haptic augmentation of the GUI

Most users do not experience the same level of fluency in their interactions with computers that they do with physical objects in their daily life. We believe that much of this results from the limitations of unimodal interaction. Previous efforts in the haptics literature to remedy those limitations have been creative and numerous, but have failed to produce substantial improvements in human performance. This paper presents a new approach, whereby haptic interaction techniques are designed from scratch, in explicit consideration of the strengths and weaknesses of the haptic and motor systems. We introduce a haptic alternative to the tool palette, called Pokespace, which follows this approach. Two studies (6 and 12 participants) conducted with Pokespace found no performance improvement over a traditional interface, but showed that participants learned to use the interface proficiently after about 10 minutes, and could do so without visual attention. The studies also suggested several improvements to our design.

A Shape Recognition Benchmark for Evaluating Usability of a Haptic Environment

This paper describes a benchmark task for evaluating the usability of haptic environments for a shape perception task. The task measures the ease with which observers can recognize members of a standard set of five shapes defined by Koenderink. Median time for 12 participants to recognize these shapes with the PHANToM was 23 seconds. This recognition time is within the range for shape recognition of physical objects using one finger but far slower than recognition using the whole hand. The results suggest haptic environments must provide multiple points of contact with an object for rapid performance of shape recognition.

Visual Comparability of 3D Regular Sampling and Reconstruction

The Body-Centered Cubic (BCC) and Face-Centered Cubic (FCC) lattices have been analytically shown to be more efficient sampling lattices than the traditional Cartesian Cubic (CC) lattice, but there has been no estimate of their visual comparability. Two perceptual studies (each with N = 12 participants) compared the visual quality of images rendered from BCC and FCC lattices to images rendered from the CC lattice. Images were generated from two signals: the commonly used Marschner-Lobb synthetic function and a computed tomography scan of a fish tail. Observers found that BCC and FCC could produce images of comparable visual quality to CC, using 30-35 percent fewer samples. For the images used in our studies, the L2 error metric shows high correlation with the judgement of human observers. Using the L2 metric as a proxy, the results of the experiments appear to extend across a wide range of images and parameter choices.