Theresa Cooke

Multimodal Similarity and Categorization of Novel, Three-Dimensional Objects

Similarity has been proposed as a fundamental principle underlying mental object representations and capable of supporting cognitive-level tasks such as categorization. However, much of the research has considered connections between similarity and categorization for tasks performed using a single perceptual modality. Considering similarity and categorization within a multimodal context opens up a number of important questions: Are the similarities between objects the same when they are perceived using different modalities or using more than one modality at a time? Is similarity still able to explain categorization performance when objects are experienced multimodally? In this study, we addressed these questions by having subjects explore novel, 3D objects which varied parametrically in shape and texture using vision alone, touch alone, or touch and vision together. Subjects then performed a pair-wise similarity rating task and a free sorting categorization task. Multidimensional scaling (MDS) analysis of similarity data revealed that a single underlying perceptual map whose dimensions corresponded to shape and texture could explain visual, haptic, and bimodal similarity ratings. However, the relative dimension weights varied according to modality: shape dominated texture when objects were seen, whereas shape and texture were roughly equally important in the haptic and bimodal conditions. Some evidence was found for a multimodal connection between similarity and categorization: the probability of category membership increased with similarity while the probability of a category boundary being placed between two stimuli decreased with similarity. In addition, dimension weights varied according to modality in the same way for both tasks. The study also demonstrates the usefulness of 3D printing technology and MDS techniques in the study of visuohaptic object processing.

Object feature validation using visual and haptic similarity ratings

The perceived similarity between objects may well vary according to the sensory modality/modalities in which they are experienced, an important consideration for the design of multimodal interfaces. In this study, we present a similarity-based method for comparing the perceptual importance of object properties in touch and in vision and show how the method can also be used to validate computational measures of object properties. Using either vision or touch, human subjects judged the similarity between novel, 3D objects which varied parametrically in shape and texture. Similarities were also computed using a set of state-of-the art 2D and 3D computational measures. Two resolutions of 2D and 3D object data were used for these computations in order to test for scale dependencies. Multidimensional scaling (MDS) was then performed on all similarity data, yielding maps of the stimuli in both perceptual and computational spaces, as well as the relative weight of shape and texture dimensions. For this object set, we found that visual subjects accorded more importance to shape than texture, while haptic subjects weighted them roughly evenly. Fit errors between human and computational maps were then calculated to assess each features perceptual validity. Shape-biased features provided good overall fits to the human visual data; however, no single feature yielded a good overall fit to the haptic data, in which we observed large individual differences. This work demonstrates how MDS techniques can be used to evaluate computational object features using the criterion of perceptual similarity. It also demonstrates a way of assessing how the perceptual validity of a feature varies as a function of parameters such as the target modality and the resolution of object data. Potential applications of this method for the design of unimodal and multimodal human---machine interfaces are discussed.

Multidimensional scaling analysis of haptic exploratory procedures

Previous work in real and virtual settings has shown that the way in which we interact with objects plays a fundamental role in the way we perceive them. This article uses multidimensional scaling (MDS) analysis to further characterize and quantify the effects of using different haptic exploratory procedures (EPs) on perceptual similarity spaces. In Experiment 1, 20 participants rated similarity on a set of nine novel, 3D objects varying in shape and texture after either following their contours, laterally rubbing their centers, gripping them, or sequentially touching their tips. MDS analysis was used to recover perceptual maps of the objects and relative weights of perceptual dimensions from similarity data. Both the maps and relative weights of shape/texture properties were found to vary as a function of the EP used. In addition, large individual differences in the relative weight of shape/texture were observed. In Experiment 2, 17 of the previous participants repeated Experiment 1 after an average of 105 days. The same patterns of raw similarity ratings, perceptual maps, dimension weights, and individual differences were observed, indicating that perceptual similarities remained stable over time. The findings underscore the role of hand movements and individual biases in shaping haptic perceptual similarity. A framework for validating multimodal virtual displays based on the approach used in the study is also presented.

The Perceptual Influence of Spatiotemporal Noise on the Reconstruction of Shape from Dynamic Occlusion

When an object moves, it covers and uncovers texture in the background. This pattern of change is sufficient to define the objects shape, velocity, relative depth, and degree of transparency, a process called Spatiotemporal Boundary Formation (SBF). We recently proposed a mathematical framework for SBF, where texture transformations are used to recover local edge segments, estimate the figures velocity and then reconstruct its shape. The model predicts that SBF should be sensitive to spatiotemporal noise, since the spurious transformations will lead to the recovery of incorrect edge orientations. Here we tested this prediction by adding a patch of dynamic noise (either directly over the figure or a fixed distance away from it). Shape recognition performance in humans decreased to chance levels when noise was placed over the figure but was not affected by noise far away. These results confirm the models prediction and also imply that SBF is a local process.

Design and Evaluation of Haptic Soft Tissue Interaction

This chapter examines the application of a psychophysical evaluation technique to quantify the fidelity of haptic rendering methods. The technique is based on multidimensional scaling analysis of similarity ratings provided by users comparing pairs of haptically-presented objects. Unbeknownst to the participants, both real and virtual deformable objects were presented. In addition, virtual objects were either rendered under high fidelity condition or under lower-fidelity condition in which filtering quality was reduced. The analysis of pairwise similarity data provides quantitative confirmation that users perceived a clear difference between real and virtual objects in the lower-fidelity, but not in the higher-fidelity condition. In the latter, a single perceptual dimension, corresponding to stiffness, sufficed to explain similarity data, while two perceptual dimensions were needed in the former condition. This demonstrates how multidimensional scaling analysis can be used in the evaluation of haptic rendering scenarios, providing perceptual maps of real and virtual objects. It offers an opportunity to visualize and quantify the perceptual effects of changes in rendering parameters.

Effects of experience and task type on unsupervised categorization of novel, 3D objects

Background: The Importance of Shape for Visual Categorization Rosch et al. (1976) and others have argued that the shape of objects is a fundamental determinant of category structure. In a previous study [2], we observed that after ten hours of visual exposure to a series of novel, 3D objects, subjects asked to perform free categorization in a sequential presentation task did so primarily on the basis of shape differences as opposed to texture or a combination of both. In contrast, no such preference was found after ten hours of haptic exposure.