Ruth Rosenholtz

Halo: a technique for visualizing off-screen objects

As users pan and zoom, display content can disappear into off-screen space, particularly on small-screen devices. The clipping of locations, such as relevant places on a map, can make spatial cognition tasks harder. Halo is a visualization technique that supports spatial cognition by showing users the location of off-screen objects. Halo accomplishes this by surrounding off-screen objects with rings that are just large enough to reach into the border region of the display window. From the portion of the ring that is visible on-screen, users can infer the off-screen location of the object at the center of the ring. We report the results of a user study comparing Halo with an arrow-based visualization technique with respect to four types of map-based route planning tasks. When using the Halo interface, users completed tasks 16-33% faster, while there were no significant differences in error rate for three out of four tasks in our study.

Using thumbnails to search the Web

We introduce a technique for creating novel, textually-enhanced thumbnails of Web pages. These thumbnails combine the advantages of image thumbnails and text summaries to provide consistent performance on a variety of tasks. We conducted a study in which participants used three different types of summaries (enhanced thumbnails, plain thumbnails, and text summaries) to search Web pages to find several different types of information. Participants took an average of 67, 86, and 95 seconds to find the answer with enhanced thumbnails, plain thumbnails, and text summaries, respectively. We found a strong effect of question category. For some questions, text outperformed plain thumbnails, while for other questions, plain thumbnails outperformed text. Enhanced thumbnails (which combine the features of text summaries and plain thumbnails) were more consistent than either text summaries or plain thumbnails, having for all categories the best performance or performance that was statistically indistinguishable from the best.

A simple saliency model predicts a number of motion popout phenomena

Visual search for a moving target among stationary distractors is more efficient than searching for a stationary target among moving distractors, and searching for a fast target among slow distractors is more efficient than vice versa. This indicates that the ease of search for a target with a particular motion is not determined simply by the difference between target and distractor velocities. We suggest a simple model for predicting ease of search for a unique motion, based upon a quantitative measure of target saliency. Essentially, search will be easier the more the target motion deviates from the general pattern of velocities in the scene. Our model predicts a number of well-known motion search phenomena, and suggests that one control for target saliency as well as target discriminability when drawing conclusions about visual system mechanisms from search experiments.

Computing Local Surface Orientation and Shape from Texture forCurved Surfaces

Shape from texture is best analyzed in two stages, analogous to stereopsis and structure from motion: (a) Computing the ‘texture distortion’ from the image, and (b) Interpreting the ‘texture distortion’ to infer the orientation and shape of the surface in the scene. We model the texture distortion for a given point and direction on the image plane as an affine transformation and derive the relationship between the parameters of this transformation and the shape parameters. We have developed a technique for estimating affine transforms between nearby image patches which is based on solving a system of linear constraints derived from a differential analysis. One need not explicitly identify texels or make restrictive assumptions about the nature of the texture such as isotropy. We use non-linear minimization of a least squares error criterion to recover the surface orientation (slant and tilt) and shape (principal curvatures and directions) based on the estimated affine transforms in a number of different directions. A simple linear algorithm based on singular value decomposition of the linear parts of the affine transforms provides the initial guess for the minimization procedure. Experimental results on both planar and curved surfaces under perspective projection demonstrate good estimates for both orientation and shape. A sensitivity analysis yields predictions for both computer vision algorithms and human perception of shape from texture.

Exploring features in a Bayesian framework for material recognition

We are interested in identifying the material category, e.g. glass, metal, fabric, plastic or wood, from a single image of a surface. Unlike other visual recognition tasks in computer vision, it is difficult to find good, reliable features that can tell material categories apart. Our strategy is to use a rich set of low and mid-level features that capture various aspects of material appearance. We propose an augmented Latent Dirichlet Allocation (aLDA) model to combine these features under a Bayesian generative framework and learn an optimal combination of features. Experimental results show that our system performs material recognition reasonably well on a challenging material database, outperforming state-of-the-art material/texture recognition systems.

Search asymmetries? what search asymmetries?

In order to establish a search asymmetry, one must run an experiment with a symmetric design and get asymmetric results. Given an asymmetric design, one expects asymmetric results, and such results do not imply an asymmetry in the search mechanisms. In this paper, I argue that a number of experiments purporting to show search asymmetries contain built-in design asymmetries. A saliency model of visual search predicts the results of these experiments, using only a simple measure of target-distractor similarity, without reliance on asymmetric search mechanisms. These results have implications for search mechanisms and for other experiments purporting to show search asymmetries.

Recognizing Materials Using Perceptually Inspired Features

Our world consists not only of objects and scenes but also of materials of various kinds. Being able to recognize the materials that surround us (e.g., plastic, glass, concrete) is important for humans as well as for computer vision systems. Unfortunately, materials have received little attention in the visual recognition literature, and very few computer vision systems have been designed specifically to recognize materials. In this paper, we present a system for recognizing material categories from single images. We propose a set of low and mid-level image features that are based on studies of human material recognition, and we combine these features using an SVM classifier. Our system outperforms a state-of-the-art system (Varma and Zisserman, TPAMI 31(11):2032–2047, 2009) on a challenging database of real-world material categories (Sharan et al., J Vis 9(8):784–784a, 2009). When the performance of our system is compared directly to that of human observers, humans outperform our system quite easily. However, when we account for the local nature of our image features and the surface properties they measure (e.g., color, texture, local shape), our system rivals human performance. We suggest that future progress in material recognition will come from: (1) a deeper understanding of the role of non-local surface properties (e.g., extended highlights, object identity); and (2) efforts to model such non-local surface properties in images.

Perceptual adaptive JPEG coding

An extension to the JPEG standard (ISO/IEC DIS 10918-3) allows spatial adaptive coding of still images. As with baseline JPEG coding, one quantization matrix applies to an entire image channel, but in addition the user may specify a multiplier for each 8/spl times/8 block, which multiplies the quantization matrix, yielding the new matrix for that block. MPEG 1 and 2 use much the same scheme, except there the multiplier changes only on macroblock boundaries. We propose a method for perceptual optimization of the set of multipliers. We compute the perceptual error for each block based upon the DCT quantization error adjusted according to the contrast sensitivity, light adaptation, and contrast masking, and pick the set of multipliers which yield maximally flat perceptual error over the blocks of the image. We investigate the bit rate savings due to this adaptive coding scheme and the relative importance of the different sorts of masking on adaptive coding.

A comparison of the use of text summaries, plain thumbnails, and enhanced thumbnails for Web search tasks

We introduce a technique for creating novel, enhanced thumbnails of Web pages. These thumbnails combine the advantages of plain thumbnails and text summaries to provide consistent performance on a variety of tasks. We conducted a study in which participants used three different types of summaries (enhanced thumbnails, plain thumbnails, and text summaries) to search Web pages to find several different types of information. Participants took an average of 67, 86, and 95 seconds to find the answer with enhanced thumbnails, plain thumbnails, and text summaries, respectively. As expected, there was a strong effect of question category. For some questions, text summaries outperformed plain thumbnails, while for other questions, plain thumbnails outperformed text summaries. Enhanced thumbnails (which combine the features of text summaries and plain thumbnails) had more consistent performance than either text summaries or plain thumbnails, having for all categories the best performance or performance that was statistically indistinguishable from the best.

Surface orientation from texture: Isotropy or homogeneity (or both)?

We examine two models for human perception of shape from texture, based on two assumptions about the surface texture: isotropy and homogeneity. Observers made orientation judgments on planar textured surfaces. Surface textures were either isotropic or anisotropically stretched or compressed. If subjects used an isotropy assumption, they would make biased orientation estimates for the anisotropic textures. In some conditions some observers showed no bias for the anisotropic textures relative to the isotropic textures. In general, even when the observers showed bias, the biases were significantly less than those predicted if the observer used only deviation from isotropy as a cue. Observers appear to use both the deviation from isotropy and a texture gradient or affine texture distortion cue for shape from texture.