Chen-Ping Yu

Modeling visual clutter perception using proto-object segmentation.

We introduce the proto-object model of visual clutter perception. This unsupervised model segments an image into superpixels, then merges neighboring superpixels that share a common color cluster to obtain proto-objects-defined here as spatially extended regions of coherent features. Clutter is estimated by simply counting the number of proto-objects. We tested this model using 90 images of realistic scenes that were ranked by observers from least to most cluttered. Comparing this behaviorally obtained ranking to a ranking based on the model clutter estimates, we found a significant correlation between the two (Spearmans ρ = 0.814, p < 0.001). We also found that the proto-object model was highly robust to changes in its parameters and was generalizable to unseen images. We compared the proto-object model to six other models of clutter perception and demonstrated that it outperformed each, in some cases dramatically. Importantly, we also showed that the proto-object model was a better predictor of clutter perception than an actual count of the number of objects in the scenes, suggesting that the set size of a scene may be better described by proto-objects than objects. We conclude that the success of the proto-object model is due in part to its use of an intermediate level of visual representation-one between features and objects-and that this is evidence for the potential importance of a proto-object representation in many common visual percepts and tasks.

Large-Scale Training of Shadow Detectors with Noisily-Annotated Shadow Examples

This paper introduces training of shadow detectors under the large-scale dataset paradigm. This was previously impossible due to the high cost of precise shadow annotation. Instead, we advocate the use of quickly but imperfectly labeled images. Our novel label recovery method automatically corrects a portion of the erroneous annotations such that the trained classifiers perform at state-of-the-art level. We apply our method to improve the accuracy of the labels of a new dataset that is 20 times larger than existing datasets and contains a large variety of scenes and image types. Naturally, such a large dataset is appropriate for training deep learning methods. Thus, we propose a semantic-aware patch level Convolutional Neural Network architecture that efficiently trains on patch level shadow examples while incorporating image level semantic information. This means that the detected shadow patches are refined based on image semantics. Our proposed pipeline can be a useful baseline for future advances in shadow detection.

Single Image Shadow Detection Using Multiple Cues in a Supermodular MRF.

We propose a single region shadow classifier based on a multikernel SVM. Our multikernel model is a linear combination of χ2 and Earth Mover’s Distance(EMD)[5] kernels that operate on texture and color histograms disjointly. This single region classifier already outperforms the more complex state of art methods, without performing MRF/CRF optimization. The local appearance of a single region is often ambiguous. Even for a human observer it can be hard to discern if a region is in shadow or not, without considering its context. Hence, it is sensible to look beyond the boundaries of a single region to decide its shadow label [1] [6]. In contrast to previous work we strive to use such contextual information sparingly. For MRF optimization reasons we prefer that most of the work is handled by the single region classifier (unary MRF potentials), with sparse pairwise connections that smooth the label changes across regions. We build on the work of [1] to propose our own improved pairwise classifiers but constrained to adjacent regions: for pairs of regions sharing the same material and same illumination condition, and for same material pairs viewed under different illumination (first lit, second in shadow). We also propose a shadow boundary classifier. Since shadow boundaries often overlap with reflectance changes confounding the effects of the illumination change, our classifier focuses on boundaries of shadows cast over surfaces with the same underlying material. We integrate our single region classifier, our pairwise classifiers, and our boundary classifier using an MRF. Confident positive predictions of the pairwise and boundary classifiers are used to define the pairwise potentials and the graph topology of the MRF. The unary potentials are defined based on the single region classifier. We want to minimize the following functional:

Efficient Video Segmentation Using Parametric Graph Partitioning

Video segmentation is the task of grouping similar pixels in the spatio-temporal domain, and has become an important preprocessing step for subsequent video analysis. Most video segmentation and supervoxel methods output a hierarchy of segmentations, but while this provides useful multiscale information, it also adds difficulty in selecting the appropriate level for a task. In this work, we propose an efficient and robust video segmentation framework based on parametric graph partitioning (PGP), a fast, almost parameter free graph partitioning method that identifies and removes between-cluster edges to form node clusters. Apart from its computational efficiency, PGP performs clustering of the spatio-temporal volume without requiring a pre-specified cluster number or bandwidth parameters, thus making video segmentation more practical to use in applications. The PGP framework also allows processing sub-volumes, which further improves performance, contrary to other streaming video segmentation methods where sub-volume processing reduces performance. We evaluate the PGP method using the SegTrack v2 and Chen Xiph.org datasets, and show that it outperforms related state-of-the-art algorithms in 3D segmentation metrics and running time.

Searching for Category-Consistent Features A Computational Approach to Understanding Visual Category Representation

This article introduces a generative model of category representation that uses computer vision methods to extract category-consistent features (CCFs) directly from images of category exemplars. The model was trained on 4,800 images of common objects, and CCFs were obtained for 68 categories spanning subordinate, basic, and superordinate levels in a category hierarchy. When participants searched for these same categories, targets cued at the subordinate level were preferentially fixated, but fixated targets were verified faster when they followed a basic-level cue. The subordinate-level advantage in guidance is explained by the number of target-category CCFs, a measure of category specificity that decreases with movement up the category hierarchy. The basic-level advantage in verification is explained by multiplying the number of CCFs by sibling distance, a measure of category distinctiveness. With this model, the visual representations of real-world object categories, each learned from the vast numbers of image exemplars accumulated throughout everyday experience, can finally be studied.

A mid-level organization of the ventral stream

Human object-selective cortex shows a large-scale organization characterized by the high-level properties of both animacy and object-size. To what extent are these neural responses explained by primitive perceptual features that distinguish animals from objects and big objects from small objects? To address this question, we used a texture synthesis algorithm to create a novel class of stimuli—texforms—which preserve some mid-level texture and form information from objects while rendering them unrecognizable. We found that unrecognizable texforms were sufficient to elicit the large-scale organizations of object-selective cortex along the entire ventral pathway. Further, the structure in the neural patterns elicited by texforms was well predicted by curvature features and by intermediate layers of a deep convolutional neural network, supporting the mid-level nature of the representations. These results provide clear evidence that a substantial portion of ventral stream organization can be accounted for by coarse texture and form information, without requiring explicit recognition of intact objects. SIGNIFICANCE STATEMENT While neural responses to object categories are remarkably systematic across human visual cortex, the nature of these responses been hotly debated for the past 20 years. In this paper, a new class of stimuli (“texforms”) is used to examine how mid-level features contribute to the large-scale organization of the ventral visual stream. Despite their relatively primitive visual appearance, these unrecognizable texforms elicited the entire large-scale organizations of the ventral stream by animacy and object size. This work demonstrates that much of ventral stream organization can be explained by relatively primitive mid-level features, without requiring explicit recognition of the objects themselves.

Clutter perception is invariant to image size

Two experiments evaluated the effect of retinal image size on the proto-object model of visual clutter perception. Experiment 1 had 20 participants order 90 small images of random-category real-world scenes from least to most cluttered. Aggregating these individual rankings into a single median clutter ranking and comparing it to a previously reported clutter ranking of larger versions of the identical scenes yielded a Spearmans ρ=.953 (p<.001), suggesting that relative clutter perception is largely invariant to image size. We then applied the proto-object model of clutter perception to these smaller images and obtained a clutter estimate for each. Correlating these estimates with the median behavioral ranking yielded a Spearmans ρ=.852 (p<.001), which we showed in a comparative analysis to be better than six other methods of estimating clutter. Experiment 2 intermixed large and small versions of the Experiment 1 scenes and had participants (n=18) again rank them for clutter. We found that median clutter rankings of these size-intermixed images were essentially the same as the small and large median rankings from Experiment 1, suggesting size invariance in absolute clutter perception. Moreover, the proto-object model again successfully captured this result. We conclude that both relative and absolute clutter perception is invariant to retinal image size. We further speculate that clutter perception is mediated by proto-objects-a preattentive level of visual representation between features and objects-and that using the proto-object model we may be able to glimpse into this pre-attentive world.

Mid-level visual features underlie the high-level categorical organization of the ventral stream

Significance While neural responses to object categories are remarkably systematic across human visual cortex, the nature of these responses has been hotly debated for the past 20 y. In this paper, a class of stimuli (texforms) is used to examine how mid-level features contribute to the large-scale organization of the ventral visual stream. Despite their relatively primitive visual appearance, these unrecognizable texforms elicited the entire large-scale organizations of the ventral stream by animacy and object size. This work demonstrates that much of ventral stream organization can be explained by relatively primitive mid-level features without requiring explicit recognition of the objects themselves. Human object-selective cortex shows a large-scale organization characterized by the high-level properties of both animacy and object size. To what extent are these neural responses explained by primitive perceptual features that distinguish animals from objects and big objects from small objects? To address this question, we used a texture synthesis algorithm to create a class of stimuli—texforms—which preserve some mid-level texture and form information from objects while rendering them unrecognizable. We found that unrecognizable texforms were sufficient to elicit the large-scale organizations of object-selective cortex along the entire ventral pathway. Further, the structure in the neural patterns elicited by texforms was well predicted by curvature features and by intermediate layers of a deep convolutional neural network, supporting the mid-level nature of the representations. These results provide clear evidence that a substantial portion of ventral stream organization can be accounted for by coarse texture and form information without requiring explicit recognition of intact objects.

Modeling categorical search guidance using a convolutional neural network designed after the ventral visual pathway

• Most of our everyday searches are for categories of things, and a growing body of evidence now exists that attention is guided to target object categories in the context of a visual search task (e.g., [1,2]). But computational models of this categorical guidance of attention are still in their infancy. In previous work we showed that a simple generative model was able to predict this guidance by learning category-consistent features (CCFs)—those features that occur both frequently and consistently across the exemplars of an object category [3]. However, this model’s prediction was limited to a single general relationship; more time is needed to first fixate a target as this target climbs levels in a subordinate-basic-superordinate category hierarchy.

Geodesic Distance Histogram Feature for Video Segmentation

This paper proposes a geodesic-distance-based feature that encodes global information for improved video segmentation algorithms. The feature is a joint histogram of intensity and geodesic distances, where the geodesic distances are computed as the shortest paths between superpixels via their boundaries. We also incorporate adaptive voting weights and spatial pyramid configurations to include spatial information into the geodesic histogram feature and show that this further improves results. The feature is generic and can be used as part of various algorithms. In experiments, we test the geodesic histogram feature by incorporating it into two existing video segmentation frameworks. This leads to significantly better performance in 3D video segmentation benchmarks on two datasets.