Andreas Veit

Learning from Noisy Large-Scale Datasets with Minimal Supervision

We present an approach to effectively use millions of images with noisy annotations in conjunction with a small subset of cleanly-annotated images to learn powerful image representations. One common approach to combine clean and noisy data is to first pre-train a network using the large noisy dataset and then fine-tune with the clean dataset. We show this approach does not fully leverage the information contained in the clean set. Thus, we demonstrate how to use the clean annotations to reduce the noise in the large dataset before fine-tuning the network using both the clean set and the full set with reduced noise. The approach comprises a multi-task network that jointly learns to clean noisy annotations and to accurately classify images. We evaluate our approach on the recently released Open Images dataset, containing ~9 million images, multiple annotations per image and over 6000 unique classes. For the small clean set of annotations we use a quarter of the validation set with ~40k images. Our results demonstrate that the proposed approach clearly outperforms direct fine-tuning across all major categories of classes in the Open Image dataset. Further, our approach is particularly effective for a large number of classes with wide range of noise in annotations (20-80% false positive annotations).

Conditional Similarity Networks

What makes images similar? To measure the similarity between images, they are typically embedded in a feature-vector space, in which their distance preserve the relative dissimilarity. However, when learning such similarity embeddings the simplifying assumption is commonly made that images are only compared to one unique measure of similarity. A main reason for this is that contradicting notions of similarities cannot be captured in a single space. To address this shortcoming, we propose Conditional Similarity Networks (CSNs) that learn embeddings differentiated into semantically distinct subspaces that capture the different notions of similarities. CSNs jointly learn a disentangled embedding where features for different similarities are encoded in separate dimensions as well as masks that select and reweight relevant dimensions to induce a subspace that encodes a specific similarity notion. We show that our approach learns interpretable image representations with visually relevant semantic subspaces. Further, when evaluating on triplet questions from multiple similarity notions our model even outperforms the accuracy obtained by training individual specialized networks for each notion separately.

Crowd Research: Open and Scalable University Laboratories

Research experiences today are limited to a privileged few at select universities. Providing open access to research experiences would enable global upward mobility and increased diversity in the scientific workforce. How can we coordinate a crowd of diverse volunteers on open-ended research? How could a PI have enough visibility into each persons contributions to recommend them for further study? We present Crowd Research, a crowdsourcing technique that coordinates open-ended research through an iterative cycle of open contribution, synchronous collaboration, and peer assessment. To aid upward mobility and recognize contributions in publications, we introduce a decentralized credit system: participants allocate credits to each other, which a graph centrality algorithm translates into a collectively-created author order. Over 1,500 people from 62 countries have participated, 74% from institutions with low access to research. Over two years and three projects, this crowd has produced articles at top-tier Computer Science venues, and participants have gone on to leading graduate programs.

Learning to Detect and Match Keypoints with Deep Architectures.

Feature detection and description is a pivotal step in many computer vision pipelines. Traditionally, human engineered features have been the main workhorse in this domain. In this paper, we present a novel approach for learning to detect and describe keypoints from images leveraging deep architectures. To allow for a learning based approach, we collect a large-scale dataset of patches with matching multiscale keypoints. The proposed model learns from this vast dataset to identify and describe meaningful keypoints. We evaluate our model for the effectiveness of its learned representations for detecting multiscale keypoints and describing their respective support regions.

Multi-agent device-level modeling framework for demand scheduling

A problem receiving increasing attention is autonomously adjusting the electricity demand of consumers in response to real-time supply. The current literature mostly operates under the assumption that it is desirable to have an autonomous energy management system. However, autonomous demand-side management requires that smart agents understand their decision space. To model the decision space, it is key to model the scheduling constraints of individual devices under the agents control. Some constraints can be set by the owners, e.g., deadlines, while others are physical constraints. Recent work in demand-side management is based on a wide variety of models. In this work, we develop a standardized, device-level modeling framework characterizing the device categories by their constraints. Our models enable researchers and practitioners to develop and compare demand-side management algorithms and programs. Further, we evaluate the scheduling complexity of the device categories and effects of different device combinations on the complexity. Our empirical results suggest that mixing different device categories can improve scheduling time.

MDSM: Generalized multiagent coordination for demand side management

One key challenge in creating a sustainable society is to make consumer demand adaptive to the supply of electricity. We propose a generalization for the multiagent coordination algorithm for partially-centralized demand side management. In this setting, a central unit buys the electricity for the whole group, while the individual agents make their own demand decisions, based on their private constraints and preferences. While the state-of-the-art algorithm achieves strong guarantees, i.e., efficiency, strict budget balance as well as weak incentive compatibility, the formulation requires a simple supply model with a 2-step increasing threshold price function. In this paper, we propose MDSM, a generalized formulation of the multiagent coordination algorithm and prove that it converges to the optimal solution for all convex piecewise linear cost functions. Further, we perform simulations based on real world consumption data to evaluate the scalability of the algorithm in respect to the number of thresholds. The results indicate that the convergence time of MDSM is not sensitive to the number of thresholds in the cost function. This means, the proposed algorithm could be used for a broader set of distributed scheduling problems.

Convolutional Networks with Adaptive Inference Graphs

Do convolutional networks really need a fixed feed-forward structure? What if, after identifying the high-level concept of an image, a network could move directly to a layer that can distinguish fine-grained differences? Currently, a network would first need to execute sometimes hundreds of intermediate layers that specialize in unrelated aspects. Ideally, the more a network already knows about an image, the better it should be at deciding which layer to compute next. In this work, we propose convolutional networks with adaptive inference graphs (ConvNet-AIG) that adaptively define their network topology conditioned on the input image. Following a high-level structure similar to residual networks (ResNets), ConvNet-AIG decides for each input image on the fly which layers are needed. In experiments on ImageNet we show that ConvNet-AIG learns distinct inference graphs for different categories. Both ConvNet-AIG with 50 and 101 layers outperform their ResNet counterpart, while using \(20\%\) and \(33\%\) less computations respectively. By grouping parameters into layers for related classes and only executing relevant layers, ConvNet-AIG improves both efficiency and overall classification quality. Lastly, we also study the effect of adaptive inference graphs on the susceptibility towards adversarial examples. We observe that ConvNet-AIG shows a higher robustness than ResNets, complementing other known defense mechanisms.