Regularizing Deep Networks with Semantic Data Augmentation

Abstract

Data augmentation is widely known as a simple yet surprisingly effective technique for regularizing deep networks. In this paper, we propose a novel semantic data augmentation algorithm to complement traditional schemes, such as flipping, translation and rotation. The proposed method is inspired by the intriguing property that deep networks are effective in learning linearized features, such that certain directions in the deep feature space correspond to meaningful semantic transformations. Consequently, translating training samples along many such directions in the feature space can effectively augment the dataset in a semantic manner. The proposed implicit semantic data augmentation (ISDA) first obtains semantically meaningful translations using an efficient sampling based method. Then, an upper bound of the expected cross-entropy (CE) loss on the augmented training set is derived, leading to a novel robust loss function. In addition, we show that ISDA can be applied to semi-supervised learning under the consistency regularization framework, where ISDA minimizes the upper bound of the expected KL-divergence between the predictions of augmented samples and original samples. Although being simple, ISDA consistently improves the generalization performance of popular deep models (ResNets and DenseNets) on a variety of datasets, e.g., CIFAR-10, CIFAR-100, ImageNet and Cityscapes.