Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, Antoine Bordes
aa r X i v : . [ c s . C L ] J u l Supervised Learning of Universal Sentence Representations fromNatural Language Inference Data
Alexis Conneau
Facebook AI Research [email protected]
Douwe Kiela
Facebook AI Research [email protected]
Holger Schwenk
Facebook AI Research [email protected]
Lo¨ıc Barrault
LIUM, Universit´e Le Mans [email protected]
Antoine Bordes
Facebook AI Research [email protected]
Abstract
Many modern NLP systems rely on wordembeddings, previously trained in an un-supervised manner on large corpora, asbase features. Efforts to obtain embed-dings for larger chunks of text, such assentences, have however not been so suc-cessful. Several attempts at learning unsu-pervised representations of sentences havenot reached satisfactory enough perfor-mance to be widely adopted. In this paper,we show how universal sentence represen-tations trained using the supervised data ofthe Stanford Natural Language Inferencedatasets can consistently outperform unsu-pervised methods like SkipThought vec-tors (Kiros et al., 2015) on a wide rangeof transfer tasks. Much like how com-puter vision uses ImageNet to obtain fea-tures, which can then be transferred toother tasks, our work tends to indicate thesuitability of natural language inferencefor transfer learning to other NLP tasks.Our encoder is publicly available . Distributed representations of words (or wordembeddings) (Bengio et al., 2003; Collobert et al.,2011; Mikolov et al., 2013; Pennington et al.,2014) have shown to provide useful features forvarious tasks in natural language processing andcomputer vision. While there seems to be a con-sensus concerning the usefulness of word embed-dings and how to learn them, this is not yet clearwith regard to representations that carry the mean-ing of a full sentence. That is, how to capture therelationships among multiple words and phrases ina single vector remains an question to be solved. In this paper, we study the task of learninguniversal representations of sentences, i.e., a sen-tence encoder model that is trained on a large cor-pus and subsequently transferred to other tasks.Two questions need to be solved in order to buildsuch an encoder, namely: what is the prefer-able neural network architecture; and how andon what task should such a network be trained.Following existing work on learning word em-beddings, most current approaches consider learn-ing sentence encoders in an unsupervised mannerlike SkipThought (Kiros et al., 2015) or FastSent(Hill et al., 2016). Here, we investigate whethersupervised learning can be leveraged instead, tak-ing inspiration from previous results in computervision, where many models are pretrained on theImageNet (Deng et al., 2009) before being trans-ferred. We compare sentence embeddings trainedon various supervised tasks, and show that sen-tence embeddings generated from models trainedon a natural language inference (NLI) task reachthe best results in terms of transfer accuracy. Wehypothesize that the suitability of NLI as a train-ing task is caused by the fact that it is a high-levelunderstanding task that involves reasoning aboutthe semantic relationships within sentences.Unlike in computer vision, where convolutionalneural networks are predominant, there are mul-tiple ways to encode a sentence using neural net-works. Hence, we investigate the impact of thesentence encoding architecture on representationaltransferability, and compare convolutional, recur-rent and even simpler word composition schemes.Our experiments show that an encoder based on abi-directional LSTM architecture with max pool-ing, trained on the Stanford Natural LanguageInference (SNLI) dataset (Bowman et al., 2015),yields state-of-the-art sentence embeddings com-pared to all existing alternative unsupervised ap-proaches like SkipThought or FastSent, while be-ng much faster to train. We establish this findingon a broad and diverse set of transfer tasks thatmeasures the ability of sentence representations tocapture general and useful information.
Transfer learning using supervised features hasbeen successful in several computer vision ap-plications (Razavian et al., 2014). Striking ex-amples include face recognition (Taigman et al.,2014) and visual question answering (Antol et al.,2015), where image features trained on ImageNet(Deng et al., 2009) and word embeddings trainedon large unsupervised corpora are combined.In contrast, most approaches for sentence repre-sentation learning are unsupervised, arguably be-cause the NLP community has not yet found thebest supervised task for embedding the semanticsof a whole sentence. Another reason is that neu-ral networks are very good at capturing the bi-ases of the task on which they are trained, butcan easily forget the overall information or seman-tics of the input data by specializing too muchon these biases. Learning models on large un-supervised task makes it harder for the model tospecialize. Littwin and Wolf (2016) showed thatco-adaptation of encoders and classifiers, whentrained end-to-end, can negatively impact the gen-eralization power of image features generated byan encoder. They propose a loss that incorporatesmultiple orthogonal classifiers to counteract thiseffect.Recent work on generating sentence embed-dings range from models that compose wordembeddings (Le and Mikolov, 2014; Arora et al.,2017; Wieting et al., 2016) to more complex neu-ral network architectures. SkipThought vectors(Kiros et al., 2015) propose an objective func-tion that adapts the skip-gram model for words(Mikolov et al., 2013) to the sentence level. By en-coding a sentence to predict the sentences aroundit, and using the features in a linear model, theywere able to demonstrate good performance on 8transfer tasks. They further obtained better resultsusing layer-norm regularization of their model in(Ba et al., 2016). Hill et al. (2016) showed thatthe task on which sentence embeddings are trainedsignificantly impacts their quality.In addition to unsupervised methods, they in-cluded supervised training in their comparison—namely, on machine translation data (using the WMT’14 English/French and English/Germanpairs), dictionary definitions and image captioningdata (see also Kiela et al. (2017)) from the COCOdataset (Lin et al., 2014). These models obtainedsignificantly lower results compared to the unsu-pervised Skip-Thought approach.Recent work has explored training sentence en-coders on the SNLI corpus and applying them onthe SICK corpus (Marelli et al., 2014), either us-ing multi-task learning or pretraining (Mou et al.,2016; Bowman et al., 2015). The results were in-conclusive and did not reach the same level as sim-pler approaches that directly learn a classifier ontop of unsupervised sentence embeddings instead(Arora et al., 2017). To our knowledge, this workis the first attempt to fully exploit the SNLI cor-pus for building generic sentence encoders. As weshow in our experiments, we are able to consis-tently outperform unsupervised approaches, evenif our models are trained on much less (but human-annotated) data.
This work combines two research directions,which we describe in what follows. First, we ex-plain how the NLI task can be used to train univer-sal sentence encoding models using the SNLI task.We subsequently describe the architectures that weinvestigated for the sentence encoder, which, inour opinion, covers a suitable range of sentenceencoders currently in use. Specifically, we exam-ine standard recurrent models such as LSTMs andGRUs, for which we investigate mean and max-pooling over the hidden representations; a self-attentive network that incorporates different viewsof the sentence; and a hierarchical convolutionalnetwork that can be seen as a tree-based methodthat blends different levels of abstraction.
The SNLI dataset consists of 570k human-generated English sentence pairs, manually la-beled with one of three categories: entailment,contradiction and neutral. It captures natural lan-guage inference, also known in previous incarna-tions as Recognizing Textual Entailment (RTE),and constitutes one of the largest high-quality la-beled resources explicitly constructed in order torequire understanding sentence semantics. We hy-pothesize that the semantic nature of NLI makesit a good candidate for learning universal sentencembeddings in a supervised way. That is, we aimto demonstrate that sentence encoders trained onnatural language inference are able to learn sen-tence representations that capture universally use-ful features. sentence encoderwith hypothesis inputsentence encoderwith premise input3-way softmax u v fully-connected layers ( u, v, | u − v | , u ∗ v ) Figure 1:
Generic NLI training scheme.
Models can be trained on SNLI in two differ-ent ways: (i) sentence encoding-based models thatexplicitly separate the encoding of the individualsentences and (ii) joint methods that allow to useencoding of both sentences (to use cross-featuresor attention from one sentence to the other).Since our goal is to train a generic sentence en-coder, we adopt the first setting. As illustrated inFigure 1, a typical architecture of this kind uses ashared sentence encoder that outputs a representa-tion for the premise u and the hypothesis v . Oncethe sentence vectors are generated, 3 matchingmethods are applied to extract relations between u and v : (i) concatenation of the two representa-tions ( u, v ) ; (ii) element-wise product u ∗ v ; and(iii) absolute element-wise difference | u − v | . Theresulting vector, which captures information fromboth the premise and the hypothesis, is fed intoa 3-class classifier consisting of multiple fully-connected layers culminating in a softmax layer. A wide variety of neural networks for encod-ing sentences into fixed-size representations ex-ists, and it is not yet clear which one best cap-tures generically useful information. We com-pare 7 different architectures: standard recurrentencoders with either Long Short-Term Memory(LSTM) or Gated Recurrent Units (GRU), con-catenation of last hidden states of forward andbackward GRU, Bi-directional LSTMs (BiLSTM)with either mean or max pooling, self-attentive network and hierarchical convolutional networks.
Our first, and simplest, encoders apply re-current neural networks using either LSTM(Hochreiter and Schmidhuber, 1997) or GRU(Cho et al., 2014) modules, as in sequence to se-quence encoders (Sutskever et al., 2014). Fora sequence of T words ( w , . . . , w T ) , the net-work computes a set of T hidden representations h , . . . , h T , with h t = −−−−→ LSTM ( w , . . . , w T ) (orusing GRU units instead). A sentence is repre-sented by the last hidden vector, h T .We also consider a model BiGRU-last that con-catenates the last hidden state of a forward GRU,and the last hidden state of a backward GRU tohave the same architecture as for SkipThoughtvectors. For a sequence of T words { w t } t =1 ,...,T , a bidirec-tional LSTM computes a set of T vectors { h t } t .For t ∈ [1 , . . . , T ] , h t , is the concatenation of aforward LSTM and a backward LSTM that readthe sentences in two opposite directions: −→ h t = −−−−→ LSTM t ( w , . . . , w T ) ←− h t = ←−−−− LSTM t ( w , . . . , w T ) h t = [ −→ h t , ←− h t ] We experiment with two ways of combining thevarying number of { h t } t to form a fixed-size vec-tor, either by selecting the maximum value overeach dimension of the hidden units (max pool-ing) (Collobert and Weston, 2008) or by consider-ing the average of the representations (mean pool-ing). The movie was great ←− h ←− h ←− h ←− h −→ h −→ h −→ h −→ h w w w w x x x xx x x x max-pooling… … u : Figure 2:
Bi-LSTM max-pooling network..2.3 Self-attentive network
The self-attentive sentence encoder (Liu et al.,2016; Lin et al., 2017) uses an attention mecha-nism over the hidden states of a BiLSTM to gen-erate a representation u of an input sentence. Theattention mechanism is defined as : ¯ h i = tanh( W h i + b w ) α i = e ¯ h Ti u w P i e ¯ h Ti u w u = X t α i h i where { h , . . . , h T } are the output hidden vec-tors of a BiLSTM. These are fed to an affine trans-formation ( W , b w ) which outputs a set of keys (¯ h , . . . , ¯ h T ) . The { α i } represent the score ofsimilarity between the keys and a learned con-text query vector u w . These weights are usedto produce the final representation u , which is aweighted linear combination of the hidden vectors.Following Lin et al. (2017) we use a self-attentive network with multiple views of the inputsentence, so that the model can learn which part ofthe sentence is important for the given task. Con-cretely, we have 4 context vectors u w , u w , u w , u w which generate 4 representations that are then con-catenated to obtain the sentence representation u .Figure 3 illustrates this architecture. The movie was great u w ←− h ←− h ←− h ←− h −→ h −→ h −→ h −→ h α α α α u w w w w Figure 3:
Inner Attention network architecture.3.2.4 Hierarchical ConvNet
One of the currently best performing models onclassification tasks is a convolutional architecturetermed
AdaSent (Zhao et al., 2015), which con-catenates different representations of the sentencesat different level of abstractions. Inspired by thisarchitecture, we introduce a faster version consist-ing of 4 convolutional layers. At every layer, a representation u i is computed by a max-poolingoperation over the feature maps (see Figure 4). …… ……… …This is the greatest movie of all timex xxx x x max-poolingmax-poolingmax-poolingmax-pooling xxxxxx u u u u u : u u u u convolutional layerconvolutional layerconvolutional layerconvolutional layer Figure 4:
Hierarchical ConvNet architecture.
The final representation u = [ u , u , u , u ] concatenates representations at different levels ofthe input sentence. The model thus captures hi-erarchical abstractions of an input sentence in afixed-size representation. For all our models trained on SNLI, we use SGDwith a learning rate of 0.1 and a weight decay of0.99. At each epoch, we divide the learning rateby 5 if the dev accuracy decreases. We use mini-batches of size 64 and training is stopped when thelearning rate goes under the threshold of − . Forthe classifier, we use a multi-layer perceptron with1 hidden-layer of 512 hidden units. We use open-source GloVe vectors trained on Common Crawl840B with 300 dimensions as fixed word embed-dings. Our aim is to obtain general-purpose sentenceembeddings that capture generic information thatis useful for a broad set of tasks. To evalu-ate the quality of these representations, we usethem as features in 12 transfer tasks. We presentour sentence-embedding evaluation procedure in ame N task C examples
MR 11k sentiment (movies) 2 ”Too slow for a younger crowd , too shallow for an older one.” (neg)CR 4k product reviews 2 ”We tried it out christmas night and it worked great .” (pos)SUBJ 10k subjectivity/objectivity 2 ”A movie that doesn’t aim too high , but doesn’t need to.” (subj)MPQA 11k opinion polarity 2 ”don’t want”; ”would like to tell”; (neg, pos)TREC 6k question-type 6 ”What are the twin cities ?” (LOC:city)SST 70k sentiment (movies) 2 ”Audrey Tautou has a knack for picking roles that magnify her [..]” (pos)
Table 1:
Classification tasks . C is the number of class and N is the number of samples.this section. We constructed a sentence evalu-ation tool called SentEval (Conneau and Kiela,2018) to automate evaluation on all the tasksmentioned in this paper. The tool uses Adam(Kingma and Ba, 2014) to fit a logistic regressionclassifier, with batch size 64.
Binary and multi-class classification
We usea set of binary classification tasks (see Table 1)that covers various types of sentence classifica-tion, including sentiment analysis (MR, SST),question-type (TREC), product reviews (CR), sub-jectivity/objectivity (SUBJ) and opinion polarity(MPQA). We generate sentence vectors and traina logistic regression on top. A linear classifier re-quires fewer parameters than an MLP and is thussuitable for small datasets, where transfer learningis especially well-suited. We tune the L2 penaltyof the logistic regression with grid-search on thevalidation set.
Entailment and semantic relatedness
We alsoevaluate on the SICK dataset for both entailment(SICK-E) and semantic relatedness (SICK-R). Weuse the same matching methods as in SNLI andlearn a Logistic Regression on top of the joint rep-resentation. For semantic relatedness evaluation,we follow the approach of (Tai et al., 2015) andlearn to predict the probability distribution of re-latedness scores. We report Pearson correlation.
STS14 - Semantic Textual Similarity
Whilesemantic relatedness is supervised in the caseof SICK-R, we also evaluate our embeddingson the 6 unsupervised SemEval tasks of STS14(Agirre et al., 2014). This dataset includes sub-sets of news articles, forum discussions, image de-scriptions and headlines from news articles con-taining pairs of sentences (lower-cased), labeledwith a similarity score between 0 and 5. Thesetasks evaluate how the cosine distance betweentwo sentences correlate with a human-labeled sim-ilarity score through Pearson and Spearman corre- lations. Paraphrase detection
The Microsoft ResearchParaphrase Corpus is composed of pairs of sen-tences which have been extracted from newssources on the Web. Sentence pairs have beenhuman-annotated according to whether they cap-ture a paraphrase/semantic equivalence relation-ship. We use the same approach as with SICK-E,except that our classifier has only 2 classes.
Caption-Image retrieval
The caption-imageretrieval task evaluates joint image and languagefeature models (Hodosh et al., 2013; Lin et al.,2014). The goal is either to rank a large collec-tion of images by their relevance with respect to agiven query caption (Image Retrieval), or rankingcaptions by their relevance for a given query image(Caption Retrieval). We use a pairwise ranking-loss L cir ( x, y ) : X y X k max(0 , α − s ( V y, U x ) + s ( V y, U x k )) + X x X k ′ max(0 , α − s ( U x, V y ) + s ( U x, V y k ′ )) where ( x, y ) consists of an image y with oneof its associated captions x , ( y k ) k and ( y k ′ ) k ′ arenegative examples of the ranking loss, α is themargin and s corresponds to the cosine similarity. U and V are learned linear transformations thatproject the caption x and the image y to the sameembedding space. We use a margin α = 0 . and contrastive terms. We use the same splits asin (Karpathy and Fei-Fei, 2015), i.e., we use 113kimages from the COCO dataset (each containing5 captions) for training, 5k images for validationand 5k images for test. For evaluation, we split the5k images in 5 random sets of 1k images on whichwe compute Recall@K, with K ∈ { , , } andmedian (Med r) over the 5 splits. For fair compari-son, we also report SkipThought results in our set-ting, using 2048-dimensional pretrained ResNet-101 (He et al., 2016) with 113k training images. ame task N premise hypothesis label SNLI NLI 560k ”Two women are embracing whileholding to go packages.” ”Two woman are holding packages.” entailmentSICK-E NLI 10k A man is typing on a machine usedfor stenography The man isn’t operating a steno-graph contradictionSICK-R STS 10k ”A man is singing a song and play-ing the guitar” ”A man is opening a package thatcontains headphones” 1.6STS14 STS 4.5k ”Liquid ammonia leak kills 15 inShanghai” ”Liquid ammonia leak kills at least15 in Shanghai” 4.6
Table 2:
Natural Language Inference and Semantic Textual Similarity tasks . NLI labels are contra-diction, neutral and entailment. STS labels are scores between 0 and 5.
Model NLI Transferdim dev test micro macro
LSTM 2048 81.9 80.7 79.5 78.6GRU 4096 82.4 81.8 81.7 80.9BiGRU-last 4096 81.3 80.9 82.9 81.7BiLSTM-Mean 4096 79.0 78.2 83.1 81.7Inner-attention 4096 82.3 82.5 82.1 81.0HConvNet 4096 83.7 83.4 82.0 80.9BiLSTM-Max 4096
Table 3:
Performance of sentence encoder ar-chitectures on SNLI and (aggregated) transfertasks. Dimensions of embeddings were selectedaccording to best aggregated scores (see Figure 5).Figure 5:
Transfer performance w.r.t. embed-ding size using the micro aggregation method.
In this section, we refer to ”micro” and ”macro”averages of development set (dev) results on trans-fer tasks whose metrics is accuracy: we compute a”macro” aggregated score that corresponds to theclassical average of dev accuracies, and the ”mi-cro” score that is a sum of the dev accuracies,weighted by the number of dev samples.
We observe in Table 3 that different mod-els trained on the same NLI corpus lead to differ-ent transfer tasks results. The BiLSTM-4096 withthe max-pooling operation performs best on bothSNLI and transfer tasks. Looking at the micro and macro averages, we see that it performs signifi-cantly better than the other models LSTM, GRU,BiGRU-last, BiLSTM-Mean, inner-attention andthe hierarchical-ConvNet.Table 3 also shows that better performance onthe training task does not necessarily translate inbetter results on the transfer tasks like when com-paring inner-attention and BiLSTM-Mean for in-stance.We hypothesize that some models are likely toover-specialize and adapt too well to the biases ofa dataset without capturing general-purpose infor-mation of the input sentence. For example, theinner-attention model has the ability to focus onlyon certain parts of a sentence that are useful forthe SNLI task, but not necessarily for the transfertasks. On the other hand, BiLSTM-Mean does notmake sharp choices on which part of the sentenceis more important than others. The difference be-tween the results seems to come from the differentabilities of the models to incorporate general in-formation while not focusing too much on specificfeatures useful for the task at hand.For a given model, the transfer quality is alsosensitive to the optimization algorithm: whentraining with Adam instead of SGD, we observedthat the BiLSTM-max converged faster on SNLI(5 epochs instead of 10), but obtained worse re-sults on the transfer tasks, most likely because ofthe model and classifier’s increased capability toover-specialize on the training task.
Embedding size
Figure 5 compares the over-all performance of different architectures, showingthe evolution of micro averaged performance withregard to the embedding size.Since it is easier to linearly separate in high di-mension, especially with logistic regression, it isnot surprising that increased embedding sizes leadto increased performance for almost all models.However, this is particularly true for some mod- odel MR CR SUBJ MPQA SST TREC MRPC SICK-R SICK-E STS14
Unsupervised representation training (unordered sentences)
Unigram-TFIDF 73.7 /80.7 - - .37/.38SIF (GloVe + WR) - - - - 82.2 - - - / -word2vec BOW † † † † † Unsupervised representation training (ordered sentences)
FastSent 70.8 78.4 88.7 80.6 - 76.8 72.2/80.3 - - .63/.64
FastSent+AE 71.8 76.7 88.8 81.5 - 80.4 71.2/79.1 - - .62/.62SkipThought 76.5 80.1 93.6 87.1 82.0
Supervised representation training
CaptionRep (bow) 61.9 69.3 77.4 70.8 - 72.2 73.6/81.9 - - .46/.42DictRep (bow) 76.7 78.7 90.7 87.2 - 81.0 68.4/76.8 - - .67/.70
NMT En-to-Fr 64.7 70.1 84.9 81.5 - 82.8 69.1/77.1 - .43/.42Paragram-phrase - - - - 79.7 - - 0.849 83.1 .71 / -BiLSTM-Max (on SST) † (*) 83.7 90.2 89.5 (*) 86.0 72.7/80.9 0.863 83.1 .55/.54BiLSTM-Max (on SNLI) † .68/.65BiLSTM-Max (on AllNLI) † Supervised methods (directly trained for each task – no transfer)
Naive Bayes - SVM 79.4 81.8 93.2 86.3 83.1 - - - - -AdaSent 83.1 86.3 95.5 93.3 - 92.4 - - - -TF-KLD - - - - - - 80.4/85.9 - - -Illinois-LH - - - - - - - - 84.5 -Dependency Tree-LSTM - - - - - - - 0.868 - -
Table 4:
Transfer test results for various architectures trained in different ways . Underlinedare best results for transfer learning approaches, in bold are best results among the models trainedin the same way. † indicates methods that we trained, other transfer models have been extractedfrom (Hill et al., 2016). For best published supervised methods (no transfer), we consider AdaSent(Zhao et al., 2015), TF-KLD (Ji and Eisenstein, 2013), Tree-LSTM (Tai et al., 2015) and Illinois-LHsystem (Lai and Hockenmaier, 2014). (*) Our model trained on SST obtained 83.4 for MR and 86.0 forSST (MR and SST come from the same source), which we do not put in the tables for fair comparisonwith transfer methods.els (BiLSTM-Max, HConvNet, inner-att), whichdemonstrate unequal abilities to incorporate moreinformation as the size grows. We hypothesizethat such networks are able to incorporate infor-mation that is not directly relevant to the objectivetask (results on SNLI are relatively stable with re-gard to embedding size) but that can neverthelessbe useful as features for transfer tasks. We report in Table 4 transfer tasks results fordifferent architectures trained in different ways.We group models by the nature of the dataon which they were trained. The first group corresponds to models trained with unsuper-vised unordered sentences. This includes bag-of-words models such as word2vec-SkipGram,the Unigram-TFIDF model, the Paragraph Vectormodel (Le and Mikolov, 2014), the Sequential De-noising Auto-Encoder (SDAE) (Hill et al., 2016)and the SIF model (Arora et al., 2017), all trainedon the Toronto book corpus (Zhu et al., 2015). Thesecond group consists of models trained with un-supervised ordered sentences such as FastSent andSkipThought (also trained on the Toronto bookcorpus). We also include the FastSent variant“FastSent+AE” and the SkipThought-LN versionthat uses layer normalization. We report results aption Retrieval Image RetrievalModel R@1 R@5 R@10 Med r R@1 R@5 R@10 Med r
Direct supervision of sentence representationsm -CNN (Ma et al., 2015) 38.3 - 81.0 2 27.4 - 79.5 3 m -CNN ENS (Ma et al., 2015) 42.8 - 84.1 2 32.6 - 82.8 3Order-embeddings (Vendrov et al., 2016) - - Pre-trained sentence representations
SkipThought + VGG19 (82k) 33.8 67.7 82.1 3 25.9 60.0 74.6 4SkipThought + ResNet101 (113k) 37.9 72.2 84.3 2 30.6 66.2 81.0 3BiLSTM-Max (on SNLI) + ResNet101 (113k) 42.4 Table 5:
COCO retrieval results . SkipThought is trained either using 82k training samples with VGG19features, or with 113k samples and ResNet-101 features (our setting). We report the average results on 5splits of 1k test images.from models trained on supervised data in the thirdgroup, and also report some results of supervisedmethods trained directly on each task for compar-ison with transfer learning approaches.
Comparison with SkipThought
The bestperforming sentence encoder to date is theSkipThought-LN model, which was trained ona very large corpora of ordered sentences. Withmuch less data (570k compared to 64M sentences)but with high-quality supervision from the SNLIdataset, we are able to consistently outperformthe results obtained by SkipThought vectors. Wetrain our model in less than a day on a single GPUcompared to the best SkipThought-LN networktrained for a month. Our BiLSTM-max trainedon SNLI performs much better than releasedSkipThought vectors on MR, CR, MPQA, SST,MRPC-accuracy, SICK-R, SICK-E and STS14(see Table 4). Except for the SUBJ dataset, italso performs better than SkipThought-LN onMR, CR and MPQA. We also observe by lookingat the STS14 results that the cosine metrics inour embedding space is much more semanticallyinformative than in SkipThought embeddingspace (pearson score of 0.68 compared to 0.29and 0.44 for ST and ST-LN). We hypothesizethat this is namely linked to the matching methodof SNLI models which incorporates a notionof distance (element-wise product and absolutedifference) during training.
NLI as a supervised training set
Our findingsindicate that our model trained on SNLI obtainsmuch better overall results than models trainedon other supervised tasks such as COCO, dictio-nary definitions, NMT, PPDB (Ganitkevitch et al.,2013) and SST. For SST, we tried exactly the same models as for SNLI; it is worth noting that SST issmaller than NLI. Our representations constitutehigher-quality features for both classification andsimilarity tasks. One explanation is that the natu-ral language inference task constrains the model toencode the semantic information of the input sen-tence, and that the information required to performNLI is generally discriminative and informative.
Domain adaptation on SICK tasks
Our trans-fer learning approach obtains better results thanprevious state-of-the-art on the SICK task - canbe seen as an out-domain version of SNLI - forboth entailment and relatedness. We obtain a pear-son score of 0.885 on SICK-R while (Tai et al.,2015) obtained 0.868, and we obtain 86.3% testaccuracy on SICK-E while previous best hand-engineered models (Lai and Hockenmaier, 2014)obtained 84.5%. We also significantly outper-formed previous transfer learning approaches onSICK-E (Bowman et al., 2015) that used the pa-rameters of an LSTM model trained on SNLI tofine-tune on SICK (80.8% accuracy). We hypothe-size that our embeddings already contain the infor-mation learned from the in-domain task, and thatlearning only the classifier limits the number ofparameters learned on the small out-domain task.
Image-caption retrieval results
In Table 5, wereport results for the COCO image-caption re-trieval task. We report the mean recalls of 5 ran-dom splits of 1K test images. When trained withResNet features and 30k more training data, theSkipThought vectors perform significantly betterthan the original setting, going from 33.8 to 37.9for caption retrieval R@1, and from 25.9 to 30.6on image retrieval R@1. Our approach pushesthe results even further, from 37.9 to 42.4 on cap-ion retrieval, and 30.6 to 33.2 on image retrieval.These results are comparable to previous approachof (Ma et al., 2015) that did not do transfer but di-rectly learned the sentence encoding on the image-caption retrieval task. This supports the claim thatpre-trained representations such as ResNet imagefeatures and our sentence embeddings can achievecompetitive results compared to features learneddirectly on the objective task.
MultiGenre NLI
The MultiNLI corpus(Williams et al., 2017) was recently releasedas a multi-genre version of SNLI. With 433Ksentence pairs, MultiNLI improves upon SNLIin its coverage: it contains ten distinct genresof written and spoken English, covering mostof the complexity of the language. We augmentTable 4 with our model trained on both SNLIand MultiNLI (AllNLI). We observe a significantboost in performance overall compared to themodel trained only on SLNI. Our model evenreaches AdaSent performance on CR, suggestingthat having a larger coverage for the training taskhelps learn even better general representations.On semantic textual similarity STS14, we arealso competitive with PPDB based paragram-phrase embeddings with a pearson score of 0.70.Interestingly, on caption-related transfer taskssuch as the COCO image caption retrieval task,training our sentence encoder on other genresfrom MultiNLI does not degrade the performancecompared to the model trained only SNLI (whichcontains mostly captions), which confirms thegeneralization power of our embeddings.
This paper studies the effects of training sentenceembeddings with supervised data by testing on12 different transfer tasks. We showed that mod-els learned on NLI can perform better than mod-els trained in unsupervised conditions or on othersupervised tasks. By exploring various architec-tures, we showed that a BiLSTM network withmax pooling makes the best current universal sen-tence encoding methods, outperforming existingapproaches like SkipThought vectors.We believe that this work only scratches the sur-face of possible combinations of models and tasksfor learning generic sentence embeddings. Largerdatasets that rely on natural language understand-ing for sentences could bring sentence embeddingquality to the next level.
References
Eneko Agirre, Carmen Banea, Claire Cardie, DanielCer, Mona Diab, Aitor Gonzalez-Agirre, WeiweiGuo, Rada Mihalcea, German Rigau, and JanyceWiebe. 2014. Semeval-2014 task 10: Multilingualsemantic textual similarity. In
Proceedings of the8th International Workshop on Semantic Evalua-tion (SemEval 2014) . Association for ComputationalLinguistics and Dublin City University, Dublin, Ire-land, pages 81–91.Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar-garet Mitchell, Dhruv Batra, C Lawrence Zitnick,and Devi Parikh. 2015. Vqa: Visual question an-swering. In
Proceedings of the IEEE InternationalConference on Computer Vision . pages 2425–2433.Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017.A simple but tough-to-beat baseline for sentence em-beddings.
International Conference on LearningRepresentations .Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin-ton. 2016. Layer normalization.
NIPS .Yoshua Bengio, Rejean Ducharme, and Pascal Vincent.2003. A neural probabilistic language model.
Jour-nal of Machine Learning Research
Proceedings of EMNLP .Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the propertiesof neural machine translation: Encoder-decoder ap-proaches. In
Eighth Workshop on Syntax, Semanticsand Structure in Statistical Translation (SSST-8) .Ronan Collobert and Jason Weston. 2008. A unifiedarchitecture for natural language processing: Deepneural networks with multitask learning. In
Pro-ceedings of the 25th international conference onMachine learning . ACM, pages 160–167.Ronan Collobert, Jason Weston, L´eon Bottou, MichaelKarlen, Koray Kavukcuoglu, and Pavel Kuksa.2011. Natural language processing (almost) fromscratch.
Journal of Machine Learning Research
LREC .Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,and Li Fei-Fei. 2009. Imagenet: A large-scale hi-erarchical image database. In
Computer Vision andPattern Recognition, 2009. CVPR 2009. IEEE Con-ference on . IEEE, pages 248–255.Juri Ganitkevitch, Benjamin Van Durme, and ChrisCallison-Burch. 2013. PPDB: The paraphrasedatabase. In
Proceedings of NAACL-HLT . Atlanta,Georgia, pages 758–764.aiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. 2016. Deep residual learning for image recog-nition. In
Conference on Computer Vision and Pat-tern Recognition (CVPR) . IEEE, page 8.Felix Hill, Kyunghyun Cho, and Anna Korhonen.2016. Learning distributed representations ofsentences from unlabelled data. arXiv preprintarXiv:1602.03483 .Sepp Hochreiter and J¨urgen Schmidhuber. 1997.Long short-term memory.
Neural computation
Journal of Ar-tificial Intelligence Research
Proceedings of the 2013 Conference onEmpirical Methods in Natural Language Process-ing . pages 891–896.Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descrip-tions. In
Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition . pages3128–3137.Douwe Kiela, Alexis Conneau, Allan Jabri, and Max-imilian Nickel. 2017. Learning visually groundedsentence representations.
Proceedings of NAACL2018 .Diederik P. Kingma and Jimmy Ba. 2014. Adam: Amethod for stochastic optimization. In
Proceed-ings of the 3rd International Conference on Learn-ing Representations (ICLR) .Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov,Richard Zemel, Raquel Urtasun, Antonio Torralba,and Sanja Fidler. 2015. Skip-thought vectors. In
Advances in neural information processing systems .pages 3294–3302.Alice Lai and Julia Hockenmaier. 2014. Illinois-lh: Adenotational and distributional approach to seman-tics.
Proc. SemEval
ICML .volume 14, pages 1188–1196.Tsung-Yi Lin, Michael Maire, Serge Belongie, JamesHays, Pietro Perona, Deva Ramanan, Piotr Doll´ar,and C Lawrence Zitnick. 2014. Microsoft coco:Common objects in context. In
European Confer-ence on Computer Vision . pages 740–755.Zhouhan Lin, Minwei Feng, Cicero Nogueira dos San-tos, Mo Yu, Bing Xiang, Bowen Zhou, and YoshuaBengio. 2017. A structured self-attentive sentenceembedding.
International Conference on LearningRepresentations . Etai Littwin and Lior Wolf. 2016. The multiverse lossfor robust transfer learning. In
Proceedings of theIEEE Conference on Computer Vision and PatternRecognition . pages 3957–3966.Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang.2016. Learning natural language inference usingbidirectional lstm model and inner-attention. arXivpreprint arXiv:1605.09090 .Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li.2015. Multimodal convolutional neural networksfor matching image and sentence. In
Proceedingsof the IEEE International Conference on ComputerVision . pages 2623–2631.Marco Marelli, Stefano Menini, Marco Baroni, LuisaBentivogli, Raffaella Bernardi, and Roberto Zam-parelli. 2014. A sick cure for the evaluation of com-positional distributional semantic models. In
Pro-ceedings of LREC . pages 216–223.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013. Distributed representa-tions of words and phrases and their compositional-ity. In
Advances in neural information processingsystems . pages 3111–3119.Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu,Lu Zhang, and Zhi Jin. 2016. How transferable areneural networks in nlp applications? arXiv preprintarXiv:1603.06111 .Jeffrey Pennington, Richard Socher, and Christopher DManning. 2014. Glove: Global vectors for wordrepresentation. In
EMNLP . volume 14, pages 1532–1543.Ali Sharif Razavian, Hossein Azizpour, Josephine Sul-livan, and Stefan Carlsson. 2014. Cnn features off-the-shelf: an astounding baseline for recognition. In
Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops . pages806–813.Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.Sequence to sequence learning with neural net-works. In
Advances in neural information process-ing systems . pages 3104–3112.Kai Sheng Tai, Richard Socher, and Christopher DManning. 2015. Improved semantic representationsfrom tree-structured long short-term memory net-works.
Proceedings of ACL .Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato,and Lior Wolf. 2014. Deepface: Closing the gapto human-level performance in face verification. In
Conference on Computer Vision and Pattern Recog-nition (CVPR) . IEEE, page 8.Ivan Vendrov, Ryan Kiros, Sanja Fidler, and RaquelUrtasun. 2016. Order-embeddings of images andlanguage.
International Conference on LearningRepresentations .ohn Wieting, Mohit Bansal, Kevin Gimpel, and KarenLivescu. 2016. Towards universal paraphrastic sen-tence embeddings.
International Conference onLearning Representations .Adina Williams, Nikita Nangia, and Samuel R Bow-man. 2017. A broad-coverage challenge corpus forsentence understanding through inference. arXivpreprint arXiv:1704.05426 .Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015.Self-adaptive hierarchical sentence model. In
Pro-ceedings of the 24th International Conference on Ar-tificial Intelligence . AAAI Press, IJCAI’15, pages4069–4076.Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut-dinov, Raquel Urtasun, Antonio Torralba, and SanjaFidler. 2015. Aligning books and movies: Towardsstory-like visual explanations by watching moviesand reading books. In
Proceedings of the IEEEinternational conference on computer vision . pages19–27. ppendix
Max-pooling visualization for BiLSTM-maxtrained and untrained
Our representationswere trained to focus on parts of a sentence suchthat a classifier can easily tell the difference be-tween contradictory, neutral or entailed sentences.In Table 8 and Table 9, we investigate howthe max-pooling operation selects the informationfrom the hidden states of the BiLSTM, for ourtrained and untrained BiLSTM-max models (forboth models, word embeddings are initialized withGloVe vectors).For each time step t , we report the number oftimes the max-pooling operation selected the hid-den state h t (which can be seen as a sentence rep-resentation centered around word w t ).Without any training, the max-pooling is rathereven across hidden states, although it seems to fo-cus consistently more on the first and last hiddenstates. When trained, the model learns to focus onspecific words that carry most of the meaning ofthe sentence without any explicit attention mecha-nism.Note that each hidden state also incorporates in-formation from the sentence at different levels, ex-plaining why the trained model also incorporatesinformation from all hidden states.Figure 6: Pair of entailed sentences A: Visualiza-tion of max-pooling for BiLSTM-max 4096 un-trained . Figure 7: Pair of entailed sentences A: Visual-ization of max-pooling for BiLSTM-max 4096 trained on NLI .Figure 8: Pair of entailed sentences B: Visualiza-tion of max-pooling for BiLSTM-max 4096 un-trained .Figure 9: Pair of entailed sentences B: Visual-ization of max-pooling for BiLSTM-max 4096 trained on NLItrained on NLI