An information theoretic view on selecting linguistic probes
AAn information theoretic view on selecting linguistic probes
Zining Zhu , , Frank Rudzicz , , , University of Toronto, Vector Institute, Surgical Safety Technologies Li Ka Shing Knowledge Institute, St Michael’s Hospital { zining,frank } @cs.toronto.edu Abstract
There is increasing interest in assessing the lin-guistic knowledge encoded in neural represen-tations. A popular approach is to attach a diag-nostic classifier – or “probe” – to perform su-pervised classification from internal represen-tations. However, how to select a good probeis in debate. Hewitt and Liang (2019) showedthat a high performance on diagnostic classifi-cation itself is insufficient, because it can beattributed to either “the representation beingrich in knowledge”, or “the probe learning thetask”, which Pimentel et al. (2020) challenged.We show this dichotomy is valid information-theoretically. In addition, we find that themethods to construct and select good probesproposed by the two papers, control task (He-witt and Liang, 2019) and control function (Pi-mentel et al., 2020), are equivalent – the errorsof their approaches are identical (modulo irrel-evant terms). Empirically, these two selectioncriteria lead to results that highly agree witheach other.
Recently, neural networks have shown substantialprogress in NLP tasks (Devlin et al., 2019; Rad-ford et al., 2019). To understand and explain theirbehavior, a natural question emerge: how muchlinguistic knowledge is encoded in these neuralnetwork systems?An efficient approach to reveal information en-coded in internal representations uses diagnosticclassifiers (Alain and Bengio, 2017). Referred toas “probes”, diagnostic classifiers are trained onpre-computed intermediate representations of neu-ral NLP systems. The performance on tasks theyare trained to predict are used to evaluate the rich-ness of the linguistic representation in encodingthe probed tasks. Such tasks include probing syn-tax (Hewitt and Manning, 2019; Lin et al., 2019;Tenney et al., 2019a), semantics (Yaghoobzadeh et al., 2019), discourse features (Chen et al., 2019;Liu et al., 2019; Tenney et al., 2019b), and com-monsense knowledge (Petroni et al., 2019; Poerneret al., 2019).However, appropriate criteria for selecting agood probe is under debate. The traditional viewthat high-accuracy probes are better is challengedby Hewitt and Liang (2019), who proposed that thehigh accuracy could be attributed to either (1) thatthe representation contains rich linguistic knowl-edge, or (2) that the probe learns the task. To cir-cumvent this ambiguity, they proposed to use theimprovement of probing task performance againsta control task (predicting random labels from thesame representations), i.e., the “selectivity” crite-rion. Recently, Pimentel et al. (2020), challengedthis dichotomy from an information theoretic view-point. They proposed to use an “information gain”criterion, which empirically is the reduction incross entropy from a “control function task” prob-ing from randomized representation.In this paper, we show the “non-exclusive-or”dichotomy raised by Hewitt and Liang (2019) isvalid information-theoretically. There is a differ-ence between the original NLP model learning thetask and the probe learning the task.In addition, we show that the “selectivity” crite-rion and the “control function” criterion are com-parably accurate. Pimentel et al. (2020) formu-lated their errors with the difference in a pair ofKL divergences. We show that the error of the“selectivity” criterion (Hewitt and Liang, 2019), ifmeasured from cross entropy loss, can be formu-lated in the difference in a pair of KL divergencesas well. When randomizations are perfect, thesetwo criteria differ by only constant terms.Empirically, on a POS tag probing task on En-glish, French and Spanish translations, we showthat the “selectivity” and the “control function” cri-teria highly agree with each other. We rank experi- a r X i v : . [ c s . C L ] S e p ents with over 10,000 different hyperparametersettings using these criteria. The Spearman correla-tion of the Hewitt and Liang (2019) vs. Pimentelet al. (2020) criteria are on par with the correla-tions of “accuracy vs. cross entropy loss” – twovery strong baselines.Overall, we recommend using control mecha-nisms to select probes, instead of relying on merelythe probing task performance. When randomiza-tion is done well, controlling the target or represen-tations are equivalent. Diagnostic probes were originally intended to ex-plain information encoded in intermediate represen-tations (Adi et al., 2017; Alain and Bengio, 2017;Belinkov et al., 2017). Recently, various prob-ing tasks have queried the representations of, e.g.,contextualized word embeddings (Tenney et al.,2019a,b) and sentence embeddings (Linzen et al.,2016; Chen et al., 2019; Alt et al., 2020; Kassnerand Sch¨utze, 2020; Maudslay et al., 2020; Chi et al.,2020).The task of probing is usually formulated as aclassifier problem, with the representations as in-put, and the features indicating information as out-put. A straightforward method to train a classifieris by minimizing cross entropy, which is the ap-proach we follow. Note that Voita and Titov (2020)derived training objectives from minimum descrip-tion lengths, resulting in cross entropy loss andsome variants.
We adopt the information theoretic formulation oflinguistic probes of (Pimentel et al., 2020), andbriefly summarize as follows.We want to probe true labels T from representa-tions R . An ideal probe should accurately reportthe code-target mutual information I ( T ; R ) , whichis unfortunately intractable. We will write I ( T ; R ) in an alternative form.Let p ( T | R ) be the unknown true conditionaldistribution, and a diagnostic probe, according tothe setting in literature (Alain and Bengio, 2017;Hewitt and Manning, 2019; Maudslay et al., 2020),is an approximation q θ ( T | R ) parameterized by θ ,then: I ( T ; R ) = H ( T ) − H ( T | R ) H ( T | R ) = − E p ( T | R ) log p ( T | R )= − E p ( T | R ) log p ( T | R ) q θ ( T | R ) q θ ( T | R )= − E p log q θ − E p log pq θ = H ( p, q θ ) − KL ( p (cid:107) q θ ) , where p and q θ stand for p ( T | R ) and q θ ( T | R ) respectively. We also use H ( p, q θ ) = − E p log q θ to represent the cross entropy for simplicity. Traditionally, people use thecross entropy loss of the diagnostic probe H ( p, q θ ) to approximate I ( T ; R ) . We can derive a source oferror by rewinding the above formulations: H ( p, q θ ) = H ( T ) − I ( T ; R ) + KL ( p (cid:107) q θ ) The first term on RHS, H ( T ) , is independentof either R or θ . Therefore, a low cross entropyloss H ( p, q θ ) can be caused by either of the twoscenarios: • High code-target mutual information I ( T ; R ) ,indicating the representation R contains richinformation about the target T . • Low KL-divergence between p ( T | R ) and q θ ( T | R ) , indicating the probe learns the task.The two scenarios exactly correspond to the di-chotomy of Hewitt and Liang (2019). A good probe
To get a good probe, we want H q θ ( T | R ) to approximate I ( T ; R ) as much aspossible. This means a good probe should min-imize KL ( p (cid:107) q θ ) , as proposed by Pimentel et al.(2020).However, empirically Pimentel et al. (2020) (aswell as many previous articles) used the cross en-tropy loss H q φ ( T | R ) to select good probes, whichis insufficient, as described above. Alternatively,Hewitt and Liang (2019) and Pimentel et al. (2020)proposed control tasks and control functions, re-spectively. The control task (Hewitt and Liang, 2019) sets ran-dom targets for the probing task. Let us use c ( T ) to indicate a “control function” applied on a token that originally has label T . The control func-tion could nullify the information of the input, ifnecessary.If we measure the difference betweencross entropy in the control task and prob-ing task H ( p ( c ( T ) | R ) , q θ c ( c ( T ) | R )) − H ( p ( T | R ) , q θ ( T | R )) , we can derive a form oferror margin in their measurements . Let us usea short-hand notations H ( p c , q θ c ) − H ( p, q θ ) forclarity. Now, what does the diff between crossentropy on control task and probing task actuallycontains? H ( p c , q θ c ) − H ( p, q θ )= ( H ( c ( T )) − I ( c ( T ); R ) + KL ( p c (cid:107) q θ c )) − ( H ( T ) − I ( T ; R ) + KL ( p (cid:107) q θ ))= ( H ( c ( T )) − H ( T )) − I ( c ( T ); R ) + I ( T ; R )+ ( KL ( p c (cid:107) q θ c ) − KL ( p (cid:107) q θ )) We already knew that H ( T ) = Const. Accordingthe definition of control function, the output c ( T ) would be independent of R . Then: H ( c ( T )) = Const p ( c ( T ) , R ) = p ( c ( T )) p ( R ) I ( c ( T ); R ) = E p ( c ( T ) , R ) p ( c ( T )) p ( R ) = ConstTherefore: H ( p c , q θ c ) − H ( p, q θ ) = I ( T ; R ) − ∆ h where ∆ h is a short-hand notation for the measure-ment error in the control task criteria: ∆ h = KL ( p (cid:107) q θ ) − KL ( p c (cid:107) q θ c ) + Const (1)When the probe fits the true distribution to a similarextent on both the control task and probing task,the error ∆ h would be small. Unfortunately, bothKL terms are intractable. Control function (Pimentel et al., 2020) introducesa random processor c ( · ) on the representation R .To measure the information gain, they used an “in-formation gain” criterion: G ( T, R, c ) = I ( T ; R ) − I ( T ; c ( R )) Note that Hewitt and Liang (2019) used accuracy insteadof cross entropy. We discuss cross entropy so as to comparethe errors against the control function (Pimentel et al., 2020)
Noticing that mutual information terms are in-tractable, they approximated the objective withthe difference between cross entropy in the con-trol function task (we refer to as “control function”henceforth) and the probing task: ˜ G ( T, R, c ) = H ( p c , q φ c ) − H ( p, q φ ) To compute the error of this approximation, theyreformulated the terms as following: G ( T, R, c )= H ( T ) − H ( T | R ) − H ( T ) + H ( T | c ( R )) The two H ( T ) terms cancel out, then: H ( T | R ) = H ( p ( T | R ) , q φ ( T | R )) − KL ( p ( T | R ) (cid:107) q φ ( T | R ))= H ( p, q φ ) − KL ( p, q φ ) H ( T | c ( R )) = H ( p ( T | c ( R )) , q φ c ( T | c ( R ))) − KL ( p ( T | c ( R )) (cid:107) q φ c ( T | c ( R )))= H ( p c , q φ c ) − KL ( p c (cid:107) q φ c ) Where we abbreviate similarly as we did for thecontrol task. Specifically, we write φ for the probeparameters of control function to tell apart from θ in the control task.Pimentel et al. (2020) showed that the errorof their approximation, ∆ p = G ( T, R, c ) − ˜ G ( T, R, c ) , can be expressed as: ∆ p = KL ( p (cid:107) q φ ) − KL ( p c (cid:107) q φ c ) (2)Again, when the probe fits the true distributionto a similar extent on both the target labels distri-bution and the probing task, the ∆ p will be small.Unfortunately, both KL terms are intractable too. From Equations 1 and 2, we showed that the selec-tivity criterion of Hewitt and Liang (2019) and the information gain criterion of Pimentel et al. (2020),if both measured in cross entropy loss, have verysimilar errors in approximating information gains.These errors, ∆ h and ∆ p respectively, appearin very similar forms . The probes selected fromthese two criteria should be highly correlated toeach other, and our experiments will confirm. When the two c ( · ) are ideal, ∆ h and ∆ p differ by onlyirrelevant terms – we include the derivations in SupplementaryMaterial. anguage train / dev / test ( t acc , f ent ) ( t acc , t ent ) ( f acc , f ent )English 17 177k / 22k / 22k 0.1615 0.1334 0.1763French 15 303k / 31k / 8k 0.0906 0.0606 0.1295Spanish 16 341k / 33k / 11k 0.1360 0.0560 0.1254 Table 1: Spearman correlations between t acc (the “selectivity” criterion (Hewitt and Liang, 2019)) and f ent (the“gain” criterion (Pimentel et al., 2020)) are on par with two “accuracy vs. cross entropy” correlations.
We use the same family of probes as Hewitt andLiang (2019) and Pimentel et al. (2020), multiplelayers perceptrons with ReLU activations, to showthe correlations of their “good probe” criteria (con-trol task and control function, respectively).Overall, we sweep the probe model hyper-parameters with a unified training scheme on threetasks (probe, control task, control function). Thecontrol task (function) setting includes labels (em-beddings) drawn from a uniform random sampleonce before all experiments. In each training, wefollow the setting of (Hewitt and Liang, 2019). Wesave the model with the best dev loss, report thetest set loss and accuracy, and average across 4different random seeds.
Data
We use the Universal Dependency (Zemanet al., 2019) dataset loaded with the Flair toolkit(Akbik et al., 2018). We examine three languages:English, French, and Spanish. For the probing task,we use POS with labels provided by SpaCy . Weuse the embedding of multilingual BERT (mBERT)implemented by huggingface (Wolf et al., 2019).If a word is split into multiple word pieces, weaverage its representations. When measuring the qualities of probes using the“selectivity” (Hewitt and Liang, 2019) or “informa-tion gain” (Pimentel et al., 2020) criterion, we showthat the rules-of-thumb for training good probeslargely agree. • Early stopping before 24,000 gradient steps(approximately 4 epochs) could inhibit probequality, but longer training procedures do notimprove the probe qualities considerably. • Smaller probes are better in general, but excep-tions exist. For example, when weight decay https://spacy.io is set to 0, probes with one hidden layer and40 hidden neurons are better in both criteria. • A small weight decay is beneficial.We include more descriptions, including compre-hensive experiment configurations and plots in theSupplementary Material.
In addition to the qualitative correlations shownabove, we compute the correlations betweenthe two criteria over a grid-search style hyper-parameter sweep of over 10,000 configurations.For each “probe, control task, control function” ex-periment set, we record the following four criteria: • t acc : Difference between probing task andcontrol t ask acc uracy. This is the “selectivity”criterion of Hewitt and Liang (2019). • f ent : Difference between control f unctionand probing task cross ent ropy. This is the“gain” criterion of Pimentel et al. (2020). • t ent : Difference between the control t ask andthe probing task cross ent ropy. • f acc : Difference between the probing taskand control f unction acc uracy.We collect all experiments of each language accord-ing to these criteria, and use Spearman correlationto test three pairs of correlations. As is reported inTable 1, the (t acc, f ent) correlations are compara-ble to two strong baselines, (t acc, t ent) and (f acc,f ent), the correlations between measurements inaccuracy and cross entropy losses. When selecting probes that better approximate I ( T ; R ) , we recommend measuring with a controlmechanism instead of relying on the traditionalcross entropy on probing task. We show bothinformation-theoretically and empirically, that con-trolling the targets and representations are equiva-lent, as long as the control mechanism is random-ized. Acknowledgement
We would like to thank Mohamed Abdalla for hisinsights and discussion. Rudzicz is supported by aCIFAR Chair in artificial intelligence.
References
Yossi Adi, Einat Kermany, Yonatan Belinkov, OferLavi, and Yoav Goldberg. 2017. Fine-grained anal-ysis of sentence embeddings using auxiliary predic-tion tasks. In
ICLR , Toulon, France.Alan Akbik, Duncan Blythe, and Roland Vollgraf.2018. Contextual String Embeddings for SequenceLabeling. In
COLING , pages 1638–1649.Guillaume Alain and Yoshua Bengio. 2017. Under-standing intermediate layers using linear classifierprobes. In
ICLR , Toulon, France.Christoph Alt, Aleksandra Gabryszak, and LeonhardHennig. 2020. Probing Linguistic Features ofSentence-Level Representations in Neural RelationExtraction. In
ACL . Association of ComputationalLinguistics.Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, HassanSajjad, and James Glass. 2017. What do Neural Ma-chine Translation Models Learn about Morphology?In
ACL , pages 861–872, Stroudsburg, PA, USA. As-sociation for Computational Linguistics.Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019.Evaluation Benchmarks and Learning Criteria forDiscourse-Aware Sentence Representations. In
EMNLP , pages 649–662, Stroudsburg, PA, USA. As-sociation for Computational Linguistics.Ethan A Chi, John Hewitt, and Christopher D Manning.2020. Finding Universal Grammatical Relations inMultilingual BERT. In
ACL . Association for Com-putational Linguistics.Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofDeep Bidirectional Transformers for Language Un-derstanding. In
NAACL , pages 4171–4186, Min-neapolis, Minnesota. Association for ComputationalLinguistics.John Hewitt and Percy Liang. 2019. Designing andInterpreting Probes with Control Tasks. In
EMNLP ,pages 2733–2743, Hong Kong, China. Associationfor Computational Linguistics.John Hewitt and Christopher D Manning. 2019. AStructural Probe for Finding Syntax in Word Repre-sentations. In
NAACL , pages 4129–4138, Minneapo-lis, Minnesota. Association for Computational Lin-guistics.Nora Kassner and Hinrich Sch¨utze. 2020. Negated andMisprimed Probes for Pretrained Language Models:Birds Can Talk, But Cannot Fly.
ACL . Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019.Open Sesame: Getting Inside BERT’s LinguisticKnowledge.
ACL BlackBoxNLP Workshop .Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.2016. Assessing the Ability of LSTMs to LearnSyntax-Sensitive Dependencies.
TACL , 4:521–535.Nelson F. Liu, Matt Gardner, Yonatan Belinkov,Matthew E. Peters, and Noah A. Smith. 2019. Lin-guistic Knowledge and Transferability of ContextualRepresentations. pages 1073–1094, Minneapolis,Minnesota. Association for Computational Linguis-tics.Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel,Adina Williams, and Ryan Cotterell. 2020. A Taleof a Probe and a Parser. In
ACL . Association forComputational Linguistics.Fabio Petroni, Tim Rockt¨aschel, Sebastian Riedel,Patrick Lewis, Anton Bakhtin, Yuxiang Wu, andAlexander Miller. 2019. Language Models asKnowledge Bases? In
EMNLP , pages 2463–2473,Hong Kong, China. Association for ComputationalLinguistics.Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay,Ran Zmigrod, Adina Williams, and Ryan Cotterell.2020. Information-Theoretic Probing for Linguis-tic Structure. In
ACL . Association of ComputationalLinguistics.Nina Poerner, Ulli Waltinger, and Hinrich Sch¨utze.2019. BERT is Not a Knowledge Base (Yet): Fac-tual Knowledge vs. Name-Based Reasoning in Un-supervised QA.Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Languagemodels are unsupervised multitask learners.
OpenAIBlog , 1(8):1–24.Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a.BERT Rediscovers the Classical NLP Pipeline. In
ACL , pages 4593–4601, Florence, Italy. Associationfor Computational Linguistics.Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang,Adam Poliak, R. Thomas McCoy, Najoung Kim,Benjamin Van Durme, Samuel R. Bowman, Dipan-jan Das, and Ellie Pavlick. 2019b. What do youlearn from context? Probing for sentence structurein contextualized word representations. In
ICLR .Elena Voita and Ivan Titov. 2020. Information-Theoretic Probing with Minimum DescriptionLength. arXiv preprint arXiv:2003.12298 .Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. HuggingFace’s Trans-formers: State-of-the-art Natural Language Process-ing.
ArXiv , abs/1910.0:1–11.adollah Yaghoobzadeh, Katharina Kann, T J Hazen,Eneko Agirre, and Hinrich Sch¨utze. 2019. Prob-ing for Semantic Classes: Diagnosing the Mean-ing Content of Word Embeddings. In
ACL , pages5740–5753, Florence, Italy. Association for Compu-tational Linguistics.Daniel Zeman, Joakim Nivre, and Mitchell et alAbrams. 2019. Universal dependencies 2.5.LINDAT/CLARIAH-CZ digital library at the Insti-tute of Formal and Applied Linguistics ( ´UFAL), Fac-ulty of Mathematics and Physics, Charles Univer-sity.
A Difference between the two criteria
In Section 3 we show that the error of the two cri-teria can both be written as difference between apair of KL divergence (modulo a constant term).Here we further simplify the terms when we as-sume the control task and functions take perfectlyrandom distributions (i.e., independent of the taskand representations, respectively). ∆ h − ∆ p = Const + KL ( p ( c ( T ) | R ) (cid:107) q θ c ( c ( T ) | R )) − KL ( p (cid:107) q θ ) − KL ( p ( T | c ( R )) (cid:107) q φ c ( T | c ( R ))) + KL ( p (cid:107) q φ ) When we use the same hyperparameter setting, q θ ( T | R ) and q φ ( T | R ) should be able to approx-imate p ( T | R ) to the same extent, so KL ( p (cid:107) q θ ) and KL ( p (cid:107) q φ ) cancel out. Additionally, followingthe definitions of the control function and controltasks, we can simplify as follows: p ( c ( T ) | R ) = p ( c ( T )) , q θ c ( c ( T ) | R ) = q θ c ( c ( T )) p ( T | c ( R )) = p ( T ) , q φ c ( T | c ( R )) = q φ c ( T ) KL ( p ( c ( T ) | R ) (cid:107) q θ c ( c ( T ) | R )) = E p ( c ( T )) p ( c ( T )) q θ c ( c ( T )) KL ( p ( T | c ( R )) (cid:107) q φ c ( T | c ( R ))) = E p ( T ) p ( T ) q φ c ( T ) Therefore, the difference between errors of crite-ria of Hewitt and Liang (2019) and Pimentel et al.(2020) are: ∆ h − ∆ p = Const − KL ( p ( T ) (cid:107) q φ c ( T ))+ KL ( p ( c ( T )) (cid:107) q θ c ( c ( T ))) (3)In short, these two criteria differ by terms depen-dent only on the randomization functions and theinherent distributions of task labels, i.e., irrelevantterms. B Experiments details
Hewitt and Liang (2019) proposed some rules-of-thumb to select good probes, including a “simpleprobe” suggestion. We sweep the hyper parametersto test whether these rules also apply when mea-suring probe qualities using the control function(Pimentel et al., 2020). yper-parameter ranges
We sweep hyper pa-rameters from the following ranges: • Learning rate: { − , × − , × − , − , × − , × − }• Maximum gradient steps: { , , , , , , ∞} .Their effects are shown in Figure 1. • Weight decay: { , . , . , . } . Their ef-fects are shown in Figures 2, 3 and 4. Whensweeping weight decay, max gradient step isset to 24000.In any configuration mentioned above, we run fourexperiments with random seeds 73, 421, 9973,361091, and average the reported results (i.e., ac-curacy and loss). Early stopping could inhibit probe quality
Early stopping, if stopped before 24,000 gradientsteps (approximately 4 epochs) may inhibit thequality of probes. In addition, we Figure 1 showshigh correlation between the “selectivity” (Hewittand Liang, 2019) and “information gain” (Pimentelet al., 2020) criteria.
Small weight decay is beneficial
We find thatsmaller weight decays (e.g., 0.01) are more benefi-cial for probes than larger weight decays. While thetwo criteria rank the capacity of probes similarly,the most simple probes tend to stand out more dis-tinctively with the “selectivity” criterion (Pimentelet al., 2020), as are shown on Figures 2, 3, and 4.
Smaller probes are not necessarily better
Wefind that while smaller probes have higher “selec-tivity” and “information gain” for mBERT repre-sentations, probes with one hidden layer and 40-80 hidden neurons are better than more simplisticprobes, as shown in Figures 5, 6 and 7. The plotsshow consistency between the two criteria. Forexample, larger models and more layers do notnecessary lead to better results. Neither are thesmallest probes with 0 hidden layers.Note that we also swept hyperparameters forFastText, where probes with less parameters donot always outperform more complex probes ineither accuracy, loss, selectivity, or informationgain. Figures 8 and 9 illustrate these observations.
C Reproducibility
On a T4 GPU card, training one epoch takes around20 seconds. Without setting maximum gradientsteps, . of experiments finish within 400epochs. We will open source our codes.of experiments finish within 400epochs. We will open source our codes.