ArXiving Before Submission Helps Everyone
AArXiving Before Submission Helps Everyone
Dmytro Mishkin ∗ Visual Recognition Group,Faculty of Electrical Engineering,Czech Technical University in Prague [email protected]
Amy Tabb writing in her personal capacityU.S.A [email protected]
Jiri Matas
Visual Recognition Group,Faculty of Electrical Engineering,Czech Technical University in Prague [email protected]
Abstract
We claim, and present evidence, that allowing arXiv publication before a conferenceor journal submission benefits researchers, especially early career, as well as thewhole scientific community. Specifically, arXiving helps professional identitybuilding, protects against independent re-discovery, idea theft and gate-keeping; itfacilitates open research result distribution and reduces inequality. The advantagesdwarf the drawbacks – mainly the relative increase in acceptance rate of papersof well-known authors – which studies show to be marginal. Analyzing the prosand cons of arXiving papers, we conclude that requiring preprints be anonymous isnearly as detrimental as not allowing them. We see no reasons why anyone but theauthors should decide whether to arXiv or not.
Releasing research papers on the arXiv open repository [1] is becoming a standard in computerscience and other fields [2, 3, 4]. We argue that arXiving provides great benefits to both individualresearchers (Section 2) and a field as a whole (Section 3). Moreover, it helps researchers in all careerstages; we fail to find circumstances in which researchers do not benefit from arXiving their work.There are opinions, however, that this open science practice should be prohibited or limited toarXiving anonymized papers only. The beliefs underpinning these views are the following: 1. waiting3-6 months until a paper is accepted at a conference is a minor inconvenience, and 2. arXiv releaseslead to an "almost" single blind conference review process, which may reduce diversity and somestudies [37] claim that it favours people from well-known institutions.We address the first contra argument in Section 2 and single blind review issues in Section 4.
We start by discussing the benefits of arXiv to, especially, early career researchers (ECRs), from theopportunities that come from early released non-anonymous preprints . We will call such practice"arXiving" in the rest of the paper. ∗ Preprint. Under review. a r X i v : . [ c s . D L ] O c t c u m u l a t i v e % o f a cc e p t a n c e Competitor birth probability=0.2 acceptedscooped c u m u l a t i v e % o f a cc e p t a n c e Competitor birth probability=0.5 acceptedscooped c u m u l a t i v e % o f a cc e p t a n c e Competitor birth probability=0.8 acceptedscooped
Figure 1: Monte-Carlo simulation (10k runs) of NeurIPS paper acceptance for the different competi-tion levels, with probability 0.2 a paper is accepted. If not accepted, with probability P comp , then asimilar idea comes to another researcher and she submits it to NeurIPS. If the competitors’ paper isaccepted earlier, the outcome is "scooped". The paper is resubmitted until "accept" or "scooped". Professional identity building instead of blank C.V.
When arXiving early, authors, particularlyECRs, can build professional identities around the work instead of waiting until acceptance. Ac-tivities include events, collaborations, and building informal networks of support outside of theirinstitution [27]. All scholarly work can be listed on a C.V. – including preprints — to show researchoutput. While having the most recent publication online might not be important for a professor with200+ publications, for the early career researcher it is the step from zero to one. The same is true forthe time – 6 months can make career life-or-death difference for an ECR.Increased early visibility through preprints allows others to become aware of new authors. For instancerecruiters, who, in our experience, contact candidates; it is not candidates sending applications. Otherexamples include fellow researchers who are looking for speakers at local meetups, non-academicconferences, workshops, departmental seminars, and so on.In the "wait until acceptance" scenario, an ECR could end up with a blank C.V. for a few years . Thisis even more significant now, when publications are required to enter the PhD program . Protection against independent re-discovery or idea theft.
Even if a paper is well-written and hasa useful contribution, there is still a high chance that it will be rejected. The review process, evenat the best conferences, is known to be quite random [24]. Low acceptance rates of 20-30% [25] atCVPR and NeurIPS-like top conferences exacerbate the decision noise problem.The “wait 3-6 months” argument does not account for the research "arms race," meaning that a papermay be scooped before acceptance. By "scooping" we mean that another researcher independentlycomes up with the same or very similar idea and publishes it earlier. We modeled the paper submissionas a Markov process, for details see Figure 1. “Wait 3-6 months” is an understatement – it is more like"wait 1-3 years and pray to be lucky." In addition to independent re-discovery, a similar idea couldbe published by the reviewer of the submission. Not because of malicious intent – most reviewersrespect the strict confidentiality of the review process – but because of unconscious idea generation,triggered by reading the submission.If an ECR waits for the paper to be accepted and it gets scooped, then there is no way to prove thatthey had the original idea months or years before. This is extremely unfair and painful, especially ifthe idea becomes influential and highly cited.On the other hand, if the researcher uploads the work on arXiv, they get a priority timestamp and canstart to get cited. Even if someone else presents a similar idea at a conference first, the affected ECRis in a much better position with arXiv.One of the examples of protection by arXiving is a story about SiLU [21] vs Swish [29] activation,which are exactly the same function. SiLU was first proposed in an arXiv paper in June 2016 byHendrycks and Gimpel and rejected from ICLR 2017 [5]. Later, in 2017, the same activation wasproposed as Swish by a Google Brain group and it was accepted to the ICLR 2018 Workshop track [6].The SiLU was unnoticed, while Swish gained popularity. The arXiv publication helped to establishpriority and give the activation function the original name and reference, SiLU, in deep learningframeworks [7, 8].
Protection against gate-keeping.
ECRs, especially from non-mainstream labs, may have difficultywriting papers such that they conform to the norms of the community, in terms of framing their ideasand expressing them with established vocabulary. This is even more true for people with non-ML/CV2ackgrounds and novel ideas. Their papers could be described as “The Puppy with 6 toes" [19] andare easily rejectable. Without arXiv, review gatekeeping is a single point of failure, which is hard topass even for experienced researchers.Famous and less famous examples are:• The most cited computer vision paper for decades, SIFT [26], took three years to get accepted, firstsubmitted in 1997, accepted as a poster in 1999, journal version published in 2004 [23].• SqueezeNet [22] was rejected from ICLR because, the “novelty of the submission is very limited,"[9], while already having wide adoption and >100 citations. Now it has almost 3000 citations onGoogle Scholar despite never having been officially published in a conference or journal.• Now famous one-cycle learning rate policy [32] and super convergence [34] papers by LeslieSmith. Smith released the original work on arXiv in 2015 [31]. The first paper was eventuallypublished in WACV 2017 [30]. Its follow-up work, the super-convergence [33] paper was rejectedfrom ICLR-2018 [10]. Then the fast.ai team read and implemented it in their framework. Later,in 2019, it was accepted at a defence-related conference [34] little-known in the computer visioncommunity.
Distribution; arXiv has become the main "new results feed," from which people discover newwork . A whole ecosystem has been evolving around arXiv. Services like arxiv-sanity [11], arxiv-vanity [12], Papers-with-Code [13], arxiv-daily-type twitter accounts and recent Papers-with-Code-arXiv integration [15, 36] allow researchers cope with the flood of published papers and supplementarymaterials. Moreover, arXiv is monitored by many practitioners, who are not concerned about whetherthe work has been endorsed by others, if there is evidence that a method is working well in practice.Work which is not on arXiv reaches much fewer people. There is also a difference if the paper isarXived before or after acceptance. According to Feldman et al . [18], "papers submitted to arXivbefore acceptance have, on average, 65% more citations in the following year compared to paperssubmitted after".
Faster and broader distribution and better explanation.
The most obvious benefit of arXiving isthe distribution of authors’ research to the community. This research and ideas are often clarified andexplained through interaction or blog posts. Such explanatory and presentation work is especiallyimportant for new ideas [20] and is logistically much more difficult for anonymous work.Public code and data releases are without the administrative overhead of keeping track of anonymityat all stages. While research results can be distributed in an anonymous preprint, the need to maintainanonymity restricts feedback and code sharing.
Open access and Funding.
While companies finance research from their own profits, most universityresearch is done on taxpayers’ money. This brings, at the very least, an obligation to share researchresults in a timely manner. Relatedly, arXiv serves an important role for public access to documentsfor grant reports and public talks.
Crowdsourcing review of preprints is the way of Open Science and it tackles more aspects thantraditional review.
Let us recall all the discussions and critique about the GPT-2 [28] and GPT-3 [17]models by OpenAI. While the opinions on GPT2/3 and its impact itself might vary, it is hard to denythat GPT2/3 started a large community-wide discussion concerning long-term impact, and machinelearning bias in general. These discussions arose on Twitter and blogs at the time that the technicalreport and later, code, was released for GPT-2, which was posted only as an OpenAI tech reportand similarly to GPT-3, released on arXiv. Would three standard reviews from NeurIPS lead to adiscussion with the same scale and impact? arXiv levels the playing field and reduces famous labs’ advantages.
Famous labs have experi-enced researchers, especially in writing papers, huge hardware resources, etc. Thus, someone from ahypothetical FAANG lab has extensive experience in how to write a “may be boring, but hard tokill” paper [19]. This acronym refers to the most prominent tech companies: Facebook, Amazon, Apple, Netflix, Google.
Пока толстый сохнет худой сдохнет ”, – "While the fat one dries, the thin one dies".Moreover,
ECRs are more dependent on having formal publications in their C.V. than seniorresearchers. If pre-submission arXiving was not allowed at major conference, ERCs would findit very difficult, unlike senior researcher, to trade early publication, code release and impact for amissed opportunity to publish at a top conference.
Single-vs-double blind studies – the bias reduction is small.
The research quoted in some anti-arXiv arguments are studies by Tomkins et al . [37] about WSDM-2017 and the study by Bharadhwaj etal . [16] by about ICLR 2019-2020. Before delving into details, let us summarize both. They report,via sophisticated statistics, that deanonymization could change the chances of acceptance by 3-4 orless percentage points. In comparison, Lawrence and Cortes [24] have found that the randomnessof the review process itself is around 50%, which is an order of magnitude bigger. We argue, thatgiven such a level of noise, it does not make any sense to trade off all the benefits arXiv provides(Sections 2, 3) for removing possible bias, which is 10x less that the randomness of the peer reviewitself.Besides, the first study contains methodological flaws, precisely explained in [35]. Specifically,Stelmakh et al . [35] have shown that "the test used by Tomkins et al . [37] can, with probability aslarge as 0.5 or higher, declare the presence of a bias when the bias is in fact absent (even when thetest is tuned to have a false alarm error rate below 0.05)." Moreover, "two factors – (d) asymmetricalbidding procedure and (e) non-random assignment of papers to referees – as is common in peer-reviewprocedures today may introduce spurious correlations in the data, breaking some key independenceassumptions and thereby violating the requisite guarantees on testing." We point the reader to [35]for more details and focus on the more recent study by Bharadhwaj et al . [16].The results can be simplified as follows. Bharadhwaj et al . have found a correlation (authorsspecifically emphasize that the study does not claim any causality) between pre-acceptance arXivrelease and acceptance rate. Specifically, releasing a paper on arXiv might increase its chances ofacceptance up to 5 percentage points in the best case for almost everyone, or decrease your chancesby 3 percentage points if the authors are totally unknown. Given that the acceptance rate at ICLR is20-25%, we argue that the benefits from having preprints, as stated in Section 2, protecting you in thevery likely case of rejection, is much more important for an ECR than a possible 3 percentage pointrejection risk increase.
Authors should not be the only ones, who ensure the unbiased review.
Instead of putting all theburden (e.g. by requiring anonymous arXiving) on the authors, effort might be directed to reviewertraining. The computer vision community already started to do this by hosting CVPR Tutorial onwriting good reviews [14]. Another direction is to adjust the review process to minimize possiblebiases. For example, the reviewers might be asked to read the paper and write the initial reviewwithout googling the paper to avoid possible influence of knowing the author names. Such process isnot uncommon in grant proposal reviewing. After the initial review, one could perform google andliterature search for the related work etc.
Double-blind review combined with banning non-anonymous preprints does not come for free. Whilepossibly removing a small bias towards "big names", it introduces other biases and limitations. Suchlimitations are especially harmful for those who are not yet fully established.Is it worthwhile to do arXiving early? We argue that the default research workflow should be thefollowing: once researchers write-up the research piece, they decide if want to submit to a conference,4elease on arXiv or do both. The choice depends on the particular situation, and only the researchersthemselves should decide how to share their work.Finally – we have likely missed some aspects of this imporant issue and we hope that this paper willbe a good starting point for a community-wide discussion. We will post updated versions on arXiv.
Broader Impact
The submission is a position paper, discussing the impact of some aspects of submission and reviewingrules on individual researchers and the research community. The topic of the paper, and thus itscontent, falls under the "Broader Impact" rubric.
References [1] https://arxiv.org/ .[2] .[3] .[4] https://psyarxiv.com/ .[5] https://openreview.net/forum?id=Bk0MRI5lg .[6] https://openreview.net/forum?id=SkBYYyZRZ .[7] .[8] https://pytorch.org/docs/master/generated/torch.nn.SiLU.html .[9] https://openreview.net/forum?id=S1xh5sYgx¬eId=SknRsfUue .[10] https://openreview.net/forum?id=H1A5ztj3b¬eId=B1BgUy6BM .[11] .[12] .[13] https://paperswithcode.com/ .[14] Cvpr 2020 tutorial "how to write a good review?". https://sites.google.com/view/making-reviews-great-again/ .[15] New arxivlabs feature provides instant access to code. https://blog.arxiv.org/2020/10/08/new-arxivlabs-feature-provides-instant-access-to-code/ , 2020.[16] Homanga Bharadhwaj, Dylan Turpin, Animesh Garg, and Ashton Anderson. De-anonymization of authorsthrough arxiv submissions during double-blind review. arXiv preprint arXiv:2007.00177 , 2020.[17] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, ArvindNeelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss,Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, ClemensWinter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, JackClark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Languagemodels are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.[18] Sergey Feldman, Kyle Lo, and Waleed Ammar. Citation count analysis for papers with preprints. arXiveprint:1805.05238 , 2018.[19] Bill Freeman. How to write a good cvpr submission. https://billf.mit.edu/sites/default/files/documents/cvprPapers.pdf , 2014.[20] Richard R Hamming.
Art of doing science and engineering: Learning to learn . CRC Press, 1997.[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv eprint:1606.08415 , 2016.[22] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer.Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size. arXiv preprintarXiv:1602.07360 , 2016.[23] Scott Krig.
Computer Vision Metrics: Textbook Edition . Springer, ISBN: 978-3-319-33762-3, 2016.[24] Neil Lawrence and Corinna Cortes. The nips experiment. http://inverseprobability.com/2014/12/16/the-nips-experiment/ , 2014.[25] Xin Li. Statistics of acceptance rate for the main ai conferences. https://github.com/lixin4ever/Conference-Acceptance-Rate , 2020.
26] D.G. Lowe. Object Recognition from Local Scale-Invariant Features. In
ICCV , 1999.[27] Beronda L. Montgomery. Building and sustaining diverse functioning networks using social media anddigital platforms to improve diversity and inclusivity.
Frontiers in Digital Humanities , 5:22, 2018.[28] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language modelsare unsupervised multitask learners. 2019.[29] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. arXiveprint:1710.05941 , 2017.[30] L. N. Smith. Cyclical learning rates for training neural networks. In , pages 464–472, 2017.[31] Leslie N. Smith. Cyclical learning rates for training neural networks. arXiv preprint arXiv:1506.01186 ,2015.[32] Leslie N Smith. A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batchsize, momentum, and weight decay. arXiv preprint arXiv:1803.09820 , 2018.[33] Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using largelearning rates. arXiv preprint arXiv:1708.07120 , 2017.[34] Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using largelearning rates. In
Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications ,2019.[35] Ivan Stelmakh, Nihar Shah, and Aarti Singh. On testing for biases in peer review. In H. Wallach,H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, editors,
Advances in NeuralInformation Processing Systems 32 , pages 5286–5296. Curran Associates, Inc., 2019.[36] Robert Stojnic. Papers with code partners with arxiv. https://medium.com/paperswithcode/papers-with-code-partners-with-arxiv-ecc362883167 , 2020.[37] Andrew Tomkins, Min Zhang, and William D. Heavlin. Reviewer bias in single- versus double-blind peerreview.
Proceedings of the National Academy of Sciences , 114(48):12708–12713, 2017., 114(48):12708–12713, 2017.