Characterising authors on the extent of their paper acceptance: A case study of the Journal of High Energy Physics
Rima Hazra, Aryan, Hardik Aggarwal, Matteo Marsili, Animesh Mukherjee
CCharacterising authors on the extent of their paper acceptance:A case study of the Journal of High Energy Physics
Rima Hazra
Indian Institute of TechnologyKharagpur, [email protected]
Aryan
Indian Institute of TechnologyKharagpur, [email protected]
Hardik Aggarwal
Indian Institute of TechnologyKharagpur, [email protected]
Matteo Marsili
ICTPTrieste, [email protected]
Animesh Mukherjee
Indian Institute of TechnologyKharagpur, [email protected]
ABSTRACT
New researchers are usually very curious about the recipe thatcould accelerate the chances of their paper getting accepted ina reputed forum (journal/conference). In search of such a recipe,we investigate the profile and peer review text of authors whosepapers almost always get accepted at a venue (Journal of HighEnergy Physics in our current work). We find authors with highacceptance rate are likely to have a high number of citations, high h -index, higher number of collaborators etc. We notice that theyreceive relatively lengthy and positive reviews for their papers.In addition, we also construct three networks – co-reviewer, co-citation and collaboration network and study the network-centricfeatures and intra- and inter-category edge interactions. We findthat the authors with high acceptance rate are more ‘central’ inthese networks; the volume of intra- and inter-category interactionsare also drastically different for the authors with high acceptancerate compared to the other authors. Finally, using the above setof features, we train standard machine learning models (randomforest, XGBoost) and obtain very high class wise precision andrecall. In a followup discussion we also narrate how apart from theauthor characteristics, the peer-review system might itself have arole in propelling the distinction among the different categorieswhich could lead to potential discrimination and unfairness andcalls for further investigation by the system admins. KEYWORDS
Peer review system, JHEP, co-reviewer network, co-citation net-work, collaboration network
ACM Reference Format:
Rima Hazra, Aryan, Hardik Aggarwal, Matteo Marsili, and Animesh Mukher-jee. 2020. Characterising authors on the extent of their paper acceptance:A case study of the Journal of High Energy Physics. In
Proceedings of the
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected].
JCDL ’20, August 1–5, 2020, Virtual Event, China © 2020 Association for Computing Machinery.ACM ISBN 978-1-4503-7585-6/20/06...$15.00https://doi.org/10.1145/3383583.3398527
ACM/IEEE Joint Conference on Digital Libraries in 2020 (JCDL ’20), Au-gust 1–5, 2020, Virtual Event, China.
ACM, New York, NY, USA, 10 pages.https://doi.org/10.1145/3383583.3398527
Publishing new research in journals/conferences is a common prac-tice in the scientific community. It is noticed that papers of fewauthors consistently get accepted in journals whereas papers ofcertain other authors get rarely accepted . An intriguing questionthus is what makes the papers of certain authors almost alwayseligible for acceptance. Is there a special recipe that they follow inpreparing their manuscripts? Does it depend on their position inthe collaboration/citation network? Does their experience or their h -index matter? Does the diversity in the topics that they workon help escalate the acceptance? The present paper attempts todelve into some of these questions and characterise authors basedon their paper acceptance profile. We base our investigations ona dataset obtained from the Journal of High Energy Physics thathas information about authors, papers written by them, citationsobtained by them and the review reports written by expert refereesfor each of their accepted paper. The overall peer review workflowfor this journal is illustrated in Figure 1. In a nutsell the workflowis as follows – once an author submits a paper, the system allocatesthe submission to an editor based on a simple keyword matchingtechnique. The editor then handles the paper and chooses one ormore competent referees who are experts in the area and can judgethe technical merit of the paper. The referee(s) in turn read thepaper and send their review report(s) to the editor. The editor readsthe review(s) and takes a decision to either accept, reject or invitethe authors to revise and resubmit. The revise and resubmit deci-sion re-instantiates the same workflow described above once againand the cycle continues until the paper is eventually accepted orrejected.We categorize the authors in this dataset into three classes basedon the fraction of their papers accepted to the journal. We calculatethe acceptance rate ( ACC ) of an author as the ratio of the number ofpapers accepted to the number of papers submitted by the authorto the journal. For each of the three categories (discussed below)we analyse a bunch of interesting features that are drawn fromthe collaboration/citation network of an author as well as the peer a r X i v : . [ c s . D L ] J un eviews received by the different accepted papers of the authors.We find that these features are considerably different across thethree ACC classes.
Figure 1: The JHEP peer review workflow.
We categorize the authors into three classes based on their accep-tance rate. Authors whose papers are consistently accepted forpublication and have high
ACC are placed in the class
ACC hiдh ;authors whose papers are rarely accepted and have low
ACC areplaced in the class
ACC low and authors who are neither in
ACC hiдh ,nor in
ACC low and have moderate
ACC are placed in
ACC mid . Weexplain the process of author categorization in details in section 3.Our main contributions are threefold.(1) Rigorous analysis of the profile and peer review based fea-tures of authors belonging to each category.(2) Analyzing inter-category and intra-category interaction andnetwork-centric properties obtained from three differentnetworks – (i) the co-reviewer network (
CRN ), (ii) the col-laboration network (
CON ) and (iii) the co-citation network(
CCN ).(3) Early prediction of an author’s category based on the profile,peer review data and network-centric features.Toward the first objective, we extract various features represent-ing an author. These features are divided into two types – (i) au-thor’s profile based features ( AP f ) and (ii) features based on peerreview data ( PE f ). Author’s profile based features ( AP f ) comprisescitation count ( C cnt ), topic diversity ( T div ), experience ( E cnt ) and h -index [10] ( H ind ). Peer review based features ( PE f ) consists ofsentiment of review text ( SNT r ), length of the review text ( L r ),reviewer diversity ( R div ) and editor diversity ( Ed div ).In addition, we extract various features – centrality values, clus-tering coefficient, core-periphery structure etc. from the three differ-ent type of networks mentioned above. These networks are definedbelow. (i) Co-reviewer network ( CRN ) : In this article, we introduce aco-reviewer network. Each author is considered as a node in thenetwork and two authors are connected by an edge if their papersare reviewed by the same reviewer. In addition, we also prepare an induced co-reviewer graph for the three different author categories. (ii) Collaboration network ( CON ) : Each author in this networkis considered as a node and two authors are connected by an edgeif they co-authored in a paper. We also prepare the induced collab-oration networks of the authors of each category. (iii) Co-citation network ( CCN ) : In this directed network, eachauthor is considered as a node and two authors are connected byan edge ( a i −→ a j ) if author a i has cited an article authored by a j .There is bidirectional edge ( a i ←→ a j ) if author a i and author a j cites each other.For our experiments, we consider the authors who have sub-mitted their paper to the Journal of High Energy Physics (JHEP)between 1997 to 2015. We consider approx. 29k papers and morethan 24k authors. We also have approx. 70k unique review reports. A nuanced analysis shows that authors in the class
ACC hiдh usuallyreceive more citations than the other two categories. We also notethat papers of the
ACC low authors receive more citation if theycoauthored with
ACC hiдh authors in some paper.
ACC hiдh authorsalways receive more positive reviews than the other two categories.An intriguing observation is that the set of referees and editors towhom the papers of the
ACC hiдh class are assigned are found to beless diverse than the other two classes. The
ACC hiдh authors aremore ‘central’ in all the networks. We finally make early predictionsof the
ACC category of an author and obtain 0.82 - 0.95 precisionand 0.82 - 0.91 recall. In a followup discussion we narrate how apartfrom the author characteristics, the peer review system itself canpotentially facilitate discrimination in the editing and the reviewingprocess of papers in the three categories which could reinforce thedistinction between the authors of these categories and calls forfurther investigation by the system admins.
The rest of the paper is organised as follows. Section 2 describesthe dataset used in this paper. Section 3 details the method for au-thor categorization. Section 4 and 5 demonstrate the author profilefeatures and peer review based features respectively. In section 6,we discuss the network features of the three category of authors.In section 7 we predict the category of the authors. In section 8discuss the potential role of the peer review system in enhancingthe distinction among the three categories of authors. Section 9presents a brief literature review. Finally, we conclude in section 10.
In our article, we consider papers submitted to the Journal of HighEnergy Physics (JHEP) in between 1997 and 2015. JHEP is oneof the leading journals in the domain of high energy physics. InJHEP, the identity of the referee remains confidential. This datasetcontains a total of 28871 papers, where the number of acceptedand rejected papers are 20384 and 6190 respectively. We also have70000 unique peer review reports. For each paper we have the title,author names, broad topics that the paper is on, publication date (incase it was accepted) and the number of citations for the acceptedpapers. In addition, this dataset contains the review text, number of https://jhep.sissa.it/jhep/ eview rounds, editor and reviewer ids (anonymised) of each paper.We also have the citation link among the papers. For the rejectedpapers, we collected the arXiv id using the Inspire search engine.We consider the cumulative number of citations obtained at theend of 2015. We present a brief statistics of the dataset in Table 1. Table 1: Dataset description.
Basic Information Count
In this section, we categorize authors’ profile into three categoriesbased on their articles’ acceptance rate (
ACC ) – (i) authors withhigh acceptance (
ACC hiдh ) (ii) authors with moderate acceptance(
ACC mid ) (iii) authors with low acceptance (
ACC low ). Acceptancerate of an author is calculated as the ratio of the number of papersaccepted to number of papers submitted by that author. We calculatearticle acceptance rate of each author for every year. In case of
ACC hiдh category, we consider only those authors who have highacceptance rate ( > . ) in at least 70% of the years over all the years. ACC low category contains authors who have very low acceptancerate ( < . ) in at least 80% of the years. We keep the rest of theauthors (not falling in the other two categories) in ACC mid category.Statistics of the unique authors are given in Table 2. The number ofaccepted and rejected papers in each class are noted in Figure 2. Thepapers of authors in the
ACC hiдh class almost always get accepted.
Table 2: Statistics of author categorization.
Author Categories
ACC hiдh
ACC mid
ACC low AP f )4.1 Citation index ( C ind ) Citation count of each author is computed by considering the totalnumber of citations an author received in their active period. Foreach category, we define citation index as the standard deviation ofcitation counts of all the authors. We compute C ind for three cate-gories ACC hiдh , ACC mid and
ACC low (see Figure see Figure 4(d)).There is a stark difference in the values of C ind among the threecategories. Authors in the class ACC hiдh have low C ind (approx.50) whereas authors in the class ACC low have high C ind (approx.101). Thus, the citation counts in the class ACC hiдh are far moreuniform across the authors compared to the
ACC low class. http://arxiv.org https://inspirehep.net Figure 2: Percentage of accepted and rejected papers of
ACC hiдh (High),
ACC mid (Moderate) and
ACC low (Low) au-thors. E cnt ) Experience of an author is defined in terms of number of papershe has published. For each category, we compute experience ofall the authors and consider the mean of these E cnt s. We observe ACC hiдh has highest mean experience (see Figure 4(a)).
ACC mid has moderate value of mean experience whereas
ACC low has verylow mean experience (see Figure 4(a)). From this, it is clear that
ACC low category contains those authors who are either new inresearch or has very few publications. T div ) We consider a topic set for each author. This topic set contains allthe topics on which an author published their papers. For eachauthor we compute topic ratio as the ratio of the total number oftopics on which he/she has written a paper to the total number ofpapers he/she published. For each category, we consider mean over topic ratio of all the authors to compute topic diversity ( T div ) (seeFigure 4(e)). Interestingly, ACC hiдh category authors have less T div (1.03) than the other two categories ( ACC mid has 1.36 and
ACC low has 1.57). We observe that
ACC low category authors publish paperson a lot of topics whereas
ACC hiдh authors focus on a relativelyless number of topics and publish a large number of papers in thosetopics. h -index ( H ind ) The h -index [10] is defined as the maximum value of h such thatan author has published h papers that have each been cited at least h times. For all the three categories, we consider mean of the H ind of all the authors. From Figure 4(b), it is clear that ACC hiдh havevery high mean H ind compared to the other two categories. Thusthe ACC hiдh class usually comprises the high impact authors.
T S ) Team size of an author is calculated as the number of contributingco-authors averaged across all the papers that the particular authorhas written. We examine mean team size for each category (see igure 3: (Left) This collaboration network includes
ACC hiдh , ACC mid and
ACC low authors. (Right) This collaboration networkincludes
ACC hiдh and
ACC mid authors only for better visualisation.
Figure 4(c)).
ACC hiдh and
ACC mid authors have mean team sizesof 2.44 and 2.15. The typical team sizes for both these classes arevery similar. On the other hand, we find that the mean team sizeof
ACC low is ∼ .
61 which is quite low compared to the other twoclasses.
P F f )5.1 Sentiment of review text ( SNT r ) We compute the sentiment score [− , ] of each review text for eachpaper . For every author we compute the average review sentimentacross all the papers (s)he has written. For every class, we take themean of these average values across all the authors in that class(see Figure 5(a)). Among the three classes, the review text bearsthe highest positive sentiment (0.15) in the ACC hiдh class. Thisis followed by
ACC mid class where the overall sentiment is 0.05.Finally, the review texts corresponding to the
ACC low class indicatethe presence of high negative sentiment (− . ) . L r ) Length of review text is computed as the number of words presentin the review text except stop-words (see Figure 5(b)). Surprisingly,we find that
ACC hiдh category receive relatively lengthier reviews(2368) compared to
ACC low (1305). It is therefore quite clear that pa-pers in the
ACC hiдh class typically receive more detailed feedbackfrom the referees compared to the
ACC low class. https://textblob.readthedocs.io/en/dev/ R div ) We use Shannon index [14] to calculate the reviewer diversity. Foreach author in a particular category, we extract the reviewer idsof all his/her published papers and add it to a global list. Thus wehave three global lists for each of the three categories. Next, foreach category, we compute the entropy of this global list. Let thesize of the global list for a category be N and let the number ofoccurrences of a reviewer r i in the list be f i . Then the entropywould be − (cid:205) ∀ i f i N loд ( f i N ) . If the value of this entropy is low thenthis would mean that the number of reviewers to whom the papersof a class go for review are very limited. In contrast, if this value ishigh for a class then it would mean that many reviewers are assignedas referees for the papers in the class (see Figure 5(c)). Surprisingly,we notice that ACC hiдh has less reviewer diversity (∼ . ) thanthe other two categories. ACC mid and
ACC low categories havereviewer diversity 7.36 and 7.34 respectively. This possibly indicatesthat for the
ACC hiдh class the set of referees are relatively morefixed and papers of authors from this group usually go to other peerauthors (in the role of referees) mostly from this group itself for areview. This, we believe, is a sign of unhealthy reviewing practice.We shall discuss more about this in section 8. Ed div ) Once again we use Shannon index [14] to calculate editor diversity.We compute this metric exactly as R div with the exception thathere the three global lists are composed of editor ids to whom thepapers are assigned (as opposed to reviewer ids in the previouscase). Here also we observe that editor diversity of ACC hiдh is quitelow ∼ .
94; on the other hand, the editor diversity of
ACC mid and
ACC low classes are relatively higher ∼ .
07 and ∼ .
01 respectively igure 4: (a) The mean experience ( E cnt ) of ACC hiдh (High),
ACC mid (Moderate) and
ACC low (Low). (b) The mean h -index forthe three categories. (c) The mean team size ( TS ) for the three categories. (d) Citation index ( C ind ) of the three categories. (e)Topic diversity ( T div ) for the three categories. (see Figure 5(d)). It seems that the same set of editors handle thepapers of the ACC hiдh class.
LQI ) Here we analyze the different emotions (positive, optimism, cheer-fulness, confusion and contentment) reflected by each word presentin the review text . Then we take the mean of the emotion val-ues of words present in a particular review text and average itover all authors in a class. We find quite a few interesting results.There are more positive emotion words in the review texts of the ACC hiдh class (0.018) compared to the
ACC low class (0.015). Fur-ther, there are more optimism related words in the review textsof the
ACC hiдh class (0.01) compared to the
ACC low class (0.004).There are more cheerfulness related words present in the reviewtexts of the
ACC hiдh class (0.0017) compared to the
ACC low class(0.0014). There are less confusion words in the review texts of the
ACC hiдh class (0.0026) compared to the
ACC low (0.0036) class. Last,there are more contentment related words in the review texts ofthe
ACC hiдh class (0.0079) compared to the
ACC low class (0.0059).
N E f ) In this section, we study the properties of the three different net-works in details.
CRN ) Recall that in a co-reviewer network each node corresponds to anauthor and two authors are connected if their papers have beenco-reviewed by the same referee. We run series of analysis on thisnetwork to investigate the differences between the three categories.
Here we compute four centrality mea-sures of the whole co-reviewer network.
Degree centrality : We compute the average degree centrality ofthe authors (see Figure 6(a)) for each category. We observe that theaverage degree centrality of authors of
ACC hiдh category is high(0.019) whereas the average degree centrality of the authors for
ACC mid ( ∼ . ACC low ( ∼ . https://github.com/Ejhfast/empath-client Betweenness centrality : We compute the average betweennesscentrality of the authors of each category. The average between-ness centrality (see Figure 6(b)) of
ACC h iдh category is marginallyhigher ( ∼ . Closeness centrality : We calculate the average closeness central-ity of authors for each category. The average closeness centrality(see Figure 7(a)) of
ACC h iдh category is higher ( ∼ . PageRank : We calculate the average PageRank score of the au-thors for each category. The average PageRank (see Figure 7(b))of
ACC h iдh category is marginally higher ( ∼ . Here we perform a k -shell decom-position of the network and inspect four different shells – the in-nermost ( k = k = k = k = ACC hiдh and
ACC mid classes compared to the
ACC low class. In contrast, the outermost shell contains the largest fractionof nodes from the
ACC low class.
Table 3: Core periphery analysis of the co-reviewer network.
Shell
ACC hiдh % ACC mid % ACC low
Innermost (180) 167 29.9 49.1 20.3Inner-mid (140) 37 13.5 78.3 8.1Outer-mid (90) 116 11.2 48.2 36.2Outermost (1) 227 7 14.5 62
Here we construct three in-duced co-reviewer networks comprising the authors in the classes
ACC hiдh , AC mid and AC low respectively. Density : We calculate the density of each induced graph to observehow densely the authors are connected among themselves throughthe common reviewers. Density of the
ACC hiдh induced graph ishigher (0.047) than others. Density of
ACC mid and
ACC low are0.016 and 0.001 respectively.
Assortativity coefficient : We compute the assortativity coeffi-cient of the three induced networks. While this coefficient for the
ACC hiдh induced graph is as high as 0.82, the same for the
ACC mid igure 5: (a) The sentiment of review text (
SNT r ) of ACC hiдh (High),
ACC mid (Moderate) and
ACC low (Low). (b) The length ofthe review text ( L r ) for the three categories. (c) Reviewer diversity ( R div ) of the three categories. (d) Editor diversity ( Ed div ) forthe three categories.Figure 6: (a) The average degree centrality of ACC hiдh (High),
ACC mid (Moderate) and
ACC low (Low) for the three networks(
CRN , CON and
CCN ). (b) The average betweenness central-ity of
ACC hiдh (High),
ACC mid (Moderate) and
ACC low (Low)for three networks. and the
ACC low induced graphs are 0.66 and 0.24 respectively. Thisindicates that the
ACC hiдh induced graph is much more homophiliccompared to the other two graphs.
Edge transitions : We finally study the edge transitions among thethree induced graphs, i.e., given a pair of induced graphs we findthe fraction of edges going from one of them to the other from theoriginal co-reviewer network. We find that
ACC hiдh and
ACC mid share almost 34.7% edges whereas
ACC hiдh and
ACC low share only4.3% edges. The fraction of edges between
ACC mid and
ACC low isaround 10.4%.
CCN ) Recall the the co-citation network has authors as its nodes andthere is an edge from author a i to a j if a i cites a paper of a j . If both Figure 7: (a) The average closeness centrality of
ACC hiдh (High),
ACC mid (Moderate) and
ACC low (Low) for three net-works (
CRN , CON and
CCN ). (b) The average PageRank of
ACC hiдh (High),
ACC mid (Moderate) and
ACC low (Low) forthree networks. a i and a j cite each other in some of their papers then there is abidirectional edge between them. We compute four centrality measuresin the co-citation network.
Degree centrality : We compute the average degree centrality ofthe authors (see Figure 6(a)) for each category. We observe thatthe average degree centrality of the authors of
ACC hiдh categoryis high compared to the average degree centrality of authors for
ACC mid and
ACC low categories.
Betweenness centrality : We compute the average betweennesscentrality of the authors for each category. The average between-ness centrality (see Figure 6(b)) of the authors of
ACC hiдh categoryis marginally higher ( ∼ . loseness centrality : We calculate the average closeness central-ity of the authors of each category. The average closeness centrality(see Figure 7(a)) of ACC hiдh category authors is higher ( ∼ . PageRank : We calculate the average PageRank score of the au-thors of each category. The average PageRank (see Figure 7(b)) of
ACC hiдh and
ACC mid categories are marginally higher than the
ACC low category.
Here again we construct threeinduced co-citation networks comprising the authors from the threeclasses –
ACC hiдh , ACC mid and
ACC low . Cross citations : We find the fraction of citations running in be-tween the classes. Notably, the largest fraction of citation edgesrun between
ACC hiдh and
ACC mid induced graphs (45%). Fractionof citation edges running between
ACC hiдh and
ACC low inducedgraphs, on the other hand, is the least (1%).
Self citations : Fraction of citation edges running within the
ACC hiдh induced graph is the highest (∼ . ) . This fraction for the ACC mid and
ACC low are 17.9% and 0.4% respectively.
Reciprocity : We compute the reciprocity within and across all thethree induced networks. Reciprocity within the
ACC hiдh inducednetwork is the highest (0.61); reciprocity in the
ACC mid inducednetwork is 0.20 and the same for the
ACC low induced network is0.08 which is the least among the three.Reciprocity in between
ACC hiдh and
ACC mid induced networksis 0.34 which is higher than between
ACC mid and
ACC low (0.11)as well as
ACC hiдh and
ACC low (0.12).
ACC low authors that are cited by
ACC hiдh authors.
Althougha rare case, here, we observe how the citations coming from the
ACC hiдh authors affect the fate of the papers written by the
ACC low authors. We separately consider those papers which are cited by
ACC hiдh authors and observe the author characteristics of suchpapers. We find that the mean citation of papers written by
ACC low authors and cited by
ACC hiдh authors is roughly double (∼ . ) the mean citation of papers (∼ . ) written by ACC low authorsthat are never cited by the
ACC hiдh authors.We further notice that the mean citation of those
ACC low authors (∼ . ) whose papers are cited by ACC hiдh authors is higherthan the mean citation of the other
ACC low authors (∼ . ) . ACC low authors cited by
ACC mid authors.
In this section,we investigate the characteristics of those
ACC low authors whosepapers are cited by
ACC mid authors. Once again, we observe thatthe mean citation of papers (∼ . ) written by ACC low authorsand cited by
ACC mid authors is much higher than the mean citationof papers (∼ . ) written by ACC low authors but never cited bythe
ACC mid authors.
CON ) Recall that the in the collaboration network each node is an authorand two authors are connected if they have co-authored a papertogether. We present a visualisation of the collaboration networkin Figure 3. The left sub-figure shows the authors in the three cate-gories as nodes of different colours. The blue nodes correspond tothe authors in the
ACC hiдh category, the red nodes correspond to the authors in the
ACC mid category and the yellow nodes corre-spond to the authors in the
ACC low category. The blue nodes areconcentrated mostly in the center of the network while the red andthe yellow nodes are scattered all across the network. This is moreclear when we draw the network of the authors corresponding tothe
ACC hiдh and the
ACC mid category. The blue nodes are largelyconcentrated at the center of the network.
We compute four centrality measuresfrom the collaboration network.
Degree centrality : We compute the average degree centrality ofthe authors (see Figure 6(a)) for each category. We observe that theaverage degree centrality of the authors in the
ACC hiдh categoryis higher (0.0039) than the average degree centrality of the authorsin the other two categories.
Betweenness centrality : We compute the average betweennesscentrality of the authors of each category. The average between-ness centrality (see Figure 6(b)) of
ACC hiдh category is higher( ∼ . Closeness centrality : We calculate the average closeness central-ity of the authors of each category. The average closeness centrality(see Figure 7(a)) of
ACC h iдh category is higher ( ∼ . PageRank : We calculate the average PageRank score of the au-thors of each category. The average PageRank (see Figure 7(b)) of
ACC h iдh category is marginally higher ( ∼ . The fraction of collaboration edgesbetween the
ACC hiдh and
ACC mid authors is 38.9% which is muchhigher than either the fraction of collaboration edges between
ACC mid and
ACC low authors (1.0%) or
ACC hiдh and
ACC low au-thors (0.2%).On the other hand, the fraction of collaboration edges withinthe
ACC hiдh authors is 26.4%, while this is 31.3% for the
ACC mid authors and 0.7% for the
ACC low authors.
ACC low authors collaborating in papers primarily writtenby
ACC hiдh authors.
In this section, we focus on those
ACC low authors who get a chance to collaborate with
ACC hiдh authors.In particular, we consider those papers which are written by amix of 20%
ACC low authors and 80%
ACC hiдh authors (i.e., paperspredominantly written by authors with high acceptance ratio).We compute various features discussed earlier for this 20%
ACC low authors when they write papers with
ACC hiдh authors and whenthey write papers without them. The feature values are noted inTable 4. Collaborations with the
ACC hiдh authors seems to heavilybenefit the
ACC low authors in terms of accrued citations as well asreview sentiments obtained from the referees.
ACC hiдh authors collaborating in papers primarily writtenby
ACC low authors.
In this section, we analyze such cases wherepapers are written by 80%
ACC low and 20%
ACC hiдh authors. Weanalyze profile features of these 80%
ACC low authors when theywrite papers with
ACC hiдh authors as well as when they writewithout them. Table 5 enumerates the important features and showsthat even having a small fraction of
ACC hiдh authors in their papercan increase the citation count and reduce the negative sentimentin the reviews of the
ACC low authors. able 4: Properties of
ACC low authors who collaborate witha high number
ACC hiдh authors.
Features Collaborated Not collaboratedwith
ACC hiдh with
ACC hiдh
Mean TS ) 4.3 3.1Citation ( C cnt ) 30 12Review text sentiment ( SNT r ) 0.23 -0.13 Table 5: Analysis of
ACC low authors who collaborate with alow number of
ACC hiдh authors.
Features Collaborated Not collaboratedwith
ACC hiдh with
ACC hiдh
Mean TS ) 4.12 2.85Citation ( C cnt ) 53.7 27.7Review text sentiment ( SNT r ) -0.39 -0.54 In our classification model, we consider AP f , PF f and N E f featuresfor the first three years of career of each author as the training data.For example, if an author published his first paper in 1996 then weconsider papers published in between 1996 and 1998 for trainingpurpose. We compute all the features of an author based on the firstthree years of career information. For testing, we leave a gap oftwo years to prevent any data leakage. After five years, we predicttheir category. We use two different classifiers – XGBoost [6] andrandom forest [3]. In order to evaluate the model, we compute classwise precision and recall. In addition, we also compute F1-score.We calculate precision as the fraction of authors who are correctlyclassified out of all the predicted authors. Recall is the fraction ofrelevant authors correctly classified by the classifier. Features : We use the author profile features ( AP f ), peer reviewbased features ( PF f ) as well as network features ( N E f ). Results : The class wise precision and recall for the XGBoost modelare noted in Table 6. The F1-score for the model is 0.84. The confu-sion matrix is tabulated in Table 7.The class wise precision and recall for the random forest modelare noted in Table 8. F1-score for this model is 0.89. We report theconfusion matrix in Table 9. The random forest model outperformsthe XGBoost model.
Table 6: Class wise precision and recall of the XGBoostmodel.
Categories Precision Recall
ACC hiдh
ACC mid
ACC low
Feature importance : Some of the important features for boththe models are degree centrality of
CCN , sentiment of review text
Table 7: Confusion matrix of the XGBoost model.
Categories
ACC hiдh
ACC mid
ACC low
ACC hiдh
ACC mid
715 8669 502
ACC low
18 987 7389
Table 8: Class wise precision and recall of the random forestmodel.
Categories Precision Recall
ACC hiдh
ACC mid
ACC low
Table 9: Confusion matrix of the random forest model.
Categories
ACC hiдh
ACC mid
ACC low
ACC hiдh
ACC mid
604 9006 276
ACC low
20 727 7647 ( SNT r ), PageRank of CCN , citation count, team size ( TS ), degreecentrality of CRN , core number, PageRank of
CRN , reciprocityof
CCN , experience, h -index ( H ind ), closeness centrality of CCN ,betweenness centrality of
CCN , reviewer diversity ( R div ). The in-dividual set of features that are important for the two models arenoted in Figure 8 (random forest) and Figure 9 (XGBoost). Figure 8: Important features for the random forest model.
So far we have investigated author characteristics that could actas early indicators of the acceptance rate of the authors. However,recall, the reviewer and editor diversity measures presented insections 5.3 and 5.4 respectively. In fact these features are alsofound to have strong predictive power in section 7. Although wehave used these features in profiling the authors, it can be easilyreasoned that they are based on the functioning of the peer reviewsystem itself. In this section we shall therefore discuss the roleof the peer review system (in any) in reinforcing the distinctionamong the three categories of authors. igure 9: Important features for XGBoost model.
To this purpose, we characterize the authors of different cate-gories in terms of the set of editors and reviewers who have everedited/reviewed their paper. We consider pairs of authors from eachcategory and compute the Jaccard overlap ( J ) of the reviewer andthe editor sets respectively. Next for each category, we calculatethe average pairwise J values. Interestingly, for the reviewer set weobserve that the average value of J for ACC hiдh authors is relativelyhigher (0.0202) compared to
ACC mid (0.0016) and
ACC low (0.0008)authors. For the editor set, the average value of J for ACC hiдh is0.0302 whereas the average value for
ACC mid and
ACC low are simi-lar (0.0137 and 0.0105 respectively). This potentially again indicatesthat there is less diversity in the editors and reviewers who areassigned to the
ACC hiдh category. However, one might argue thatthis could as well be an artefact of the authors in the
ACC hiдh category collaborating more heavily among themselves comparedto the other two categories and therefore it is obvious that theywould tend to have more overlap in the reviewer and editor sets.In order to verify if this is actually an artefact, we next considerfor each category the pairs of authors who have never collaborated(i.e., never co-authored a paper together). For such pairs of authorsin a category, we calculate the J of their editor and reviewer setsagain. In particular, we identify the % of author pairs having J inthe range [ . , ] and author pairs having J exactly 1. We notethe percentage overlap values in Table 10. For both the editor andthe reviewer sets we observe that even if the authors have nevercollaborated they tend to get more similar referees and editors inthe ACC hiдh category compared to the other two categories. Thisresult indicates that the initial observation that we made was notan artefact and that the peer-review system indeed enables a lessdiverse referee and editor set for the
ACC hiдh authors. We presenta visualisation of this phenomenon in Figure 10. In Figure 10 (Up),the green coloured nodes represent the reviewers and the blue, thered and the yellow nodes correspond to the authors in the
ACC hiдh , ACC mid and
ACC low categories respectively. There is a directededge from a reviewer to an author if the reviewer had reviewedone or more papers of the author (i.e., a directed bipartite network).The visualisation again indicates that there are ‘patches’ of clus-ters of unique reviewers around authors of the
ACC hiдh category.Similarly, in Figure 10 (Down) the sky blue colored nodes representthe editors and the blue, the red and the yellow nodes correspondto the authors in the
ACC hiдh , ACC mid and
ACC low categoriesrespectively. There is a directed edge from an editor to an authorif the editor had edited one or more papers of the author. Similar patches of clusters also appear here. Overall, we believe that thismight lead to potential discrimination and unfairness and shouldtherefore be further investigated by the system admins.
Figure 10: (Up) This network shows the relationship be-tween hundred top cited authors and their reviewer fromthree different categories. There is a directed edge from areviewer to an author if the reviewer had reviewed one ormore papers of the author. (Down) This network shows therelationship between hundred top cited authors and theireditors from three different categories. There is a directededge from an editor to an author if the editor had edited oneor more papers of the author.able 10: Percentage of author pairs having Jaccard overlapof editor and reviewer set in [ . , ] and exactly 1. Categories Editor set Reviewer set J ( [ . , ] ) J ( = J ( [ . , ] ) J ( = ACC hiдh
ACC mid
ACC low
As an additional investigation we choose author pairs acrosscategories and observe how their editor sets overlap. If we chooseauthor pairs with one from
ACC hiдh and another from
ACC mid the J value in [ . , ] for the editor set is 0.13%. Similarly, if we chooseauthor pairs with one from ACC mid and another from
ACC low the J value in [ . , ] for the editor set is 0.57%. However, what ismost intriguing is that if we choose author pairs with one from ACC hiдh and another from
ACC low the J value in [ . , ] for theeditor set is 0%. This indicates that the editors who are assigned tothe ACC hiдh category of authors are almost never assigned to the
ACC low category users. Once again this could be indicative of apotential unfairness situation in the peer-review system and needsto be carefully investigated further.
Peer review system plays an important role in the acceptance ofa research paper in a journal. Quality peer review system helpsauthors to improve themselves. There are lots of debates on thequality [11] and bias in a peer review system [7, 9, 12]. Jeffersonet al. [11] investigated the quality of editorial peer review. Theyclaimed that measuring the quality of peer review require hugeco-operation of authors. Sikdar et al. [13] studied reviewer-reviewerinteraction network to predict the long term citation of a paper.They also studied whether the peer review system can be improved.In [12], the authors investigated anomalies in a peer review system.They computed different features from the editor and the reviewerinformation available. In [8] the authors investigated the existenceof gender bias in a peer review system. Another interesting studyby Tomkins et al. [15] showed that a single blind reviewing systemgives disproportionate advantage to the papers of famous authorsand authors from highly reputed institutions. In similar lines theauthors in [16] proposed how to improve a single blind reviewprocess.Earlier research also explored various author profile based fea-tures such as experience, citation count, h -index, research topicdiversity to quantify research productivity/success of an author [5].The productivity of an author [1] had been defined as the extentof his/her contribution (publications) to the scientific community.Most of the earlier research focused on whether such author profilebased features are sufficient to justify ones research productivity.In [4], the authors explored the productivity of authors and theircitations considering publications in the Proceedings of the Okla-homa Academy of Science (POAS). They found that authors withhigh productivity are not highly cited. Bayer et al. [2] computedcitation count to measure the productivity and found that it is less correlated with the quality of researcher’s academic career but thereis no correlation with his/her IQ.Our work is very different from the above studies. We utiliseauthor profile information, peer review information and three dif-ferent networks to predict the class of an author based on his/heracceptance rate.
10 CONCLUSION
We categorize the authors into three classes based on their accep-tance rate in the journal. We characterise these classes of authorsbased on their profile, the peer reviews their papers received andthree different networks. The authors with high acceptance rateseem to be markedly different in terms of many of these characteris-tic features. Finally, using these features we show that it is possibleto predict the acceptance rate class early for any author.In future we would like to investigate in more details the reasonsfor the differences in the reviewer and editor diversities across theclasses. In specific this problem can be posed as an anomaly/biasdetection where we plan to use state-of-the-art techniques to un-derstand the precise reasons for such uneven diversity across theclasses.
11 ACKNOWLEDGEMENTS
We thank Media Lab SISSA for providing us with the necessaryJHEP data for the analysis. RH and AM thank Simons Foundationfor financial support through the Simons Associateship Programme.
REFERENCES [1] Giovanni Abramo and Ciriaco Andrea D’Angelo. 2018. How do you define andmeasure research productivity?
CoRR abs/1810.12830 (2018). arXiv:1810.12830http://arxiv.org/abs/1810.12830[2] Alan E. Bayer and John Folger. 1966. Some Correlates of a Citation Measure ofProductivity in Science.
JSTOR
39 (1966). Issue 4. https://doi.org/10.2307/2111920[3] Leo Breiman. 2001. Random Forests.
Mach. Learn.
45, 1 (Oct. 2001), 5–32.[4] Tony Bremholm. 2005. Productivity and Citation Frequency in the Proceedingsof the Oklahoma Academy of Science , 1921-2000.[5] Yi Bu, Ying Ding, Jian Xu, Xingkun Liang, Gege Gao, and Yiming Zhao. 2018.Understanding success through the diversity of collaborators and the milestoneof career.
JASIST
69, 1 (2018), 87–97. https://doi.org/10.1002/asi.23911[6] Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree BoostingSystem. In
Proceedings of KDD . 785–794.[7] Laura J. Falkenberg and Patricia A. Soranno. 2018. Reviewing Reviews: AnEvaluation of Peer Reviews of Journal Article Submissions.
Limnology andOceanography Bulletin
27, 1 (2018), 1–5.[8] Markus Helmer, Manuel Schottdorf, Andreas Neef, and Demian Battaglia. 2017.Gender bias in scholarly peer review. eLife
Scientometrics
113 (2017), 633–650. Issue 1.[10] Hirsch JE. 2005. An index to quantify an individual’s scientific research output.
PNAS
102 (07 2005), 46.[11] Tom Jefferson, Elizabeth Wager, and Frank Davidoff. 2002. Measuring the Qualityof Editorial Peer Review.
JAMA
Proceedings of CIKM . 2245–2250.[13] Sandipan Sikdar, Matteo Marsili, Niloy Ganguly, and Animesh Mukherjee. 2017.Influence of Reviewer Interaction Network on Long-term Citations: A Case Studyof the Scientific Peer-review System of the Journal of High Energy Physics. In
Proceedings of JCDL . 179–188.[14] Ian F. Spellerberg and Peter J. Fedor. 2003. A tribute to Claude Shannon(1916–2001) and a plea for more rigorous use of species richness, species di-versity and the ‘Shannon–Wiener’Index.
Global ecology and biogeography
PNAS
114 (2017), 12708–12713. Issue 48.[16] Alina Tvina, Ryan Spellecy, and Anna Palatnik. 2019. Bias in the Peer ReviewProcess Can We Do Better?