How Visualization PhD Students Cope with Paper Rejections
HHow Visualization PhD Students Cope with Paper Rejections
Shivam Agarwal †1 , Shahid Latif †2 , and Fabian Beck † Equal contributionpaluno, The Ruhr Institute of Software TechnologyUniversity of Duisburg-Essen, Germany A BSTRACT
We conducted a questionnaire study aimed towards PhD students inthe field of visualization research to understand how they cope withpaper rejections. We collected responses from 24 participants andperformed a qualitative analysis of the data in relation to the providedsupport by collaborators, resubmission strategies, handling multiplerejects, and personal impression of the reviews. The results indicatethat the PhD students in the visualization community generallycope well with the negative reviews and, with experience, learnhow to act accordingly to improve and resubmit their work. Ourresults reveal the main coping strategies that can be applied forconstructively handling rejected visualization papers. The mostprominent strategies include: discussing reviews with collaboratorsand making a resubmission plan, doing a major revision to improvethe work, shortening the work, and seeing rejection as a positivelearning experience.
NTRODUCTION
To appear in Celebrating the Scientific Value of Failure (FailFest) Workshop at IEEE VIS 2020.
Facing paper rejections in academia is common and sometimeshas adverse effects on individuals. Existing research studies theeffect of paper rejections on scholars [4, 7] and provide some guide-lines on how to cope with such experiences ( e.g. , [3, 9]). However,the studies were conducted for management and statistical sciencesand the results might not necessarily generalize to the visualizationcommunity due to the diversity in reviewing models and culture.Additionally, these studies did not focus on PhD students specifi-cally. We are interested in this group because we want to understand,among other things, if and how PhD students are negatively affectedby paper rejections as well as what are their strategies to cope with it.PhD students, as new and inexperienced members of the community,might suffer more from negative reviews and paper rejections. Ifbadly timed, a single rejected paper can lead to significant delaysin completing the PhD dissertation or even threaten its successfulcompletion.Within visualization research, there exists literature that outlinescommon reasons of a paper rejection and provides guidelines forpreparing a manuscript for submission [8]. Additionally, seniorresearchers of the visualization community have made suggestionson how to deal with paper rejections. Elmqvist [5,6] described somestrategies on how to handle paper rejections while acknowledgingthat young PhD students are most vulnerable. Taking a wider per-spective, Shneiderman [10] laid his thoughts on a new paradigm ofresearch providing guidance to the students, junior and senior re-searchers, and policymakers. However, we lack research that studieshow PhD students in the visualization research community perceiveand handle paper rejections. [email protected] [email protected] [email protected] In this paper, we aim at understanding and revealing the strategiesof visualization PhD students in coping with paper rejections. Weconducted an online study to gather responses from our target usergroup, i.e. , PhD students working in visualization research com-munity. Based on the responses, we performed a qualitative dataanalysis and report our findings. We provide our questionnaire, rawdata, and coded responses as a supplemental material [1].
TUDY D ESIGN
We decided to use a short online questionnaire for the study toreach visualization PhD students internationally. To not bias theparticipants towards certain answers and to also discover unexpectedstrategies, the questionnaire used open-ended questions for address-ing the main research questions. Hence, a mostly qualitative analysisis required to identify common themes and noteworthy exceptions.To capture the full breadth of the community, we strove towardsdiversity of participants with respect to the country of affiliation andlevel of experience (among PhD students).
The online questionnaire consists of three sections: general infor-mation , publication history , and handling paper rejections . Thequestions in the first section focus on knowing the stage of partici-pants’ doctoral studies, countries of their universities, and field ofstudy during their bachelor and master studies. The second sectionhas two questions and aims at gauging the research experience of par-ticipants in terms of submitted and rejected number of peer-reviewedresearch articles. The third and final section captures the experienceof participants with the papers that were rejected and re-submitted.All questions in the third section were open-ended questions andparticipants could write as much as they wanted. In particular, weasked the following four questions:• Q1:
How did your lab-mates/colleagues/supervisors help tohandle paper rejections?•
Q2:
What strategic decisions did you take to re-submit therejected papers?•
Q3:
How did you handle the papers that got rejected morethan once? (if any)•
Q4:
How did your impression of paper reviews change overtime?These questions focus on capturing the possible ways or steps thatstudents may have taken to cope when they receive a paper rejection.The questionnaire was designed to be completed in approximately25–30 minutes.
The target participants were the PhD students from the visualiza-tion community who have submitted research articles to peer-reviewvenues and have already experienced one or more paper rejections.This target group ranges from first year PhD students to the ones a r X i v : . [ c s . H C ] S e p igure 1: Stage of PhD studies of the participants.Figure 2: Participants by country of affiliation. who have recently (in the past 6 months) graduated or left the PhDprogram. The questionnaire was posted to the IEEE VIS and In-fovis.org mailing lists as well as to the Discord channel used byEuroVis 2020 and Euro Graphics 2020. We also forwarded the ques-tionnaire within our professional network by email and advertised iton Twitter.In total, we received 43 responses until the submission date. Inthis paper, as responses have been still coming in during data anal-ysis, we focus on the first 24 responses and plan on including thelater responses in an extended version of this paper. The followinganalysis and descriptions are purely based on the first 24 responses. To analyze the responses, we followed an open coding process. Theprocess began with two of the authors independently coding thefirst five responses, followed by a discussion on a merged set ofauthoritative codes. This resulted in a list of 19 codes. In the nextstage, we divided the remaining responses into two equal parts (9and 10 responses in these groups) and each coder used the list ofalready established 19 codes to do the coding. During this secondstage of coding, both coders still came across new types of com-ments and occasionally had to define new codes. These codes werethen immediately added to the shared list of codes. Notifying theother coder allowed him to use the newly defined code as well. Thisprocess, once again, resulted in some partly redundant codes thatwere again merged and resolved in another follow-up meeting. Thisrefinement left us with a set of 28 different codes. Now, both codersused this set of codes to validate each other’s coding. The resultingconflicts during the validation phase were then resolved in subse-quent meetings, which also led to a further consolidation to 25 codes(cf. Table 1).
Figure 3: Number of submitted and rejected articles per participant.
ESULTS
Based on the assigned codes (Table 1), we report the results perquestion (Q1–Q4), each discussing the most frequently mentionedtypes of answers and noteworthy individual examples. We begin withsummarizing the general information and publication history of theparticipants. We numbered the participants with identifiers P1–P24.Individual participants are listed when referring to groups of fiveparticipants or fewer; else, we just provide the count of participants,but details can be found in the supplemental material [1].
Among the 24 participants, about 84 percent of the participants haddone a master degree in computer science, and almost 63 percent hadcompleted a bachelors in computer science. As shown in Fig. 1, amajority of the participants (22) were PhD students ranging from firstyear to sixth year and the ones who graduated in the past six months.One respondent (P2) had left the PhD program and one participant(P22) had graduated two years ago. Participants were enrolled atuniversities in 10 different countries, with half of the responsescoming from Germany and USA ( Fig. 2). None of the participantclaimed of being enrolled with multiple universities. Among theparticipants, the number of submitted and rejected peer-reviewedarticles ranged from 0 to 26 and 0 to 12 respectively (Fig. 3). Oneparticipant (P8, a second year student) did not submit any articleto date. Another participant (P21, a first year student) had notexperienced any rejection. But since both of them provided valuablecomments in their responses, we decided to still include them in ouranalysis.
As seen from Table 1, the most mentioned support strategies withinthe collaboration groups of participants were to discuss reviews (19participants) and make a concrete resubmission plan (9 participants) able 1: List of codes along with their frequency of occurrence perquestion and total (sorted by total).
Codes FrequencyQ1 Q2 Q3 Q4 Total improvement 4 13 11 4 32downgrade work – 16 5 – 21discuss reviews 19 – 1 – 20impression gets better – – – 12 12see rejection as positive – 1 4 5 10resubmission plan 9 – – – 9similar venue scope – 9 – – 9painful reviews at first – – – 9 9informal discussion 6 – – 2 8impression remains the same – – – 7 7submit to journal – 6 – – 6fair reviews 1 – – 5 6high-level reflection of reviews 3 – – 2 5discuss reviews quickly 3 – – – 3contradictory reviews – – – 3 3different venue scope – 2 – – 2disagree with reviews – – – 2 2no special discussion 1 – – – 1lack of support from coauthors 1 – – – 1encouragement from supervisor 1 – – – 1dealing with rejections meeting 1 – – – 1supervisor’s decision – 1 – – 1post on arxiv – 1 – – 1unclear reviews – – – 1 1get early feedback – 1 – – 1
Total 49 50 21 52 172 with collaborators of the paper. Six participants reported to havehad informal discussions ( e.g. , vis-`a-vis during a coffee or lunchbreak, or via instant messaging) with their colleagues. They either discussed the reviews quickly (P3, P9, and P14) or celebrated thesuccess of their peers and commiserated with the others who faceda rejection (P15 and P24). For getting support within the workgroup, the similarity of research topics seems to play a role as P14commented: “I just talked briefly with my supervisor about it. SinceI’m the only person in my lab doing visualization research, the otherones don’t know about the problems so much in detail.” Towards improving the submission, participant P7 complainedabout the lack of support from coauthors in revising or reworkingthe paper and mentioned that they did not have background in visual-ization research. Four participants (P2, P7, P10, and P12) discussedthe exact improvements and feasibility of these improvements fortheir paper with coauthors. Three experienced students (P10, P13,and P19) stated that they did a high-level reflection of reviews tounderstand what went wrong and to find areas of improvement.
When it comes to resubmitting rejected papers, prominent strategiesare either downgrading work to resubmit in a lower-ranked venue (16participants) or making improvements in the work (13 participants).The improvement might involve considerably extending the workwith additional contributions (P12) or by including a user study(P1, P3, and P6). Both P1 and P12 stated that they converted shortpapers to full papers. The participants who decided to go for an improvement either resubmit their papers to a similar scoped venue (9 participants) or to a different scoped venue (P7 and P15). Itcould be better to switch to a different venue for better aligningcontributions with the target audience as P15 stated: “In the caseof one paper the reviewers commented that the novelty/contribution to the visualization community was limited, so here we submittedthe application-based paper to a workshop with a better audiencefit [. . . ].”
Being afraid that somebody else might publish similar researchfirst, participant P24 mentioned that they had posted the paper on arXiv and described it as: “[. . . ] if we were scared of being ‘scooped’we would put it on arXiv and go for the next venue. If no externalpressure, we hold on to next year’s visualization-related conference.”
The responses also indicated few submission strategies of thepapers. Participant P7 reported his/her tendency of submitting aposter to get early feedback : “I tend to like to do posters, because Ican try to think about how to present the work visually, and in verbaldiscussions with others. This is generally helpful for me.” Anotherstrategy is to submit to a journal (6 participants) rather than waitingfor the next conference cycle. P4 responded that he/she prefers todirectly submit to journals as it provides more review cycles (majorand/or minor revisions) in contrast to conferences where a majorrevision is considered as a reject. Participant P14 also favored thisstrategy as some journals in visualization community ( e.g. , TVCG)offers the presentation of accepted work at any major visualizationconference.
Eleven participants mentioned that they had made further improve-ments before resubmitting the papers that were rejected more thanonce. Similar to handling first rejections of their papers, the partici-pants reported to downgrade work by resubmitting to lower-ratedvenues (P7, P15, P19, P20, and P23). None of the participantsmentioned to have abandoned their work after getting it rejectedmultiple times. However, participant P7 highlighted that shorteningthe work could be a good strategy to avoid spending more time ona project. Four participants (P7, P14, P15, and P20) mentionedthat they had seen rejection as a positive learning experience whilehandling papers that got multiple rejects. The response of participantP7 summarizes this positive outlook: “My attitude is ‘this will get in somewhere, eventually’. Ihave seen my papers improve dramatically after revisingand resubmitting, so I tend to think of rejections as agood thing, because the paper wasn’t ready and will onlyget better each time I try again. If it were to be on atopic where I didn’t feel like making substantial changes,I might stop at submitting a short paper or a poster, but Ihaven’t encountered this yet.” – P7However, participant P14 also highlighted the difficulty while han-dling papers that got multiple rejections as: “It was really hard tonot give up. There were many tears. But in the end, you have tostand up again and improve your work.” . Eleven participants eitherdid not provide any answer to the question or mentioned that, untilnow, they had not faced multiple rejections of the same paper.
Nine participants highlighted that it was painful to read the reviewsat first, especially, on the notification day. Considering how thesubjective impression of the reviews changes over time, half of theparticipants (12) mentioned that their impression got better , whileseven participants reported that their impression remained the same .Four participants (P5, P6, P7, and P15) said that they saw rejectionas positive . A response from participant P6 summarizes a strategyto avoid taking reviews personally: “ [The impression of the reviews]fairly remained the same. I always took it constructively and knowthat rejections are part of the science process. It’s not [a] reflectionon me personally or my skills. ”.Five participants (P3, P12, P14, P20, and P22) reported that over-all they had gotten fair reviews while two of them (P3 and P22) disagreed with reviews on some points. Three participants (P1, P7,nd P14) reported that sometimes reviewers had provided contra-dictory reviews , while participant P18 mentioned receiving unclearreviews . Participant P13 mentioned a strategy of not consideringeach point in a review; instead he/she had learned to do a high-levelreflection of reviews with experience: “My perception of specific reviews did not change muchwith time, but my general view on reviews over the yearschanged a lot, in such a way that I am not any longer sim-ply considering each point in the review, but am askingmyself more often, what went wrong in the communica-tion of the content.” – P13Two participants (P15 and P24) reported doing informal discussion of the reviews. Elaborating on their responses, participant P24 saidthat “[. . . ] Talking with peers who also submitted papers on reviewday also adds to stress if your paper wasn’t accepted.”
On the otherhand, participant P15 mentioned that going through the reviews withfriends as a social event helps him/her to remain positive, especiallywhen some comments in the reviews are disheartening.
TUDY L IMITATIONS
The geographical diversity of the participants is limited to 10 coun-tries from Europe and North America only. Moreover, 12 out of24 (50%) responses are from Germany and USA. We believe thismight be the result of using our personal networks to circulate thequestionnaire. Therefore, the analysis results may be dominated bythe work culture of visualization PhD students in Europe and NorthAmerica.Since the questions in the third section of our questionnaire wereopen-ended and broad, we provided some examples for each ques-tion to specify along which lines participants can think of whileanswering the questions. On the one hand, those examples weregood to encourage participants to share their thoughts in case theyfelt lost. But this might have influenced the responses to some extenton the other hand.Along similar lines, we might have biased the participants byadvertising the study in a way that implied that they might feel frus-trated about rejected papers—this was done to attract the attention ofmore participants. In general, just focusing on rejected papers takesa perspective that cuts short examples where papers were improvedin a short revision cycle and then directly accepted. Since we hadto compromise on the extensiveness of the questionnaire, we can-not make statements about this or other aspects of the submissionprocess and strategies.
ISCUSSION
Based on the responses, we can infer that PhD students in the fieldof visualization research take the reviews seriously. The studentsseem to discuss reviews and formulate a resubmission plan andcontinue to make further improvements in their work. The studentsfrequently use the strategy of downgrading work while resubmitting,either by shortening the length or opting for a lower-rated venue.It seems that the strategies work well as they eventually get theirpapers published. This also indicates that visualization researchcommunity has many venues and tracks where the improved workeventually gets published. The strategies reported by visualizationPhD students—discussing reviews, improving the work, and resub-mitting it to another venue—align well with the suggestions madeby Shneiderman [10, Section 6.6] and Elmqvist [5].Although the responses of PhD students indicate that reviews arepainful at first, they try to see rejection as a positive event. Somestudents mentioned that they try not to take the reviews personallyand highlighted that rejections do not mean incompetency, whichresonates with the advice by Elmqvist [6]. Other students alsomentioned that engaging in informal discussions with colleagues and friends helps them cope with paper rejection. Knowing about thesestrategies can be helpful especially for young PhD students whohave limited knowledge about the normalcy of receiving rejectionsin a peer-review research process. It maybe a good idea to talk aboutrejections, both formally and informally, and discuss strategies onhow to cope with them. The discussions potentially creates a socialsupport system that helps in coping with rejections and matches wellwith the recommendation by Day [3].There were few responses which highlighted the occurrencesof unclear and contradictory reviews . Even though these factorscontribute to the painful experience on the day of notification, PhDstudents in visualization research usually handle them positively. Asa result, many students highlighted that their impression of reviewsgets better with time. Some students reported that they had used theirjudgment and sometimes had disagreed with few comments in thereviews. Additionally, some experienced students mentioned doinga high-level reflection of reviews trying to understand the issues thatmight have misdirected the reviewers. These strategies can also behelpful for finding areas of improvement and formulating a plan ofaction, especially when the reviews are unclear, have contradictoryviews, or contain insufficient information to back the review rating.The high-level reflection of reviews can be done in different ways.Elmqvist suggested summarizing the reviews [5] while Shneider-man shared personal experiences of handling difficult reviews andhighlighted using one’s own judgment to continue improving thework [10, Section 6.6].Since nine respondents (37.5%) indicated to have experiencedemotional distress on the day of rejection notification after readingthe reviews, it may be indicative of a harsh textual connotationbeing used in the reviews. We argue that conscious efforts can bemade by reviewers to proactively minimize the painful rejectionexperience. A more polite text tone over a harsh comment maybe used to communicate the same review. Although not alwayspossible, concrete suggestions for improving the work can be clearlyhighlighted. Moreover, alternative reviewing models can be exploredand adopted that have helped in reducing the harsh tone; Besanc¸on et al. [2] concluded in a recent survey that reviewers use a morepolite tone in open and non-anonymized peer reviewing model.
ONCLUSION AND F UTURE W ORK
Based on the responses we can infer that most PhD students in thevisualization research have a positive outlook to cope with paperrejections. However, sometimes the positive outlook does not corre-late with the absence of a painful experience while facing rejections( e.g. , P14). We hope that knowing about the strategies of others canhelp the students who are facing paper rejections. Also, we hopethat initiating discussions on the normalcy of paper rejections inacademia and talking about known strategies in coping with rejec-tions can be helpful especially to the young PhD students joining aresearch team.As part of future work, we would be interested in discerning thestrategies of students having different levels of experience. Anotheraspect would be to validate the results of this analysis by gettingexpert feedback of experienced reviewers from the community. Also,we plan to extend the analysis to the complete set of responses. A CKNOWLEDGMENTS
We wish to thank all the researchers who participated in our studyand shared their thoughts and experiences. We are indebted toBen Shneiderman and Niklas Elmqvist who pointed us to valuableresources on the subject matter. R EFERENCES [1] S. Agarwal. Supplementary material: How visualization PhD studentscope with paper rejections, 2020. doi: 10.17605/osf.io/vgj362] L. Besanc¸on, N. R¨onnberg, J. L¨owgren, J. P. Tennant, and M. Cooper.Open up: a survey on open and non-anonymized peer reviewing.
Re-search Integrity and Peer Review , 5(1):1–11, 2020. doi: 10.1186/s41073-020-00094-z[3] N. E. Day. The silent majority: Manuscript rejection and its impact onscholars.
Academy of Management Learning & Education , 10(4):704–718, 2011. doi: 10.5465/amle.2010.0027[4] M. S. Edwards and N. M. Ashkanasy. Emotions and failure in academiclife: Normalising the experience and building resilience.
Journal ofManagement & Organization , 24(2):167–188, 2018. doi: 10.1017/jmo.2018.20[5] N. Elmqvist. Dealing with rejection. https://sites.umiacs.umd.edu/elm/2016/10/25/dealing-with-rejection/ , 2016. ac-cessed: July 2020.[6] N. Elmqvist. Above all, persistence. https://sites.umiacs.umd.edu/elm/2018/08/16/above-all-persistence/ , 2018. ac-cessed: July 2020.[7] S. A. Horn. The social and psychological costs of peer review: Stressand coping with manuscript rejection.
Journal of Management Inquiry ,25(1):11–26, 2016. doi: 10.1177/1056492615586597[8] T. Munzner. Process and pitfalls in writing information visualizationresearch papers. In
Information Visualization: Human-Centered Issuesand Perspectives , pp. 134–153. 2008. doi: 10.1007/978-3-540-70956-5 6[9] D. B. Rubin. Converting rejections into positive stimuli. In
Past,Present, and Future of Statistical Science , pp. 593–603. 2014. doi: 10.1201/b16720-55[10] B. Shneiderman.