CComments are welcome ∗Asier Minondo † February 17, 2020
Abstract
Scholars present their new research at seminars and conferences, and send drafts topeers, hoping to receive comments and suggestions that will improve the quality of theirwork. Using a dataset of papers published in economics journals, this article measureshow much peers’ individual and collective comments improve the quality of research.Controlling for the quality of the research idea and author, I find that a one standarddeviation increase in the number of peers’ individual and collective comments increasesthe quality of the journal in which the research is published by 47%.
JEL : A14, I23
Keywords : production of science, peer effects, research seminars, economics. ∗ I thank Juan de Lucio, Benedikt Heid and Francisco Requena for very valuable comments and sugges-tions. I also thank the feedback from participants in the research seminar at the University of Valencia. Igratefully acknowledge financial support from the Spanish Ministry of Science, Innovation and Universities(RTI2018-100899-B-I00, co-financed with FEDER) and the Basque Government Department of Education,Language policy, and Culture (IT885-16). † Deusto Business School, University of Deusto, Camino de Mundaiz 50, 20012 Donostia - San Sebastián(Spain). Email: [email protected] a r X i v : . [ ec on . GN ] F e b Introduction
Scientific progress is fueled by new ideas. During the process of transforming new ideasinto research outputs scholars rely on their peers to identify weaknesses in their work, andto find alternative models, methodologies and databases that can improve the quality oftheir research. Considering the time scholars devote to present draft versions of their papersat conferences and research seminars, and discuss ideas with colleagues, it is reasonableto expect that peers’ comments and suggestions should improve the quality of research.However, despite its alleged importance, no study has quantified this contribution yet. Thegoal of this paper is to fill this gap.I build a dataset of papers published in economics journals. Based on the acknowledgmentsection of the paper, I record all the scholars that gave comments on the paper, and theseminars and conferences at which the paper was presented. To obtain unbiased estimateson how these individual comments, seminars, and conferences contributed to the quality ofthe paper, I control for the quality of the author and research idea. First, Minondo (2020)shows that high-quality scholars are more likely to be invited to present their work at aresearch seminar. It also reasonable to expect that papers written by high-quality scholarsare more likely to be accepted at conferences. Furthermore, high-quality scholars may receivemore comments on their work because they have more opportunities to interact with otherscholars at seminars and conferences, or because their work is more likely to be followed.Second, it seems reasonable to expect that scholars will choose their most promising projectwhen deciding what paper they will present at a research seminar and what draft they willsend to a colleague.To control for the quality of the author, I use the quality of the institution she is affiliatedwith. In some specifications, I also use author fixed effects. To control for the quality of theresearch idea, I use a feature of the job placement process of PhD candidates in economics.During their last academic year, future PhD graduates in economics select a project, amongtheir contemporaneous research ideas, as their job market paper. This paper is the tool PhDcandidates use to show their research skills to potential employers. Since PhD candidateswant to maximize job offers, they select as job market paper their highest quality project.Thus, the fact that a paper was selected as job market paper provides a signal for the initialquality of a research project. I retrieved information from 2067 PhD candidates in economics,from the top US economics departments, that entered the labor market between 2000 and2018. When the PhD candidate enters the job market, I identify her job market paper andthe additional projects she could also have selected as her job market paper. I follow the jobmarket paper and additional projects until they are published. These publications constitute2y estimation sample.Using the job market status of a paper to control for the quality of the research idea, andauthor’s fixed effect, I find that a one standard deviation increase in the number of individualcomments in a paper that received the average number of individual comments, increasesthe impact factor of the journal in which the paper is published by 16%. A one standarddeviation increase in the number of research seminars in a paper that was presented at theaverage number of seminars increases the impact factor of the journal by 31%. Presentingthe paper at conferences has no impact on the quality of the journal in which the paperis published, once I control for the number of individual comments and seminars. I findthat comments given by high-quality scholars have a larger positive impact on the qualityof a paper than comments received from non-top scholars. Likewise, presenting the paperat a top economics department has a larger positive impact on the quality of the journal inwhich the paper is published than presenting the paper in a non-top economics department.Receiving comments from other scholars and presenting at research seminars have similareffects on theoretical and empirical papers.This paper is related to the literature exploring how knowledge is produced (Stephan,2010; Fortunato et al., 2018) and, in particular, how peers contribute to that process. Azoulayet al. (2010); Waldinger (2012); Borjas and Doran (2015); Agrawal et al. (2017); Jaravelet al. (2018); Bosquet et al. (2019) analyzed how the premature death, migration, or arrivalof scientists affect collaborators’ and other peers’ productivity. My paper contributes to thisliterature by analyzing another channel by which peers’ can affect the quality of a scholar’soutput: the individual and collective feedback on ongoing research projects. Our findingthat peers’ feedback has a large positive effect on the quality of research is in line with Oettl(2012) who found that a scholar’s output quality decreases after the death of a co-author ifthe co-author was helpful to other colleagues.My analysis is also linked with studies that have analyzed how conferences and meetingscontribute to the flow of ideas, to increase the probability of publication, and to enhance thevisibility of a paper. Iaria et al. (2018) found that the ban on Central scientists from partici-pating at international conferences during and after World War I was associated with a dropin citations between Allied and Central scientists. Using data from the Joint MathematicsMeetings between 1990 and 2009, Head et al. (2019) showed that a mathematician is morelikely to cite the work of another mathematician if they coincided in the same conference.This probability increases if the two scholars coincided in the session in which the cited paperwas presented. Using data from a major political science conference that was canceled in2012, Lopez de Leon and McQuillin (2018) concluded that the probability that a paper is3ited increases by five percentage points over a period of four years if it was presented at theconference. Gorodnichenko et al. (2019) found that presenting a paper at major conferencesin economics increases the probability of publishing it in a high-quality journal and enhancesits visibility. I find that presenting a paper at a leading economic conference is associatedwith publishing it at a high-quality journal. However, this positive association becomes sta-tistically insignificant once I control for the number of individual comments received by apaper, and the number of research seminars at which it was presented. This paper is close toBrown (2005), who analyzed whether presentations at research seminars, conferences, andcomments received from colleagues increase the likelihood that a paper receives a revise andresubmit decision at an accounting journal; and whether individual and collective commentsincrease the number of citations received by papers published in three leading accountingjournals. He finds that presenting at research seminars is the only variable that is positivelycorrelated with receiving an invitation to revise and resubmit, and the number of citationsreceived by a paper. I add to this paper analyzing whether the number of individual andcollective comments increase the quality of the journal in which a paper is published. Fur-thermore, I control for the quality of the research idea and author fixed effects, and explorewhether some comments and presentations have a higher impact than others.The remainder of the paper is organized as follows. Section 2 describes the dataset andpresents some summary statistics. Section 3 discusses the results of the regression analyses,and Section 4 concludes.
The sample is composed by PhD candidates from the top 41 US economics departmentsthat entered the labor market between 2000 and 2018. To identify the top US economicsdepartments I use the ranking elaborated by Ideas. Every year, during the fall term,economics departments announce their job market candidates. From the department’s webpage, I recorded each PhD candidate’s job market paper and the projects that she couldalso have selected as job market paper. These were projects whose sole author was the PhDcandidate, or were written with other PhD students. I excluded the papers co-authored withscholars that already had a PhD. I followed the job market paper and papers that couldalso have been selected as a job market paper until they were published. I use the 10-year ranking of US economics departments published in June 2019. The latest ranking isavailable at https://ideas.repec.org/top/top.usecondept.html I included a paper written with a senior scholar if the job market paper was written with the same seniorscholar. Table A.1 in the Appendix reports the economics depart-ments and the PhD candidate cohorts included in the sample. It also reports, for each PhDprogram, the number of graduates from which I could retrieve information, and the numberof potential job market paper projects that became journal articles. There are differencesin the number of PhD candidate cohorts included in the sample across US economics de-partments. Those differences are explained by the possibility of accessing the information of“old” cohorts. Economics departments provide information about the PhD candidates thatenter the labor market in the current year. Few departments also provide links to previousyears’ job market candidates. To retrieve information for older cohorts, I used the InternetArchive Library ( https://archive.org/about/ ). In some cases, the library has a fairlycomplete record of the different versions of the web site over time. However, in many cases,the information is scant, or there is no copy archived. This explains why I could retrieveinformation for “very old” PhD candidates (i.e., 2000) for some economics departments (e.g.,UC Berkeley or MIT), whereas I could only retrieve information about the most recent cohortfor others (e.g., Ohio State). I measure the quality of a paper with the Scimago Journal Ranking (SJR) of the journalin which it was published. Similar to Smeets et al. (2006), I measure the quality of aPhD candidate by the quality of her placement after graduation. To measure the qualityof the placement, I use the worldwide economics institutions ranking elaborated by Ideas. If a paper has multiple authors I add up the quality of individual authors. I computethe individual comments received by a paper counting the scholars that are listed in theacknowledgments section of the paper. I also compute the number of comments given bytop 10 scholars. I count the seminars and conferences at which the paper was presented. I did not include the editors of the journals in the list of scholars that provided comments. I also excludedthe acknowledgments for research assistance, sharing data, or facilitating access to data. There is no correlation between non-archived web sites and the quality of the economics departments. This ranking is built using the average number of weighted citations received in the selected year by thedocuments published in the journal the three previous years. If an author reports more than one affiliation I select her latest academic affiliation. I use the 10-year ranking of institutions published in May 2019. The latest ranking is available at https://ideas.repec.org/top/top.inst.all10.html . The Ideas ranking provides specific scores for thetop 5 institutions (494 institutions). For each percentile between 6 and 10, it lists, randomly, the institutionslocated at that percentile. To provide a score for institutions located between the 6th and the 10th percentile,I ran a regression with the institutions that have a specific score. The dependent variable is the score (inlogs) and the independent variables the percentile in which the institution is located (in logs) and a constant.I use the estimated coefficients to calculate a score for percentiles 6, 7, 8, 9 and 10. If an institution is notat the top 10, I assign it the score of an institution located at the 55th percentile. I use the Ideas’ author ranking for November 2019. The most recent ranking is available at https://ideas.repec.org/top/top.person.all.html Table A.2 in the Appendix provides information about the construction and characteris-tics of the estimation sample. I retrieved information from 2067 PhD candidates that enteredthe job market between 2000 and 2018. These job market candidates were working on 5118projects that could have been selected as job market papers. Among those projects, 2070were selected as job market papers. By December 2019, 551 of the job market candidates(27%) had published their job market paper or another paper they could also have selectedas job market paper in a journal included in the SJR. This percentage is in line with theresults of previous studies that highlighted the low “publication productivity” of PhD grad-uates (Conley and Onder, 2014). A total of 806 out of 5118 potential papers, 16%, hadbeen published by December 2019. 47% of these publications were job market papers. Thispercentage is larger than the share of job market papers among potential projects (40%).18% of publications had more than one author, and 12% were published in a top 5 economicsjournal. I have information on the number of individual comments for all papers in the sam-ple. However, there are some articles that use formulas such as “we acknowledge numerous seminar participants”, “ several audiences”, or “seminar and conference participants”. Sincethe number of seminars could not be computed for these publications, the main estimationsample drops from 806 to 685 articles.Table 1 reports some summary statistics on the individual and collective comments re-ceived by a publication. It provides statistics for all publications, published job marketpapers, and other publications. The median publication received 9 individual comments.The distribution is not skewed: the average is 10 and the standard deviation is 7. The min-imum number of comments received by a publication is zero, whereas the maximum is 49.There are 61 publications, out of 685, with no individual comments. The median publicationwas presented at 1 seminar only. The maximum number of seminars at which a publicationwas presented was 23. There are 263 publications, out of 685, that were not presented atany seminar. Note that the distribution of seminars per publication is skewed, since the To identify the top 10 institutions, I use the ranking built by Ideas mentioned above. Note that the number of job market papers is larger than the number of job market candidates, sincesome PhD students have more than one job market paper. Our percentage is even lower than the 40% figure reported by Conley and Onder (2014), due to thelarger presence of recently graduated students in our sample, whose papers may be still waiting a editorialdecision. American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Eco-nomics, and Review of Economic Studies. able 1: Summary statistics of the estimation sample
Median Mean SD Min MaxAll publicationsIndividual comments 9 10 7 0 49Seminars 1 3 5 0 23Conferences 0 1 1 0 15Job market papersIndividual comments 12 12 8 0 49Seminars 4 6 6 0 23Conferences 0 1 2 0 15OthersIndividual comments 7 8 6 0 32Seminars 1 1 2 0 21Conferences 0 1 1 0 6average number of presentations is much larger than the median. Finally, the median pub-lication was not presented at any conference. The average and the standard deviation is 1.There is a paper that was presented at 15 different conferences, whereas 411 publications,out of 685, were not presented at any conference. Job market papers received more individ-ual comments, and were presented at more research seminars than non job market papers.Specifically, the median job market paper received 5 more individual comments, and waspresented at 3 more seminars than a non job market paper.Panel A of Figure 1 plots a scatter diagram of the relationship between the number ofindividual comments received by a paper and the quality of the journal in which it waspublished. Job market papers are identified by blue dots and non job market papers by redhollow squares. There is a positive correlation between the number of individual commentsreceived by a paper and quality of the journal in which it was published. In Panel B, I addthe seminars and conferences in which a paper was presented, and plot a scatter diagramfor the relationship between the number of times a paper was presented and quality of thejournal in which it was published. There is also a positive correlation between the numberof presentations and quality of the journal in which the paper was presented.These scatter diagrams suggest that peers’ individual and collective comments improvethe quality of a paper. However, these correlations may be capturing the positive associationbetween the quality of the scholar and number of individual and collective comments frompeers; or the quality of the research idea and comments received from peers. In the next7 igure 1:
Scatter diagramsA. Journal quality vs. Number of personal comments J ou r na l qua li t y B. Journal quality vs. Number of presentations J ou r na l qua li t y Note: The quality of the journal is measured by the Scimago Journal Ranking. Presentations is the sum ofresearch seminars and conferences at which a paper was presented. section, I explore the contribution of individual and collective comments to the final quality8f a paper once I control for the quality of the author and research idea.
To estimate the contribution of peer’s individual and collective comments to the quality ofa paper, I estimate the following regression equation: ln Q pajt = β ln I pajt + β ln S pajt + β ln C pajt + β ln Q ajt + β J M P pajt + γ j + γ t + (cid:15) patj (1)where Q pajt is the quality of paper p written by author a who did her doctoral studies atuniversity j and entered the job market at year t . I pajt is the number of individual commentsreceived by paper p ; S pajt and C pajt are the number of seminars and conferences at whichpaper p was presented, respectively. Q ajt is the quality of the author and J M P pajt is anindicator variable that turns 1 if the paper was a job market paper. (cid:15) patj is the disturbanceterm.Since some economics departments may have more social ties with journal editors thanothers (Colussi, 2018), I control for the economics department at which the candidate didher PhD ( γ j ). Due to the time elapsing between submitting a paper and being acceptedfor publication at a journal, papers from ”younger” PhD candidate cohorts are less likely tobe included in the estimation sample. This may create a sample selection problem in thedependent variable. To address this problem, I introduce cohort fixed effects ( γ t ). Theyalso control for other cohort-specific factors that may affect the probability of publishingin a high-quality journal, such as the quality of other PhD candidates that entered thejob market in the same year, or the number of PhD candidates that decided to pursue anacademic career.Table 2 presents the estimates for the impact of the number of comments given by peers’individually and collectively at research seminars and conferences. I cluster standard errorsat the author level. First, I estimate Equation (1) with the number of individual commentsvariable only (column (1) of Table 2). This estimation uses the full sample of publications:806. As expected, the ln Comment coefficient is positive and very precisely estimated. Since the number of individual comments, seminars, and conferences enter in logs in Equation (1), I add1 to the number of comments, seminars, and conferences variables in order to keep the observations withzero values in the estimation sample. Results are robust to using an inverse hyperbolic sine transformationof the variables (Bellarare and Wichman, 2019). able 2: Contribution of individual comments, seminars, and conferences to the quality of a paper (1) (2) (3) (4) (5) (6) (7)ln Comment 0.553 a a a a b (0.052) (0.059) (0.060) (0.057) (0.119)ln Seminar 0.428 a a a a a (0.044) (0.050) (0.048) (0.047) (0.112)ln Conference 0.337 a -0.055 -0.049 -0.025 -0.250(0.069) (0.070) (0.068) (0.065) (0.151)ln Author(s) quality 0.093 a a (0.022) (0.021)Job market paper 0.494 a c (0.076) (0.141)Observations 806 685 685 685 685 685 276R-square 0.376 0.345 0.258 0.400 0.421 0.456 0.282Author(s) FE No No No No No No YesNote: The dependent variable is the journal’s log impact factor. Estimations in Columns (1) to (6) includecohort and PhD institution fixed effects (not reported). Standard errors clustered at the author level are inparentheses. a, b, c: statistically significant at 1%, 5%, and 10%, respectively. This result indicates that receiving more individual comments is positively correlated withpublishing in a high-ranked journal. For example, a one standard deviation increase inthe number of comments (7 comments), for a paper that received the average number ofcomments (10 comments), increases the quality of the journal in which the paper is publishedby 29% [((ln(17)-ln(10))*0.553]. This increase would lift a paper published in a journallocated in the 2nd quartile of the SJR Economics and Econometrics category (e.g., CESifoEconomics Studies; SJR score: 0.851) to a journal located in the 1st quartile (e.g., Journalof Industrial Economics, SJR score: 1.059).In column (2) the number of seminars is the only independent variable. Note that thenumber of observations is lower than in column (1) since, as mentioned above, there aresome papers that do not provide a valid list of seminars. As expected, presenting the paperat research seminars is positively correlated with publishing the paper at a high-rankedjournal. For example, a one standard deviation increase in the number of presentations(5 seminars) for a paper that was presented at the average number of research seminars(3 seminars), increases the quality of the journal in which the paper is published by 42%[((ln(8)-ln(3))*0.428]. There is a positive correlation between the number of conferences inwhich a paper was presented and quality of the journal in which the paper was published(column (3)). Specifically, a one standard deviation increase in the number of conferences101 conference), for a paper that was presented at the average number of conferences (1conference), increases the quality of the journal by 23% [((ln(2)-ln(1))*0.337].Column (4) presents the results when the specification includes all peers’ contributionvariables: individual comments, research seminars, and conferences. The ln Comment andthe ln Seminar coefficients remain positive and very precisely estimated. However, both coef-ficients have a lower point value than in previous estimations. This result indicates that thereis a positive correlation between the number of individual comments received by a paper andnumber of seminars and conferences at which it is presented. Interestingly, the conferencecoefficient is close to zero. This result indicates that the positive association between thenumber of conferences in which a paper is presented and quality of the journal in which itis published disappears once I control for the number of individual comments received bya paper and seminars in which it was presented. According to the coefficients reported incolumn (4), a one standard deviation increase in the number of individual comments and re-search seminars, for a paper that has an average number of comments and seminars, increasesthe quality of the journal in which the paper is published by 49% [((ln(17)-ln(10))*0.397 +(ln(8)-ln(3))*0.288].In column (5), I introduce the quality of the author as an additional regressor. Asexpected, the quality of the author is positively correlated with quality of the journal inwhich the paper is published. There is also a reduction in the ln Comment and ln Seminar coefficients’ point estimates, suggesting that these coefficients were partially capturing thepositive correlation between the quality of the author and journal. Column (6) presents theresults when I control for the quality of the research idea. The job market paper coefficient ispositive and very precisely estimated. According to the coefficient reported in column (6), thequality of journals in which job market papers were published was, on average, 63% higherthan the quality of the journals in which the rest of projects were published (exp .494). The ln Comment and ln Seminar coefficients remain positive and precisely estimated. However,their point values, specially for ln Seminar , are lower than in column (5). This is consistentwith the argument that scholars choose to present their most promising projects when theyare invited to give a research seminar. Even when I control for the quality of the author andresearch idea, a one standard deviation increase in the number of comments and seminars,for a paper with average values of these variables, still increases the quality of the journal inwhich the paper is published by 34% [((ln(17)-ln(10))*0.348 + (ln(8)-ln(3))*0.156].Finally, column (7) reports the estimations when the regression equation includes author I also analyzed whether papers with more than one author had a larger quality than solo papers. Thecoefficient for multi-authored papers was imprecisely estimated. Inthis specification, I identify peers’ contribution to the quality of a paper with the variationin the number of individual and collective comments among papers written by the same au-thor, who were devised and began to be developed during the same period, and whose initialquality was identified by the author. Although I do not have a natural experiment that gen-erates a random variation in the number of individual and collective comments received bya paper, I argue that, conditional on author fixed effects and the initial quality of the paper,the variation is mostly random. This enables me to lean towards a causal interpretation ofestimates.The sample in column (7) only includes scholars that published more than one of theprojects that were initiated when they were doing their doctoral studies. This leads toa large reduction in the number of observations. Despite this drop, and the increase instandard errors, the ln Comment and ln Seminar coefficients remain positive and preciselyestimated. This result confirms that peers’ individual and collective comments improve thequality of a paper. A one standard deviation increase in the number of comments, for apaper that received the average number of comments, increases the quality of the journal inwhich the paper is published by 16% [((ln(17)-ln(10))*0.305]; and a one standard deviationincrease in the number of seminars, for a paper that was presented in the average number ofseminars, increases the quality of the journal by 31% [((ln(8)-ln(3))*0.317]. For example, thecombined effect of these increases, 47%, would lift a paper published in Review of Economicsand Statistics (8.363) to the quality level of The American Review (11.889). Presenting atconferences does not raise the quality of the journal in which the paper is published.In previous estimations, I assumed that all individual and collective comments con-tributed equally to increase the quality of a paper. However, it seems reasonable to expectthat individual comments from top scholars, or comments received at presentations at topeconomics departments, or leading conferences, contribute more to improve the quality of aresearch project.Table 3 presents the results when individual and collective comments are distinguishedby quality. Column (1) shows that comments given by top 10 scholars have a much largerpositive correlation with quality of the paper than comments offered by other scholars. The quality of the author, and the PhD institution and cohort fixed effects are removed from theregression equation since they are collinear with author fixed effects. able 3: Peers’ contribution by quality(1) (2) (3) (4) (5) (6) (7)ln Comment top 10 0.628 a a a a c (0.067) (0.070) (0.070) (0.068) (0.132)ln Comment rest 0.094 0.074 0.100 c a a a a b (0.050) (0.050) (0.049) (0.049) (0.139)ln Seminar rest -0.035 -0.080 -0.061 -0.134 c a b b a a -0.095 -0.088 -0.077 -0.298 c (0.073) (0.072) (0.070) (0.067) (0.154)ln Author(s) quality 0.073 a a (0.021) (0.021)Job market paper 0.489 a c (0.074) (0.141)Observations 806 685 685 685 685 685 276R-square 0.410 0.352 0.260 0.435 0.447 0.481 0.301Author(s) FE No No No No No No Yes Note: The dependent variable is the journal’s log impact factor. Estimations in Columns (1) to (6) includecohort and PhD institution fixed effects (not reported). Standard errors clustered at the author level are inparentheses. a, b, c: statistically significant at 1%, 5%, and 10%, respectively.
Column (2) reports that presenting the paper at a top 10 economics department has astrong positive association with publishing the paper at a high-ranked journal. However,presenting the paper at a non-top 10 economics department has no correlation with qualityof the paper. Presenting the paper at a major economics conference (American EconomicAssociation, European Economic Association, and the Royal Economic Society) has a strongpositive correlation with quality of the journal in which the paper is published. Presentingat other conferences also has a positive coefficient, although its point value is lower. Thequality of the author (column (5)) and the job market status of the paper (column (6))increase the quality of the journal in which the paper is published. In these specifications,the comments offered by top scholars, giving a seminar at top departments, or presenting13he paper at a leading conference have a stronger positive association with quality of thepaper than comments by non-top scholars or presenting the paper at non-top departmentsor conferences. When I control for author fixed effects (column (7)), comments given by top10 scholars have a larger positive impact on quality of the paper than comments given bynon-top scholars. However, the difference between the coefficients is not large. I find thatpresenting the paper at a top 10 economics department has a positive effect on the quality ofthe paper. However, presenting the paper in a non-top economics department has no effect onthe quality of the journal in which the paper is published. The coefficient for top conferencesis positive, but imprecisely estimated. Surprisingly, presenting at a non-top conference hasa negative effect on the quality of the journal in which the paper is published. As explained above, there is an important number of publications (121 out of 806) thatacknowledged the comments received by participants at research seminars and conferences,but did not list the institutions at which these seminars were hold, or the name of the con-ferences. To test the robustness of my results, I re-estimate all specifications with the wholesample (806 observations instead of 685) and removing the number of seminars and confer-ences variables from the regression equation. The estimates for the ln Comment coefficientshould be taken with caution. Since the number of individual comments is correlated withthe number of seminars and conferences, the ln Comment coefficient may also capture theeffect that seminars and conferences have on the quality of the journal in which a paperis published. Table A.3 in the Appendix confirms that individual comments have a strongpositive effect on the quality of the journal in which the paper is published (columns (1)to (4)). Estimates also confirm that the individual comments given by top scholars have astronger effect on the quality of the journal in which the paper is published than commentsprovided by non-top scholars (columns (5) to (8)).Finally, I analyze whether some type of papers benefit more from individual and collectivecomments than others. Following the methodology used in Card et al. (2020), I classifypapers as theoretical, empirical, structural, or experimental based on the counting of somespecific words.
61% of papers in the sample are experimental, and 36% are theoretical.I select these categories and compare whether peers’ comments have a larger impact onempirical than on theoretical papers. I expand Equation (1) with a dummy variable that I also ran regressions using top 5 as the quality threshold for scholars and seminars. Results, not reported,are qualitatively and quantitatively similar to those presented in Table 3. The words used to identify a theoretical paper are proposition, theorem, lemma, proof, model, and theory;for an empirical paper: data, standard error, table, regression, difference-in-differences, and empirical; fora structural paper: structural, BLP, maximum likelihood, mixture, simulation, and calibration; and for anexperimental paper: field experiment, RCT, laboratory, subjects, and survey. The category with the largestnumber of words determines the type the paper belongs to.
A scholar’s knowledge is limited and, therefore, is unaware of all the elements that maycontribute to improve the quality of her research. To discover these elements, she relies onpeers, who at research seminars, conferences, or through conversations, identify limitationsin the research project and suggest avenues to improve it. In this paper, I measured howmuch these comments and suggestions improve the quality of research. Since the numberof suggestions a paper receives is not independent from the quality of the research idea andauthor, I use a sample of papers that enables me to control for these variables: the researchprojects of job market candidates in economics. I find that a one standard deviation increasein the number of individual comments and research seminars increases the quality of thejournal in which the paper is published by 47%. I find that comments provided by topscholars have a stronger positive effect on the quality of the paper than comments given bynon-top scholars. I also show that while presenting a paper at top economics departmentshas a strong positive effect on the quality of the paper, presenting at non-top economicsdepartments has no effect. I find that presenting at conferences, even at the top ones, isnot associated with publishing in a high-ranked journal, once I control for the number ofindividual comments and seminars. Peers’ comments have similar effect on theoretical andempirical papers.My results confirm that peers’ individual and collective comments have a large positiveeffect on the quality of research projects, specially when they come from top scholars orare received when presenting the paper at a top economics department. From a policyperspective, these results justify the use of public funding to organize research seminars,15nteract with other scholars, and finance stays at top economics departments.
References
Agrawal, A., McHale, J., and Oettl, A. (2017). How stars matter: Recruiting and peer effectsin evolutionary biology.
Research Policy , 46(4):853 – 867.Azoulay, P., Graff Zivin, J. S., and Wang, J. (2010). Superstar Extinction.
The QuarterlyJournal of Economics , 125(2):549–589.Bellarare, M. F. and Wichman, C. J. (2019). Elasticities and the inverse hyperbolic sinetransformations.
Oxford Bulletin of Economics and Statistics , forthcoming.Borjas, G. J. and Doran, K. B. (2015). Which peers matter? The relative impacts of collabo-rators, colleagues, and competitors.
The Review of Economics and Statistics , 97(5):1104–1117.Bosquet, C., Combes, P.-P., Henry, E., and Mayer, T. (2019). Peer effects in academicresearch: senders and receivers.
Sciences Po Economics Discussion Paper 2019-16 .Brown, L. D. (2005). The importance of circulating and presenting manuscripts: evidencefrom the accounting literature.
The Accounting Review , 80(1):55–83.Card, D., DellaVigna, S., Funk, P., and Iriberri, N. (2020). Are referees and editors ineconomics gender neutral?
The Quarterly Journal of Economics , 135(1):269–327.Colussi, T. (2018). Social ties in academia: A friend is a treasure.
The Review of Economicsand Statistics , 100(1):45–50.Conley, J. P. and Onder, A. S. (2014). The research productivity of new PhDs in economics:The surprisingly high non-success of the successful.
Journal of Economic Perspectives ,28(3):205–16.Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Pe-tersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., Vespignani, A., Waltman, L., Wang, D.,and Barabási, A.-L. (2018). Science of science.
Science , 359(6379).Gorodnichenko, Y., Pham, T., and Talavera, O. (2019). Conference presentations and aca-demic publishing. Working Paper 26240, National Bureau of Economic Research.16ead, K., Li, Y. A., and Minondo, A. (2019). Geography, ties, and knowledge flows: Evidencefrom citations in mathematics.
Review of Economics and Statistics , 101(4):713–727.Iaria, A., Schwarz, C., and Waldinger, F. (2018). Frontier knowledge and scientific pro-duction: Evidence from the collapse of international science.
The Quarterly Journal ofEconomics , 133(2):927–991.Jaravel, X., Petkova, N., and Bell, A. (2018). Team-specific capital and innovation.
AmericanEconomic Review , 108(4-5):1034–73.Lopez de Leon, F. L. and McQuillin, B. (2018). The role of conferences on the pathwayto academic impact: Evidence from a natural experiment.
Journal of Human Resources ,Online publication ahead of print.Minondo, A. (2020). Who presents and where? an analysis of research seminars in USeconomics departments. arXiv 2001.10561 .Oettl, A. (2012). Reconceptualizing stars: Scientist helpfulness and peer performance.
Man-agement Science , 58(6):1122–1140.Smeets, V., Warzynski, F., and Coupé, T. (2006). Does the academic labor market initiallyallocate new graduates efficiently?
Journal of Economic Perspectives , 20(3):161–172.Stephan, P. E. (2010). The economics of science. In Hall, B. H. and Rosenberg, N., editors,
Handbook of The Economics of Innovation, Vol. 1 , volume 1, pages 217 – 273. Elsevier.Waldinger, F. (2012). Peer effects in science: Evidence from the dismissal of scientists inNazi Germany.
The Review of Economic Studies , 79(2):838–861.17 able A.1:
Graduate programs in economics included in the sample
University Cohort Graduates PublicationsArizona State 2016, 2018 12 0Boston 2014, 2015, 2016, 2017, 2018 84 11Boston College 2018 5 0Brown 2014, 2015, 2016, 2018 30 12Chicago 2008, 2010, 2011, 2013, 2015, 2018 93 40Columbia 2017, 2018 39 4Cornell 2017, 2018 38 3Duke 2016, 2017, 2018 31 8George Washington 2013, 2014, 2015, 2016, 2017, 2018 39 10Georgetown 2015, 2016, 2017, 2018 18 4Harvard 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 196 76Iowa State 2017 3 0Johns Hopkins 2011, 2012, 2015, 2016, 2017 34 13MIT 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008,2010, 2011, 2012, 2016 , 2017, 2018 242 192Maryland 2015, 2016, 2017, 2018 45 2Michigan 2018 17 0Michigan State 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 84 35Minnesota 2016, 2017, 2018 38 1New York 2017, 2018 33 0Northwestern 2000, 2001, 2003, 2004, 2005, 2006, 2007, 2008 78 61Notre Dame 2011, 2012, 2013, 2015, 2016, 2018 20 12Ohio State 2018 10 4Oregon 2007, 2008, 2010, 2011, 2012, 2013, 2015, 2016, 2018 31 15Penn State 2017 8 0Pittsburgh 2014, 2015, 2016, 2017, 2018 23 13Princeton 2014, 2015, 2016, 2017, 2018 94 16Rutgers 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 47 8Southern California 2016, 2017, 2018 23 7Stanford 2008, 2011, 2012, 2014, 2015, 2016, 2017, 2018, 2019 129 63Texas Austin 2015, 2016, 2018 30 4UC Berkeley 2000, 2002, 2003, 2004, 2005, 2006, 2007, 2009, 2011,2018 187 128UC Davis 2018 7 1UC Irvine 2014, 2015, 2016, 2018 45 31UC Los Angeles 2017, 2018 33 3UC San Diego 2016, 2017, 2018 46 14UC Santa Barbara 2016 12 6UC Santa Cruz 2014, 2015, 2016, 2017, 2018 29 3Vanderbilt 2014, 2015, 2016, 2018 22 9Virginia 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2009,2011, 2016, 2017, 2018 55 14Wisconsin-Madison 2018 16 0Yale 2015, 2016, 2018 41 9Note: Cohort is the year when job market candidates were announced. able A.2: Information about the sample
Job market candidates 2067Potential papers 5118Job market candidates with a publication in a SJR journal 551Publications in a SJR journal 806Main estimation sample 685
Table A.3: