Contest models highlight inherent inefficiencies of scientific funding competitions
CContest models highlight inherent inefficienciesof scientific funding competitions
Kevin Gross ∗ Department of StatisticsNorth Carolina State UniversityRaleigh, NC USA
Carl T. Bergstrom † Department of BiologyUniversity of WashingtonSeattle, WA USA (Dated: January 4, 2019)Scientific research funding is allocated largely through a system of soliciting and ranking com-petitive grant proposals. In these competitions, the proposals themselves are not the deliverablesthat the funder seeks, but instead are used by the funder to screen for the most promising researchideas. Consequently, some of the funding program’s impact on science is squandered because ap-plying researchers must spend time writing proposals instead of doing science. To what extentdoes the community’s aggregate investment in proposal preparation negate the scientific impact ofthe funding program? Are there alternative mechanisms for awarding funds that advance sciencemore efficiently? We use the economic theory of contests to analyze how efficiently grant proposalcompetitions advance science, and compare them with recently proposed, partially randomized al-ternatives such as lotteries. We find that the effort researchers waste in writing proposals may becomparable to the total scientific value of the research that the funding supports, especially whenonly a few proposals can be funded. Moreover, when professional pressures motivate investigatorsto seek funding for reasons that extend beyond the value of the proposed science (e.g., promotion,prestige), the entire program can actually hamper scientific progress when the number of awards issmall. We suggest that lost efficiency may be restored either by partial lotteries for funding, or byfunding researchers based on past scientific success instead of proposals for future work.
INTRODUCTION
Over the past fifty years, research funding in theUnited States has failed to keep pace with growth inscientific activity. Funding rates in grant competitionshave plummeted (Fig. S1, [1–4]) and researchers spendfar more time writing grant proposals than they did inthe past [5]. A large survey of top U.S. universities foundthat, on average, faculty devote 8% of their total time —and 19% of their time available for research activities— towards preparing grant proposals [6]. Anecdotally,medical school faculty may spend fully half their time ormore seeking grant funding [5, 7]. While the act of writ-ing a proposal may have some intrinsic scientific value[8] — perhaps by helping an investigator sharpen ideas— much of the effort given to writing proposals is ef-fort taken away from doing science [9]. With respect toscientific progress, this time is wasted [10].Frustrated with the inefficiencies of the current fund-ing system, some researchers have called for an overhaulof the prevailing funding model [9, 11–18]. In particular,Fang & Casadevall [16] recently suggested a partial lot-tery, in which proposals are rated as worthy of fundingor not, and then a subset of the worthy proposals are ∗ [email protected] † [email protected] randomly selected to receive funds. Arguments in favorof a partial lottery include reduced demographic and sys-temic bias, increased transparency, and a hedge againstthe impossibility of forecasting how scientific projects willunfold [16]. Indeed, at least three funding organizations— New Zealand’s Health Research Council and their Sci-ence for Technological Innovation program, as well as theVolkswagen Foundation [19] — have recently begun usingpartial lotteries to fund riskier, more exploratory science.Compared with a proposal competition, a lottery per-mits more proposals to qualify for funding and thus low-ers the bar that applicants must clear. A lottery alsooffers a lower reward for success, as a successful proposalreceives a chance at funding, not a guarantee of funding.Thus we expect that investigators applying to a partiallottery will invest less time and fewer resources in writ-ing a proposal. To a first approximation, then, a proposalcompetition funds high-value projects while wasting sub-stantial researcher time on proposal preparation, whereasa partial lottery would fund lower-value projects on aver-age but would reduce the time wasted writing proposals.It is not obvious which system will have the greater netbenefit for scientific progress.In this article, we study the merits and costs of tra-ditional proposal competitions versus partial lotteries bysituating both within the rich economic theory of con-tests . In this theory, competing participants make costlyinvestments (“bids”) in order to win one or more prizes[20, 21]. Participants differ in key attributes, such as a r X i v : . [ phy s i c s . s o c - ph ] J a n ability and opportunity cost, that determine their op-timal strategies. In an economics context, contests areoften used by the organizer as a mechanism to elicit ef-fort from the participants. For example, TopCoder andKaggle are popular contest platforms for tech firms (theorganizers) to solicit programming or data-analysis effortfrom freelance workers (the participants). However, be-cause the participants’ attributes influence their optimalstrategies, the bids that participants submit reveal thoseattributes. Thus, screening of participants often arises asa side effect.In funding competitions, the organizer is a fundingbody and the participants are competing investigators.Investigators pitch project ideas of varying scientificvalue by preparing costly proposals. However, unlikea traditional economic contest, the funding body’s pri-mary objective is to identify the most promising science,using proposals to screen for high-value ideas. The fund-ing body has little interest in eliciting work during thecompetition itself, as the proposals are not the deliver-ables the funder seeks. All else equal, the funding agencywould prefer to minimize the work that goes into prepar-ing proposals, to leave as much time as possible for in-vestigators to do science. In this case, how should thefunder organize the contest to support promising sciencewithout squandering much of the program’s benefit ontime wasted writing proposals?Below, we pursue this question by presenting and ana-lyzing a contest model for scientific funding competitions.We first use the model to assess the efficiency of proposalcompetitions for promoting scientific progress, and askhow that efficiency depends on how many proposals arefunded. We next explore how efficiency is impacted whenextra-scientific incentives such as professional advance-ment motivate scientists to pursue funding, and comparethe efficiency of proposal competitions versus partial lot-teries. Finally, we reflect on alternative ways to improvethe efficiency of funding competitions without adding in-tentional randomness to the award process. All of ouranalyses focus on equilibrium behavior, and thus per-tain most directly to long-standing funding competitionsfor which researchers can acquire experience that informstheir future actions. A CONTEST MODEL FOR SCIENTIFICFUNDING COMPETITIONS
Our model draws upon a framework for contests de-veloped by Moldovanu & Sela [20]. In our application,a large number of scientists (or research teams) competefor grants to be awarded by a funding body. The fundercan fund a proportion p of the competing investigators.We call p the payline, although p could be smaller thanthe proportion of investigators who are funded if someinvestigators do not enter the competition.Project ideas vary in their scientific value, which wewrite as v , where v ≥
0. In this case, scientific value com- bines the abilities of the investigator and the promise ofthe idea itself. Although we do not assign specific unitsto v , scientific value can be thought of as some measure ofscientific progress, such as the expected number of pub-lications or discoveries. We assume that the funder seeksto advance science by maximizing the scientific value ofthe projects that it funds, minus the value of the sciencethat investigators forgo while writing proposals. How-ever, the funder cannot observe the value of a projectidea directly. Instead the funder evaluates proposals forresearch projects, and awards grants to the top-rankedproposals. Assume that proposals can be prepared todifferent strengths, denoted x ≥
0, with a larger value of x corresponding to a stronger proposal. A scientist witha project idea of value v must decide how much effortto invest in writing a proposal, that is, to what strength x her proposal should be prepared. In our model, thisdecision is made by a cost-benefit optimization.On the benefit side, if a proposal is funded, the inves-tigator receives a reward equal to the scientific value ofthe project, or v . This reward is public, in the sense thatit benefits both the investigator and the funder. Receiv-ing a grant may also bestow an extra-scientific reward onthe recipient, such as prestige, promotion, or professionalacclaim. Write this extra-scientific reward as v ≥ η ( x ) bethe equilibrium probability that a proposal of strength x is funded; η ( x ) will be a non-decreasing function of x .Thus, in expectation, an investigator with a project ofvalue v who prepares a proposal of strength x receives abenefit of ( v + v ) η ( x ).Preparing a grant proposal also entails a disutility cost,equal to the value of the science that the investigatorcould have produced with the time and resources investedin writing. Let c ( v, x ) give the disutility cost of preparinga proposal of strength x for a project of value v . Here,we study the case where c ( v, x ) is a separable function of v and x , so we set c ( v, x ) = g ( v ) h ( x ). Proposal compe-titions are effective screening devices because it is easierto write a strong proposal about a good idea than abouta poor one. Therefore, g ( v ) is a decreasing function of v , i.e., g (cid:48) ( v ) <
0. For a given idea, it takes more workto write a stronger proposal, and thus h (cid:48) ( x ) >
0. Fi-nally, we assume that preparing a zero-strength proposalis tantamount to opting out of the competition, whichcan be done at zero cost. Thus h (0) = 0.Preparing a proposal has some scientific value of itsown through the sharpening of ideas that writing a pro-posal demands [8]. Let k ∈ [0 ,
1) be the proportion ofthe disutility cost c ( v, x ) that an investigator recoups byhoning her ideas. We call the recouped portion of thedisutility cost the intrinsic scientific value of writing aproposal. The portion of the disutility cost that cannotbe recouped is scientific waste.All told, the total benefit to the investigator of prepar-ing a proposal to strength x is ( v + v ) η ( x ) + kc ( v, x ),and the total cost is c ( v, x ). The difference between thebenefit and the cost is the investigator’s payoff. The in-vestigator’s optimal proposal (or, in economic terms, her“bid”) maximizes this payoff (Fig. 1): b ( v ) = arg max x { ( v + v ) η ( x ) − (1 − k ) c ( v, x ) } . (1)For simplicity, we assume that variation among projectsis captured entirely in the distribution of v , which wewrite as F ( v ). We assume that v and k have commonvalues shared by all investigators. In the appendix weshow that our results extend to cases where v or k varyamong investigators, as long as they are perfectly corre-lated with v . B ene f i t o r c o s t FIG. 1.
An investigator prepares her grant proposalto the strength that maximizes her payoff.
The bluecurve shows the expected benefit to the investigator, whichis determined by the project’s value, any extra-scientific re-ward that the investigator receives from getting the grant,the probability of receiving funding, and the intrinsic value ofwriting a proposal. The red curve shows the disutility cost ofpreparing a proposal. The investigator’s payoff is the differ-ence between the benefit and the cost. The vertical line showsthe bid (eq. 1) — the proposal strength that maximizes thepayoff. At the bid, the ratio of the payoff (given by the lengthof the solid vertical line) to the cost (given by the length ofthe dashed vertical line) gives the investigator’s return on herinvestment.
The challenge in finding the payoff-maximizing bid b ( v )is that the equilibrium probability of funding, η ( x ), mustbe determined endogeneously, in a way that is consistentwith both the payline p and the distribution of bids thatinvestigators submit. In Appendix S1, we follow Hoppe et al. [22] to show that, at equilibrium, the bid functionis given by b ( v ) = h − (cid:20) − k (cid:90) v v + tg ( t ) ξ (cid:48) ( t ) dt (cid:21) . (2)In eq. 2, ξ ( v ) = η ( b ( v )) is the equilibrium probabilitythat an idea of value v is funded. The particular formof ξ ( v ) depends on how much randomness is introducedduring the review process, which we discuss below.By comparison, Moldovanu & Sela [20] considered acontest with a small number of competitors, in whichthe contest’s judges observe x directly. In their set-up,each contestant is uncertain about the strength of her competition (that is, her competitors’ types, v ), but shecan be certain that the strongest bid will win the topprize. In our case, we assume that the applicant pool islarge enough that the strength of the competition (i.e.,the distribution of v among the applicants) is predictable.However, the funding agency does not observe x directly,but instead convenes a review panel to assess each pro-posal’s strength. Variability among reviewers’ opinionsthen introduces an element of chance into which propos-als get funded. Scientific efficiency
We use the model to explore how efficiently the grantcompetition advances science. From the perspective ofan individual investigator, the investigator’s return onher investment (ROI) is the ratio of her payoff to thecost of her bid:Investigator’s ROI = ( v + v ) η ( b ( v )) − (1 − k ) c ( v, b ( v )) c ( v, b ( v )) . (3)An investigator will never choose to write a proposal thatgenerates a negative payoff, because she can always ob-tain a payoff of 0 by opting out. (If the investigator optsout, eq. 3 evaluates to 0/0, in which case we define herROI to be 0.) Thus, an investigator’s equilibrium ROImust be ≥ p (cid:90) vη ( b ( v )) dF ( v ) , (4)and the average scientific waste per funded proposal is1 p (cid:90) (1 − k ) c ( v, b ( v )) dF ( v ) . (5)We will refer to the difference between these two quanti-ties as the scientific gain (or loss, should it be negative)per funded proposal, which is our measure of the fundingprogram’s scientific efficiency.Note that while an investigator will never enter a grantcompetition against her own self interest, there is noguarantee that the scientific value per funded proposalwill exceed the scientific waste. This is because the inves-tigator’s payoff includes private, extra-scientific rewardsobtained by winning a grant ( v ), and (in our account-ing, at least) these extra-scientific rewards do not benefitthe funding agency. If extra-scientific motivations forwinning grants are large enough, investigators may entera grant competition even when doing so decreases theirscientific productivity. If enough investigators are moti-vated accordingly, then the scientific progress sacrificedto writing proposals could exceed the scientific value ofthe funding program. In this case, the grant competi-tion would operate at a loss to science, and the fund-ing agency could do more for science by eschewing theproposal competition and spreading the money evenlyamong active researchers in the field, or by giving themoney to researchers selected entirely at random. ANALYSIS AND NUMERICAL RESULTS
We illustrate the model’s behavior by choosing a fewpossible sets of parameter values. Our parameter choicesare not directly informed by data. Thus, while the nu-merical examples illustrate the model’s possible behav-ior, we highlight the results that are guaranteed to holdin general. Throughout, we use the following baselineset of parameters. We assume that the project values, v ,have a triangular distribution ranging from v min = 0 . v max = 1 with a mode at v min , such that low-valueideas are common and high-value ideas are rare (i.e., F ( v ) = 1 − (16 / − v ) ). For the cost function, wechoose c ( v, x ) = x /v . We choose a convex dependenceon x to suggest that the marginal cost of improving aproposal increases as the proposal becomes stronger. Weassume that the intrinsic scientific value of writing a pro-posal allows investigators to recoup k = 1 / v = 0), and then introduceextra-scientific benefits ( v = 0 . copula [26] to specify the joint distribution ofa proposal’s actual quantile, and its quantile as assessedby the funding agency’s review panel. A bivariate copulais a probability distribution on the unit square that hasuniformly distributed marginals, as all quantiles must.We use a Clayton copula [27], which allows for accurateassessment of weak proposals, but noisier assessment ofstrong proposals (Fig. S2). This choice is motivated bythe pervasive notion that review panels can readily dis-tinguish strong proposals from weak ones, but struggleto discriminate among strong proposals [16, 25, 28]. AClayton copula has a single parameter ( θ ) that controlshow tightly its two components are correlated. Ratherarbitrarily, we use θ = 10 in the baseline parameter set.The Clayton copula has the important property that aproposal’s probability of funding increases monotonicallyas its strength increases, regardless of the payline. Thus,we exclude the possibility that panels systematically fa-vor weaker proposals. By using a copula, we implicitly assume that η ( x ) depends on x only through its rank. InAppendix S1, we show how a copula leads to an equationfor ξ (cid:48) ( v ), which can then be plugged in to eq. 2.Fig. 2 shows numerical results for the baseline param-eters at generous ( p = 45%) and low ( p = 15%) paylines.In this particular case, investigators’ payoffs fall fasterthan costs as paylines drop, leading to a reduced ROIfor everyone at the lower payline (Fig. 2B). We will ar-gue below that every investigator’s ROI must inevitablyfall when the payline becomes small (see Figs. S3–S4 foradditional examples). B ene f i t o r c o s t Proposal strength (x) A benefit, p = 45%benefit, p = 15%cost B I n v e s t i ga t o r ' s R O I Quantile of scientific value
45% payline orlottery line15% payline orlottery line
FIG. 2.
Diminishing paylines reduce investigators’ re-turns on their investments in a proposal competition.
A: Equilibrium benefit (blue or green) and cost (red) curvesfor an investigator with a project at the 90th percentile of v in a proposal competition. The blue and green curves showthe benefit at a 45% and 15% payline, respectively. Verti-cal lines show the investigator’s equilibrium bid, with solidportions giving the investigator’s payoff, and dashed portionsshowing the cost. The corner in the benefit curves appears atthe strongest proposal submitted. B: An investigator’s ROI(eq. 3) in a proposal competition with 45% (solid line) or15% (dashed line) paylines, as a function of the quantile ofthe scientific value of her project, F ( v ). These curves alsogive the investigator’s ROI in a partial lottery with a 45%or 15% lottery line, and any payline. These results use thebaseline parameters. From the funding agency’s perspective, with our base-line parameters, both the average scientific value andaverage waste per funded proposal increase as the pay-line falls, for paylines below 50% (Fig. 3A). However, asthe payline decreases, waste escalates more quickly thanscientific value, reducing the scientific gain per fundedproject (Fig. 3B). This same result also appears in ouralternative parameter sets (Fig. S5–S6). We will arguebelow that the decline in scientific efficiency at low pay-lines is an inevitable if unfortunate characteristic of pro-posal competitions.Clearly, quantitative details of the model’s predictionsdepend on the parameter inputs. To understand therobustness of these predictions, it helps to study thecase where panels discriminate perfectly among propos-als. While perfect discrimination is obviously unrealisticin practice, it yields a powerful and general set of resultsthat illuminate how the model behaves when discrimi-nation is imperfect. Numerical results for perfect dis- A Avg. scientific value or wasteper funded proposal valuewaste, v = = −0.050.000.050.100.150.200.250.30 B
10% 30% 50%Scientific gain or lossper funded proposal v = = Payline or lottery line
FIG. 3.
Both decreasing paylines and extra-scientificrewards to investigators reduce the scientific effi-ciency of the funding program.
A: Both the averagescientific value per funded proposal (blue line, eq. 4) and theaverage waste per funded proposal (red lines, eq. 5) are higherfor lower paylines, for the baseline parameter set and paylines ≤ v = 0); the dashed red line shows costs when investigatorsare additionally motivated by private, extra-scientific rewards( v = 0 . v = 0, solid line) andpresence ( v = 0 .
25, dashed line) of extra-scientific benefitsto investigators. Values of other parameters are as specifiedin the main text. Identical results hold if the horizontal axisis re-interpreted as the proportion of proposals that qualifyfor a lottery, regardless of the payline. crimination under the baseline parameter set appear inFigs. S7–S8.At equilibrium under perfect assessment, every projectabove a threshold value v (cid:63) = F − (1 − p ) will receivefunding, and no project idea below this threshold willbe funded. Investigators with projects of value v > v (cid:63) all prepare proposals to the identical strength x (cid:63) = h − [( v + v (cid:63) ) / ((1 − k ) g ( v (cid:63) ))], and are funded with cer-tainty. Investigators with projects of value v < v (cid:63) optout (Fig. S7). All of the subsequent results follow. (De-tails appear in Appendix S1.) First, as paylines drop, allinvestigators realize either a diminishing or zero ROI, be-cause investigators who remain in the competition mustpay a higher cost for a reduced payoff. Second, the av-erage scientific value per funded proposal must increaseas paylines drop, because only the highest-value projectsare funded under low paylines. Third, in the limitingcase when only one of many proposals can be funded(technically, the limit as p approaches 0 from above), thescientific value and scientific waste associated with thelast funded project converge, and science is no better offthan if no grant had been given at all (Fig. S8).With perfect assessment, there is no general relation-ship between the scientific efficiency of a proposal com-petition and the payline that holds across the full rangeof paylines (but see Hoppe et al. [22] for a sharp resultwhen the cost function is independent of v ). Of course, we wouldn’t expect scientific efficiency to decline mono-tonically with a falling payline, because there are likelyto be some low-value projects that can be weeded out atlow cost. However, our last result above guarantees thatthe scientific gain per funded proposal must eventuallyvanish as the payline declines to a single award.Returning to the reality of imperfect discrimination, aslong as review panels do not systematically favor weakerproposals, noisy assessment changes little about thesequalitative results. That is, investigators’ ROIs will dropas paylines fall, the average scientific value per fundedproposal will increase as paylines decrease, and the sci-entific efficiency of the proposal competition must even-tually decline as the payline approaches a single award.But efficiency need not drop to zero. Perhaps counter-intuitively, imperfect discrimination is a saving grace atlow paylines. Noisy assessment discourages top inves-tigators from pouring excessive effort into grant-writingas paylines fall, because the marginal benefit of writingan even better grant becomes small when review panelsstruggle to discriminate among top proposals. Indeed,noisy assessment, unlike perfect discrimination, allows aproposal competition to retain a positive impact on sci-ence even with a single funded grant (compare Figs. 3and S8). This result hints at the salutary nature of ran-domness at low paylines, which we will see more vividlywhen we consider lotteries below.Thus far, we have considered the case where investi-gators are motivated only by the scientific value of theprojects proposed ( v = 0). Now suppose that investi-gators are additionally motivated by the extra-scientificbenefits of receiving a grant, such as professional ad-vancement or prestige ( v > LOTTERIES
Our model can also be used to analyze the efficiencyof a partial lottery for advancing science. Suppose thata fraction q ≥ p of proposals qualify for the lottery, andeach qualifying proposal is equally likely to be chosen forfunding. Call q the “lottery line”. Now, the investigator’spayoff is ( p/q )( v + v ) η l ( x ) − (1 − k ) c ( v, x ), where η l ( x )is the equilibrium probability that the proposal qualifiesfor the lottery. In Appendix S1, we show that the inves-tigator’s bid is given by b ( v ) = h − (cid:20) pq − k (cid:90) v v + tg ( t ) ξ (cid:48) l ( t ) dt (cid:21) (6)where ξ l ( v ) = η l ( b ( v )).Our major result for lotteries is that measures of sci-entific efficiency — expressions 3, 4, and 5 — depend onthe lottery line q but are independent of the payline p (proofs appear in Appendix S1). This result follows fromthe fact that, in a lottery, each investigator’s benefit andcost are proportional to p . Thus, an investigator’s ROIand the scientific efficiency of the funding program aredetermined by the lottery line, but are not affected bythe payline. To illustrate, Fig. 4 compares an investiga-tor’s costs and benefits in a proposal competition with45%, 30%, and 15% paylines vs. a partial lottery with a q = 45% lottery line and the same three paylines. Thekey feature of Fig. 4 is that the investigator’s benefitcurve in a partial lottery scales in such a way that herROI is the same for any payline ≤ q . Consequently, apartial lottery with a lottery line of q and any payline ≤ q achieves the same scientific efficiency as a proposalcompetition with a payline of q .Thus, our numerical results showing the investigator’sROI (Fig. 2B) or the scientific efficiency (Fig. 3) in afunding competition also show the efficiency of a lotterywith the equivalent lottery line. That is, a lottery inwhich 45% of applicants qualify for the lottery has thesame scientific efficiency as a proposal competition witha 45% payline, regardless of the fraction of proposals cho-sen for the lottery that are ultimately funded. Thus, alottery can restore the losses in efficiency that a proposalcompetition suffers as paylines become small. B ene f i t o r c o s t Proposal strength (x)A: Proposal competition benefit, p = 45%benefit, p = 30%benefit, p = 15%cost B ene f i t o r c o s t Proposal strength (x)B: Qualifying lottery benefit, p = 45%benefit, p = 30%benefit, p = 15%cost
FIG. 4.
An investigator’s ROI falls as the payline dropsin a proposal competition, but is independent of thepayline in a partial lottery.
A: Benefit (blue or green) andcost (red) curves for an investigator with a project at the 90thpercentile of v in a proposal competition. The dark blue, lightblue, and green curves shows benefits with a 45%, 30%, and15% payline, respectively. Vertical lines show the investiga-tor’s bid, with the length of the solid portion giving the payoff,and the length of the dashed portion giving the cost. The in-vestigator’s ROI declines as the payline decreases. (Note alsothat her effort does not vary monotonically with the payline.)B: The same investigator’s benefit and cost curves in a par-tial lottery with a 45% lottery line. The investigator’s ROIis the same for all paylines. These results use the baselineparameter set given in the main text. In Appendix S1, we also analyze a more general typeof lottery in which proposals are placed into one of a small number of tiers, with proposals in more selectivetiers awarded a greater chance of funding [13, 17, 29]. Ina multi-tier lottery, the efficiency is entirely determinedby the number of tiers and the relative probabilities offunding in each, and is independent of the payline. Nu-merical results (Fig. S9) illustrate that the scientific valueand waste of a multi-tier lottery fall in between those ofa proposal competition and a single-tier lottery. Thus,a multi-tier lottery offers an intermediate design thatwould partially reduce the waste associated with prepar-ing proposals, while still allowing review panels to rewardthe best proposals with a higher probability of funding.
DISCUSSION
Our major result is that proposal competitions are in-evitably and inescapably inefficient mechanisms for fund-ing science when the number of awards is smaller than thenumber of meritorious proposals. The contest model pre-sented here suggests that a partially randomized schemefor allocating funds — that is, a lottery — can restorethe efficiency lost as paylines fall, albeit at the expense ofreducing the average scientific value of the projects thatare funded.Why does a lottery disengage efficiency from the pay-line, while a proposal competition does not? For inves-tigators, proposal competitions are, to a first approxi-mation, all-or-nothing affairs — an investigator only ob-tains a substantial payoff if her grant is funded. At highpaylines (or, more precisely, when the number of awardsmatches the number of high-value projects), investiga-tors with high-value projects can write proposals thatwin funding at modest cost to themselves. As the num-ber of awards dwindles, however, competition stiffens.Depending on the details of the assessment process, aninvestigator with a high-value project must either workharder for the same chance of funding, or work just ashard for a smaller chance of funding. Either way, the re-turn on her investment declines sharply. Thus, a contestis most efficient at the payline that weeds out low-valueprojects, but does not attempt to discriminate amongthe high-value projects (e.g., Fig. S6). At lower pay-lines, however, the effort needed to signal which projectsare most valuable begins to approach the value of thoseprojects, making the funding program less worthwhile.In a lottery, investigators do not compete for awards per se , but instead compete for admission to the lottery.The value to the investigator of being admitted to the lot-tery scales directly with the number of awards. It turnsout that both the investigator’s expected benefit and hercosts of participation scale directly with the payline, andthus the payline has no effect on efficiency. (In AppendixS1, we follow Hoppe et al. [22] to show that this scalingcan be explained by the economic principle of revenueequivalence.) If there are fewer awards than high-valueprojects, a lottery that weeds out the low-value projectsbut does not attempt to discriminate among high-valueprojects will facilitate scientific progress more efficientlythan a contest.Unfortunately, empirical comparisons between the ef-ficiencies of funding competitions versus partial lotteriesdo not yet exist, to the best of our knowledge. However,two recent anecdotes support our prediction that thewaste in proposal competitions is driven by the strate-gic dynamics of the contest itself. First, in 2012, theU.S. National Science Foundation’s Divisions of Envi-ronmental Biology and Integrative Organismal Systemsswitched from a twice-annual, one-stage proposal compe-tition to a once-annual, two-stage competition, in part toreduce applicants’ workload. However, the switch failedto reduce the applicants’ aggregate workload meaning-fully [30], and the two-stage mechanism was subsequentlyabandoned. Second, in 2014, the National Health andMedical Research Council of Australia streamlined theprocess of applying for their Project Grants, cutting thelength of an application in half [31]. However, researchersspent more time, not less, preparing proposals after theprocess had been streamlined, both individually and inaggregate [31]. Both of these experiences are consistentwith our prediction that, in a proposal competition, theeffort applicants expend is dictated by the value of fund-ing to the applicants and the number of awards available,but does not depend on the particular format of the pro-posals.A lottery is a radical alternative, and may be polit-ically untenable [32]. If a lottery is not viable, an al-ternative approach to restoring efficiency is to design acontest where the effort given to competing for awardshas more direct scientific value. For example, a contestthat rewards good science in its completed form — as op-posed to rewarding well-crafted proposals that describefuture science — motivates the actual practice of goodscience, and will be less wasteful at low paylines [9, 14].Program officers could be given the discretion to allo-cate some funds by proactively scouting for promisingresearchers or projects. Of course, a contest based oncompleted science or scouting has its own drawbacks, in-cluding rich-getting-richer feedback loops, a risk of newbarriers to entry for investigators from historically under-represented demographic groups, and the Goodhart’s lawphenomenon, whereby a metric that becomes a targetceases to be a good metric [33]. Nevertheless, it is tan-talizing to envision a world in which the resources thatuniversities currently devote to helping researchers writeproposals are instead devoted to helping researchers doscience.This analysis also shows that extra-scientific profes-sional incentives to pursue grant funding can damage thescientific efficiency of a proposal competition. As many ofthese extra-scientific incentives arise from administratorsusing grant success as a primary yardstick of professionalachievement, perhaps one major benefit of adding ex-plicit randomness to the funding mechanism would be tocompel administrators to de-emphasize grant success inprofessional evaluations. Alternatively, to the degree that administrators value and reward grant success becauseof the associated overhead funds that flow to the univer-sity, funding agencies could reduce waste by distributingoverhead separately from funding awards. Instead, per-haps overhead could be allocated based partially on therecent past productivity of investigators at qualifying in-stitutions, among other possible criteria. Disengagingoverhead from individual grants would encourage admin-istrators to value grants for the science those grants en-able (as opposed to the overhead they bring), while allo-cating overhead based on institutions aggregate scientificproductivity would motivate universities to help their in-vestigators produce good science.Funding agencies often have pragmatic reasons to em-phasize the meritocratic nature of their award processes.However, our model also suggests that downplaying el-ements of a funding competition’s structure that intro-duce randomness to funding decisions can increase scien-tific waste. When applicants fail to recognize the degreeto which the contest is already a lottery, they will over-invest effort in preparing proposals, to the detriment ofscience.This model does not account for all of the costs orscientific benefits of a proposal competition, includingthe costs of administering the competition, the time lostto reviewing grant proposals, or the benefit of buildingscientific community through convening a review panel.Nonetheless, we suggest that the direct value of the sci-ence supported by funding awards and the disutility costsof preparing grant proposals are the predominant scien-tific benefits and costs of the usual proposal competition[13, 30], and provide a useful starting point for a moredetailed accounting.Our model also makes several simplifying assumptions,each of which may provide scope for interesting futurework. First, researchers pay a time cost to prepare aproposal, but receive money if the proposal is funded. Inour model, we have converted both time and money intoscientific productivity, in order to place both on a com-mon footing. To be more explicit, though, scientific pro-ductivity requires both time and money (among other re-sources), and researchers may have vastly different needsfor both. In the Appendix, we show that our model canbe formally extended to encompass researchers’ differentneeds for time and money if the marginal rate of tech-nical substitution (that is, the rate at which time andmoney can be exchanged without altering scientific pro-ductivity) is exactly correlated with the project’s scien-tific value. Our main results still hold in this case, aslong as researchers with the best ideas do not value timeso greatly that they write the weakest proposals. A moregeneral exploration of researchers’ heterogeneous needsfor time and money — and of how researchers may ad-just their portfolio of scientific activities when time ormoney is scarce — provide ample opportunity for futurework.Second, our model assumes that the distribution ofthe scientific value ( v ) across possible projects is ex-ogeneous to the structure of the funding competition.This may not be the case if, for instance, a partial lot-tery encourages participation by investigators with un-conventional views, reduces the psychological stigma ofprevious rejection [16], or discourages investigators ei-ther who have succeeded under the traditional proposal-competition format or who perceive a lottery as riskier.In reality, such feedback loops may endogenize the dis-tribution of v . Third, our model does not consider thesavings that may accrue to investigators if they can sub-mit a revised version of a rejected proposal to a differ-ent or subsequent competition. To a first approxima-tion, submissions to multiple funders have the effect ofincreasing p , which can then be interpreted more gen-erally as the proportion of ideas that get funded acrossall available funding programs. Iterations of revision andresubmission to the same funding program are likely tohave more complex effects on efficiency and waste. Fi-nally, our model is silent regarding whether many smallor few large grants will promote scientific progress mostefficiently, and is likewise silent about the factors thatwill influence this comparison.To be sure, much more can be done to embellish this model. However, the qualitative results — that proposalcompetitions become increasingly inefficient as paylinesdrop, and that professional pressure on investigators topursue funding exacerbates these inefficiencies — are in-herent to the structure of contests. Partial lotteries andcontests that reward past success present radical alter-natives for allocating funds, and are sure to be contro-versial. Nevertheless, whatever their other merits anddrawbacks, these alternatives could restore efficiency indistributing funds that has been lost as those funds havebecome increasingly scarce. ACKNOWLEDGMENTS
We thank M. Lachmann for early discussions, A. Bar-nett and T. Bergstrom for useful feedback on an earlierdraft, and B. Moldovanu for helping us articulate clearlythe tension between proposal competitions and lotteries.KG thanks the University of Washington Department ofBiology for visitor support. [1] Stephen Cole, Leonard Rubin, and Jonathan R Cole.
Peer review in the National Science Foundation: Phaseone of a study . National Academy of Sciences, 1978.[2] National Institutes of Health. NIH funding facts. https://report.nih.gov/fundingfacts/fundingfacts.aspx ,2017. Accessed Dec. 29, 2017.[3] DEB Science Staff. DEB Numbers: FY 2016wrap-up. https://debblog.nsfbio.com/2017/01/06/deb-numbers-fy-2016-wrap-up/ , 2017. Accessed Oct.11, 2017.[4] National Science Foundation. Report to the National Sci-ence Board on the National Science Foundation’s meritreview process, Fiscal Year 2016, 2017. NSB-2017-26.[5] Nicholas Geard and Jason Noble. Modelling academicresearch funding as a resource allocation problem. In , 2010.[6] Albert N Link, Christopher A Swann, and Barry Boze-man. A time allocation study of university faculty.
Eco-nomics of Education Review , 27(4):363–374, 2008.[7] Robert Siliciano. Testimony before the Senate Commit-tee on Appropriations: Subcommittee on Labor, Healthand Human Services, Education, and Related Agencies.U.S. Senate Committee on Appropriations, Washington,DC, March 19, 2007.[8] Adrian G. Barnett. Ask the researcher: The experienceof applying for health and medical research funding inaustralia — survey results. https://eprints.qut.edu.au/107238/ , 2013. Accessed Apr. 19, 2018.[9] Bruce Alberts, Marc W Kirschner, Shirley Tilghman, andHarold Varmus. Rescuing US biomedical research fromits systemic flaws.
Proceedings of the National Academyof Sciences , 111(16):5773–5777, 2014. [10] Danielle L Herbert, Adrian G Barnett, Philip Clarke, andNicholas Graves. On the time spent preparing grant pro-posals: an observational study of australian researchers.
BMJ Open , 3(5):e002800, 2013.[11] Michele Pagano. American Idol and NIH grant review.
Cell , 126(4):637–638, 2006.[12] Peter A Lawrence. Real lives and white lies in the fundingof scientific research.
PLoS Biology , 7(9):e1000197, 2009.[13] Nicholas Graves, Adrian G Barnett, and Philip Clarke.Funding grant proposals for scientific research: retrospec-tive analysis of scores by members of grant review panel.
BMJ , 343:d4797, 2011.[14] John PA Ioannidis. More time for research: fund peoplenot projects.
Nature , 477(7366):529–531, 2011.[15] Johan Bollen, David Crandall, Damion Junk, Ying Ding,and Katy B¨orner. From funding agencies to scientificagency.
EMBO Reports , 15(2):131–133, 2014.[16] Ferric C. Fang and Arturo Casadevall. Research funding:the case for a modified lottery. mBio , 7(2):e00422–16,2016.[17] Shahar Avin. Centralized funding and epistemic explo-ration.
The British Journal for the Philosophy of Science ,page axx059, 2018.[18] Krist Vaesen and Joel Katzav. How much would each re-searcher receive if competitive government research fund-ing were distributed equally among researchers?
PLoSONE , 12(9):e0183967, 2017.[19] Shahar Avin. Policy considerations for random allocationof research funds.
RT. A Journal on Research Policy andEvaluation , 6(1), 2018.[20] Benny Moldovanu and Aner Sela. The optimal allocationof prizes in contests.
American Economic Review , pages542–558, 2001. [21] Milan Vojnovi´c.
Contest theory: Incentive mechanismsand ranking methods . Cambridge University Press, 2016.[22] Heidrun C Hoppe, Benny Moldovanu, and Aner Sela. Thetheory of assortative matching based on costly signals.
The Review of Economic Studies , 76(1):253–281, 2009.[23] Stephen Cole, Leonard Rubin, and Jonathan R Cole.
Peer review in the National Science Foundation: Phasetwo of a study . National Academy of Sciences, 1981.[24] Mikael Fogelholm, Saara Leppinen, Anssi Auvinen, JaniRaitanen, Anu Nuutinen, and Kalervo V¨a¨an¨anen. Paneldiscussion does not improve reliability of peer review formedical research grant proposals.
Journal of Clinical Epi-demiology , 65(1):47–52, 2012.[25] Elizabeth L Pier, Markus Brauer, Amarette Filut, AnnaKaatz, Joshua Raclaw, Mitchell J Nathan, Cecilia EFord, and Molly Carnes. Low agreement among reviewersevaluating the same nih grant applications.
Proceedingsof the National Academy of Sciences , 115(12):2952–2957,2018.[26] Roger B Nelsen.
An introduction to copulas . Springer,2nd edition, 2006.[27] David G Clayton. A model for association in bivariate lifetables and its application in epidemiological studies of fa-milial tendency in chronic disease incidence.
Biometrika ,65(1):141–151, 1978.[28] Samuel M Scheiner and Lynette M Bouchie. The pre-dictive power of NSF reviewers and panels.
Frontiers inEcology and the Environment , 11(8):406–407, 2013.[29] Elise S Brezis. Focal randomisation: An optimal mech-anism for the evaluation of r&d projects.
Science andPublic Policy , 34(10):691–698, 2007.[30] Luba Katz, Daniel Litwok, and Jessie Gertels. Evaluationof the NSF DEB and IOS switch to preliminary proposalreview: Final report. Technical report, Abt Associates,2017.[31] Adrian G Barnett, Nicholas Graves, Philip Clarke, andDanielle Herbert. The impact of a streamlined fund-ing application process on application time: two cross-sectional surveys of australian researchers.
BMJ open , 5(1):e006912, 2015.[32] Adrian G Barnett. Funding by lottery: Political problemsand research opportunities. mBio , 7(4):e01369–16, 2016.[33] Charles AE Goodhart. Problems of monetary manage-ment: The UK experience. In
Monetary Theory andPractice , pages 91–121. Springer, 1984.[34] NIH Office of Extramural Research. Actual of NIHR01 equivalent success rate FY 1970 – 2008 and esti-mates of NIH R01 success rates, FY 1962 – 1969 fromCGAF. http://report.nih.gov/FileLink.aspx?rid=796&ver=1 . Accessed Dec. 29, 2017, 2008. APPENDIX S1: MATHEMATICAL DETAILS
Bid function
The bid function b ( v ) can be found following steps thatare identical to the derivation of the first part of Propo-sition 8 in Hoppe et al. [22]. We repeat that derivationhere, almost fully verbatim.Begin by dividing the investigator’s payoff function byby (1 − k ) g ( v ) to rescale the investigator’s optimizationproblem (eq. 1) : b ( v ) = arg max x (cid:26) ( v + v )(1 − k ) g ( v ) η ( x ) − h ( x ) (cid:27) . (S1)Consider two projects of values v and v , v > v , withequilibrium bids b ( v ) and b ( v ), respectively. The inves- tigator with a project of value v should not submit abid as if her project had value v , and similarly the in-vestigator whose project has value v should not submita bid as if her project had value v . This yields:( v + v )(1 − k ) g ( v ) η ( b ( v )) − h ( b ( v )) ≥ ( v + v )(1 − k ) g ( v ) η ( b ( v )) − h ( b ( v ))( v + v )(1 − k ) g ( v ) η ( b ( v )) − h ( b ( v )) ≥ ( v + v )(1 − k ) g ( v ) η ( b ( v )) − h ( b ( v )) . Rearrange each inequality to isolate h ( b ( v )) − h ( b ( v ))and divide through by v − v to give ( v + v )(1 − k ) g ( v ) ( η ( b ( v )) − η ( b ( v ))) v − v ≤ h ( b ( v )) − h ( b (ˆ v )) v − v ≤ ( v + v )(1 − k ) g ( v ) ( η ( b ( v )) − η ( b ( v ))) v − v . Take the limit as v → v to give ddv h ( b ( v )) = v + v (1 − k ) g ( v ) ddv η ( b ( v )) . (S2)Essentially, after rescaling investigators’ benefits andcosts so that the cost function ( h ( x )) is the same for allinvestigators, eq. S2 says that, at equilibrium, an investi-gator’s marginal (re-scaled) cost and marginal (re-scaled)benefit of preparing an infinitesimally stronger proposalare equal [22]. Proceeding with the derivation, multi-ply both sides of eq. S2 by dv to separate variables andintegrate from 0 to v to obtain (cid:90) v dh ( b ( t )) = (cid:90) v v + t (1 − k ) g ( t ) ξ (cid:48) ( t ) dt (S3)where ξ ( v ) = η ( b ( v )). Then use h ( b (0)) = h (0) = 0 togive (cid:82) v dh ( b ( t )) = h ( b ( v )), and thus h ( b ( v )) = (cid:90) v v + t (1 − k ) g ( t ) ξ (cid:48) ( t ) dt. (S4)Take h − on both sides to complete the derivation.To make our derivation somewhat more general than inthe main text, suppose that both v and k are themselvesfunctions of v , such that the investigator’s optimizationproblem now becomes b ( v ) = arg max x (cid:26) ( v ( v ) + v )(1 − k ( v )) g ( v ) η ( x ) − h ( x ) (cid:27) . (S5)We require the key condition that the quantity ( v ( v )+ v )(1 − k ( v )) g ( v ) is a strictly increasing function of v , so that investigators with higher value projects will submitstronger proposals. This condition rules out the possibil-ity that, say, variation in v or k is large enough thateither replaces v as the primary correlate of proposalstrength. When this key condition holds, the derivationabove proceeds as before, leading to a bid function of b ( v ) = h − (cid:20)(cid:90) v v ( t ) + t (1 − k ( t )) g ( t ) ξ (cid:48) ( t ) dt (cid:21) . (S6)In this case, variation in v or k among investigators maychange the value of investigators’ bids, but it does notchange the rank of investigators’ bids (that is, higher-value projects are still associated with stronger propos-als). Because we use a copula to capture noisy assessmentof proposals, an investigator’s probability of funding de-pends only on the rank of the investigator’s bid, and thusthe portfolio of funded projects does not change.Finally, we can extend the model to accommodate re-searchers’ differing needs for time and money, as longas the marginal rate of technical substitution for time vs.money is also a function of v . In this case, we re-interpretthe cost function c ( v, x ) = g ( v ) h ( x ) as the time cost ofwriting a proposal, and we assume that the monetary costof writing a proposal is negligible. Let φ be a conversionfactor that converts time into scientific productivity, suchthat the disutility cost of writing a proposal in terms oflost productivity is φc ( v, x ). Moreover, suppose φ is alsoa function of v . Then, we can write the investigator’soptimization problem as b ( v ) = arg max x { ( v ( v ) + v ) η ( x ) − (1 − k ( v )) φ ( v ) g ( v ) h ( x ) } . (S7)1As before, we require the key condition that ( v ( v )+ v )(1 − k ( v )) φ ( v ) g ( v ) is a strictly increasing function of v ,to ensure that proposal strength is positively corre-lated with scientific value at equilibrium. Under thiscondition, the same steps give a bid function of b ( v ) = h − (cid:20)(cid:90) v v ( t ) + t (1 − k ( t )) φ ( t ) g ( t ) ξ (cid:48) ( t ) dt (cid:21) . (S8)To demonstrate that b ( v ) maximizes the investiga-tor’s payoff (as opposed to minimizing it), we followMoldovanu & Sela’s [20] “pseudo-concavity” argument.This argument requires that b (cid:48) ( v ) >
0, which we estab-lish first. To do so, note that h − in eq. 2 is an increasingfunction (because h is an increasing function), and that( v + t ) /g ( t ) > b (cid:48) ( v ) >
0, it suffices to show that ξ (cid:48) ( v ) > b (cid:48) ( v ) >
0, the pseudo-concavity argument of Moldovanu & Sela [20] proceedsas follows. Let (cid:36) ( v, x ) = ( v + v ) η ( x ) − (1 − k ) g ( v ) h ( x )be the payoff associated with a project of value v and aproposal of quality x . Let (cid:36) x = ∂(cid:36) ( v, x ) /∂x . We claimthat (cid:36) x > x < b ( v ), and (cid:36) x < x > b ( v ).These claims, together with the continuity of b ( v ), estab-lish that x = b ( v ) maximizes (cid:36) ( v, x ).We first show that (cid:36) x > x < b ( v ). Choose avalue of x < b ( v ), and let v (cid:63) be the value of a projectthat will generate a bid of x , that is, b ( v (cid:63) ) = x . Because b (cid:48) ( v ) >
0, it follows that v (cid:63) < v . Simple differentiationgives (cid:36) x ( v, x ) = ( v + v ) η (cid:48) ( x ) − (1 − k ) g ( v ) h (cid:48) ( x ) . (S9)Now differentiate (cid:36) x with respect to v to give the mixedderivative (cid:36) xv ( v, x ) = η (cid:48) ( x ) − (1 − k ) g (cid:48) ( v ) h (cid:48) ( x ) . (S10)Under the assumptions of our model, η (cid:48) ( x ) > g (cid:48) ( v ) <
0, and h (cid:48) ( x ) >
0; thus, (cid:36) xv >
0. Therefore, (cid:36) x is anincreasing function of v , and thus (cid:36) x ( v (cid:63) , x ) < (cid:36) x ( v, x ).By virtue of the fact that b ( v (cid:63) ) = x , we have (cid:36) x ( v (cid:63) , x ) =0. Therefore, (cid:36) x ( v, x ) > (cid:36) x < x > b ( v ) follows similarly. Copulas for noisy assessment
A bivariate copula is simply a bivariate probability dis-tribution on the unit square with uniform marginal dis-tributions [26]. Let U = F ( v ) be the actual quantileof a proposal, and let W be the assessed quantile. Thejoint distribution of U and W is given by the copula C ( u, w ) = Pr { U ≤ u, W ≤ w } . Given a value of U , theconditional distribution of W given U is C W | U ( u, w ) =Pr { W ≤ w | U = u } = ∂ C ( u, w ) /∂u . (Here we use the factthat U is uniformly distributed on the unit interval.) Tofind ξ ( v ), evaluate C W | U at u = F ( v ) and w = 1 − p tofind 1 − ξ ( v ), the probability that an idea of value v is notfunded. Take the complement to find ξ ( v ). Differentiatewith respect to v to find ξ (cid:48) ( v ), which can then be pluggedin to eq. 2.The distribution function for a Clayton copula [27] is[26, § C ( u, w ) = (cid:0) u − θ + w − θ − (cid:1) − /θ . (S11)The parameter θ ≥ U and W , with larger values of θ givingstronger associations (i.e., more accurate assessment ofgrant proposals). Alternative parameter sets
To complement the example in the main text, we shownumerical results for two alternative parameter sets. Inthe first alternative set, scientific value is uniformly dis-tributed across projects, the disutility cost increases lin-early with proposal quality, and assessment is less precisethan we assume in the baseline parameter set. In this set, v is uniformly distributed between 1/3 and 1. We use c ( v, x ) = xe − v for the cost function, and we use θ = 5 inthe Clayton copula (Fig. S2B).The second alternative parameter set captures a sce-nario where the pool of possible project values is bi-modal, with many minimal-value projects and equallymany maximal-value projects. In this alternative set, v ranges from 1/2 to 1. To construct the distributionof v , let Y be a beta random variable with both shapeparameters equal to 1/2. Thus, Y has a symmetric,U-shaped distribution on the unit interval. Then v isgiven by (1 + Y ) /
2. In this parameter set, we choose c ( v, x ) = (1 . − v ) x , and we set θ = 7 . k = 1 / Perfect discrimination
In the perfect discrimination case, we require thatthere is a maximum possible value of v , which we write v max . The previous derivation of the bid function doesnot work for perfect discrimination, because b ( v ) becomesa step function and thus is not differentiable. Instead, let v (cid:63) denote the threshold value, that is, v (cid:63) = F − (1 − p ).Under perfect discrimination, the threshold investigator2will break even regardless of her bid. Thus, η ( x ) = g k ( v (cid:63) ) h ( x ) v + v (cid:63) x ≤ x (cid:63) x > x (cid:63) (S12)where x (cid:63) = h − (cid:20) v + v (cid:63) g k ( v (cid:63) ) (cid:21) . (S13)Consequently, it is straightforward to show that the bidfunction is b ( v ) = (cid:40) x (cid:63) v > v (cid:63) v < v (cid:63) . (S14)The following results are all immediate. First, as thepayline drops, v (cid:63) increases, and hence x (cid:63) increases. (Re-call that g is a strictly decreasing function, and h − isa strictly increasing function.) Thus, investigators with v > v (cid:63) experience a reduced payoff and increased costs,leading to a reduced ROI. Second, the average scientificbenefit per funded proposal, which can be written as (cid:90) v max v (cid:63) v dF ( v ) (cid:30) (cid:90) v max v (cid:63) dF ( v ) , (S15)increases as v (cid:63) increases. Third, as p approaches 0 fromabove, v (cid:63) approaches v max from below. Thus, in thelimit, the bid function approacheslim p → + b ( v ) = h − (cid:20) v + v max g k ( v max ) (cid:21) v = v max v < v max . (S16)Thus, the payoff to all investigators approaches 0 as p approaches 0 from above. Lotteries
We consider the more general case of a multi-tier lot-tery. Proposals deemed worthy of funding are placedinto one of z tiers, with tier 1 representing the highest-ranked proposals, etc. Write the proportion of propos-als in tier i as q i , and let π i represent the probabil-ity that a proposal placed in tier i is funded, where1 ≥ π > π > . . . > π z >
0. We assume that the fund-ing agency determines q , q , . . . , q z and π , π , . . . , π z inadvance. Because the payline is still p , we must have (cid:80) zi =1 q i π i = p . The single-tier lottery proposed by Fang& Casadevall [16] and others is a special case with z = 1,with a probability of funding π = p/q in that tier.In a tiered lottery, the investigator’s maximizationproblem becomes b ( v ) = arg max x (cid:40) ( v + v ) z (cid:88) i =1 π i η i ( x ) − (1 − k ) c ( v, x ) (cid:41) (S17) where η i ( x ) is the probability that a proposal of quality x is placed in tier i . A similar derivation to the steps ineq. S2–S4 yields the bid function b ( v ) = h − (cid:34) z (cid:88) i =1 π i − k (cid:90) v v + tg ( t ) ξ (cid:48) i ( t ) dt (cid:35) . (S18)where ξ i ( v ) = η i ( b ( v )).We now show that the efficiency of a lottery dependsentirely on the structure of the lottery, and is indepen-dent of the payline. For multi-tier lotteries, we requirethat the ratios of the π i ’s — the probabilities of fundingin each tier — are fixed. To establish these ratios, write κ i = π i /π . The condition (cid:80) zi =1 q i π i = p implies thatthe probability that a proposal in tier i is funded is π i = pκ i (cid:80) i κ i q i , (S19)as long as p ≤ (cid:80) i κ i q i . (If p > (cid:80) i κ i q i , then we wouldhave π > p , and thusthe payline p cancels out of the efficiency calculations ineqq. 3–5. The investigator’s benefit from entering thecompetition is ( v + v ) (cid:80) zi =1 π i η i ( x ). A simple substitu-tion shows that this benefit is proportional to p :( v + v ) z (cid:88) i =1 π i η i ( x ) = ( v + v ) z (cid:88) i =1 pκ i η i ( x ) (cid:80) j κ j q j = p ( v + v ) (cid:80) i η i ( x ) κ i (cid:80) j κ j q j . To show that the investigator’s cost is proportional to p , we have c ( v, b ( v )) = g ( v ) h ( b ( v ))= g ( v ) z (cid:88) i =1 π i − k (cid:90) v v + tg ( t ) ξ (cid:48) i ( t ) dt = p g ( v ) (cid:80) i κ i (cid:82) v v + tg ( t ) ξ (cid:48) i ( t ) dt (1 − k ) (cid:80) j κ j q j . It thus follows that the investigator’s ROI (eq. 3), theaverage value per funded grant (eq. 4), and the averagewaste per funded grant (eq. 5) are all independent of p .Hoppe et al. [22] provide an argument based on theeconomic principle of revenue equivalence that explainswhy costs are independent of the payline in a lottery.This argument applies both to proposal competitions andto lotteries of any structure, and it applies regardlessof whether panels discriminate perfectly among propos-als, or not. The argument is most easily explained in asingle-tier lottery with perfect discrimination, so we con-sider that setting. First, for revenue equivalence to apply,we need to re-scale the model so that only benefits varyamong investigators. That is, re-scale the investigator’sequilibrium benefit function to pq ( v + v )(1 − k ) g ( v ) ξ l ( v ) (S20)3and write her cost function as h ( x ). Having re-scaledthe investigator’s benefits and costs, the principle of rev-enue equivalence suggests that the (re-scaled) cost paidby an investigator will be exactly equal to the negativeexternality that her entrance into the competition cre-ates, that is, the amount by which her entrance decreasesthe aggregate benefit of the competing investigators [22].With perfect discrimination, the threshold investigator(the last one to qualify for the lottery) has project value v (cid:63) = F − (1 − q ); every investigator with v > v (cid:63) quali-fies for the lottery, while every investigator with v < v (cid:63) opts out. Consider an investigator with an idea of value v . If v < v (cid:63) , then this investigator’s entrance into thecompetition has no effect on other investigators’ benefit,and thus the cost she pays is 0 (i.e., she opts out). Ifthe investigator has an idea of value v > v (cid:63) , then sheknocks one threshold investigator out of the lottery. Noother investigator’s benefit changes. Thus, the new in-vestigator’s presence decreases the aggregate benefit ofthe other investigators by pq ( v + v (cid:63) )(1 − k ) g ( v (cid:63) ) . (S21)This negative externality is exactly the re-scaled cost thatshe pays, i.e., h ( b ( v )) = pq ( v + v (cid:63) )(1 − k ) g ( v (cid:63) ) . (S22)Multiplying by g ( v ) undoes the re-scaling to give the ac-tual cost paid: c ( v, b ( v )) = g ( v ) h ( b ( v )) = g ( v ) pq ( v + v (cid:63) )(1 − k ) g ( v (cid:63) ) . (S23)Two observations explain why the cost paid by the in-vestigator is directly proportional to p . First, the identityof the threshold investigator — the one who is knockedout of the lottery when an investigator with a higher-value project enters — is determined by q , not p . (In aproposal competition, the threshold investigator is deter-mined by p .) Second, the threshold investigator’s benefit— and hence the negative externality imposed by thenewly arriving investigator — is directly proportional to p . Thus, the (re-scaled) cost paid by any investigator willalso be directly proportional to p . Multiplying the costby g ( v ) to undo the rescaling does not change the directproportionality to p .With noisy assessment, and/or in a multi-tiered lot-tery, the negative externality that an investigator im-poses on the field, and hence the (re-scaled) cost thatshe pays at equilibrium, integrates the amount by whichher entry decreases the benefit of every investigator witha project value less than hers. (This is one way to under-stand the bid functions in eq. 2, 6, and S18.) In a lottery,everyone’s benefit is proportional to p , and hence eachinvestigator’s negative externality, and the cost that shepays, is proportional to p as well.4 SUPPORTING INFORMATION FIGURES lllllllllllllllllllllllllllllllllllllllllllllllllllllll S u cc e ss r a t e ll NIH estimatesactual
FIG. S1. Funding rates of NIH R01 and equivalents fromFY 1962 – FY 2016. Data from FY 1962 – 2008 includeR01, R23, R29, and R37 proposals, as reported by NIH’sOffice of Extramural Research [34]. 1962 – 1969 data areNIH estimates. Data for FY 2009 – 2016 include R01 andR37 proposals, as reported by [2]. (R01 and R37 provide thevast majority of proposals for earlier years.) Data includenew applications, supplements, and renewals, and the successrate is calculated as the number of proposals funded dividedby the number of proposals reviewed. l l l ll l llll ll ll ll l ll l ll ll l ll l ll l lll ll ll ll ll lll ll l lll lllll l l ll ll ll ll l ll ll lll l lll l ll lll ll ll ll ll l lllll ll lll llll l ll lll lll ll l l llll ll lll llll l llll ll llll lll ll llll l ll ll ll l ll l lll l ll l llll ll ll lll ll ll ll ll lll ll ll ll lll lll l ll l l ll l lll l lllll l l llll l lll l ll llllll l ll ll ll l ll ll lll l ll llll llll llll l ll ll ll l ll l lll l lll lll ll l llll lll lll ll ll ll l lll ll l ll ll l ll llll l ll l ll l ll l lll ll ll ll l lll ll ll ll lll l ll l ll llll l lll ll ll ll l l lll lll lll lll l ll ll q = A ll lll ll lll lll ll l lll ll l l l ll l ll ll llll ll l ll ll l llll l ll lll lll l llll l lll l l l ll l lll ll l l l ll llll llllll lll ll lll l lllllll lll l llll ll ll ll lll llll ll l lll ll ll lll ll l ll l l lll l l lll ll ll l ll lll l l ll lll llll l l lll lll l ll lll l lll llll l l ll lll l l ll lll ll llll ll ll ll lll l llll lll l l ll ll lll l ll l ll ll lll ll lll lll l lll llll ll ll ll llll ll l lll ll l ll l l ll l l lll lll ll l lll ll l lll ll ll ll ll ll ll l ll llll l l lll l ll llll ll ll l l lll ll ll l ll ll ll ll llll ll l lll ll ll lll ll l l ll ll lllll q = B Actual quantile A ss e ss ed quan t il e FIG. S2. Random samples from copula distributions usedto model error in assessment of grant proposals. A: Claytoncopula with θ = 10. B: Clayton copula with θ = 5. Bluedashed lines give the median of the assessed quantile as afunction of the actual quantile. B ene f i t o r c o s t Proposal strength (x) A benefit, p = 45%benefit, p = 15%cost B I n v e s t i ga t o r ' s R O I Quantile of scientific value
45% payline orlottery line15% payline orlottery line
FIG. S3. Parallel results to Fig. 2, except with the first alter-native parameter set given in the SI. B ene f i t o r c o s t Proposal strength (x) A benefit, p = 45%benefit, p = 15%cost B I n v e s t i ga t o r ' s R O I Quantile of scientific value
45% payline orlottery line15% payline orlottery line
FIG. S4. Parallel results to Fig. 2, except with the secondalternative parameter set. A Avg. scientific value or wasteper funded proposal valuewaste, v = = −0.050.000.050.100.150.200.25 B
10% 30% 50%Scientific gain or lossper funded proposal v = = Payline or lottery line
FIG. S5. Parallel results to Fig. 3, except with the first alter-native parameter set given in the SI. Note that the verticalaxis in panel A does not extend to 0. A Avg. scientific value or wasteper funded proposal valuewaste, v = = −0.10.00.10.20.30.4 B
20% 60% 100%Scientific gain or lossper funded proposal v = = Payline or lottery line
FIG. S6. Parallel results to Fig. 3, except with the secondalternative parameter set given in the SI. In this figure, dataare shown for paylines ranging from p = 0 .
001 to p = 0 . B ene f i t o r c o s t Proposal strength (x) A benefit, p = 45%benefit, p = 15%cost B I n v e s t i ga t o r ' s R O I Quantile of scientific value
45% payline orlottery line15% payline orlottery line
FIG. S7. Parallel results to Fig. 2, except with perfect as-sessment of grant quality. All other parameter values are thesame as in Fig. 2. A Avg. scientific value or wasteper funded proposal valuewaste, v = = −0.2−0.10.00.10.20.3 B
10% 30% 50%Scientific gain or lossper funded proposal v = = Payline or lottery line
FIG. S8. Parallel results to Fig. 3, except with perfect assess-ment of proposal strength. All other parameter values are thesame as in Fig. 3. Note that the vertical axis in panel A doesnot extend to 0. A Avg. scientific value or wasteper funded proposal valuewaste, v = = −0.050.000.050.100.150.200.25 B