PRINCIPIA: a Decentralized Peer-Review Ecosystem
Andrea Mambrini, Andrea Baronchelli, Michele Starnini, Daniele Marinazzo, Manlio De Domenico
PPRINCIPIA: a Decentralized Peer-Review Ecosystem
Andrea Mambrini, ∗ Andrea Baronchelli, † Michele Starnini, ‡ Daniele Marinazzo, § and Manlio DeDomenico ¶ Optidrome Limited City University of London ISI Foundation University of Ghent Fondazione Bruno Kessler (Dated: August 21, 2020)
Peer review is a cornerstone of modern scientific endeavor. However, there is grow-ing consensus that several limitations of the current peer review system, from lack ofincentives to reviewers to lack of transparency, risks to undermine its benefits. Here,we introduce the PRINCIPIA ab framework for peer-review of scientific outputs (e.g.,papers, grant proposals or patents). The framework allows key players of the scientificecosystem – including existing publishing groups – to create and manage peer-reviewedjournals, by building a free market for reviews and publications. PRINCIPIA’s refereesare transparently rewarded according to their efforts and the quality of their reviews.PRINCIPIA also naturally allows to recognize the prestige of users and journals, withan intrinsic reputation system that does not depend on third-parties. PRINCIPIA re-balances the power between researchers and publishers, stimulates valuable assessmentsfrom referees, favors a fair competition between journals, and reduces the costs to accessresearch output and to publish. CONTENTS
I. Introduction 1A. State of peer review 2B. Variants and alternatives 3C. Aims of PRINCIPIA 4II. The peer-review market 4A. Differences with the current peer-review system 5B. Fee market 5III. System architecture 5A. Person 5B. Journal 5C. Paper 6IV. Consensus 7A. Modifying a journal 7B. Joining a journal 7C. Accepting a paper for review 7D. Deciding whether a paper should be published 7E. How to split he review fee between reviewers 8V. Reputation system 9A. Journal score 9B. User score 9C. Editorial board score 9D. Incentives/disincentives of the reputation system 9VI. A minimal decentralized peer-review system 9A. System Architecture 9 ∗ [email protected] † [email protected] ‡ [email protected] § [email protected] ¶ [email protected] a b [email protected] B. Peer-review process 10C. Reputation system and incentives 10VII. Implementation as a blockchain-powered solution 11VIII. Conclusion and outlook 12References 13
I. INTRODUCTION
Peer review is the practice of evaluating scientific out-put by experts in the relevant domain of knowledge. Itprovides a validation mechanism that is crucial to reducethe likelihood of publishing misleading results or falseclaims, adding value to scientific output. The impact ofpeer-review is pivotal at all scales of the scientific pro-cess. It can affect academic careers, and – through topjournals and funding agencies – it might drive the rise orfall of academic trends or even research fields.Despite its crucial role, the current peer review comeswith many disadvantages that make it a highly debatedmatter across decades (Culliton, 1996; Editorial, 2003;Tennant et al., 2017; Editorial, 2019a). A remarkableexample is the public criticism to funding decisions ofthe National Science Foundation, leading to a statisticalanalysis of the underlying evaluative procedures whichshowed no significant evidence of impact on funding (Coleet al., 1977).Rationalizing peer review is challenging: the work ofreviewers is essentially based on voluntary contributionwith virtually no incentives to report’s quality. At thesame time, academic publishing is a tremendously prof-itable business where the workforce 1) produces valuable a r X i v : . [ c s . D L ] A ug content for free, 2) performs valuable tasks without anyreward and 3) guarantees a constant demand. In fact,the whole peer-review system induces a social dilemmaat the individual level that is barely balanced by cur-rent incentives, such as the journal impact factor. Whyshould peers cooperate for the public good benefits? Whyshould they do it, if their work is done without rewardswhile publishers take all benefits, e.g., from subscriptionsor open access policies? Why authors should be fair orproduce high-quality scientific output if their peers mightopt for not cooperating?Any rational answer to such questions would lead tosub-optimal individual decisions, resulting in an unreli-able scientific ecosystem and a tragedy of the commons.However, decades of results from behavioral organizationand decision theory, experimental economics and surveyresearch show that rationality is very often far from be-ing a good model of human behavior (Loomes and Sug-den, 1982; Simon, 1986; Tversky and Kahneman, 1989;Arthur, 1994; Simon, 1995; Kahneman, 2003).Some studies have shown that increased public goodbenefits, together with editorial decisions accountingfor reputational information (reputation bias), can helpthe evolution of high-quality contributions from authorswhile reviewers still lack the right incentives (Righi andTakács, 2017). It is emblematic the provocative study byCampanario about the unreasonable resistance encoun-tered by some studies – among the most-cited articles ofall times – during peer review (Campanario, 1996).Nevertheless, peer-review is still perceived by all keyplayers, from authors to publishers, as an effective way toimprove the quality of a manuscript by means of a collec-tive effort, opposed to isolated ones by few authoritativeindividuals. In fact, knowledge becomes scientific onlywhen the community reaches a consensus on it by meansof academic scrutiny, refinement and validation (Edito-rial, 2019a). A. State of peer review
The potential disadvantages of peer review have beenwidely discussed for decades. In 1982, 12 articles – pub-lished by academics from prestigious and highly pro-ductive American psychology departments in highly re-garded American psychology journals – where resubmit-ted to the same journals which previously published them18 to 32 months earlier, after changing authors and theiraffiliations. The result was that 9 out of 12 submis-sions were not recognized as resubmissions, while 8 out9 those studies were rejected because of serious method-ological flaws (Blissett, 1982). A study similar in spirit,one decade after, highlighted the same absence of reli-ability in peer review of medical research (Ernst et al.,1993).Research interest on peer review has recently in- creased, yet the field remains fragmented across differ-ent research communities (Grimaldo et al., 2018). TheGlobal State of Peer Review report for 2013–2017 , pub-lished by a platform helping academics to track andrecognize their contributions to the scientific endeavor,has highlighted that such contributions are unevenly dis-tributed across countries and institutions. Scholars inleading science countries, such as the United States andJapan, write 2 reviews per each of their submitted arti-cles, against the 0.6 written by their colleagues in emerg-ing countries such as China and Poland. Moreover, thelatter tend to accept more frequently peer review requestsand complete them faster than the former ones, althoughtheir reviews are on average shorter than the ones fromscholars in wealthy locations. According to the same re-port, reviewers spend globally about 68.5 million hourseach year, providing short reviews (477 words on aver-age) within 16.4 days (median value): surprisingly, 10%of reviewers account for 50% of peer reviews, and 75%of editors claim the hardest part of their job is findingwilling reviewers. It seems that for 41% scholars, peerreview is part of their job, although 71% of researchershave to decline review requests because the article is notwithin their area of expertise and 42% declare to be toobusy.The excessive academic burdening, the lack of train-ing in reviewing and the excessive number of requests,coupled to potential human biases, might lead to poorreviews which fail to prevent the publication of stud-ies that do not deserve consideration from the academiccommunity (Editorial, 2019b). One possible source of bi-ases in the peer-review process resides in the connections– such as co-authorship – between creators and evalua-tors of scientific work. Interestingly, even belonging todifferent professional networks, or “schools of thought”,might lead to substantive disagreements between scien-tists in the peer-review process (Teplitskiy et al., 2018).One possible solution to this issue could be the adop-tion of “double-blind” peer-review, in which reviewers arenot aware of the names and affiliations of paper authors,as opposed to the “single-blind” system in which onlynames of reviewers are hidden. For papers submittedto computer science conferences, single-blind reviewershave been showed to be significantly more likely thantheir double-blind counterparts to recommend for accep-tance papers from famous authors, top universities, andtop companies (Tomkins et al., 2017).Even editors, as humans, tend to take decisions thatmight be affected by their current burdening, resultinginto an overall peer review system with more disadvan-tages than advantages, and a scientific ecosystem sup-ported by wrong incentives. It is emblematic the lack https://publons.com/community/gspr of policies to recognize researchers for the production ofvaluable data set – even if the trend is slowly chang-ing (Gorgolewski et al., 2013) – and for reproducibilitystudies, resulting in movements against “data lechers” –researchers who use data produced and shared by othercolleagues – and in defining research parasitism as a treatto science (Longo and Drazen, 2016). In reality, thesecollateral effects might be a treat only to selfish indi-viduals, whereas the whole collectivity can only benefitfrom reproducibility and replicability to advance humanknowledge (Bergstrom, 2016; Greene et al., 2017; Edito-rial, 2018a; Teytelman, 2018). Recent quantitative evi-dence from neuroscience supports this argument (Milhamet al., 2018).It is clear how concerns related to peer review reflectmore general issues affecting the scientific ecosystem.Note that, despite all limitations, peer-review scores togrant applications have been showed to be at least par-tially predictive of the success of funded projects, quan-tified by their total time-adjusted citation output, whilethe amount of funds awarded per application is not (Galloet al., 2014). Therefore, improving the peer-review pro-cess also will ultimately impact the allocation of billionsof dollars in research funds. As we will see, opening thewhole peer review process, and making it transparent toanyone, might only improve the current state of the art.Remarkably, high-reputation journals and research com-munities are already recognizing data sharing as part ofthe scientific process, requiring access to data used in ar-ticles they publish (Taichman et al., 2016; McNutt et al.,2016; Gewin, 2016; Editorial, 2018b).Similarly, it seems plausible to think that opening thepeer-review system – in a way that accounts for poten-tial human biases and findings from behavioral sciences,game theory and complexity science – might be the rightway to go. Other potential benefits of opening the pro-cess include valuable examples for early-career academicsand the availability of huge databases which might be an-alyzed by scientists to identify potential biases and pro-pose methods to eradicate them. B. Variants and alternatives
Since 2008 the American Physical Society recognizesits outstanding referees for their work and, more re-cently, Elsevier and Nature Publishing Group (Edito-rial, 2019c) are adopting similar strategies.Some initiatives have experimented novel and alterna-tive schemes for peer review, from publishing reports https://journals.aps.org/OutstandingReferees For instance: Biology Direct (2006), The EMBO Journal (2009), – i.e., the content of the review – and reviewers’ iden-tity (Editorial, 2018c) – i.e., the names of reviewers –to crowd-based approaches (List, 2017). In the latter,manuscripts are made available by editors and the crowdis given a certain amount of time to respond. Surpris-ingly, each paper received several comments – even indetails buried in supplementary material and support-ing information – considered informative by editors andallowing for a rapid editorial response. Quantitatively,the crowd responded much faster (days versus months)and provided more-comprehensive collective feedbacks,appreciated by authors (Editorial, 2019a).Other initiatives like the one by the journal eLife , sim-ilar in spirit, obtained a good response to pilot projectsfrom the research community, while measuring higher ac-ceptance rates for late-career researchers compared totheir early- and mid-career colleagues . The EuropeanJournal of Neuroscience reported that transparent re-view at the end of 2016 led to better and faster reports .Nevertheless, researchers from different disciplines areresponding differently, with more than 70% in Evolu-tion and Ecology opting for publishing reports when thechoice is given, against the 50% in Physics (Editorial,2016). A group of biologists built an online platform tokeep track of this type of experiments, providing a directway to compare trials with respect to multiple criteria .A common factor is transparency in peer-review re-ports, which could avoid superficial reviews or too harshtone either from referees or from authors. By opening thereview process, it is also possible to avoid abuses fromeditors and predatory behavior from publishers (e.g., theones that do not send the manuscript for review or thataccept it even with low-quality/unreliable reviews withthe only goal to charge fees). Moreover, while many re-searchers are in favor of opening reviews, opening iden-tities is seen as potentially dangerous, fearing that bydisclosing names the reviewers might be incentivized toweaken criticisms or authors might be incentivized to re-taliate against them (Ross-Hellauer et al., 2017). Never-theless, the British Medical Journal reported that open-ing reviews and reviewers’ identities, simultaneously, didnot change the quality of the peer reviews, suggest-ing that reviewers were not intimidated (van Rooyenet al., 2010). Therefore, a good trade-off might be toopen reviews while asking to opt-in for open identi-ties: anonymity, in fact, does not compromise the pro-cess (Bravo et al., 2019) and might avoid unfair use in eLife (2011), F1000 Research (2012), PeerJ (2013) and NatureCommunications (2016) https://elifesciences.org/inside-elife/262c4c71/peer-review-first-results-from-a-trial-at-elife https://reimaginereview.asapbio.org/ subsequent evaluation of the authors for grants, jobs,awards or promotions. According to a recent research,the potential benefits of published reviews are multi-ple (Polka et al., 2018), from encouraging constructivecomments and training examples for young researchers to preserving arguments and ideas that characterize howfields evolve, from building trust based on transparencyto recognizing the important work done by reviewers.Several journals and platforms are in the process ofadopting these principles one way or another. Pubpeeris a platform dedicated to the discussion of papers follow-ing their publication in a journal or as preprints. Eventhough the majority of comments address the (still veryimportant) problem of scientific misconduct and majorissues changing the conclusions of the studies, there isspace for costructive discussions between authors andreaders, and even public reviews of preprints. This as-pect is crucial and several journals (including major out-lets such as PLOS and ELife) are now embracing com-ments on papers submitted to the journals and posted aspreprints, and including them in the process of editorialevaluation. Platforms of preprint and open reviews canalso serve the purpose of overlay journals, which includepapers already made public, commented and revised, incollections curated by the editorial board.Other approaches aim at decoupling the function ofjournals (most importantly, dissemination) from the cer-tification provided by peer review. Several platforms,such as the Peerage of Science or RUBRIQ , offer peer-review services and then forward revised manuscript tojournals, in exchange of some fee or for free. Other plat-forms, such as F1000Research , combine a open peer-review system with publication services, without edito-rial bias. However, the major limitation of these decou-pled approaches is the lack of engagement of reviewersinvolved in the process, that can eventually lead to thefailure of such initiatives. As we will show in the fol-lowing, we argue that principles for a open, transparentpeer-review process can work better if anchored to a solidreputation system, by which participants in the systemare both rewarded and responsibilized. C. Aims of PRINCIPIA
PRINCIPIA aims at improving the current peer-reviewsystem in the following aspects:1. Referees are remunerated. peerageofscience.org rubriq.com https://f1000research.com/
2. Remuneration of referees depends on the quality oftheir reviews.3. New journals are easy and free to start.4. A new journal inherits the reputation of itsfounders (i.e. editorial board).5. All journals are open access and cost-effective.Point 1. and 2. will restore a balance of power betweenresearchers and publishers while guaranteeing a improvedquality and fairness of reviews. Point 3., 4. will increasethe offer and flexibility of the journal landscape by mak-ing it extremely easy to bootstrap a new journal. Finally,point 5. guarantees a lower entry barrier both for authorsaiming to publish in a journal and to scientists interestedin accessing published papers.
II. THE PEER-REVIEW MARKET
Every person using PRINCIPIA is uniquely identifiedby its RSA-4096 public key.Any group of people can join and form a journal byserving as member of the editorial board. Thus, in PRIN-CIPIA a journal is identified by a collection of publickeys. The editorial board has the following roles: • It reviews the paper submitted to the journal anddecide whether a paper should be accepted or not • It decides whether other people should be allowedto join the editorial board • It sets the rules of the journal (see Section III.B)These actions are performed using the consensus rulesexplained in Section IV.The PRINCIPIA framework assigns a reputation scoreto a journal according to the people participating to theeditorial board. The mechanism is explained in sectionV.In order to submit a paper to a journal the authorsmust publish it (see Section VII for details on the pos-sible implementation). Then they perform a request ofpublication to the journal by attaching the hash of theirpaper and a bid for a review fee . The editorial board willvote using the consensus mechanism explained in SectionIV.C and decide whether to accept the bid and considerthe paper for review. If the vote is positive then a set ofreviewers is randomly generated from the people in theeditorial board. The reviewers will perform their reviewand publish it. According to the rules explained in Sec-tion IV.D the paper will be accepted or rejected. Afterthe authors have received the written reviews they willwrite the final version, publish it and submit it again tothe same reviewers for a final acceptance vote. The re-view fee will be split between the journal’s fee and anamount that will be distributed to the referees accordingto the mechanism awarding the good reviews explainedin Section IV.E.
A. Differences with the current peer-review system
There are many substantial differences between thecurrent peer review system and PRINCIPIA. A journal inPRINCIPIA is uniquely identified by its editorial board.This creates a liquid system in which the reputation ofa publication is identified by who was serving as edito-rial board in the moment of the publication. This ismore flexible than the current system in which a jour-nal’s reputation is identified by its history and will helpnew quality journal to be created more easily since itwill reduce the difficulties to bootstrap a journal’s repu-tation. Of course established journal with a strong brandcan participate to PRINCIPIA and their reputation willstill be recognized outside of PRINCIPIA’s system. Onthe other hand, inside the system their reputation will bebased just on their editorial board’s reputation, exactlyas for any other journal.Another remarkable difference with the current systemis that referees will not work for free. They will receivepart of the review fee for the work they performed by re-viewing the paper. Moreover they will be incentivized bywriting detailed and high quality reviews by the systemexplained in Section IV.E.In PRINCIPIA the equilibrium between authors andjournals is re-established. In fact, while authors can lookfor the most suitable journals to publish their work, weenvision a system where journals bid on papers to at-tract those authors of (initially perceived) high qualityscientific products.
B. Fee market
In PRINCIPIA fees are not fixed and are instead de-cided by the market. There are two kind of fees: the review fee which is bid by authors to have their paperreviewed by a journal, and a joining fee which is bid bya person to join the editorial board of a journal.Authors will have to bid a review fee in order to havetheir paper reviewed by a journal. This is different fromthe current system in which fees are paid just in casethe paper gets published and the publication fee is fixed.By moving the payment at a review level we think au-thors will be incentivized to just submit papers on whichthey have enough confidence. Moreover this will discour-age journals to accept as many papers as possible just tocollect the publication fees. We believe this system willincrease the quality of the work submitted and will freethe reviewers from reviewing many low quality papers.Morover paying for reviews instead of paying for publi- cations allows to allocate a payment to the stage wheremost of the work is performed (in fact the publicationof a paper, after it has been reviewed, has a low costcompared to to cost of performing the review).Keeping the review fee variable and assigned througha bid creates a market in which journal competes for thebest papers. An author might propose the same qualitypaper to few journals and eventually decide to submit itto the journal that allows him to bid the lowest reviewfee (or which would even pay him to review to have thechance to review the paper). Of course this also open tothe possibility for a low quality paper to be accepted forreview because the authors have bid an high review fee.This is possible, but disincentivized as being acceptedfor review does not guarantee publication. In fact if thepaper is not good, the editorial board will not be incen-tivized to accept it for publication as it would reduce thereputation of the journal (see Section V).A similar situation happens with the fee to join a jour-nal as editorial board member. On one hand journalsmigh be tempted to accept in the editorial board some-body with a bad reputation if he/she bids an high joiningfee. On the other hand the reputation system will disin-centivize those behaviours. Of course having a market-based joining fee will allow journal to compete for thebest people to join the program committe. A person withlow reputation might have to pay to join a good journal,but a person with high reputation might instead end upbeing paid to join the program committe of a journal.
III. SYSTEM ARCHITECTURE
In this section we describe in details the entity of thePRINCIPIA framework
A. Person
A person in PRINCIPIA is a public key ( p pk ). Becauseof the reputation system we will explain in section V, it isvery important that each public key is strongly connectedto a real person. Therefore keys need to be validated byreliable institutions, similarly to what happens in digitalsignature released by governments. B. Journal
A journal is a collection of public keys, a reference toits ancestor journal and the value of its parameters. Theparameters of a journal are the following: • Percentage of review fee to keep in the journal ( f j ) • Whether reviewers should be anonymous or not( a j ) JOURNAL
JOURNAL_HASH: 32A...134JOURNAL_SCORE: 10
JOURNAL
JOURNAL_HASH: 12B...1A3JOURNAL_SCORE: 5PROGRAM_COMMITEE_SCORE: MEAN(6.66, 1.14, 7.25) = 5 S p e n t m o n t h i n t h e e d i t o r i a l b o a r d S p e n t m o n t h s i n t h e e d i t o r i a l b o a r d PERSONUSER_SCORE = 10/18 + 5/18 = 6.66PERSONUSER_SCORE = 1.14PERSONUSER_SCORE = 7.25
Figure 1 Schematic illustration of the interplay between journals and actors in PRINCIPIA. • Maximum time to perform a review ( t j ) • Number of reviewers for each paper ( n j ) • Size of the qualified majority needed to accept apaper for review ( r j ) • Size of the qualified majority needed to spend thejournal’s balance. ( p j ) • Size of the qualified majority needed to modify thejournal ( m j ) [this must be higher than p j ]When a journal is created it requires the signature ofall the members of the editorial board.Any of the element of the journal can be changed bycreating a new journal and setting the old journal as its ancestor journal . In order for the creation of a new jour-nal to be valid, the signature of a certain amount of mem-bers of the ancestor journal should be present accordingto the rules described in Section IV. This means that anychange of the journal parameters or to the journal’s edi-torial board triggers the creation of a new journal. It isnot possible to publish on an ancestor journal, but theold publication will still be connected to the old journal.This way each publication is always connected to the sta-tus of the journal at the moment of the publication. Each journal has a journal balance consisting on thefunds from the review fees, excluding the part which goesto the reviewers. The balance can be spent to cover theexpenses of the journal (e.g advertisement or to ask a per-son to join the journal). Spending the funds of the jour-nal’s wallet requires the signatures of a qualified majorityof the editorial board (specified by the journal’s param-eter p j ). Since the majority needed to spend the journalbalance is lower than the majority needed to modify ajournal, every time a new journal is created, the balanceof the journal wallet of its ancestor can (and should) betransfered to the wallet of the new journal. See Fig. 1 fora schematic illustration. C. Paper
A papers is a published document (see Section VII fordetails on possible implementation) identified by its hashand signed by the authors. Review and publication of apaper goes through the procedure explained in sectionIV
PERSON
Bid to join joining fee bid: 2 ETH
JOURNAL
JOURNAL_HASH: 32A...134JOURNAL_BALANCE: 142 ETH
JOURNAL
JOURNAL_HASH: 14D...1A3ANCESTOR_HASH: 32A...134JOURNAL_BALANCE: 144 ETH
Joining bid returned and theperson does not join thejournal.
Journaldecides not acceptthe new member Vote on whether toaccept the bid to join.Requires a qualifiedmajorityJournal deciedsto acceptthe new member
A new journal is generated withthe new member of theeditorial board in and a newbalance.
Figure 2 Schematic illustration of the process to extend an existing editorial board in PRINCIPIA.
IV. CONSENSUSA. Modifying a journal
A modification of a journal triggers the creation of anew journal which has as ancestor the old journal (as de-scribed in Section III.B). Modifications are approved bya qualified majority defined as a journal parameter m j .Adding or removing a person to the editorial board isa modification to a journal. Changing one of the jour-nal’s parameters (for example the number of reviewersfor each paper) is also a modification to a journal. Theonly journal modification which does not require a qual-ified majority is when a member of the editorial boarddecides to leave the journal, this can be done unilaterallywithout any approval from the journal. See Fig. 2 for aschematic illustration of this process. B. Joining a journal
In order for a person to join a journal it needs to bid a joining fee . This will trigger a journal modification eventthat needs to be approved by a qualified majority ( m j )of the journal. The amount bid will be deposited to the journal’s wallet in case the person is allowed to join thejournal, otherwise the amount is returned to the bidder. C. Accepting a paper for review
A person can ask a journal to review its paper by bid-ding a review fee. A paper is accepted for review by aqualified majority defined as a journal parameter r j . Ifthe paper is accepted for review, and the authors confirmhe wants to proceed with the review, then n j reviewersare selected at random and they will review the paper. Incase the journal want to keep the identity of the review-ers anonymous (as of the parameter a j ), a system basedon ring signature will be used to assign the reviewers. D. Deciding whether a paper should be published
Each reviewer will review the paper and assign a score s u between and . If the average score s is greater than then the paper is accepted for publication. IPFSIPFS
AUTHORPUB_KEY: A32...412 JOURNALJOURNAL_HASH: 32A...134
Vote on whether to acceptthe bid and review the paper
Bid rejected Bid accepted
Bid returned toauthor A random set ofreviewers isselected fromthe programcommitee
Outcome ofreviews isuploaded on IPFSUnreviewed paper uploaded on IPFS
UNREVIEWD PAPERHASH: 5D...12
Review fee is splitbetween journal andreviewers
Author reads the reviewsand prepares the final version Paper accepted
REVIEWS FOR PAPER5D...12
Paper notpublished. Reviewfee not returned
PaperrejectedFinal version according to reviewssubmitted for publicationJournal acceptsthe final version
FINAL VERSION HASH: AA2...431AUTHOR: A32...412JOURNAL: 32A...134
Final versionpublished on IPFS
AUTHORPUB_KEY: A32...412JOURNALJOURNAL_HASH: 32A...134
Bid to publish paper_hash: 5d...12review_fee bid: 0.5 ETH
Figure 3 Schematic illustration of the review process leading to paper acceptance/rejection and reward for the reviewers inPRINCIPIA.
E. How to split he review fee between reviewers
The review split is divided in this way: • f j percentage goes to the journal’s wallet • The remaining is split between each reviewer u .Each reviewers receive an amount of funds equalt to r u where r u is calculated using the following equa-tion r u = f r · (1 − f j ) · (cid:16) | s u − | (cid:80) u | s u − | − | s u − s | (cid:80) u | s u − s | (cid:17) The idea is to reward "consensus", i.e. to reward re-viewers that agrees with the average score of the otherreviewers, while at the same time discourage scores closeto the and forcing reviewers to take a position.An illustration of the overall process is shown in Fig. 3. V. REPUTATION SYSTEM
Each user in PRINCIPIA has a reputation score whichdepends on journal in which he/she served as member ofthe editorial board. At the same time each journal hasa reputation score which depends on the impact of theresearch it published. Since a modification to a journal(including change in the editorial board) triggers the cre-ation of a new journal, which will therefore have a scorethat does not depends on its ancestor journal, we alsodefine a reputation score for the editorial board, whichwill be the main score to judge the quality of a journal.
A. Journal score
The journal’s score is inspired to the weighted impactfactor described in (Alguliyev et al., 2015). The scoreis calculated as the number of citations of all the paperspublished in that journal divided by the number of paperspublished by the journal. The number of citations isweighted by the editorial board score (see Section V.C)of journals who published the citing papers.Notice that a modification of the paper triggers thecreation of a new journal (see Section III.B) so this scoretakes into account just the papers published in the jour-nal, ignoring the ancestor and descendant journals. Alsonotice that the weight is calculated using the editorialboarde score, rather than the journal’s score.
B. User score
The score of a user serving as reviewer is calculated asthe weighted average of the journal’s score of the jour-nal in which he or she served as member of the edito-rial board. The average is weighted by the time he orshe spent in the committee. So for example a user whoserved 6 months in a journal having journal score of 10and 12 months in a journal having score 5, will have ascore of ·
10 + · . C. Editorial board score
The editorial board score is a score assigned to theeditorial board of a journal. It is the main score to judge the quality of a journal. It is calculated as the averagescore of the users serving in the editorial board.
D. Incentives/disincentives of the reputation system
This reputation system gives reputation to the usersaccording to the reputation of the journal in which theyserved as editorial board. This incentivize users to joinjournal with an high reputation.At the same time the reputation of a journal dependson the impact of the paper published, which should en-courage a journal to publish good quality papers.The reputation of a user does not depend on the paperhe/she published as an author. PRINCIPIA untight thereputation of a person as an author from their reputationas a reviewer. We want to focus on improving the peerreview system and reward users who can discern highquality papers from low quality ones. We believe this iswhat makes a person a good editorial board member andwe want the system to create a race between journals toget the best of them. We believe this will increase thequality of the work published.
VI. A MINIMAL DECENTRALIZED PEER-REVIEWSYSTEM
While journals are pivotal actors of the peer-reviewecosystem and need to be part of it, it might be dif-ficult to include them in the PRINCIPIA at the begin-ning. Therefore, one might consider a two-step process inwhich i) the peer-review system is decentralized, and ii)new journals are created. In this Section, we propose analternative decentralized peer-review system fully com-patible with PRINCIPIA, without including the forma-tion of new journals.
A. System Architecture
The architecture of the system is similar to the one pro-posed in Section III. Each person (scientists) in PRIN-CIPIA is a public key. Each scientist in the peer-reviewprocess can have one of two roles: author of the papersubmitted for review or reviewer of that paper.Each scientist i has associated a few keywords indicat-ing their expertise, a reputation score RS i , and a reviewfee R i . The keywords of each scientist are initially val-idated by their publication record and updated on thebasis of the papers reviewed. The RS is initially equalfor everyone and completely independent by the scien-tist’s publication record. The review fee R i of scientist i is the price they ask to perform a review. It will bedependent on their RS i (see Section VI.B): the higher RS i , the higher R i fees can be asked.0A papers is a document identified by its hash andsigned by the authors. Authors submit a paper to thepeer-review system with associated keywords indicatingthe field, by bidding a paper review fee R p . B. Peer-review process
Once a paper is submitted, the system tries to assignthe paper to potential reviewers, by matching the R p bidby the authors with the review fees asked in the marketby scientists with respect to the paper’s field (i.e. alsomatching paper’s keywords with potential reviewers’ key-words). That is, the system tries to solve the equation (cid:80) ni =1 R i (cid:39) R p by choosing a pool of n (cid:62) reviewers.These reviewers are also chosen in order to have mixed RS , compatibly with the R p bid by the authors. The sys-tem also limits the maximum number of reviews assignedto each scientists, to avoid excessive burden to single re-viewers and to avoid reviewers asking low R fees to gettoo many reviews. If a match is possible, the paper isaccepted for review and assigned to the reviewers.The peer-review process proceeds as it follows: In step1, selected reviewers perform review independently andassign the paper a score, representing its quality. In step2, selected reviewers read the reports of other reviewersand assign them a score, judging the quality of the re-ports. For each reviewer i , its average report score iscomputed. If it is above a certain threshold, the reviewercollects the R i fee asked. If not, that R i fee is returned tothe paper’s authors. In this way, there are no incentivesto give low or high scores to reports of other reviewersto collect their fees. Then, the reputation score RS ofeach reviewer is updated with the average score obtainedby their report. In this way, reviewers have incentives toproduce high-quality reports, see Section VI.C. Note thatstep 2 is a further burden for reviewers with respect tothe current peer-review process, but relatively light sincereviewers have already reviewed the paper in step 1 andso they can easily judge other report’s quality.Once step 2 is concluded, the average peer-review scoreof the paper S p is computed. If it is above a certainthreshold, which can be different for different fields, thepaper is accepted by the system as “peer-reviewed”. Oth-erwise it is withdraw and disappears. Further interac-tions among the authors and the reviewers could be con-sidered, as in the current peer-review process, to achieveconsensus, if necessary.Once a paper is accepted by the system, it becomespublic together with its associated score S p , the (anony-mous) reports, the scores assigned to each report indi-cating its quality, and the reputation score RS of the n reviewers. These elements (paper’s score, reports, re-ports’ scores, and reputation scores of reviewers) certifythe quality of the paper. At this point, authors can sub-mit the paper for publication to journals and journals can bid to publish papers they found interesting, basedon the quality of the paper certified by the peer-reviewsystem. The authors can also decide to leave the paperin the system and not to publish it in any journal. C. Reputation system and incentives
The RS of each scientist is at the core of the peer-review system. It represents their prestige as a reviewer,in the same way the h-index or number of citation mightrepresent their prestige as an author. Note that thesetwo quantities are completely independent.Scientists have the incentive to increase their RS in or-der to ask higher R fees, see below. As described in Sec-tion VI.B, the RS increases for each review performed bythe average score collected by the report of the reviewer.Therefore, reviewers have two incentives to produce high-quality reports: i) to collect the R fee (which is not payedfor low-quality reports), and ii) to obtain high scores andincrease their RS . These incentives are expected to solveone of the most important problems of the current peer-review system, low-quality reports.Coherently with PRINCIPIA, the R fees and R p bidsare not fixed but decided by the market, that is, reviewers(authors) can ask (bid) whatever fee they like. The peer-review system is an actual market that matches the de-mand from peer-review service from authors to the sup-ply of peer-review service provided by reviewers.Reviewers have the incentive to perform reviews to in-crease their RS . In order to obtain papers for review,they have the incentive to ask low R fees to be easilymatched with the R p fees offered by a paper’s authors.Note that the matching is crucially dependent on the RS of reviewers: if the R fee asked by a reviewer is notaligned with their RS , they do not obtain any paper toreview. Therefore, reviewers have the incentive to ask a R fee aligned with their RS . Once their RS increases,they can ask for higher R fees. Note that the systemlimits the maximum number of reviews for scientist, sonone has incentives to ask too low R fees. To sum up,reviewers will ask a R fee determined by the market fortheir RS and field of expertise (associated keywords). A R fee asked higher than the market value would not getany paper to review.At the same time, authors have the incentive to bidhigh R p fee to attract reviewers with high RS , who cancharge high R fees. However, note that this does notguarantee a positive peer-review outcome, with a goodpaper’s score. Again, authors will bid a R p fee deter-mined by the market, depending on the quality of thereviewers they want, certified by the RS of the review-ers. A R p fee bid lower than the market value would notobtain any reviewer.To help the matching process, the system can suggesta “fair" R p bid to associate to the paper in order to secure1reviewers, based on the current R fees asked in the marketfor the paper’s specific field. VII. IMPLEMENTATION AS A BLOCKCHAIN-POWEREDSOLUTION
Implementating PRINCIPIA requires a solution to thefollowing issues: • How to process the payments for fees (review feeand journal joining fee) • Where to store papers and reviews • How to implement and where to run the implemen-tation of the protocolA possible solution would be to set up a no-profit foun-dation which would handle all the aspects above. Thefoundation would develop the code, host the platform,deal with the storage of papers and review and processall the payments in the main currencies. This would besimilar to the model behind Wikipedia.Such a centralized approach will expose non-trivial fi-nancial problems as the foundation would have to processlarge amount of money in fees. Morover having the wholeacademic publishing system in the hand of a foundationmight lead to conflicts of interest in the way the founda-tion is managed which might lead to censorship, sabotageand lack of transparency.We think that PRINCIPIA would benefit from an im-plementation running on platform enabling smart con-tracts, like for example Ethereum and from using adecentralized storage to store papers and reviews, for ex-ample IPFS (Benet, 2014), a peer to peer distributedstorage system in which files are identified and retrievedby hashIf the protocol is implemented using smart contractson Ethereum, all the payments can be automatically pro-cessed using any token supported by the Ethereum plat-form (including Ether) so the foundation would not haveto deal with processing the payments. Moreover the pro-cess would be completely transparent as the code of thesmart contracts could be inspected and all the steps ofthe protocol would be public and indefinitely stored onthe blockchain.Using IPFS for storage would allow to cut down thecost of storage and lead to a system which is more cen-sorship resistant compared to a centralized approach.With a blockchain-based solution a foundation wouldstill be needed in order to fund the development of thesystem and support institutions in starting to adopt theuse of PRINCIPIA. On the other hand on the long term https://github.com/ethereum/wiki/wiki/White-Paper the foundation would have no control on the systemtherefore avoiding centralization of power and conflict ofinterests.Blockchain solutions also come with drawbacks, e.g.performance, cost of development and flexibility .Blockchain should be used just when specific trust issuesneed to be addressed. We envision PRINCIPIA becom-ing the system on which the whole peer review systemcan take place, therefore we think the additional com-plexity which a blockchain-solution would bring is bal-anced off by the system becoming censorship resistant,permissionless, transparent and decentralized.According to a recent report (Science and van Rossum,2017), some of the most prominent issues currently facedby scholarly communication – e.g., costs, openness, anduniversal accessibility to scientific information – might besolved by adequate processes based on blockchain. In facta number of different blockchain-based for peer reviewshave been proposed recently.Pluto networks aims at providing an ecosystem inwhich research can be published and reviewed. Paperscan be published by any actor in the system, and jour-nals do not exist. We think this process has limitationsas it opens up to the possibility for a reviewer to reviewa work outside of her expertise. Moreover removing com-pletely the concept of journal would not allow the currenteditors to participate in the system, leading to difficultiesin bootstrapping the system.ScienceRoot aims at defining a system for researchfunding and publications based on blockchain. Whilehaving some elements in common with PRINCIPIA, Sci-enceRoot does not tackle some of the problems whichPRINCIPIA aims at solving. There isn’t a reputationsystem so the problem of concentration of powers in fewjournal is not solved. Moreover ScienceRoot lack a sys-tem of incentives to incentives good reviews and the cre-ation of good journals.Blockchain For Peer Review is also developing ablockchain-based system for peer review. Also this sys-tem does not tackle all the problems PRINCIPIA is aim-ing to solve, for example there isn’t a liquid reputationsystem, and the system is also lacking a process for renu-merating reviewers.Ledger Journal is a peer-reviewed journal whichpublish research articles on the subjects of cryptocur-rency. Ledger is not aiming at replacing the whole peerreview system, despite that it contains some interestingelements which are including in PRINCIPIA as well. For https://arxiv.org/abs/1904.13093 https://pluto.network/ https://ledgerjournal.org/ojs/index.php/ledger VIII. CONCLUSION AND OUTLOOK
The public debate around the peer-review systemthrives, spurring a plethora of new proposals to improvethe current model and overcome its limitations. Inter-estingly, some of these proposals are implemented by blockchain technology , as we suggest here. Far fromcovering the extensive, fast-growing literature in the field,this white paper presents a novel framework for peer-review, open to contribution from all key players of thescientific ecosystem. Our proposal is built around fewprinciples aimed at creating a market for peer-review ser-vices and publications. PRINCIPIA allows reviewers tobe rewarded, thus improving the quality of their reviews,and opens the publishing market by lowering entry bar-riers. While we solidify these principles in a coherentarchitecture, the details of such implementation remainopen for discussion.We propose two different yet partially compatible im-plementation of PRINCIPIA, a whole system includinga detailed description of how new, liquid journals work,and a minimal implementation of the peer-review systemindependent by the creation of new journals. Both pro-posals are built around two key ingredients: an intrinsicreputation system and a decentralized market for peer-review services. These ingredients are aimed at over-coming the main limitations of the current system, aswell as potential limitations of other frameworks. As dis-cussed in the introduction, indeed, several proposals in-clude – at least partially – the principles presented here,in particular the idea of a transparent, open peer-reviewsystem. While some are thriving, others, such as AxiosReview, failed due to a lack of engagement of scientistsinvolved in the peer-review process. We believe that theintrinsic reputation system proposed here, which rewardand responsibilize reviewers, can engage scientists in join-ing PRINCIPIA. In the long term, the reputation gainedthrough reviews could form part of the overall reputationof a scientists, in the same way citations are related toreputation as an author.The creation of a market for peer-review, on the otherhand, will incentivize i) reviewers to improve the qualityof their reports, being rewarded for it, and ii) authors tojudge the quality of their papers by bidding an appro-priate review fees. This is another interesting differencewith respect to the current system, where the rationalfor authors is always to submit a paper to journals withthe highest reputation, since fees are asked only in caseof acceptance, while peer-review process is not paid – atleast in the majority of STEM-related journals.The two proposed architectures implement these in-gredients in slightly different ways, each one with its ownstrengths and limitations. The whole system includingjournals anchors the reputation of users to the reputa-tion of journals they serve, which in turns depends onthe citations collected by papers published on such jour-nals. Therefore, the reputation of a user is ultimately https://github.com/aletheia-foundation/aletheia-whitepaper/blob/master/WHITE-PAPER.md REFERENCES
B. J. Culliton, Nature Medicine , 1161 (1996), URL https://doi.org/10.1038/nm1196-1161 .Editorial, Nature Immunology , 297 (2003), URL https://doi.org/10.1038/ni0403-297 .J. P. Tennant, J. M. Dugan, D. Graziotin, D. C. Jacques,F. Waldner, D. Mietchen, Y. Elkhatib, L. B. Collister, C. K.Pikas, T. Crick, et al., F1000Research , 1151 (2017), ISSN2046-1402, URL https://f1000research.com/articles/6-1151/v2 .Editorial, Nature Catalysis , 279 (2019a), URL https://doi.org/10.1038/s41929-019-0276-9 .S. Cole, L. Rubin, and J. R. Cole, Scientific American ,34 (1977), ISSN 00368733, 19467087, URL .G. Loomes and R. Sugden, The Economic Journal ,805 (1982), ISSN 00130133, 14680297, URL .H. A. Simon, The Journal of Business , S209 (1986),ISSN 00219398, 15375374, URL .A. Tversky and D. Kahneman, in Multiple Criteria DecisionMaking and Risk Analysis Using Microcomputers (SpringerBerlin Heidelberg, 1989), pp. 81–126, URL https://doi.org/10.1007/978-3-642-74919-3_4 .W. B. Arthur, The American Economic Review , 406(1994), ISSN 00028282, URL .H. A. Simon, Political Psychology , 45 (1995),ISSN 0162895X, 14679221, URL .D. Kahneman, American Economic Review , 1449 (2003),URL https://doi.org/10.1257/000282803322655392 .S. Righi and K. Takács, Scientometrics , 587 (2017), URL https://doi.org/10.1007/s11192-017-2244-y .J. M. Campanario, Journal of the American Societyfor Information Science , 302 (1996), URL https://doi.org/10.1002/(sici)1097-4571(199604)47:4<302::aid-asi6>3.0.co;2-0 .M. Blissett, Behavioral and Brain Sciences , 203 (1982), URL https://doi.org/10.1017/s0140525x00011262 .E. Ernst, T. Saradeth, and K. L. Resch, Nature , 296(1993), URL https://doi.org/10.1038/363296a0 .F. Grimaldo, A. Marusic, and F. Squazzoni, PLOS ONE ,1 (2018), URL https://doi.org/10.1371/journal.pone.0193148 .Editorial, Nature Biomedical Engineering , 159 (2019b),URL https://doi.org/10.1038/s41551-019-0374-7 .M. Teplitskiy, D. Acuna, A. Elamrani-Raoult, K. Ko-rding, and J. Evans, Research Policy , 1825(2018), URL https://ideas.repec.org/a/eee/respol/v47y2018i9p1825-1841.html .A. Tomkins, M. Zhang, and W. D. Heavlin, Proceedings of the National Academy of Sciences , 12708 (2017), URL .K. J. Gorgolewski, D. S. Margulies, and M. P. Milham, Fron-tiers in Neuroscience (2013), URL https://doi.org/10.3389/fnins.2013.00009 .D. L. Longo and J. M. Drazen, New England Journal ofMedicine , 276 (2016), URL https://doi.org/10.1056/nejme1516564 .C. Bergstrom, New England Journal of Medicine , 1895(2016), URL https://doi.org/10.1056/nejmc1602586 .C. S. Greene, L. X. Garmire, J. A. Gilbert, M. D. Ritchie,and L. E. Hunter, Nature Genetics , 483 (2017), URL https://doi.org/10.1038/ng.3830 .Editorial, Nature Communications (2018a), URL https://doi.org/10.1038/s41467-018-05227-z .L. Teytelman, Nature , 411 (2018), URL https://doi.org/10.1038/d41586-018-06008-w .M. P. Milham, R. C. Craddock, J. J. Son, M. Fleischmann,J. Clucas, H. Xu, B. Koo, A. Krishnakumar, B. B. Biswal,F. X. Castellanos, et al., Nature Communications (2018),URL https://doi.org/10.1038/s41467-018-04976-1 .S. A. Gallo, A. S. Carpenter, D. Irwin, C. D. McPartland,J. Travis, S. Reynders, L. A. Thompson, and S. R. Glisson,PLOS ONE , 1 (2014), URL https://doi.org/10.1371/journal.pone.0106474 .D. B. Taichman, J. Backus, C. Baethge, H. Bauchner, P. W.de Leeuw, J. M. Drazen, J. Fletcher, F. A. Frizelle,T. Groves, A. Haileamlak, et al., New England Journalof Medicine , 384 (2016), URL https://doi.org/10.1056/nejme1515172 .M. McNutt, K. Lehnert, B. Hanson, B. A. Nosek, A. M.Ellison, and J. L. King, Science , 1024 (2016), URL https://doi.org/10.1126/science.aad7048 .V. Gewin, Nature , 117 (2016), URL https://doi.org/10.1038/nj7584-117a .Editorial, Nature Communications (2018b), URL https://doi.org/10.1038/s41467-018-06012-8 .Editorial, Nature Physics , 199 (2019c), URL https://doi.org/10.1038/s41567-019-0473-0 .Editorial, Nature , 527 (2018c), URL https://doi.org/10.1038/d41586-018-06045-5 .B. List, Nature , 9 (2017), URL https://doi.org/10.1038/546009a .Editorial, Nature Communications (2016), URL https://doi.org/10.1038/ncomms13626 .T. Ross-Hellauer, A. Deppe, and B. Schmidt, PLOS ONE , e0189311 (2017), URL https://doi.org/10.1371/journal.pone.0189311 .S. van Rooyen, T. Delamothe, and S. J. W. Evans, BMJ ,c5729 (2010), URL https://doi.org/10.1136/bmj.c5729 .G. Bravo, F. Grimaldo, E. López-Iñesta, B. Mehmani, andF. Squazzoni, Nature Communications (2019), URL https://doi.org/10.1038/s41467-018-08250-2 .J. K. Polka, R. Kiley, B. Konforti, B. Stern, and R. D. Vale,Nature , 545 (2018), URL https://doi.org/10.1038/d41586-018-06032-w .R. M. Alguliyev, R. M. Aliguliyev, and N. Ismayilova, CoRR abs/1506.02783 (2015), eprint 1506.02783, URL http://arxiv.org/abs/1506.02783 .J. Benet, CoRR abs/1407.3561 (2014), eprint 1407.3561,URL http://arxiv.org/abs/1407.3561 .D. Science and J. van Rossum (2017), URL https://digitalscience.figshare.com/articles/Blockchain_for_Research/5607778 . C. Furlanello, M. D. Domenico, G. Jurman, and N. Bus-sola (2017), eprint arXiv:1707.06552, URL https://arxiv.org/abs/1707.06552 .A. Extance, Nature , 301 (2017), URL https://doi.org/10.1038/d41586-017-08589-4https://doi.org/10.1038/d41586-017-08589-4