A System Dynamics Analysis of National R&D Performance Measurement System in Korea
RRESEARCH ARTICLE
A System Dynamics Analysis of National R&D PerformanceMeasurement System in Korea
Taekho You a and Woo-Sung Jung a,b a Department of Industrial and Management Engineering, POSTECH, Pohang 37673, Korea; b Department of Physics, POSTECH, Pohang 37673, Korea
ABSTRACT
Peer review is one of useful and powerful performance measurement process. InKorea, it needs to increase quality of R&D performance, but bibliometric evalua-tion and lack of peers have opposite effect. We used system dynamics to describeKorean R&D performance measurement system and ways to increase performancequality. To meet a desired R&D performance quality, increasing fairness and qualityof evaluation is needed. Size of peer pool decreased because of the specialization ofR&D projects and the
Sangpi process both, and it is critical to acquire both fairnessand quality. Also, shortening evaluation period affect to R&D performance qual-ity, by causing workloads increase, limiting long-term and innovative R&D projects,and decreasing evaluation quality. Previous evaluation policies do a role like micro-controlling the R&D’s activities, but increasing the size of peer pool and changingevaluation period would make a change to quality and fairness of evaluation.
KEYWORDS
R&D performance measurement system; Peer review process; System dynamics;
Sangpi process; Bibliometric evaluation.
1. Introduction
Nations invest huge amount of taxes to the R&D projects for science and technologydevelopment. It is important to measure the project’s performance, but it is a difficultwork. Measurement criteria, increasing number of R&D projects, science and technol-ogy development itself, conflict of interest, and correlation between those phenomenamake it more complex. Peer review is the professional tool for evaluating scientificwork. Peer review is effective to evaluate scientific work qualitatively because it relieson professional peer’s decision. It is used widely such as prizing, performance mea-surement, and evaluating research group or institution(Hemlin, 1996).The Korean government decided to enhance bibliometric evaluation since 2006. Formeasuring R&D performance and managing it the bibliometric indices are used such asSCI impact factor, number of citations, number of papers, and patents. And researcherset their annual goal using these indices. After 2006, R&D performance has grownremarkably, especially bibliometric indices increased about 30% annually. However, thebibliometric evaluation is criticized that the qualitative performance is not grown withquantitative one. The government has been trying to improve qualitative performanceof R&D activity, from diverging R&D investment strategy to managing researchers.Changing R&D performance measurement system is also one of these improvement
CONTACT Woo-Sung Jung. Email: [email protected] a r X i v : . [ c s . D L ] J a n olicies. It is now on debating how to change the system, for example, improving peerreview process and abolishing Sangpi processhas been discussed for a long time. In thispaper, we focus on describing and analyzing R&D performance measurement systemand issues with increasing qualitative performance.We use system dynamics to represent and analyze issues of R&D performance mea-surement system in Korea. System dynamics is a useful way to describe issues of acomplex system. System dynamics describes the system structure using subsystem di-agram, causal loop diagram or stock and flow diagram with feedback loops. It givesinsight to the dynamic behavior of the issue that occurs in the system(Forester, 1961;Sterman, 2000). Major mechanisms are represented to feedback loops that are rein-forced or counteracted with positive or negative loops, respectively. The changes ofR&D performance measurement system will accompany side effects. Therefore, min-imizing these effects is necessary to decision making. To do this, it is important toinvestigate cause and effect relations in the system. System dynamics offers a chanceto understand dynamics of R&D performance measurement system and political de-cision to the system (Rodrigues and Bowers, 1996).Rest of this paper is organized as following: in section 2, literature review of peerreview in R&D performance measurement system is introduced, because peer reviewis one of powerful and widely-used process to measure performance quality. In section3, a system dynamics approach of R&D performance measurement system in Koreawill be introduced. Issues of the system and causal loop diagram about them will bedescribed here. Finally, we summarize results of the system dynamics analysis andgive insight to the system.
2. Peer review
For evaluation, peer review is a powerful and effective process because it is determinedby professional peers. The problem of peer review, however, is also that it is highlyrely on professional peers. It is not easy to object to a decision although peers areindependent(Bornmann, 2011; Roy, 1985), so peer selection is important to make afair evaluation. Even though all know it, previous works show that bias exist suchas gender, nationality, prestige, and conservatism. For example, Wenneras and Wold(1997) and Sandstr¨om and H¨allsten (2008) shows that female and young researchersshould make more efforts to get funds in Swedish Medical Research Council (MFR).While CV, bibliography and a research proposal are reviewed by a committee withoutconflict of interest, and bibliometric indices has no difference between two, men hashigher success rate. Wenneras and Wold (1997) notes that peers who have relationshipgive higher score than others, and ’male gender’ tend to have this bonus more thanwomen.In the case of interdisciplinary research, peer review is more complex. Different as-sessment by different peers for their own research interests, difficulty in judgement, andpolicy make peers do biased evaluation(Langfeldt, 2006; Laudel, 2006; Pautasso andPautasso, 2010). Usually interdisciplinary research has more participants, participantsare international, or the project are top-down for governmental policy. Bias drivenpeer selection is easier and decision not related to project quality can be possible.Conflict of interest is another aspect of the disturbing evaluation. While regulationfor conflict of interest, the the evaluation is performed not related to research perfor-2ance itself. For example, peers often have a different view about the research topicand question. Or they often lie in opposite situation like major competitor. Most ofthe case peers are needed to be objective, but conflict of interest makes them difficultto be objective.These works imply two thing. First, peer’s relationship is critical factor to affect-ing evaluation results. If any relation exist peers suffered to give a bonus(Martinson,Anderson, and De Vries, 2005; Wenneras and Wold, 1997), because they know aboutthe work more than others or even they already grouped together. Second is that eventhough independent peers are selected for the evaluation, bias that exist in peer’s mindaffects to performance evaluation. It is hard to find and classify it, thus hoping forpeers to acting fairly(Martinson et al., 2005).The reason of bias in peer review can be summarized as following: peer individual,peer-researcher relation, and inter-peer relation. Variations to avoid bias are suggested,and we refer Lee, Sugimoto, Zhang, and Cronin (2013) as they give a summary forbias of academic peer review. They also summarize types of peer review applied, suchas double-blind peer review or open peer review. The fundamental of peer review isthat achieving both fairness and its quality, simply said. For example, double-blindpeer review achieves it by hiding information about peers and authors and open peerreview try opposite way by open information to the both. They have weakness, sodeciding which type is proper one is a heuristic problem.
While peer review is criticized by its reliability, validity and bias as other studies show,it is useful tool for the alternative of bibliometric evaluation. Opthof and Leydesdorff(2010) and Bertocchi, Gambardella, Jappelli, Nappi, and Peracchi (2015) suggest a cor-relation of evaluation results between bibliometric evaluation and peer review. Opthofand Leydesdorff (2010) studied that reports of CWTS and leading scientists’s publi-cation have same ranking structure of Netherland’s Medical Physics research. Whiledebates exist about this correlation after the study, such as van Raan, van Leeuwen,Visser, van Eck, and Waltman (2010), Bornmann (2010), and Waltman, van Eck, vanLeeuwen, Visser, and van Raan (2011), we can agree that they are not independentlyevaluated. Bertocchi et al. (2015) studied performance difference between peer reviewof Economics, History, Management and Statistics in Italy and its paper’s journalindices. Even though these research sub-areas have different bibliometric indices, thebibliometric ranking and peer review result give same classification results. The dif-ference between two result have no statistically significant, whether peer review isdouble-blinded or not. While these studies show evaluation results are statisticallyno different, however, we treat them as different one because the two evaluations hasdifferent effect in the system.In Korea, a peer selection policy to prevent bias and acquiring fairness is
Sangpi process. The
Sangpi process is that exclude possible peers who are related to theR&D participants at the peer selection step. Before the selecting peer, an ego-centricnetwork about R&D participants were made; if peer-peer or peer-researcher has anyrelation, they are linked. After that, peers within a distance are excluded. From doingthis, a peer is excluded if peer is in same university or institution, they graduate sameschool, or they participated in the same project before with one of R&D participants.Controlling size of distance peers who are professional and have no relation can beselected, but practical problem arise. As R&D fields are highly specialized network3ize in a field is decreased. If distance increases, peers selected are too far from theproject so it could not get reliable evaluation results. If distance decreases, peersselected are may have relation to R&D project participants, so fairness problem arise.The issue about
Sangpi process is to abolish and keeping it, described in Fig. 1(a).
3. Analysis of R&D Performance Measurementment System in Korea
As a start we use subsystem diagram to summarize practical issues. Note that to sum-marize we used reports in two Korean Institute, the Science and Technology PolicyInstitute (STEPI) and Korea Institute of S&T Evaluation and Planning (KISTEP),written in Korean. Fig. 1 is a subsystem diagram of the National evaluation system inKorea. As an international trend to increase R&D funds, the Korean government alsoincreases number of R&D programs. Also they decided to enhance bibliometric eval-uation since 2006, which determine a R&D work’s achievement by number of papers,impact factor, number of patents, etc. Bibliometric evaluation system is effective tocontrol all R&D programs, because scoring R&D performance by numerical indices iseasy to compare them. However, bibliometric evaluation gives two effect to the R&Dperformance evaluation system. First, they used evaluation results to comparing R&Dswithout considering their research fields. Bibliometric indices are different values in re-search fields and comparing between two other works is dangerous. Second, researcherstake more administrative workload. At the project selection step, researchers proposetheir annual goals such as how much they public papers or patents each year. Theyshould summarize and present their annual work more and more because competitionbetween researchers increases and evaluation quality decrease.Fig. 1(a) shows an issue about
Sangpi process. As we mentioned, the size of peer poolis controlled by the distance, and maximum number of peers are decreased when R&Dproject is more specialized. The limitation of peer pool and budget, the size of peer pooldecreased and achieving evaluation quality and fairness is hard. So there has voicedof abolishing
Sangpi process. But if abolishing, unfair peers would be selected morethan before, so the problem would not be solved clearly. System dynamics analysis ofthe process is described more in Fig. 2.R&Ds becomes interdisciplinary, and they are needed to produce more qualitativeperformance. There has voiced that evaluation system blocks for researchers to produceby creative and qualitative performance. The issue is about the annual evaluation,shown in Fig. 1(b). The annual evaluation has been adopted to manage ongoing R&Dprojects. However, increasing the number and size of reports make it hard to control.It eventually decrease time to focusing on R&D work, leading performance qualitydecreases. Some of creative performance cannot be produced annually and annualevaluation is the opposite evaluation process to doing these R&Ds. Fig. 3 describedmore about the issue.As a summary, we show two issues in Korean R&D performance evaluation system,
Sangpi process and annual evaluation. The system dynamics analysis for the systemwill be given in Section 3.2, but additional suggestions are given in Fig. 1(c). Forexample, increasing budget can contribute to solving such as lack of peer pool sizeand quality increase by give plenty of time with incentive to peers. However, they area part of solving problems and do not affect dynamics of the evaluation system to bechanged. 4 i g u r e . Sub s y s t e m d i ag r a m o f t h e R & D p e r f o r m a n ce m e a s u r e m e n t s y s t e m i ss u e s igure 2. Cycles describing the qualitative R&D evaluation
A causal loop diagram of the
Sangpi process and the annual report about R&D workis described in Fig. 2. We draw two loops, one is a negative feedback loop for the
Sangpi process and the other is a positive feedback loop for the annual report. Bothloops affect to evaluation quality and fairness, and R&D performance quality and sat-isfaction for the evaluation results is controlled by two policies. Note that there will beother uncountable factors for evaluation. For example, an incentive to successful R&Dproject(Squazzoni, Bravo, and Tak´acs, 2013) will give motivation to the researchers,but we do not handle these factors to focus on issues.The role of
Sangpi process is to exclude peers who are not professional and haveany relation to the R&D project. With the
Sangpi process reliable evaluation resultswith no bias can be achieved, but this process also decreases the number of possiblepeers. If size of peer pool is enough it causes no problems, but if not it will be hardto select professional and fair peers.So the loop B1 depends on size of peer pool. As Korean ministry restricts peersto Korean researchers, there is limited peer pool in the system. Accompany withthat, R&D projects are highly specialized, finding professional peers without relationbecomes a hard work. Thus quality of R&D evaluation decrease, and the unsatisfaction6 igure 3.
Cycle describing the challengeable R&D works of evaluation results leads unsatisfaction for competitive researchers. They would insistthat it is needed to repeal
Sangpi process to monitor R&D project strongly.To control the R&D projects, Korean government ministry requires more reportsto R&D researchers, such as the annual report or increase the detail of each report. Itwould lead to increase of workloads to both R&D researchers and the ministry. Morework would decrease project work time, which would cause decrease of performancequality. And to handle more reports, the ministry evaluates performance by bibliomet-ric indicators. The procedure is described in the loop R1. Since it is a positive feedbackloop, it expects that exponential growth of workloads and performance decrease, butthe limited budget and time controls this exponential growth while we do not draw inthe Fig. 2.Repealing
Sangpi process is not the solution since the process is essential to controlpeer evaluation process. As we mentioned, size of peer pool is an important factorto
Sangpi process. Regulation of peer pool should be expanded, including foreignresearchers and experts in industrial area. For example, for the commecialized R&Dprograms patent attorney or lawyer can be appropriate to evaluate R&D performance.As Marsh, Jayasinghe, and Bond (2008) noted that external peers have rare experiencefor evaluation and training them is another important issue. For doing this, more fundsto the peer review process and selecting appropriate peers by its type of R&D programwill be needed. The more the size of peer pool increases, the more professional andfair evaluation are achieved.To focus on another issue of annual evaluation, we draw another causal loop diagramin Fig. 3. We start by evaluation period, which is the duration between two sequentialevaluation in a R&D project. This value can be varied from each day to whole R&Dproject period that means only projection selection and final evaluation exist. However,it is not controlled by other variables, but decided by the policy in each country’sevaluation strategy and for considering R&D characteristics. We draw it as a balancing7oop B1 in Fig. 3, because the evaluation period cannot be changed easily, like a stablestate, and evaluation period affect to itself eventually.Evaluation period affect to quality of R&D outputs by three ways as we draw thesereinforcing loops in Fig. 3. These are about research works, researcher’s career, andevaluation quality itself. First, as evaluation period are shortened, R&D participantsshould produce more reports. Making each report is time-consuming work, so it wouldlimit time to focus on R&D activities relative to long evaluation period. Second, re-searchers should produce their research outcomes in a short time if evaluation periodbecomes shorten. Moreover, if evaluation is done by bibliometric indices, they neednumber of papers, patents, or any achievements in the period. In this case, the threatsmake researchers to avoid some of disciplines and so-called innovative R&D projectbecause they need time to produce outcomes and have higher probability to failure.The last loop is about evaluation quality. Demanding work for R&D evaluation in-crease if evaluation period is shortened. We can easily infer that increasing work withlimited resources means decrease of evaluation quality. If evaluation process does notwork normally, the probability for abusing would increase.As we mentioned, evaluation period has been optimized and decided by the policy.Changing evaluation period will give tremendous effect to the performance measure-ment system, because it does not mean the only changes of period itself but wouldimply changes of R&D management policy, strategy, and researcher’s career man-agement. Nevertheless, we suggest increasing period to ensure that researchers tryinnovative R&D works. To do that, qualitative evaluation is also needed to determinetheir work’s value if they failed. Fig. 1(c) would be good strategies to concrete theidea.
4. Conclusion
In this work we analyzed Korean R&D performance measurement system, showingthat bibliometric evaluation increase workload and negative effect for researchers totry creative works from the positive feedback loop. It is the ”truth deficit” of the eval-uation. For the trend of increasing interdisciplinary research and demand for creativeperformance, enhancing bibliometric evaluation is not proper. And Korean special-ized process, named
Sangpi , exists to limit the size of peer pool for increasing fairnessand evaluation quality, but it also has negative effect to them. From this we suggestthat repealing the process is not the solution so increasing the size of peer pool withexternal peers will be needed.We analyze issues by system dynamics approach, and propose these political sug-gestions as a possible policy. The system dynamics approach provides information forproject management. The results would give insight to make a political decision, butit needs to analyze the decision’s impact quantitatively. However, policy test is not asimple work, even starting by measuring quantities of each variables is hard to achieve.As it is not an easy work to apply these policy suggestion in the real world, a moreprecise modeling of the system and simulation would be proceded.Peer review is becoming more important process to measuring performance qual-itatively, both project management and academic area. We proposed a system dy-namical approach to the R&D performance evaluation in Korea, it will be benefit toany complex system of evaluation. The approach can be expanded to analyze var-ious administrative systems. It is needed to improve with political decision makingsince most decisions of administrative systems are irreversible. Although simulation8f such system is hard to establish, mapping causal relation considering stakeholdersand their issues is helpful to decision making. In academic area, bias like nationalityand conservatism is one good expansion to apply the analysis.
Acknowledgement
This work was supported by Basic Science Research Program through the Na-tional Research Foundation of Korea (NRF) funded by the Ministry of Education(2016R1D1A1B03932590).
References
Graziella Bertocchi, Alfonso Gambardella, Tullio Jappelli, Carmela A Nappi, and Franco Per-acchi. Bibliometric evaluation vs. informed peer review: Evidence from Italy.
ResearchPolicy , 44(2):451–466, March 2015.Lutz Bornmann. Towards an ideal method of measuring research performance: Some commentsto the Opthof and Leydesdorff (2010) paper.
Journal of Informetrics , 4(3):441–443, July2010.Lutz Bornmann. Scientific Peer Review.
Annual Review of Information Science and Technol-ogy , 45:199–245, 2011.W J Forester.
Industrial Dynamics, Massachusetts . MIT Press, 1961.Sven Hemlin. Research on research evaluation.
Social Epistemology , 10(2):209–250, January1996.Liv Langfeldt. The policy challenges of peer review: Managing bias, conflict of interests andinterdisciplinary assessments.
Research Evaluation , 15(1):31–41, April 2006.Grit Laudel. Conclave in the Tower of Babel: How peers review interdisciplinary researchproposals.
Research Evaluation , 15(1):57–68, April 2006.Carole J Lee, Cassidy R Sugimoto, Guo Zhang, and Blaise Cronin. Bias in peer review.
Journalof the Association for Information Science and Technology , 64(1):2–17, January 2013.Herbert W Marsh, Upali W Jayasinghe, and Nigel W Bond. Improving the Peer-ReviewProcess for Grant Applications: Reliability, Validity, Bias, and Generalizability.
AmericanPsychologist , 63(3):160–168, April 2008.Brian C Martinson, Melissa S Anderson, and Raymond De Vries. Scientists behaving badly.
Nature , 435(7043):737–738, June 2005.Tobias Opthof and Loet Leydesdorff. Caveats for the journal and field normalizations in theCWTS (” Leiden” ) evaluations of research performance.
Journal of Informetrics , 4(3):423–430, July 2010.Marco Pautasso and Cesare Pautasso. Peer reviewing interdisciplinary papers.
EuropeanReview , 18(2):227–237, May 2010.Alexandre Rodrigues and John Bowers. The role of system dynamics in project management.
International Journal of Project Management , 14(4):213–220, January 1996.R Roy. Funding science: The real defects of peer review and an alternative to it.
Science , 1985.Ulf Sandstr¨om and Martin H¨allsten. Persistent nepotism in peer-review.
Scientometrics , 74(2):175–189, January 2008.Flaminio Squazzoni, Giangiacomo Bravo, and K´aroly Tak´acs. Does incentive provision increasethe quality of peer review? An experimental study.
Research Policy , 42(1):287–294, February2013.John Sterman.
Business Dynamics: Systems Thinking and Modeling for a Complex Worldwith CD-ROM . McGraw-Hill Education, February 2000.Anthony F J van Raan, Thed N van Leeuwen, Martijn S Visser, Nees Jan van Eck, and Ludo altman. Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics ,4(3):431–435, July 2010.Ludo Waltman, Nees Jan van Eck, Thed N van Leeuwen, Martijn S Visser, and Anthony F Jvan Raan. On the correlation between bibliometric indicators and peer review: reply toOpthof and Leydesdorff.
Scientometrics , 88(3):1017–1022, September 2011.C Wenneras and A Wold. Nepotism and sexism in peer-review.
Nature , 387(6631):341–343,1997., 387(6631):341–343,1997.