Publication of Decision Model Source Code: Attitudes of Health Economics Authors

Abstract

Computer decision models serve as the analytic foundation for most health economic evaluations, the prominence of which continues to grow. Cohen et al. [1], Cohen and Wong [2], and Sampson and Wrightson [3] have argued that authors should publish the decision model ‘source code’, by which we mean the model’s human-readable computer instructions or its underlying component files with formulas in the case of spreadsheet-implemented models. They explained that releasing source code would boost model credibility and allow other researchers to adapt existing computer code to answer similar and related questions, thus increasing the efficiency of health economics. Others (e.g., Padula et al. [4]) have argued against publication of source code, citing intellectual property concerns and the potential for models to be misused to promote misleading claims. We conducted a survey of health economic article authors regarding their willingness to publicly release their model source code. The Center for the Evaluation of Value and Risk in Health (CEVR) at Tufts Medical Center (Boston, MA, USA) has developed an online ‘Open-Source Model Clearinghouse’. Funded by the Bill and Melinda Gates Foundation, the clearinghouse encourages investigators to publicly post their models, and helps others to locate those models and download the code. As the clearinghouse website (http://www.GHCEA Regis try.org) explains, CEVR asks each author to provide summary information about their model (e.g., disease or condition modeled, intervention, etc.) and limits clearinghouse contents to “computer simulations developed to support original economic assessments of health interventions.” CEVR does “not evaluate the correctness or quality of posted models,” but instead aims to “facilitate ‘crowdsourcing’ of model review by individuals interested in each particular model’s problem area.” Nor does CEVR “require model documentation,” as we aim to allow the health economics community to establish de facto documentation standards that balance the need for adequate clarity with the amount of work attending model publication. We emailed primary authors of articles describing original cost per disability-adjusted life-year (DALY) averted analyses, published in peer-reviewed journals from 2010 to 2017. The Tufts Global Health CEA Registry (www. GHCEA Regis try.org) catalogs all such articles published in English. We targeted cost-per-DALY article authors for this survey because the open-source registry work is supported by the Bill and Melinda Gates Foundation, which is also supporting our cataloging of the cost-per-DALY literature, and which has an interest in promoting the sharing of information most salient to low and middle-income countries. We asked authors if they would post their code (or executable models) in the clearinghouse. We sent an initial email and one reminder email between August 8, 2018 and September 6, 2018. We invited only those authors answering in the affirmative to our initial survey to follow up by actually posting their models. We asked authors who said they would not post their code to identify concerns that factored into their decision. We sent emails to 337 authors and received 89 bounceback messages, resulting in a final sample of 248 distinct authors. We received 18 responses (7.3%). Five authors agreed to post their code in the clearinghouse. Of the five, four ultimately submitted models. The other author in this group declined to post code, stating that their model “needs to be refined further before publication.” The 13 responding authors who declined to post their code selected the following reasons for their decision (responses not mutually exclusive): need to document code * Joshua T. Cohen [email protected]