Computing the Expected Value of Sample Information Efficiently: Expertise and Skills Required for Four Model-Based Methods
Natalia R. Kunst, Edward Wilson, Fernando Alarid-Escudero, Gianluca Baio, Alan Brennan, Michael Fairley, David Glynn, Jeremy D. Goldhaber-Fiebert, Chris Jackson, Hawre Jalal, Nicolas A. Menzies, Mark Strong, Howard Thom, Anna Heath
11 Computing the Expected Value of Sample Information Efficiently: Expertise and Skills Required for Four Model-Based Methods
Natalia R. Kunst, MSc,
Edward CF Wilson, PhD, Fernando Alarid-Escudero, PhD, Gianluca Baio, PhD, Alan Brennan, PhD, Michael Fairley, BE(Hons), David Glynn, PhD, Jeremy D. Goldhaber-Fiebert, PhD, Chris Jackson, PhD, Hawre Jalal, MD, PhD, Nicolas A. Menzies, PhD, Mark Strong, PhD, Howard Thom, PhD, Anna Heath, PhD, and on behalf of the Collaborative Network for Value of Information (ConVOI) University of Oslo; Yale University School of Medicine; Amsterdam UMC; LINK Medical Research; Health Economics Group, Norwich Medical School, University of East Anglia; Center for Research and Teaching in Economics (CIDE); University College London; School of Health and Related Research (ScHARR), University of Sheffield; Stanford Health Policy, Centers for Health Policy and Primary Care and Outcomes Research, Stanford University; University of York; MRC Biostatistics Unit, University of Cambridge; University of Pittsburgh; Harvard TH Chan School of Public Health; University of Bristol; The Hospital for Sick Children; University of Toronto
Abstract Objectives
Value of information (VOI) analyses can help policy-makers make informed decisions about whether to conduct and how to design future studies. Historically, a computationally expensive method to compute the Expected Value of Sample Information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, four EVSI approximation methods have made such analyses more feasible and accessible. We provide practical recommendations for analysts computing EVSI by evaluating these novel methods.
Methods
Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, analyst’s expertise and skills, and software required for four recently developed approximation methods. Information was also collected on the strengths and limitations of each approximation method.
Results
All four EVSI methods require a decision-analytic model’s probabilistic sensitivity analysis (PSA) output. One of the methods also requires the model to be re-run to obtain new PSA outputs for each EVSI estimation. To compute EVSI, analysts must be familiar with at least one of the following skills: advanced regression modeling, likelihood specification, and Bayesian modeling. All methods have different strengths and limitations, e.g., some methods handle evaluation of study designs with more outcomes more efficiently while others quantify uncertainty in EVSI estimates. All methods are programmed in the statistical language R and two of the methods provide online applications. Conclusion
Our paper helps to inform the choice between four efficient EVSI estimation methods, enabling analysts to assess the methods’ strengths and limitations and select the most appropriate EVSI method given their situation and skills.
Highlights The Expected Value of Sample Information (EVSI) can be used to prioritize research and design future studies to reduce decision uncertainty for policy-makers. Four recently published methods have overcome the computational issues associated with EVSI analysis but practical guidance on using and distinguishing these methods is lacking. Because the four methods use different approaches to estimate EVSI, requiring different expertise and skills, members of the Collaborative Network on Value of Information (ConVOI) reviewed these four EVSI computation methods to understand their required skills and inputs. They also identified the strengths and limitations of these methods and provide step-by-step guides. By comparing these methods, ConVOI provides practical guidance for analysts looking to compute EVSI and use it for study design. Analysts now have useful information to confidently select the most appropriate EVSI estimation method for their application and expertise.
Introduction
Decisions on which research studies to fund are intrinsically economic in nature; that is, public, private and third sector (research charity) funders have finite resources and their decisions bear an opportunity cost. Therefore, it is important to prioritise research studies that are expected to yield the greatest benefit for every dollar or euro spent. Typically, research priorities are set through consultation with experts, decision- or policy-makers and other key stakeholders (e.g., the James Lind Alliance [1]) [2]. Study outcomes are then selected for a number of reasons, such as ensuring timely collection of data. Most research designs focus on clinical outcomes and rarely consider economic implications. The sample size for most randomized controlled trials (RCTs) is determined using power calculations which explicitly manage type I and II errors for a statistical test of the selected primary outcome. However, this approach fails to take into account the opportunity cost of research: a large (and expensive) trial could be proposed while that knowledge could have minimal value to society. In contrast, Value of Information analysis (VOI) values an RCT and other types of research studies in terms of how much they reduce decision uncertainty (i.e., the probability of making a sub-optimal decision) about the best treatment for use in a population of interest. If a sub-optimal decision is made, then potential health improvements are foregone. The probability of making the wrong decision is multiplied by the size of the loss incurred by that decision to generate the expected loss associated with making a sub-optimal decision [3]. New data are expected to reduce decision uncertainty, and so reduce the expected loss. The Expected Value of Sample Information (EVSI) measures the “expected reduction in expected loss" from a given research study. Scaled up to the relevant population this can be expressed in health terms of life years, or quality-adjusted life years (QALYs), or in monetary units [4, 5]. The difference between the EVSI and the cost of the research is the Expected Net Benefit of Sampling (ENBS). An increasing number of authorities and Health Technology Assessment (HTA) agencies acknowledge the importance of VOI as part of a cost-effectiveness analysis and are recommending it [6-9]. Thus, EVSI methods will soon be required from analysts who conduct cost-effectiveness analyses for HTA agencies. Until recently, EVSI calculations were extremely computationally expensive, potentially taking weeks or months [10], as they required nested simulation methods [11]. Hence, most VOI analyses were restricted to computing the Expected Value of Perfect Information (EVPI) or the Expected Value of Partial Perfect Information (EVPPI). However, new algorithms and associated software [12-20] enable the efficient computation of EVSI for realistic decision models. While these methods have lowered the computational barriers to VOI analyses, they differ in their approach, requiring different expertise and skills. However, there is no structured comparison of the practical steps required to use them. Thus, it is challenging for analysts to determine which method is appropriate for their situation and expertise. Additionally, each method has different strengths and limitations that could make it more suitable for a given decision problem. The Collaborative Network for Value of Information (ConVOI) [21] is an international group of researchers with interests in the application and development of methods for VOI calculation. This manuscript provides practical guidance and good practice recommendations for computing EVSI using four recently developed approximation methods. For each method, we: 1) provide a step-by-step guide to its use, 2) compare the expertise and skills required to implement it, and 3) highlight its strengths and limitations.
Four EVSI approximation methods
This paper focuses on four recently developed estimation methods, developed by Strong, Oakley (13) (regression-based method [RB]), Menzies (14) (importance sampling method [IS]), Jalal and Alarid-Escudero (16) building on work from Jalal, Goldhaber-Fiebert (15) (Gaussian approximation method [GA]), and Heath, Manolopoulou (17) (moment matching method [MM]). These calculation methods were selected as they place limited restrictions on the complexity of the underlying decision-analytic model and/or data collection exercise when calculating EVSI. The recommendations presented in this paper were developed and reviewed by ConVOI to aid analysts looking to compute EVSI.
Inputs Required for Efficient EVSI Calculation
The considered EVSI approximation methods have diverse requirements and make different assumptions. However, all require a decision-analytic model on which a probabilistic sensitivity analysis is conducted. The IS and MM methods also require calculation of the EVPPI for the parameter(s) to be evaluated and updated in the proposed study. If studies informing different parameters or groups of parameters are considered, then EVPPI would need to be computed for each. The following section briefly outlines these requirements and assumptions.
Decision-Analytic Model
VOI requires an objective function that should be optimized to determine the best course of action [22]. Economic evaluations in health care typically define this in terms of net health or monetary benefit for each of 𝑇 interventions [23], which uses a “willingness-to-pay” threshold to put the health consequences on the same scale as the costs for the different interventions [23]. The function that computes this usually takes the form of a decision-analytic model. Decision-analytic models are mathematical models often used for cost-effectiveness analyses. They draw on set of inputs that we denote 𝜽 . These inputs could include information on the prevalence of a disease, the effectiveness of treatments, background mortality, health-related utility weights and costs. The model can take a number of forms, foremost among them are decision trees, Markov models and microsimulation models [24]. Conceptually, the decision-analytic model maps a set of inputs to the output, net benefit (either in monetary or health units). Assuming risk neutrality, the intervention with the highest expected net benefit should be implemented in the wider population. Probabilistic Sensitivity Analysis
Probabilistic sensitivity analysis (PSA), sometimes known as probabilistic analysis or uncertainty analysis, is performed to propagate input parameter uncertainty to the model output under each decision alternative and thereby quantify decision uncertainty in a cost-effectiveness analysis [25]. PSA simultaneously varies all parameters for which there is meaningful uncertainty. Uncertainty in the model inputs is characterized using probability distributions, 𝑝(𝜽) [26]. PSA is often conducted using Monte Carlo methods where 𝑆 parameter sets are drawn from 𝑝(𝜽) , for the whole set of model parameters 𝜽 = (𝜃 , … , 𝜃 𝑃 ) . The decision-analytic model is evaluated at all 𝜽 𝒔 , 𝑠 =1, … , 𝑆 to estimate the costs and health outcomes of each of the 𝑇 strategies. This produces a distribution for the net monetary benefit for each strategy, which we denote NB 𝑡𝜽 , for 𝑡 = 1, … , 𝑇 . In non-linear models, PSA is required to generate the expected net monetary benefit of each treatment strategy, E 𝜃 [NB 𝑡𝜽 ] . PSA results can be presented in the form of a cost-effectiveness acceptability curve (CEAC), cost-effectiveness acceptability frontier (CEAF), expected loss curves (ELCs), and cost-effectiveness plane [3, 6]. Furthermore, PSA is key to determining the value of potential future research using VOI methods. To compute EVSI, we require the PSA simulations of the model inputs 𝜽 𝒔 and the corresponding simulations for the net monetary benefit NB 𝑡𝜽 𝒔 for 𝑠 = 1, … , 𝑆 and 𝑡 = 1, … , 𝑇 . These simulations should be saved in a matrix or spreadsheet form, often called a PSA dataset, where the columns contain first the input parameters and then the outputs from a decision-analytic model. Each row then contains the parameter sets drawn from their distributions, and their corresponding simulated results from a decision-analytic model output. We have provided an example of the PSA simulations in Supplementary Materials. While PSA results are necessary for all the four efficient EVSI calculation methods, two of the methods (RB and GA) require the above mentioned PSA matrix from a traditional cost-effectiveness analysis while the other two (IS and MM) require an augmented PSA simulation matrix that is presented in the following section. Expected Value of Partial Perfect Information
The EVPPI computes the value of eliminating all uncertainty about a subset of the model parameters [27-29]. Specifically, the model parameters are split into two subsets 𝜽 = (𝜙, 𝜓) , where we propose to gather further information about the model parameters 𝜙 . Typically, a proposed study does not collect information about all the underlying parameters in a decision-analytic model and therefore, 𝜓 indicates the parameters that will not be directly informed by the proposed data collection. Mathematically, the EVPPI for the parameters 𝜙 is defined as EVPPI = E 𝜙 [max 𝑡 E 𝜓|𝜙 [NB 𝑡𝜽 ]] − max 𝑡 E 𝜃 [NB 𝑡𝜽 ], (1) where the inner expectation in the first term of equation (1) calculates the net monetary benefit for each intervention conditional on 𝜙 . The second term calculates the value of the decision made with current information, i.e., the expected net benefit of the treatment with the highest expected net benefit. The word “dataset” is not used in the traditional sense; here, the PSA dataset simply contains simulated values from distributions representing the uncertainty in the parameter estimates.
The PSA outcomes are simulated using probability distributions for 𝜙 and the EVPPI is estimated by computing the net monetary benefit for each intervention conditional 𝜙 = 𝜙 𝑠 for 𝑠 = 1, … , 𝑆 . Several approximation methods are available to estimate the conditional net monetary benefit for each value 𝜙 𝑠 [27, 30-34] and have been formally assessed in [28] and [29]. Software is also available to perform these calculations [35-37]. The MM and IS methods require a PSA simulation matrix with 𝑇 additional columns that contain Monte Carlo estimates of the expected net benefits E 𝜓|𝜙 [NB 𝑡𝜽 ] , conditional on 𝜙 𝑠 for 𝑠 = 1, … , 𝑆 , but averaged over the uncertain values of 𝜓 . We denote these simulations 𝜂 𝑡𝑠 . EVSI Methods
EVSI for a proposed research strategy that collects additional data, denoted 𝑿 , is defined as EVSI = 𝐸 𝑿 [max 𝑡 E 𝜽|𝑿 [NB 𝑡𝜽 ]] − max 𝑡 E 𝜽 [NB 𝑡𝜽 ]. (2) In this setup, 𝑿 are observ able , but not yet observ ed — and possibly never will be. At this stage, we consider the possibility of collecting data in the future and we compute EVSI to determine whether we should defer making a decision on the optimal intervention, among the 𝑇 possible options, and instead invest money and time collecting 𝑿 . In line with a full Bayesian approach (which underpins the ideas behind the VoI analysis), the distribution of 𝑿 is defined by 𝑝(𝑿, 𝜽) = 𝑝(𝜽)𝑝(𝑿|𝜽) where 𝑝(𝜽) is the marginal distribution of 𝜽 and 𝑝(𝑿|𝜽) is the sampling distribution of the data. In a full Bayesian setting, 𝑝(𝜽) is a “prior” distribution — in the sense that it represents the current level of uncertainty on the model parameters, before observing the new data 𝑿 . In reality, this distribution is defined by the PSA procedure and can represent the result of a Bayesian update given observed data that is used to construct the current economic model. We note that 𝑿 will give information on the subset of model parameters 𝜙 , where 𝜙 could be the whole set 𝜽 . By definition, this means that 𝜓 and 𝑿 are independent given 𝜙 and that 𝑝(𝑿|𝜽) = 𝑝(𝑿|𝜙) . The second term of equation (2) can be estimated from the initial PSA. To obtain simulations for 𝑿 , we simulate potential study outcomes, 𝑿 𝒔 , 𝑠 = 1, … , 𝑆 , for each row of the PSA dataset, i.e., we generate a single sample from 𝑝(𝑿|𝜽 𝒔 ) for 𝑠 = 1, … , 𝑆 . Traditionally, EVSI has been computationally demanding and methodologically challenging because a large number of simulations are needed to estimate the posterior mean of the net benefit for each 𝑿 𝒔 . This requires a Bayesian model for the distribution of the data and the parameters, where the posterior mean is given by 𝜇 𝑡 (𝑿 𝒔 ) = E 𝜽|𝑿 𝒔 [NB 𝑡𝜽 ]. (3) The EVSI approximation methods in this paper estimate 𝜇 𝑡 (𝑿) with a reduced computational burden. For computational simplicity and stability, VOI calculations should be undertaken using incremental net benefit defined, without loss of generality, as INB 𝑡𝜽 = NB 𝑡∗𝜽 − NB 𝑡𝜽 , for 𝑡 = 2, … , 𝑇 . We could also use the opportunity loss, defined as the incremental net benefit of all treatments from the optimal strategy 𝑡 ∗ . By estimating the posterior mean of the incremental net benefit, we only need to add 𝑇 − 1 columns containing the posterior mean conditional on 𝑿 𝒔 to our PSA dataset. Once these simulations, denoted 𝜇 𝑡 (𝑿 𝒔 ) , are available, the EVSI can be estimated by
1𝑆 ∑
𝑆𝑠=1 max 𝑡 {𝜇 𝑡 (𝑿 𝒔 )} − max 𝑡 {1𝑆 ∑ 𝑆𝑠=1 𝜇 𝑡 (𝑿 𝒔 )}, (4) where 𝜇 (𝑿 𝒔 ) = 0 for all 𝑠 = 1, … , 𝑆 . Expertise Required for EVSI Calculation
We have identified three skills that the analyst may require to compute EVSI using the approximation methods reviewed in this manuscript. None of the methods require all three skills but each method requires the mastery of at least one of the skills presented. Table 1 summarises the required skills and inputs for each method.
Regression Methods
Regression methods that model the relationship between a set of predictors and an outcome of interest are required for both the RB and GA methods. Both methods use regression metamodeling to model the incremental net benefit as a function of model inputs or quantities related to model inputs. The two methods differ as they require alternative covariates. Both of these EVSI methods model relatively complex relationships using models that require little information about the functional relationship between the independent and dependent variables. To account for potential nonlinear relationships between these variables, flexible regression methods characterized using Generalized Additive Models (GAMs) [38] are the predominant regression methods for EVSI calculation. Standard software is available to fit GAMs [39-41], but the EVSI estimate can be affected by the structure of the GAM model and, thus, an understanding of these models is required. Gaussian Processes [42] have also been suggested for more complex problems [13], so knowledge of these methods may also be required. If regression methods are used to estimate EVSI, it is important to assess whether the regression model has correctly captured the relationship between the independent and dependent variables [13]. Regression methods assume that the residual error is uncorrelated with the fitted values from the regression, has zero mean and trivial covariance structure. These assumptions can be confirmed by observing no groups, systematic features, or outliers in the plot of the residuals against the fitted values [43].
Specification of data generating distribution
EVSI computes the expected value of collecting information in a future research study. The future research study would collect a potential data set, 𝑿 , with 𝑁 𝒪 different clinical, health or economic outcomes from 𝑁 participants. For example, a clinical trial would collect the primary and secondary outcomes from each participant. All EVSI computation methods require the simulation of potential data sets (the GA method requires study data simulation to compute the effective sample sizes of the priors of the 𝜙 only if these cannot be estimated directly, c.f. Bayesian Updating). As outlined above, this is achieved by specifying the assumed data generating distribution 𝑝(𝑿|𝜙) . Thus, analysts must have knowledge of probability distributions. For the RB method, the simulated data sets 𝑿 𝒔 for 𝑠 = 1, … , 𝑆 must be summarized to reflect how the data would be analyzed following the trial. This requires expertise in standard methods for the statistical analysis of study results, for example, maximum likelihood procedures. A variation of the GA method can be also be applied if a summary statistic is available. For the IS method, the analytic likelihood for 𝑿 must be specified and coded as a function. Likelihood is 0 used in its proper statistical meaning and, therefore, this specification requires statistical expertise. Typically, this likelihood specification will require an analyst who has knowledge of statistical theory, function development and coding. Bayesian Updating
Bayesian methods are a statistical paradigm in which conclusions are formally updated as evidence accumulates [44]. A full Bayesian model consists of probability distributions specified for the model inputs 𝑝(𝜽) , the prior distribution defined using the PSA distributions, and the sampling distribution for the proposed data collection. Based on these distributions, Bayesian methods update the distribution of the parameters conditional on the data 𝑿 to produce a posterior distribution. In practice, this is undertaken using specialized software such as BUGS [45], JAGS [46] or Stan [47]. The MM and, in some cases, GA methods are explicitly based on Bayesian analysis, which requires expertise in these programs and in the specification of Bayesian models. This is advantageous if the final trial analysis will be performed according to Bayesian principles as the statistical analysis plan has been developed as part of the design process. Additionally, the GA method is based on the prior effective sample size [48]. The prior effective sample size, denoted 𝑛 , is the number of participants that would need to be studied to obtain the level of information in the prior. In some Bayesian models, typically based on conjugate prior-likelihood pairs, the prior effective sample size can be estimated directly from the parameters of the distributions [48]. If this is not possible, then Jalal and Alarid-Escudero (16) present two algorithms to compute the prior effective sample size (see the Supplementary Materials). The first of these is based on summary statistics and the second on a full Bayesian analysis. Table 1. The skills and inputs required to compute EVSI with the regression-based, importance sampling, Gaussian approximation and moment matching methods Inputs
1 Decision-analytic model
2 Probabilistic sensitivity analysis
3 Simulations of the expected net benefit conditional on 𝜙 (required to compute EVPPI) Skills
1 Regression methods
2 Specification of likelihoods * 3 Bayesian updating * GA, Gaussian approximation method; IS, importance sampling method; MM, moment matching method; RB, regression-based method; indicates that the skill/input is required; * indicates that the skill/input may be required. Calculating EVSI
To highlight how these skills and inputs are used to compute EVSI, we have presented step-by-step guides to each of the four methods considered. These algorithms are provided in Boxes 1-4. Box 1. Step-by-step guide for regression-based method
INB, incremental net benefit; PSA, probabilitic sensitivity analysis; θ, set of inputs in a decision-analytic model; p(θ), probability distributions to characterize uncertainty in the model inputs; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, new data proposed to be collected; μ, posterior mean; W(X), a summary measure for the data.
2 Box 2. Step-by-step guide for importance sampling method
INB, incremental net benefit; EVPPI, Expected Value of Partial Perfect Information; PSA, probabilitic sensitivity analysis; θ, set of inputs in a decision-analytic model; p(θ), probability distributions to characterize uncertainty in the model inputs; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ) φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, new data proposed to be collected; μ, posterior mean; η, PSA simulations used to compute EVPPI.
3 Box 3. Step-by-step guide for Gaussian approximation method
INB, incremental net benefit; PSA, probabilitic sensitivity analysis; θ, set of inputs in a decision-analytic model; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ) φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, new data proposed to be collected; μ, posterior mean.
4 Box 4. Step-by-step guide for moment matching method
INB, incremental net benefit; EVPPI, Expected Value of Partial Perfect Information; EVSI, Expected Value of Sample Information; PSA, probabilitic sensitivity analysis; θ, set of inputs in a decision-analytic model; p(θ), probability distributions to characterize uncertainty in the model inputs; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ) φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; X, new data proposed to be collected; μ, posterior mean; η, PSA simulations used to compute EVPPI; Q, number of times PSA is performed. Strengths and Limitations
As each method is different, the most suitable method for EVSI calculation will depend on the decision-analytic model, the sampling distribution for the data, the expertise of the analyst and the amount of computation time available. The following section highlights the strengths and limitations of these methods to help analysts select the most appropriate method. A summary of these strengths and limitations is presented in Table 2.
Regression-Based Method
Strengths
The decision-analytic model does not need to be rerun to produce EVSI estimates, so EVSI can be computed by an analyst with access to the PSA results only. Standard statistical software includes procedures to fit flexible regression models, particularly GAM regression, making this method relatively simple to implement once the summary statistics
𝑊(𝑿 𝒔 ) are available. Furthermore, EVSI estimates across different sample sizes for the future data are obtained at a constant computational cost. Finally, if the model is judged to fit well, an estimate of the uncertainty in the EVSI estimate can be obtained using an algorithm developed by Strong, Oakley (13). An online application [36] was developed to compute the EVPPI, using the regression-based method [31]. This tool, called Sheffield Accelerated Value of Information (SAVI), fits a regression model between the INB 𝑡𝜽 𝒔 and the parameters of interest 𝜙 . Due to the similarity between the EVPPI and EVSI calculation methods, SAVI can be used to compute EVSI, once the future data sets have been summarized 𝑊(𝑿 𝒔 ) , 𝑠 = 1, … , 𝑆 . This is achieved by augmenting the PSA matrix with (a) column(s) containing the data summaries. This must be saved and uploaded into SAVI and EVSI is then equal to the “EVPPI” calculated for the column(s) containing the data summary. Limitations
For complex studies where the number of collected outcomes 𝑁 𝒪 is greater than five or six, it can be challenging to fit a sufficiently accurate regression model. The simulated study data must also be correctly summarised using 𝑊(𝑿) to obtain accurate EVSI estimates. This can be challenging and time-consuming in more complex studies. The relationship between the incremental net benefit and
𝑊(𝑿) must also be well-approximated by the regression model to ensure accurate EVSI estimation. Importance Sampling Method
Strengths
This method only requires the PSA results to compute the EVSI (i.e., access to the original decision-analytic model is not necessary). Furthermore, the IS method can be used irrespective of the number of study outcomes 𝑁 𝒪 with similar complexity, once the likelihood function for the potential data to be collected has been defined. Limitations
The accuracy of the EVSI estimation relies heavily on the appropriate specification of the analytic likelihood and the accurate estimation of the EVPPI. Because the most efficient EVPPI estimation methods are based on regression methods [29, 31], the analyst must be confident assessing the accuracy of regression models before proceeding with EVSI estimation. Furthermore, the IS method can have computational issues for large sample sizes of the future data as the likelihood tends to 0 with increasing sample size, leading to inaccurate EVSI estimation [10]. Finally, uncertainty in the EVSI estimation procedure cannot be estimated using this method.
Gaussian Approximation Method
Strengths
Estimation of EVSI with this method does not require to rerun the decision-analytic model; thus, access to PSA results is sufficient. This method estimates EVSI using flexible non-parametric (typically GAM) linear regression models. Although GA method requires an estimate of the prior effective sample size 𝑛 for the proposed data collection, 𝑛 can be estimated directly from previous studies. Furthermore, this method estimates EVSI across different sample sizes of the proposed data collection exercise at minimal computational cost, once the prior effective sample size has been estimated. Thus, the optimal sample size for proposed data collection can be obtained cheaply. Finally, the method can estimate the uncertainty in the EVSI estimation procedure using the same approach as in [13] based on [49]. An online repository is available for the GA method (https://zenodo.org/record/3263876), which contains a function that estimates the fitted values 𝜇 𝑡 (𝑿 𝒔 ) from a GAM regression model [38] and estimates of 𝑛 for 𝑝 = 1, … , 𝑃 . Thus, implementation for this method can be automated if estimates for the effective sample size are available for each parameter and if a GAM regression model accurately captures the relationship between the incremental net benefit and the parameters of interest. 7 Limitations
This estimation method can lead to inaccurate EVSI estimation if the prior effective sample size is small. Relatively intensive simulation methods may also be required to estimate the prior effective sample size if it cannot be obtained directly. The GA method also relies on a regression model, which may perform poorly if the regression model does not capture the relationship between the net monetary benefit and the parameters 𝜙 . Moreover, if the number of collected outcomes 𝑁 𝒪 is over five or six, flexible GAM regression methods become challenging [31]. Although the analyst can use linear regression model instead, this model may not be sufficiently capture the relationship, leading to inaccurate EVSI estimates. Moment Matching Method
Strengths
This method uses the same nested simulation structure as the gold standard nested Monte Carlo methods. This means that if an analyst has already developed the nested Monte Carlo method for EVSI estimation, it can be easily adapted for this method. Furthermore, this method has been extended [19] to estimate EVSI for multiple alternative sample sizes with a fixed additional computational cost. This extended method also provides a measure of uncertainty around the final EVSI estimate. The MM method can estimate EVSI irrespective of 𝑁 𝒪 the number of outcomes considered. The EVSI package in R has been developed to implement the MM method based on a Bayesian decision-analytic model. The manual is available [50] and the EVSI package can be installed in R using the command: devtools::install_github("annaheath/EVSI") Limitations
The MM method requires simulated study data and relies on performing a PSA 𝑄 times, with
30 < 𝑄 <50 . Thus, the decision-analytic model must be rerun a significant number of times. Therefore, in computationally expensive decision-analytic models, such as microsimulation models, this method requires significant computational power. Next, if the original PSA simulation size is small, this method is inaccurate as it is based on an estimate of the variance of NB 𝑡𝜽 𝒔 , which must be sufficiently accurate. The MM method can also lead to inaccurate EVSI estimates when the sample size of the proposed study is less than 10. Finally, this method requires an accurate estimate of EVPPI and is more accurate for studies that will have significant impact on the underlying uncertainty in the decision-analytic model, i.e., the EVPPI of 𝜙 needs to be high compared to the value of reducing all model uncertainty (i.e., EVPI), ideally greater than 40%. 8 Table 2. Selected strengths and limitations of the four EVSI approximation methods Strengths
1 Estimates EVSI for complex studies collecting a large number of outcomes
2 Only requires the PSA results
3 Uses non-parametric (typically GAM) regression
4 Estimates EVSI for different study sizes with same computational cost
5 Quantifies uncertainty in estimate Limitations
1 Requires simulated study data *
2 Requires accurate EVPPI estimation
3 Can be computationally challenging to estimate EVSI for proposed studies with a large sample size
4 Requires simulated study data to be summarized in a low dimensional statistic
5 Struggles if proposed study has more than 5 outcomes
6 May estimate inaccurate EVSI if the proposed study has small prior effective sample size
7 May estimate inaccurate EVSI if the proposed study has small sample size GA, Gaussian approximation method, IS, importance sampling method, MM, moment matching method; RB, regression-based method; indicates that the characteristic is required for the given method; * indicates that the characteristic may be required for the given method. Real-World Examples
To aid the implementation of the reviewed EVSI methods, we have created a comprehensive GitHub repository that presents the code used to compute EVSI in the original publications, available at https://github.com/convoigroup. This repository also contains a suite of practical examples of EVSI computation that demonstrate EVSI calculations across several real-world examples using common decision-analytic model structures, such as Markov models. We provide both an example that was previously published in the literature [51] and hypothetical examples. These examples are all developed in 9 the statistical computing language R . Thus, our GitHub repository demonstrates how the reviewed methods can be used in practice and will help analysts to implement VOI methods in their own work. Presenting EVSI results
Once EVSI results are available, the EVSI package in R contains several graphical displays to present EVSI and related quantities to practitioners and stakeholders. Results of an EVSI analysis can be loaded into the EVSI package, irrespective of the computation method used. These graphics can then be displayed in R directly or explored using a dynamic graphical display launched from within R . For analysts unfamiliar with R , EVSI results can be loaded into an online interface and these graphics can be explored online at https://egon.stats.ucl.ac.uk/projects/EVSI/Test/. The optimal sample size estimated using VOI methods can also be presented in form of a curve of optimal sample size (COSS) [52]. Conclusion
VOI analysis has the potential to guide policy-makers in the prioritization and design of future research studies, thereby improving decision-making. Increasingly, HTA agencies are acknowledging this potential and are recommending VOI analyses to determine whether/what potential future research is needed. In this study, members of ConVOI have provided a practical guide and good practice recommendations to facilitate the implementation of these methods. Our recommendations outline the inputs, analyst skills, and software required to use each of the EVSI approximation methods. We have also highlighted the strengths and limitations of each method. These recommendations are supported by a recent review that compared properties of these four recently developed, efficient computation methods across three decision-analytic models [10]. An accompanying GitHub repository includes R code demonstrating all four EVSI methods in real-world examples. Thus, this guide helps analysts choose the method that is the most suitable for their application and skills, and our code repository provides practical support to aid the implementation of these methods. This increases the feasibility and accessibility of the EVSI methods as they become an important and required tool. Funding
NRK was funded by the Research Council of Norway (276146) and LINK Medical Research. EW was funded by Norwich Medical School. FAE was funded by the National Cancer Institute (U01- CA-199335) as part of the Cancer Intervention and Surveillance Modeling Network (CISNET). GB was partially 0 funded by a research grant sponsored by Mapi/ICON at University College London. AB was funded through Higher Education Funding Council for England and a range of UK and International research grant award bodies. MF was funded by the Stanford Interdisciplinary Graduate Fellowship. DG has no funding to declare. JDGF was funded in part by a grant from Stanford’s Precision Health and Integrated Diagnostics Center (PHIND). CJ was funded the UK Medical Research Council programme MCUU00002/11. MS has no funding to declare. HJ was funded by NIH/NCATS grant 1KL2TR0001856. NM was supported by National Institutes of Health (NIH) [R01AI112438-02.]. HT was funded by the NIHR Biomedical Research Centre at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol. AH was funded by the Canadian Institute of Health Research through the PERC SPOR iPCT grant. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.
Contributions
NK has made substantial contributions to: Concept and design, Acquisition of data, Analysis and interpretation of data, Drafting of the manuscript, Critical revision of paper for important intellectual content, Administrative, technical, or logistic support. EW has made substantial contributions to: Concept and design, Analysis and interpretation of data, Drafting of the manuscript, Critical revision of paper for important intellectual content. DG has made substantial contributions to: Concept and design, Acquisition of data, Analysis and interpretation of data, Critical revision of paper for important intellectual content. FAE, GB, AB, MF, JDGF, CJ, HJ, NAM, MS, and HT have made substantial contributions to: Concept and design, Analysis and interpretation of data, Critical revision of paper for important intellectual content. AH has made substantial contributions to: Concept and design, Analysis and interpretation of data, Drafting of the manuscript, Critical revision of paper for important intellectual content, Supervision. All authors approved the final draft.
References
Supplementary Materials
Accompanying the manuscript:
Computing the Expected Value of Sample Information Efficiently: Expertise and Skills Required for Four Model-Based Methods
Table of Contents We provide an example of a PSA dataset in Table S1 and an example of an augmented dataset in Table S2. Table S1. An example of a PSA dataset from a traditional cost-effectiveness analysis
QALY, quality-adjusted life years; θ, set of inputs in a decision-analytic model; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; p.dr.t, input parameter probability of distant recurrence for strategy t =1,…, T ; hr.dr.t, input parameter hazard ratio of distant recurrence for strategy t =1,…, T ; p.tox.t, input parameter probability of treatment toxicity for strategy t =1,…, T ; u.ndr, input parameter utility weigth for non distant recurrence health state; u.dr, input parameter utility weigth for distant recurrence health state; u.d.tox, input parameter utility weight decrement due to treatment toxicity; c.dr, input parameter cost of distant recurrence health state; c.tox, input parameter cost of treatment toxicity; qaly.t, model outcome QALYs for strategy t =1,…, T ; cost.t, model outcome costs for strategy t =1,…, T . Table S2. An example of an augmented PSA dataset θ, set of inputs in a decision-analytic model; S, number of parameter sets that are drawn from p(θ) in PSA; T, number of considered interventions; θ=(φ,ψ), φ are model parameters for which we are aiming to collect further information and ψ are parameters that will not be directly informed by the proposed data collection; nmb.t, net monetary benefit for strategy t =1,…, T ; evppi.t, expected value of partial perfect information for strategy t =1,…, T. This section proposes three alternative methods for calculating the prior effective sample size. The effective sample size 𝑛 is the number of patients that would have to provide data to generate the “amount” of information in the prior. In some settings, 𝑛 can be found from the parameters prior distribution. For example, 𝑛 for a beta prior 𝐵𝑒𝑡𝑎(𝑎, 𝑏) coupled with a binomial likelihood 𝑥 ∼ 𝐵𝑖𝑛(𝑛, 𝑝) is equal to 𝑎 + 𝑏 . This comes from conjugacy as the posterior distribution is
𝐵𝑒𝑡𝑎(𝑎 + 𝑥, 𝑏 + (𝑛 − 𝑥)) . This is the case for specific conjugate pairs listed in Table S3.
Table S3. Identifying 𝑛 from parameters’ prior distribution Prior Likelihood Effective Sample Size
𝐵𝑒𝑡𝑎(𝑎, 𝑏)
Binomial 𝑎 + 𝑏
𝐺𝑎𝑚𝑚𝑎(𝑎, 𝑏)
Exponential 𝑎 𝐺𝑎𝑚𝑚𝑎(𝑎, 𝑏)
Poisson 𝑁𝑜𝑟𝑚𝑎𝑙(𝑎, 𝑏)
Mean in Normal with known variance 𝜎 𝜎 𝑏 𝐼𝑛𝑣𝑒𝑟𝑠𝑒𝐺𝑎𝑚𝑚𝑎(𝑎, 𝑏)
Variance in Normal with known mean 𝑎 This method uses a Gaussian approximation to compute 𝑛 based on a summary statistic 𝑊(𝑿 𝑠 ) . Specifically, the data must be simulated and summarised following the method described in the Specification of data generating distribution section. We can assume that this summary statistic is approximately Gaussian and that the prior for the parameter of interest is also approximately Gaussian. In this setting,
𝑉𝑎𝑟 (𝑊(𝐗 𝑠 )) = 𝜎 𝑛 + 𝜎 𝑛 , where 𝜎 is an unknown variance. However, we know that 𝑉𝑎𝑟 (𝜙) =
𝑉𝑎𝑟 (𝐗)𝑛 so we can solve for 𝑛 ; 𝑛̂ = 𝑛 ( 𝑉𝑎𝑟 (𝑊(𝐗 𝑠 )) 𝑉𝑎𝑟 (𝜙) − 1),
9 where
𝑉𝑎𝑟 (𝑊(𝐗 𝑠 )) and 𝑉𝑎𝑟 (𝜙) are the variances of the summary statistic and the prior, respectively. Therefore, if we have calculated the summary statistic by summarising a dataset of size 𝑛 , we can compute the 𝑛 using this formula. However, in these cases, we need to compute a summary statistic that is “on the same scale” as the parameter. For example, the prior in a beta-binomial is typically a probability between zero and 1, while the binomial likelihood is generally the distribution of the number of successes. Therefore, the summary statistic must convert the data into the proportion of successes, which can be easily achieved by dividing the number of successes by the total sample. In some setting this may be non-trivial. For these settings, we propose an indirect Markov chain Monte Carlo (MCMC) approach. In this case, we calculate the posterior mean for the parameter of interest 𝝓 for each simulated value of 𝑿 𝑠 , denoted 𝜇 𝛟𝑠 . The prior effective sample size can then be estimated from the variance of 𝜇 𝛟𝑠 using the following formula: 𝑛̂ = 𝑛 ( 𝑉𝑎𝑟 (𝜙) 𝑉𝑎𝑟 (𝜇 𝛟 ) − 1). More information about calculating 𝑛 can be found in [1] alongside code to estimate 𝑛 within R.3 Supplementary References