G.A.K. van Voorn

AdViSHE: A Validation-Assessment Tool of Health-Economic Models for Decision Makers and Model Users.

BackgroundA trade-off exists between building confidence in health-economic (HE) decision models and the use of scarce resources. We aimed to create a practical tool providing model users with a structured view into the validation status of HE decision models, to address this trade-off.MethodsA Delphi panel was organized, and was completed by a workshop during an international conference. The proposed tool was constructed iteratively based on comments from, and the discussion amongst, panellists. During the Delphi process, comments were solicited on the importance and feasibility of possible validation techniques for modellers, their relevance for decision makers, and the overall structure and formulation in the tool.ResultsThe panel consisted of 47 experts in HE modelling and HE decision making from various professional and international backgrounds. In addition, 50 discussants actively engaged in the discussion at the conference workshop and returned 19 questionnaires with additional comments. The final version consists of 13 items covering all relevant aspects of HE decision models: the conceptual model, the input data, the implemented software program, and the model outcomes.ConclusionsAssessment of the Validation Status of Health-Economic decision models (AdViSHE) is a validation-assessment tool in which model developers report in a systematic way both on validation efforts performed and on their outcomes. Subsequently, model users can establish whether confidence in the model is justified or whether additional validation efforts should be undertaken. In this way, AdViSHE enhances transparency of the validation status of HE models and supports efficient model validation.

Improving Model Validation in Health Technology Assessment: Comments on Guidelines of the ISPOR-SMDM Modeling Good Research Practices Task Force

__Abstract__ In 2012, a joint task force from ISPOR and SMDM produced a set of guidelines on modeling good practices for research [1]. These articles are highly influential in the field of health technology assessment, and rightly so. Therefore, we would like to discuss some concerns they raise about the definition and interpretation of an important aspect, namely, model validation.

ADVISHE: A new tool to report validation of health-economic decision models

Background: Modelers and reimbursement decision makers could both profit from a more systematic reporting of the efforts to validate health-economic (HE) models. Objectives: Development of a tool to systematically report validation efforts of HE decision models and their outcomes. Methods: A gross list of model validation techniques was collected using a literature review, including sources outside the HE field. A panel then selected the most important items. Based on the Delphi method, the panel members could score items in three e-mail rounds. Participants were HE modelling experts, covering various nationalities and work environments. They could comment on relevance, feasibility and formulation of the items and received feedback on comments from others. This resulted in a draft tool of selected items, which was tested and improved in two further rounds. In addition, the Dutch National Health Care Institute commented on usefulness for decision makers, while a separate group of 50 HE experts could comment during a workshop at ISPOR Montreal 2014. Results: 35 Validation techniques were identified and grouped into four categories: conceptual model validation, computerized model validation, data validation and operational validation. Around 30 HE experts commented in each of the first three Delphi rounds, resulting in a 15 item draft tool. The Dutch health care advisory institute suggested to add one more item. Participants from the ISPOR workshop delivered 19 filled-in questionnaires. A fourth round resulted in 17 responses. This led to a refined version containing 16 items, which is currently sent out for a final, fifth round. Conclusions: When filled out by the modellers, AdVISHE (Assessment of the ValIdation Status of Health- Economic decision models) supports model users in assessing the validation status of a model It will be useful as part of reimbursement dossiers, by providing systematic and transparent insight into the validation efforts performed and their results.

Operational validation of health economic decision analytic models

Objectives: To validate health economic (HE) models by means of statistical comparison of model outcomes against empirical observations. Such a comparison is structured and the applicability of several existing validation techniques is discussed, with a special focus on statistical testing. When standard methods (95%-confidence intervals) are used several problems, both of a technical and philosophical nature, are encountered. These problems are discussed. A new statistical approach is consequently proposed. Methods: The proposed method can be applied to validate HE models when the uncertainty around the input parameters of the model is assessed via probabilistic sensitivity analysis (PSA). It is based on the idea of establishing a level of accuracy in advance for the empirical observations and model outcomes should meet. If the model result falls within the limits determined by the pre-required accuracy, then the model result is considered valid. The number of valid results obtained in a PSA defines a measure of the reliability of the model. Embodying the method in a Bayesian framework allows defining such a reliability measure with statistical properties. Results: Existing approaches suffer from technical and interpretational problems. In addition, these methods are lacking a measure of overall reliability. Our new method (1) departs from classical statistical techniques, circumventing the noted problems, (2) can be used for both cohort and patient-level models and (3) makes use of all PSA outcomes. The method is demonstrated with the help of a case study in a published diabetes model (MICADO). Conclusions: Standard statistical techniques have to be applied very carefully on the comparison of model outcomes to empirical observations. They suffer from several problems. A new promising Bayesian approach is proposed that solves some of these issues. Our new method allows stepwise validation of the model as new data becomes available, which may increase the models validation status.