Laura Bojke

Good Practice Guidelines for Decision-Analytic Modelling in Health Technology Assessment A Review and Consolidation of Quality Assessment

The use of decision-analytic modelling for the purpose of health technology assessment (HTA) has increased dramatically in recent years. Several guidelines for best practice have emerged in the literature; however, there is no agreed standard for what constitutes a ‘good model’ or how models should be formally assessed. The objective of this paper is to identify, review and consolidate existing guidelines on the use of decision-analytic modelling for the purpose of HTA and to develop a consistent framework against which the quality of models may be assessed.The review and resultant framework are summarised under the three key themes of Structure, Data and Consistency. ‘Structural’ aspects relate to the scope and mathematical structure of the model including the strategies under evaluation. Issues covered under the general heading of ‘Data’ include data identification methods and how uncertainty should be addressed. ‘Consistency’ relates to the overall quality of the model.The review of existing guidelines showed that although authors may provide a consistent message regarding some aspects of modelling, such as the need for transparency, they are contradictory in other areas. Particular areas of disagreement are how data should be incorporated into models and how uncertainty should be assessed.For the purpose of evaluation, the resultant framework is applied to a decision-analytic model developed as part of an appraisal for the National Institute for Health and Clinical Excellence (NICE) in the UK. As a further assessment, the review based on the framework is compared with an assessment provided by an independent experienced modeller not using the framework.It is hoped that the framework developed here may form part of the appraisals process for assessment bodies such as NICE and decision models submitted to peer review journals. However, given the speed with which decision-modelling methodology advances, there is a need for its continual update.

The Internet for weight control in an obese sample: results of a randomised controlled trial

BackgroundRising levels of obesity coupled with the limited success of currently available weight control methods highlight the need for investigation of novel approaches to obesity treatment. This study aims to determine the effectiveness and cost-effectiveness of an Internet-based resource for obesity management.MethodsA randomised controlled trial conducted in a community setting, where obese volunteers (n = 221) were randomly assigned to Internet group (n = 111) or usual care group (n = 110). Objective measures of weight and height were obtained. Questionnaires were used to collect dietary, lifestyle, physical activity and quality of life data. Data were collected at baseline, six months and 12 months.ResultsData were collected on 54 (49%) participants in the Internet group and 77 (70%) participants in the usual care group at 12 months. Based on analysis conducted on all available data, the Internet group lost 1.3 kg, compared with 1.9 kg weight loss in the usual care group at 12 months, a non-significant difference (difference = 0.6 kg; 95% CI: -1.4 to 2.5, p = 0.56). No significant differences in change in secondary outcome measures between the two groups at six or 12 months were revealed. Total costs per person per year were higher in the Internet group than the usual care group (£992.40 compared to £276.12), primarily due to the fixed costs associated with setting up the website, and QALYs were similar (0.78 and 0.77) for both groups.ConclusionThis trial failed to show any additional benefit of this website in terms of weight loss or secondary outcome measures compared with usual care. High attrition and low compliance limits the results of this research. The results suggest that the Internet-based weight control resource was not a cost-effective tool for weight loss in the obese sample studied.Trail RegistrationISRCTN 58621669

Characterizing structural uncertainty in decision analytic models: a review and application of methods.

BACKGROUND The characterization of uncertainty is critical in cost-effectiveness analysis, particularly when considering whether additional evidence is needed. In addition to parameter and methodological uncertainty, there are other sources of uncertainty which include simplifications and scientific judgments that have to be made when constructing and interpreting a model of any sort. These have been classified in a number of different ways but can be referred to collectively as structural uncertainties. MATERIALS AND METHODS Separate reviews were undertaken to identify what forms these other sources of uncertainty take and what other forms of potential methods to explicitly characterize these types of uncertainties in decision analytic models. These methods were demonstrated through application to four decision models each representing one of the four types of uncertainty. RESULTS These sources of uncertainty fall into four general themes: 1) inclusion of relevant comparators; 2) inclusion of relevant events; 3) alternative statistical estimation methods; and 4) clinical uncertainty.Two methods to explicitly characterize such uncertainties were identified: model selection and model averaging. In addition, an alternative approach, adding uncertain parameters to represent the source of uncertainty was also considered.The applications demonstrate that cost-effectiveness may be sensitive to these uncertainties and the methods used to characterize them. The value of research was particularly sensitive to these uncertainties and the methods used to characterize it. It is therefore important, for decision-making purposes, to incorporate such uncertainties into the modeling process. CONCLUSION Only parameterizing the uncertainty directly in the model can inform the decision to conduct further research to resolve this source of uncertainty.

Preventive strategies for group B streptococcal and other bacterial infections in early infancy: cost effectiveness and value of information analyses

Objective To determine the cost effectiveness of strategies for preventing neonatal infection with group B streptococci and other bacteria in the UK and the value of further information from research. Design Use of a decision model to compare the cost effectiveness of prenatal testing for group B streptococcal infection (by polymerase chain reaction or culture), prepartum antibiotic treatment (intravenous penicillin or oral erythromycin), and vaccination during pregnancy (not yet available) for serious bacterial infection in early infancy across 12 maternal risk groups. Model parameters were estimated using multi-parameter evidence synthesis to incorporate all relevant data inputs. Data sources 32 systematic reviews were conducted: 14 integrated results from published studies, 24 involved analyses of primary datasets, and five included expert opinion. Main outcomes measures Healthcare costs per quality adjusted life year (QALY) gained. Results Current best practice (to treat only high risk women without prior testing for infection) and universal testing by culture or polymerase chain reaction were not cost effective options. Immediate extension of current best practice to treat all women with preterm and high risk term deliveries without testing (11% treated) would result in substantial net benefits. Currently, addition of culture testing for low risk term women, while treating all preterm and high risk term women, would be the most cost effective option (21% treated). If available in the future, vaccination combined with treating all preterm and high risk term women and no testing for low risk women would probably be marginally more cost effective and would limit antibiotic exposure to 11% of women. The value of information is highest (�67m) if vaccination is included as an option. Conclusions Extension of current best practice to treat all women with preterm and high risk term deliveries is readily achievable and would be beneficial. The choice between adding culture testing for low risk women or vaccination for all should be informed by further research. Trials to evaluate vaccine efficacy should be prioritised.

A Framework for Addressing Structural Uncertainty in Decision Models

Decision analytic models used for health technology assessment are subject to uncertainties. These uncertainties can be quantified probabilistically, by placing distributions on model parameters and simulating from these to generate estimates of cost-effectiveness. However, many uncertain model choices, often termed structural assumptions, are usually only explored informally by presenting estimates of cost-effectiveness under alternative scenarios. The authors show how 2 recent research proposals represent parts of a framework to formally account for all common structural uncertainties. First, the model is expanded to include parameters that encompass all possible structural choices. Uncertainty can then arise because these parameters are estimated imprecisely from data, for example, a treatment effect of doubtful significance. Uncertainty can also arise if there are no relevant data. If there are relevant data, uncertainty can be addressed by averaging expected costs and effects generated from probabilistic analysis of the models with and without the parameter. The weights used for averaging are related to the predictive ability of each model, assessed against the data. If there are no data, additional parameters can often be informed by eliciting expert beliefs as probability distributions. These ideas are illustrated in decision models for antiplatelet therapies for vascular disease and new biologic drugs for the treatment of active psoriatic arthritis.

Laparoscopic fundoplication compared with medical management for gastro-oesophageal reflux disease: cost effectiveness study

Objective To describe the long term costs, health benefits, and cost effectiveness of laparoscopic surgery compared with those of continued medical management for patients with gastro-oesophageal reflux disease (GORD). Design We estimated resource use and costs for the first year on the basis of data from the REFLUX trial. A Markov model was used to extrapolate cost and health benefit over a lifetime using data collected in the REFLUX trial and other sources. Participants The model compared laparoscopic surgery and continued proton pump inhibitors in male patients aged 45 and stable on GORD medication. Intervention Laparoscopic surgery versus continued medical management. Main outcome measures We estimated quality adjusted life years and GORD related costs to the health service over a lifetime. Sensitivity analyses considered other plausible scenarios, in particular size and duration of treatment effect and the GORD symptoms of patients in whom surgery is unsuccessful. Main results The base case model indicated that surgery is likely to be considered cost effective on average with an incremental cost effectiveness ratio of £2648 (€3110; US

Golimumab for the Treatment of Psoriatic Arthritis: A NICE Single Technology Appraisal.

4385) per quality adjusted life year and that the probability that surgery is cost effective is 0.94 at a threshold incremental cost effectiveness ratio of £20 000. The results were sensitive to some assumptions within the extrapolation modelling. Conclusion Surgery seems to be more cost effective on average than medical management in many of the scenarios examined in this study. Surgery might not be cost effective if the treatment effect does not persist over the long term, if patients who return to medical management have poor health related quality of life, or if proton pump inhibitors were cheaper. Further follow-up of patients from the REFLUX trial may be valuable. Trial registration ISRCTN15517081.

Methods to elicit experts' beliefs over uncertain quantities: application to a cost effectiveness transition model of negative pressure wound therapy for severe pressure ulceration.

The National Institute for Health and Clinical Excellence (NICE) invited the manufacturer of golimumab (Schering-Plough/Centocor) to submit evidence for the clinical and cost effectiveness of this drug for the treatment of active and progressive psoriatic arthritis (PsA) in patients who have responded inadequately to previous disease-modifying anti-rheumatic drugs (DMARDs). The Centre for Reviews and Dissemination and the Centre for Health Economics at the University of York were commissioned to act as the Evidence Review Group (ERG) to critically appraise the evidence presented by the manufacturer. This article provides a description of the company submission, the ERG review and the resulting NICE guidance. The ERG critically reviewed the evidence presented in the manufacturer’s submission and identified areas requiring clarification, for which the manufacturer provided additional evidence.The main clinical effectiveness data were derived from a single phase III randomized controlled trial (GO-REVEAL) that compared golimumab with placebo for the treatment of active and progressive patients who were symptomatic despite the use of previous DMARDs or NSAIDs. The 14-week data showed that, compared with placebo, golimumab 50 mg significantly improved joint disease response as measured by American College of Rheumatology (ACR) 20 (relative risk [RR] 5.73, 95%CI 3.24, 10.56) and Psoriatic Arthritis Response Criteria (PsARC) [RR 3.45, 95% CI 2.49, 4.87], and significantly improved skin disease response as measured by Psoriasis Area and Severity Index (PASI) 75 (RR 15.95, 95% CI 4.62, 59.11). The 24-week absolute data showed that these treatment benefits were maintained. There was a significant improvement in patients’ functional status as measured by Health Assessment Questionnaire change from baseline at 24 weeks (−0.33; p < 0.001). The open-label extension data showed that these beneficial effects were also maintained at 52 and 104 weeks. The ERG identified several issues relating to the clinical effectiveness results. Analyses of the 24-week data were less robust, failing to adjust for treatment contamination due to patient crossover at week 16. It was also unclear if these results were generalizable to clinical practice.No randomized controlled trial compared the effectiveness of different biologic therapies head-to-head. To compare the effectiveness of the biologics etanercept, infliximab, adalimumab and golimumab, the manufacturer conducted a network meta-analysis, including the comparator palliative care (usual care including use of NSAIDs or DMARDs). The ERG considered the assumption of exchangeability between the trials for the purpose of the network meta-analysis to be acceptable and the statistical approach to be reliable. The results indicated somewhat lower efficacy with golimumab than with comparator biologics.The ERG identified a number of issues relating to the cost-effectiveness results. The manufacturer calculated incremental cost-effectiveness ratios (ICERs) incorrectly by comparing golimumab with palliative care instead of the most cost-effective alternative (etanercept). Despite the manufacturer’s claim that golimumab was a cost-effective treatment option, the manufacturer’s own model showed that golimumab was unlikely to be cost effective, relative to currently accepted thresholds, when the ICERs were correctly calculated using an incremental analysis (i.e. comparing each treatment to the next best alternative). None of the sensitivity analyses carried out by the manufacturer or the ERG regarding uncertainty in the estimates of clinical effectiveness, the acquisition and administration cost of drugs, the cost of treating psoriasis and the utility functions estimated to generate health outcomes changed this conclusion.However, a key area in determining the cost effectiveness of biologics was whether they should be treated as a class. The ERG concluded that if all biologics were considered equally effective, then etanercept, adalimumab and golimumab had almost equal costs and equal QALYs, and all had an ICER of about £15 000 per QALY versus palliative care, whilst infliximab, with a higher acquisition cost, was dominated by the other biologics.The Appraisal Committee altered its position between the Appraisal Consultation Document and the Final Appraisal Determination. It ultimately recommended that golimumab be provided as an option for the treatment of active and progressive PsA in adults only if (i) it is used as described for other tumour necrosis factor inhibitor treatments in ‘Etanercept, infliximab and adalimumab for the treatment of psoriatic arthritis’ (NICE clinical guideline 199); and (ii) the manufacturer provides the 100mg dose of golimumab at the same cost as the 50mg dose.

Identifying research priorities: the value of information associated with repeat screening for age-related macular degeneration.

We can use decision models to estimate cost effectiveness, quantify uncertainty regarding the adoption decision and provide estimates of the value of further research. In many cases, the existence of only limited data with which to populate a decision model can mean that a cost-effectiveness analysis either does not proceed or may misrepresent the degree of uncertainty associated with model inputs. An example is the case of negative pressure wound therapy (NPWT) used to treat severe pressure ulceration, for which the evidence base is limited and sparse. There is, however, substantial practical experience of using this treatment and its comparators. We can capture this knowledge quantitatively to inform a cost-effectiveness model by eliciting beliefs from experts. This paper describes the design and conduct of an elicitation exercise to generate estimates of multiple uncertain model inputs and validate analytical assumptions for a decision model on the use of NPWT. In designing the exercise, the primary focus was the use of elicitation to inform decision models (multistate models), where representations of uncertain beliefs need to be probabilistically coherent. This paper demonstrates that it is feasible to collect formally elicited evidence to inform decision models.

Clinical and economic evaluation of laparoscopic surgery compared with medical management for gastro-oesophageal reflux disease: 5-year follow-up of multicentre randomised trial (the REFLUX trial)

The authors report an analysis that was developed as part of a pilot study examining the use of decision analysis and value-of-information methods to inform research prioritization decisions for the UK health care system. This analysis was conducted to inform decision makers whether additional research on screening for age-related macular degeneration (AMD) would be worthwhile and to demonstrate the benefits and feasibility of using such analytic methods to inform policy decision within the time-lines demanded by existing procedures. A probabilistic decision model was developed to establish the cost-effectiveness of a policy of repeat screening for AMD using the Amsler grid followed by treatment with photodynamic therapy (PDT) compared with 2 alternatives: PDT without screening (self-referral) and no screening or treatment. Screening for AMD appears to be cost-effective on the basis of existing evidence; however, the decision to implement a policy of screening is somewhat uncertain, with a probability that screening is cost-effective of 0.87 and 0.72 for the 20/40 and 20/80 models, respectively, at a threshold of £30,000 per quality-adjusted life-year. The expected value of perfect information (EVPI) associated with this decision is substantial (£6.9 million for the 20/40 model and £14.5 million for the 20/80 model), with a sizeable EVPI associated with the effect of PDT on quality of life. The analysis demonstrates that EVPI analysis can be implemented in a timely fashion to inform the type of research prioritization decisions faced by any health care system. This case study also illustrates the need to account for any structural uncertainties appropriately.