Jonathan Karnon

Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6.

A model’s purpose is to inform medical decisions and health care resource allocation. Modelers employ quantitative methods to structure the clinical, epidemiological, and economic evidence base and gain qualitative insight to assist decision makers in making better decisions. From a policy perspective, the value of a model-based analysis lies not simply in its ability to generate a precise point estimate for a specific outcome but also in the systematic examination and responsible reporting of uncertainty surrounding this outcome and the ultimate decision being addressed. Different concepts relating to uncertainty in decision modeling are explored. Stochastic (first-order) uncertainty is distinguished from both parameter (second-order) uncertainty and from heterogeneity, with structural uncertainty relating to the model itself forming another level of uncertainty to consider. The article argues that the estimation of point estimates and uncertainty in parameters is part of a single process and explores the link between parameter uncertainty through to decision uncertainty and the relationship to value-of-information analysis. The article also makes extensive recommendations around the reporting of uncertainty, both in terms of deterministic sensitivity analysis techniques and probabilistic methods. Expected value of perfect information is argued to be the most appropriate presentational technique, alongside cost-effectiveness acceptability curves, for representing decision uncertainty from probabilistic analysis.

Model Parameter Estimation and Uncertainty: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6

A models purpose is to inform medical decisions and health care resource allocation. Modelers employ quantitative methods to structure the clinical, epidemiological, and economic evidence base and gain qualitative insight to assist decision makers in making better decisions. From a policy perspective, the value of a model-based analysis lies not simply in its ability to generate a precise point estimate for a specific outcome but also in the systematic examination and responsible reporting of uncertainty surrounding this outcome and the ultimate decision being addressed. Different concepts relating to uncertainty in decision modeling are explored. Stochastic (first-order) uncertainty is distinguished from both parameter (second-order) uncertainty and from heterogeneity, with structural uncertainty relating to the model itself forming another level of uncertainty to consider. The article argues that the estimation of point estimates and uncertainty in parameters is part of a single process and explores the link between parameter uncertainty through to decision uncertainty and the relationship to value of information analysis. The article also makes extensive recommendations around the reporting of uncertainty, in terms of both deterministic sensitivity analysis techniques and probabilistic methods. Expected value of perfect information is argued to be the most appropriate presentational technique, alongside cost-effectiveness acceptability curves, for representing decision uncertainty from probabilistic analysis.

A systematic review of economic analyses of telehealth services using real time video communication

BackgroundTelehealth is the delivery of health care at a distance, using information and communication technology. The major rationales for its introduction have been to decrease costs, improve efficiency and increase access in health care delivery. This systematic review assesses the economic value of one type of telehealth delivery - synchronous or real time video communication - rather than examining a heterogeneous range of delivery modes as has been the case with previous reviews in this area.MethodsA systematic search was undertaken for economic analyses of the clinical use of telehealth, ending in June 2009. Studies with patient outcome data and a non-telehealth comparator were included. Cost analyses, non-comparative studies and those where patient satisfaction was the only health outcome were excluded.Results36 articles met the inclusion criteria. 22(61%) of the studies found telehealth to be less costly than the non-telehealth alternative, 11(31%) found greater costs and 3 (9%) gave the same or mixed results. 23 of the studies took the perspective of the health services, 12 were societal, and one was from the patient perspective. In three studies of telehealth to rural areas, the health services paid more for telehealth, but due to savings in patient travel, the societal perspective demonstrated cost savings. In regard to health outcomes, 12 (33%) of studies found improved health outcomes, 21 (58%) found outcomes were not significantly different, 2(6%) found that telehealth was less effective, and 1 (3%) found outcomes differed according to patient group. The organisational model of care was more important in determining the value of the service than the clinical discipline, the type of technology, or the date of the study.ConclusionDelivery of health services by real time video communication was cost-effective for home care and access to on-call hospital specialists, showed mixed results for rural service delivery, and was not cost-effective for local delivery of services between hospitals and primary care.

Modeling Using Discrete Event Simulation A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force–4

Discrete event simulation (DES) is a form of computer-based modeling that provides an intuitive and flexible approach to representing complex systems. It has been used in a wide range of health care applications. Most early applications involved analyses of systems with constrained resources, where the general aim was to improve the organization of delivered services. More recently, DES has increasingly been applied to evaluate specific technologies in the context of health technology assessment. The aim of this article was to provide consensus-based guidelines on the application of DES in a health care setting, covering the range of issues to which DES can be applied. The article works through the different stages of the modeling process: structural development, parameter estimation, model implementation, model analysis, and representation and reporting. For each stage, a brief description is provided, followed by consideration of issues that are of particular relevance to the application of DES in a health care setting. Each section contains a number of best practice recommendations that were iterated among the authors, as well as among the wider modeling task force.

Trends in healthcare incident reporting and relationship to safety and quality data in acute hospitals: results from the National Reporting and Learning System

Background: Internationally, there is increasing recognition of the need to collect and analyse data on patient safety incidents, to facilitate learning and develop solutions. The National Patient Safety Agency (NPSA) for England and Wales has been capturing incident data from acute hospitals since November 2003. Objectives: This study analyses patterns in reporting of patient safety incidents from all acute hospitals in England to the NPSA National Reporting and Learning System, and explores the link between reporting rates, hospital characteristics, and other safety and quality datasets. Methods: Reporting rates to the NPSA National Reporting and Learning System were analysed as trends over time, from the point at which each hospital became connected to the system. The relationships between reporting rates and other safety and quality datasets were assessed using correlation and regression analyses. Results: Reporting rates increased steadily over the 18 months analysed. Higher reporting rates correlated with positive data on safety culture and incident reporting from the NHS Staff Survey, and with better risk-management ratings from the NHS Litigation Authority. Hospitals with higher overall reporting rates had a lower proportion of their reports in the “slips, trips and falls” category, suggesting that these hospitals were reporting higher numbers of other types of incident. There was no apparent association between reporting rates and the following data: standardised mortality ratios, data from other safety-related reporting systems, hospital size, average patient age or length of stay. Conclusions: Incident reporting rates from acute hospitals increase with time from connection to the national reporting system, and are positively correlated with independently defined measures of safety culture, higher reporting rates being associated with a more positive safety culture.

Calibrating Models in Economic Evaluation: A Seven-Step Approach

In economic evaluation, mathematical models have a central role as a way of integrating all the relevant information about a disease and health interventions, in order to estimate costs and consequences over an extended time horizon. Models are based on scientific knowledge of disease (which is likely to change over time), simplifying assumptions and input parameters with different levels of uncertainty; therefore, it is sensible to explore the consistency of model predictions with observational data. Calibration is a useful tool for estimating uncertain parameters, as well as more accurately defining model uncertainty (particularly with respect to the representation of correlations between parameters). Calibration involves the comparison of model outputs (e.g. disease prevalence rates) with empirical data, leading to the identification of model parameter values that achieve a good fit.This article provides guidance on the theoretical underpinnings of different calibration methods. The calibration process is divided into seven steps and different potential methods at each step are discussed, focusing on the particular features of disease models in economic evaluation. The seven steps are (i) Which parameters should be varied in the calibration process? (ii) Which calibration targets should be used? (iii) What measure of goodness of fit should be used? (iv) What parameter search strategy should be used? (v) What determines acceptable goodness-of-fit parameter sets (convergence criteria)? (vi) What determines the termination of the calibration process (stopping rule)? (vii) How should the model calibration results and economic parameters be integrated?The lack of standards in calibrating disease models in economic evaluation can undermine the credibility of calibration methods. In order to avoid the scepticism regarding calibration, we ought to unify the way we approach the problems and report the methods used, and continue to investigate different methods.

Modeling Using Discrete Event Simulation

Discrete event simulation (DES) is a form of computer-based modeling that provides an intuitive and flexible approach to representing complex systems. It has been used in a wide range of health care applications. Most early applications involved analyses of systems with constrained resources, where the general aim was to improve the organization of delivered services. More recently, DES has increasingly been applied to evaluate specific technologies in the context of health technology assessment. The aim of this article is to provide consensus-based guidelines on the application of DES in a health care setting, covering the range of issues to which DES can be applied. The article works through the different stages of the modeling process: structural development, parameter estimation, model implementation, model analysis, and representation and reporting. For each stage, a brief description is provided, followed by consideration of issues that are of particular relevance to the application of DES in a health care setting. Each section contains a number of best practice recommendations that were iterated among the authors, as well as the wider modeling task force.

Model-based cost-effectiveness analysis of interventions aimed at preventing medication error at hospital admission (medicines reconciliation).

RATIONALE Medication errors can lead to preventable adverse drug events (pADEs) that have significant cost and health implications. Errors often occur at care interfaces, and various interventions have been devised to reduce medication errors at the point of admission to hospital. The aim of this study is to assess the incremental costs and effects [measured as quality adjusted life years (QALYs)] of a range of such interventions for which evidence of effectiveness exists. METHODS A previously published medication errors model was adapted to describe the pathway of errors occurring at admission through to the occurrence of pADEs. The baseline model was populated using literature-based values, and then calibrated to observed outputs. Evidence of effects was derived from a systematic review of interventions aimed at preventing medication error at hospital admission. RESULTS All five interventions, for which evidence of effectiveness was identified, are estimated to be extremely cost-effective when compared with the baseline scenario. Pharmacist-led reconciliation intervention has the highest expected net benefits, and a probability of being cost-effective of over 60% by a QALY value of pound10 000. CONCLUSIONS The medication errors model provides reasonably strong evidence that some form of intervention to improve medicines reconciliation is a cost-effective use of NHS resources. The variation in the reported effectiveness of the few identified studies of medication error interventions illustrates the need for extreme attention to detail in the development of interventions, but also in their evaluation and may justify the primary evaluation of more than one specification of included interventions.

Selecting a decision model for economic evaluation: a case study and review

The increased use of modelling techniques as a methodological tool in the economic evaluation of health care technologies has, in the main, been limited to two approaches – decision trees and Markov chain models. The former are suited to modelling simple scenarios that occur over a short time period, whilst Markov chain models allow longer time periods to be modelled, in continuous time, where the timing of an event is uncertain. In the context of economic evaluation, a less well developed technique is discrete event simulation, which may allow even greater flexibility.Taking the economic evaluation of adjuvant therapies for breast cancer as an illustrative example, the process of building a decision tree, a Markov chain model, and a discrete event simulation model are described. The potential benefits and problems of each approach are discussed.The suitability of the modelling techniques to economic evaluations of health care programmes in general is then discussed. This section aims to illustrate the areas in which the alternative modelling methods may be most appropriately employed.

Performance, treatment pathways, and effects of alternative policy options for screening for developmental dysplasia of the hip in the United Kingdom

Aims: To compare, using a decision model, performance, treatment pathways and effects of different newborn screening strategies for developmental hip dysplasia with no screening. Methods: Detection rate, radiological absence of subluxation at skeletal maturity and avascular necrosis of the femoral head, as favourable and unfavourable treatment outcomes respectively, were compared for the following strategies: clinical screening alone using the Ortolani and Barlow tests; the addition of static and dynamic ultrasound examination of the hips of all infants (universal ultrasound) or restricted to infants with defined risk factors (selective ultrasound); “no screening” (that is, clinical diagnosis only). Results: Universal or selective ultrasound detects more more affected children (76% and 60% respectively) than clinical screening alone (35%), results in a higher proportion of affected children with favourable treatment outcomes (92% and 88% respectively) than clinical screening alone (78%) or no screening (75%), and the highest proportion of these achieved without recourse to surgery (64% and 79% respectively) compared with clinical screening alone (18%). However, ultrasound based strategies are also associated with the highest number of unfavourable treatment outcomes arising in unaffected children treated following a false positive screening result. The detection rate of clinical screening alone becomes similar to that reported for universal ultrasound when based on studies using experienced examiners (80%) rather than junior medical staff (35%). Conclusion: From the largely observational data available, ultrasound based screening strategies appear to be most sensitive and effective but are associated with the greatest risk of potential adverse iatrogenic effects arising in unaffected children.