Andrew J. Vickers

Assessing the performance of prediction models: a framework for traditional and novel measures.

The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic [ROC] curve), and goodness-of-fit statistics for calibration. Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions. We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation). We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

Statistics Notes: Analysing controlled trials with baseline and follow up measurements

In many randomised trials researchers measure a continuous variable at baseline and again as an outcome assessed at follow up. Baseline measurements are common in trials of chronic conditions where researchers want to see whether a treatment can reduce pre-existing levels of pain, anxiety, hypertension, and the like. Statistical comparisons in such trials can be made in several ways. Comparison of follow up (post-treatment) scores will give a result such as “at the end of the trial, mean pain scores were 15 mm (95% confidence interval 10 to 20 mm) lower in the treatment group.” Alternatively a change score can be calculated by subtracting the follow up score from the baseline score, leading to a statement such as “pain reductions were 20 mm (16 to 24 mm) greater on treatment than control.” If the average baseline scores are the same in each group the estimated treatment effect will be the same using these two simple approaches. If the treatment is effective the statistical significance of the treatment effect by the two methods will depend on the correlation between baseline and follow up scores. If the correlation is low using the change score will …

Chronic kidney disease after nephrectomy in patients with renal cortical tumours: a retrospective cohort study

BACKGROUND Chronic kidney disease is a graded and independent risk factor for substantial comorbidity and death. We aimed to examine new onset of chronic kidney disease in patients with small, renal cortical tumours undergoing radical or partial nephrectomy. METHODS We did a retrospective cohort study of 662 patients with a normal concentration of serum creatinine and two healthy kidneys undergoing elective partial or radical nephrectomy for a solitary, renal cortical tumour (</=4 cm) between 1989 and 2005 at a referral cancer centre. Glomerular filtration rate (GFR) was estimated with the abbreviated Modification in Diet and Renal Disease Study equation. Separate analysis was undertaken, with chronic kidney disease defined as GFR lower than 60 mL/min per 1.73 m(2) and GFR lower than 45 mL/min per 1.73 m(2). FINDINGS 171 (26%) patients had pre-existing chronic kidney disease before surgery. After surgery, the 3-year probability of freedom from new onset of GFR lower than 60 mL/min per 1.73 m(2) was 80% (95% CI 73-85) after partial nephrectomy and 35% (28-43; p<0.0001) after radical nephrectomy; corresponding values for GFRs lower than 45 mL/min per 1.73 m(2) were 95% (91-98) and 64% (56-70; p<0.0001), respectively. Multivariable analysis showed that radical nephrectomy remained an independent risk factor for patients developing new onset of GFR lower than 60 mL/min per 1.73 m(2) (hazard ratio 3.82 [95% CI 2.75-5.32]) and 45 mL/min per 1.73 m(2) (11.8 [6.24-22.4]; both p<0.0001). INTERPRETATION Because the baseline kidney function of patients with renal cortical tumours is lower than previously thought, accurate assessment of kidney function is essential before surgery. Radical nephrectomy is a significant risk factor for the development of chronic kidney disease and might no longer be regarded as the gold standard treatment for small, renal cortical tumours.

Decision Curve Analysis: A Novel Method for Evaluating Prediction Models

Background. Diagnostic and prognostic models are typically evaluated with measures of accuracy that do not address clinical consequences. Decision-analytic techniques allow assessment of clinical outcomes but often require collection of additional information and may be cumbersome to apply to models that yield a continuous result. The authors sought a method for evaluating and comparing prediction models that incorporates clinical consequences, requires only the data set on which the models are tested, and can be applied to models that have either continuous or dichotomous results. Method. The authors describe decision curve analysis, a simple, novel method of evaluating predictive models. They start by assuming that the threshold probability of a disease or event at which a patient would opt for treatment is informative of how the patient weighs the relative harms of a false-positive and a false-negative prediction. This theoretical relationship is then used to derive the net benefit of the model across different threshold probabilities. Plotting net benefit against threshold probability yields the “decision curve.” The authors apply the method to models for the prediction of seminal vesicle invasion in prostate cancer patients. Decision curve analysis identified the range of threshold probabilities in which a model was of value, the magnitude of benefit, and which of several models was optimal. Conclusion. Decision curve analysis is a suitable method for evaluating alternative diagnostic and prognostic strategies that has advantages over other commonly used measures and techniques.

Empirical Study of Data Sharing by Authors Publishing in PLoS Journals

Background Many journals now require authors share their data with other investigators, either by depositing the data in a public repository or making it freely available upon request. These policies are explicit, but remain largely untested. We sought to determine how well authors comply with such policies by requesting data from authors who had published in one of two journals with clear data sharing policies. Methods and Findings We requested data from ten investigators who had published in either PLoS Medicine or PLoS Clinical Trials. All responses were carefully documented. In the event that we were refused data, we reminded authors of the journals data sharing guidelines. If we did not receive a response to our initial request, a second request was made. Following the ten requests for raw data, three investigators did not respond, four authors responded and refused to share their data, two email addresses were no longer valid, and one author requested further details. A reminder of PLoSs explicit requirement that authors share data did not change the reply from the four authors who initially refused. Only one author sent an original data set. Conclusions We received only one of ten raw data sets requested. This suggests that journal policies requiring data sharing do not lead to authors making their data sets available to independent investigators.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration.

In medicine, numerous decisions are made by care providers, often in shared decision making, on the basis of an estimated probability that a specific disease or condition is present (diagnostic setting) or a specific event will occur in the future (prognostic setting) in an individual. In the diagnostic setting, the probability that a particular disease is present can be used, for example, to inform the referral of patients for further testing, to initiate treatment directly, or to reassure patients that a serious cause for their symptoms is unlikely. In the prognostic context, predictions can be used for planning lifestyle or therapeutic decisions on the basis of the risk for developing a particular outcome or state of health within a specific period (13). Such estimates of risk can also be used to risk-stratify participants in therapeutic intervention trials (47). In both the diagnostic and prognostic setting, probability estimates are commonly based on combining information from multiple predictors observed or measured from an individual (1, 2, 810). Information from a single predictor is often insufficient to provide reliable estimates of diagnostic or prognostic probabilities or risks (8, 11). In virtually all medical domains, diagnostic and prognostic multivariable (risk) prediction models are being developed, validated, updated, and implemented with the aim to assist doctors and individuals in estimating probabilities and potentially influence their decision making. A multivariable prediction model is a mathematical equation that relates multiple predictors for a particular individual to the probability of or risk for the presence (diagnosis) or future occurrence (prognosis) of a particular outcome (10, 12). Other names for a prediction model include risk prediction model, predictive model, prognostic (or prediction) index or rule, and risk score (9). Predictors are also referred to as covariates, risk indicators, prognostic factors, determinants, test results, ormore statisticallyindependent variables. They may range from demographic characteristics (for example, age and sex), medical historytaking, and physical examination results to results from imaging, electrophysiology, blood and urine measurements, pathologic examinations, and disease stages or characteristics, or results from genomics, proteomics, transcriptomics, pharmacogenomics, metabolomics, and other new biological measurement platforms that continuously emerge. Diagnostic and Prognostic Prediction Models Multivariable prediction models fall into 2 broad categories: diagnostic and prognostic prediction models (Box A). In a diagnostic model, multiplethat is, 2 or morepredictors (often referred to as diagnostic test results) are combined to estimate the probability that a certain condition or disease is present (or absent) at the moment of prediction (Box B). They are developed from and to be used for individuals suspected of having that condition. Box A. Schematic representation of diagnostic and prognostic prediction modeling studies. The nature of the prediction in diagnosis is estimating the probability that a specific outcome or disease is present (or absent) within an individual, at this point in timethat is, the moment of prediction (T= 0). In prognosis, the prediction is about whether an individual will experience a specific event or outcome within a certain time period. In other words, in diagnostic prediction the interest is in principle a cross-sectional relationship, whereas prognostic prediction involves a longitudinal relationship. Nevertheless, in diagnostic modeling studies, for logistical reasons, a time window between predictor (index test) measurement and the reference standard is often necessary. Ideally, this interval should be as short as possible without starting any treatment within this period. Box B. Similarities and differences between diagnostic and prognostic prediction models. In a prognostic model, multiple predictors are combined to estimate the probability of a particular outcome or event (for example, mortality, disease recurrence, complication, or therapy response) occurring in a certain period in the future. This period may range from hours (for example, predicting postoperative complications [13]) to weeks or months (for example, predicting 30-day mortality after cardiac surgery [14]) or years (for example, predicting the 5-year risk for developing type 2 diabetes [15]). Prognostic models are developed and are to be used in individuals at risk for developing that outcome. They may be models for either ill or healthy individuals. For example, prognostic models include models to predict recurrence, complications, or death in a certain period after being diagnosed with a particular disease. But they may also include models for predicting the occurrence of an outcome in a certain period in individuals without a specific disease: for example, models to predict the risk for developing type 2 diabetes (16) or cardiovascular events in middle-aged nondiseased individuals (17), or the risk for preeclampsia in pregnant women (18). We thus use prognostic in the broad sense, referring to the prediction of an outcome in the future in individuals at risk for that outcome, rather than the narrower definition of predicting the course of patients who have a particular disease with or without treatment (1). The main difference between a diagnostic and prognostic prediction model is the concept of time. Diagnostic modeling studies are usually cross-sectional, whereas prognostic modeling studies are usually longitudinal. In this document, we refer to both diagnostic and prognostic prediction models as prediction models, highlighting issues that are specific to either type of model. Development, Validation, and Updating of Prediction Models Prediction model studies may address the development of a new prediction model (10), a model evaluation (often referred to as model validation) with or without updating of the model [1921]), or a combination of these (Box C and Figure 1). Box C. Types of prediction model studies. Figure 1. Types of prediction model studies covered by the TRIPOD statement. D = development data; V = validation data. Model development studies aim to derive a prediction model by selecting predictors and combining them into a multivariable model. Logistic regression is commonly used for cross-sectional (diagnostic) and short-term (for example 30-day mortality) prognostic outcomes and Cox regression for long-term (for example, 10-year risk) prognostic outcomes. Studies may also focus on quantifying the incremental or added predictive value of a specific predictor (for example, newly discovered) (22) to a prediction model. Quantifying the predictive ability of a model on the same data from which the model was developed (often referred to as apparent performance [Figure 1]) will tend to give an optimistic estimate of performance, owing to overfitting (too few outcome events relative to the number of candidate predictors) and the use of predictor selection strategies (2325). Studies developing new prediction models should therefore always include some form of internal validation to quantify any optimism in the predictive performance (for example, calibration and discrimination) of the developed model and adjust the model for overfitting. Internal validation techniques use only the original study sample and include such methods as bootstrapping or cross-validation. Internal validation is a necessary part of model development (2). After developing a prediction model, it is strongly recommended to evaluate the performance of the model in other participant data than was used for the model development. External validation (Box C and Figure 1) (20, 26) requires that for each individual in the new participant data set, outcome predictions are made using the original model (that is, the published model or regression formula) and compared with the observed outcomes. External validation may use participant data collected by the same investigators, typically using the same predictor and outcome definitions and measurements, but sampled from a later period (temporal or narrow validation); by other investigators in another hospital or country (though disappointingly rare [27]), sometimes using different definitions and measurements (geographic or broad validation); in similar participants, but from an intentionally different setting (for example, a model developed in secondary care and assessed in similar participants, but selected from primary care); or even in other types of participants (for example, model developed in adults and assessed in children, or developed for predicting fatal events and assessed for predicting nonfatal events) (19, 20, 26, 2830). In case of poor performance (for example, systematic miscalibration), when evaluated in an external validation data set, the model can be updated or adjusted (for example, recalibrating or adding a new predictor) on the basis of the validation data set (Box C) (2, 20, 21, 31). Randomly splitting a single data set into model development and model validation data sets is frequently done to develop and validate a prediction model; this is often, yet erroneously, believed to be a form of external validation. However, this approach is a weak and inefficient form of internal validation, because not all available data are used to develop the model (23, 32). If the available development data set is sufficiently large, splitting by time and developing a model using data from one period and evaluating its performance using the data from the other period (temporal validation) is a stronger approach. With a single data set, temporal splitting and model validation can be considered intermediate between internal and external validation. Incomplete and Inaccurate Reporting Prediction models are becoming increasingly abundant in the medical literature (9, 33, 34), and policymakers are incre

Prostate-specific antigen and prostate cancer: prediction, detection and monitoring

Testing for prostate-specific antigen (PSA) has profoundly affected the diagnosis and treatment of prostate cancer. PSA testing has enabled physicians to detect prostate tumours while they are still small, low-grade and localized. This very ability has, however, created controversy over whether we are now diagnosing and treating insignificant cancers. PSA testing has also transformed the monitoring of treatment response and detection of disease recurrence. Much current research is directed at establishing the most appropriate uses of PSA testing and at developing methods to improve on the conventional PSA test.

Prostate Cancer–Specific Mortality After Radical Prostatectomy for Patients Treated in the Prostate-Specific Antigen Era

PURPOSE The long-term risk of prostate cancer-specific mortality (PCSM) after radical prostatectomy is poorly defined for patients treated in the era of widespread prostate-specific antigen (PSA) screening. Models that predict the risk of PCSM are needed for patient counseling and clinical trial design. METHODS A multi-institutional cohort of 12,677 patients treated with radical prostatectomy between 1987 and 2005 was analyzed for the risk of PCSM. Patient clinical information and treatment outcome was modeled using Fine and Gray competing risk regression analysis to predict PCSM. RESULTS Fifteen-year PCSM and all-cause mortality were 12% and 38%, respectively. The estimated PCSM ranged from 5% to 38% for patients in the lowest and highest quartiles of predicted risk of PSA-defined recurrence, based on a popular nomogram. Biopsy Gleason grade, PSA, and year of surgery were associated with PCSM. A nomogram predicting the 15-year risk of PCSM was developed, and the externally validated concordance index was 0.82. Neither preoperative PSA velocity nor body mass index improved the models accuracy. Only 4% of contemporary patients had a predicted 15-year PCSM of greater than 5%. CONCLUSION Few patients will die from prostate cancer within 15 years of radical prostatectomy, despite the presence of adverse clinical features. This favorable prognosis may be related to the effectiveness of radical prostatectomy (with or without secondary therapy) or the low lethality of screen-detected cancers. Given the limited ability to identify contemporary patients at substantially elevated risk of PCSM on the basis of clinical features alone, the need for novel markers specifically associated with the biology of lethal prostate cancer is evident.

A Contemporary Prostate Cancer Grading System: A Validated Alternative to the Gleason Score.

BACKGROUND Despite revisions in 2005 and 2014, the Gleason prostate cancer (PCa) grading system still has major deficiencies. Combining of Gleason scores into a three-tiered grouping (6, 7, 8-10) is used most frequently for prognostic and therapeutic purposes. The lowest score, assigned 6, may be misunderstood as a cancer in the middle of the grading scale, and 3+4=7 and 4+3=7 are often considered the same prognostic group. OBJECTIVE To verify that a new grading system accurately produces a smaller number of grades with the most significant prognostic differences, using multi-institutional and multimodal therapy data. DESIGN, SETTING, AND PARTICIPANTS Between 2005 and 2014, 20,845 consecutive men were treated by radical prostatectomy at five academic institutions; 5501 men were treated with radiotherapy at two academic institutions. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS Outcome was based on biochemical recurrence (BCR). The log-rank test assessed univariable differences in BCR by Gleason score. Separate univariable and multivariable Cox proportional hazards used four possible categorizations of Gleason scores. RESULTS AND LIMITATIONS In the surgery cohort, we found large differences in recurrence rates between both Gleason 3+4 versus 4+3 and Gleason 8 versus 9. The hazard ratios relative to Gleason score 6 were 1.9, 5.1, 8.0, and 11.7 for Gleason scores 3+4, 4+3, 8, and 9-10, respectively. These differences were attenuated in the radiotherapy cohort as a whole due to increased adjuvant or neoadjuvant hormones for patients with high-grade disease but were clearly seen in patients undergoing radiotherapy only. A five-grade group system had the highest prognostic discrimination for all cohorts on both univariable and multivariable analysis. The major limitation was the unavoidable use of prostate-specific antigen BCR as an end point as opposed to cancer-related death. CONCLUSIONS The new PCa grading system has these benefits: more accurate grade stratification than current systems, simplified grading system of five grades, and lowest grade is 1, as opposed to 6, with the potential to reduce overtreatment of PCa. PATIENT SUMMARY We looked at outcomes for prostate cancer (PCa) treated with radical prostatectomy or radiation therapy and validated a new grading system with more accurate grade stratification than current systems, including a simplified grading system of five grades and a lowest grade is 1, as opposed to 6, with the potential to reduce overtreatment of PCa.

Increasing value and reducing waste: addressing inaccessible research

The methods and results of health research are documented in study protocols, full study reports (detailing all analyses), journal reports, and participant-level datasets. However, protocols, full study reports, and participant-level datasets are rarely available, and journal reports are available for only half of all studies and are plagued by selective reporting of methods and results. Furthermore, information provided in study protocols and reports varies in quality and is often incomplete. When full information about studies is inaccessible, billions of dollars in investment are wasted, bias is introduced, and research and care of patients are detrimentally affected. To help to improve this situation at a systemic level, three main actions are warranted. First, academic institutions and funders should reward investigators who fully disseminate their research protocols, reports, and participant-level datasets. Second, standards for the content of protocols and full study reports and for data sharing practices should be rigorously developed and adopted for all types of health research. Finally, journals, funders, sponsors, research ethics committees, regulators, and legislators should endorse and enforce policies supporting study registration and wide availability of journal reports, full study reports, and participant-level datasets.