Thomas A. Gerds

Assessing the performance of prediction models: a framework for traditional and novel measures.

The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic [ROC] curve), and goodness-of-fit statistics for calibration. Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions. We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation). We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

The Performance of Risk Prediction Models

For medical decision making and patient information, predictions of future status variables play an important role. Risk prediction models can be derived with many different statistical approaches. To compare them, measures of predictive performance are derived from ROC methodology and from probability forecasting theory. These tools can be applied to assess single markers, multivariable regression models and complex model selection algorithms. This article provides a systematic review of the modern way of assessing risk prediction models. Particular attention is put on proper benchmarks and resampling techniques that are important for the interpretation of measured performance. All methods are illustrated with data from a clinical study in head and neck cancer patients.

Assessment of survival prediction models based on microarray data

MOTIVATION In the process of developing risk prediction models, various steps of model building and model selection are involved. If this process is not adequately controlled, overfitting may result in serious overoptimism leading to potentially erroneous conclusions. METHODS For right censored time-to-event data, we estimate the prediction error for assessing the performance of a risk prediction model (Gerds and Schumacher, 2006; Graf et al., 1999). Furthermore, resampling methods are used to detect overfitting and resulting overoptimism and to adjust the estimates of prediction error (Gerds and Schumacher, 2007). RESULTS We show how and to what extent the methodology can be used in situations characterized by a large number of potential predictor variables where overfitting may be expected to be overwhelming. This is illustrated by estimating the prediction error of some recently proposed techniques for fitting a multivariate Cox regression model applied to the data of a prognostic study in patients with diffuse large-B-cell lymphoma (DLBCL). AVAILABILITY Resampling-based estimation of prediction error curves is implemented in an R package called pec available from the authors.

A genome-wide association study of men with symptoms of testicular dysgenesis syndrome and its network biology interpretation

Background Testicular dysgenesis syndrome (TDS) is a common disease that links testicular germ cell cancer, cryptorchidism and some cases of hypospadias and male infertility with impaired development of the testis. The incidence of these disorders has increased over the last few decades, and testicular cancer now affects 1% of the Danish and Norwegian male population. Methods To identify genetic variants that span the four TDS phenotypes, the authors performed a genome-wide association study (GWAS) using Affymetrix Human SNP Array 6.0 to screen 488 patients with symptoms of TDS and 439 selected controls with excellent reproductive health. Furthermore, they developed a novel integrative method that combines GWAS data with other TDS-relevant data types and identified additional TDS markers. The most significant findings were replicated in an independent cohort of 671 Nordic men. Results Markers located in the region of TGFBR3 and BMP7 showed association with all TDS phenotypes in both the discovery and replication cohorts. An immunohistochemistry investigation confirmed the presence of transforming growth factor β receptor type III (TGFBR3) in peritubular and Leydig cells, in both fetal and adult testis. Single-nucleotide polymorphisms in the KITLG gene showed significant associations, but only with testicular cancer. Conclusions The association of single-nucleotide polymorphisms in the TGFBR3 and BMP7 genes, which belong to the transforming growth factor β signalling pathway, suggests a role for this pathway in the pathogenesis of TDS. Integrating data from multiple layers can highlight findings in GWAS that are biologically relevant despite having border significance at currently accepted statistical levels.

Marginal adaptation of three different zirconium dioxide three-unit fixed dental prostheses

STATEMENT OF PROBLEM Marginal adaptation is important for the long-term success of dental restorations. Data on the marginal discrepancy of zirconia-based fixed dental prostheses made with different computer-aided design/computer-aided manufacturing technology is needed. PURPOSE The purpose of this study was to evaluate the marginal adaptation of different zirconia 3-unit fixed dental prostheses at different fabrication stages and after artificial aging. MATERIAL AND METHODS Twenty-four zirconia 3-unit fixed dental prostheses (DCS, Procera, and VITA YZ-Cerec; n=8) were fabricated using different manufacturing systems and conventionally cemented with glass ionomer cement on human teeth. Each group was aged in a masticatory simulator with thermal cycling. The marginal gaps were examined on epoxy replicas for frameworks and for restorations before and after cementation, and after masticatory simulation, at x 250 magnification. Marginal adaptation was assessed using geometric means of the marginal gap values with 95% confidence intervals. Differences between the manufacturing systems and the effect of artificial aging were tested using repeated-measures ANOVA and post hoc paired and unpaired t tests with Bonferroni-Holm correction (alpha=.05). RESULTS The geometric mean (95% confidence limits) marginal gap values (mum) for frameworks and for restorations before cementation, after cementation, and after masticatory simulation were, respectively: DCS: 86 (80-93), 86 (83-90), 86 (78-94), and 84 (79-90); Procera: 82 (74-89), 89 (81-97), 89 (84-95), and 88 (82-94); and VITA YZ-Cerec: 64 (57-72), 67 (61-77), 76 (71-82), and 78 (76-80). The repeated-measures ANOVA showed significant group and stage effects (P<.05). Group VITA YZ-Cerec showed significantly smaller marginal gap values than groups DCS and Procera at framework (P<.05) and before-cementation (P<.05) stages. The VITA YZ-Cerec group showed significantly smaller marginal gap values than the Procera group after cementation (P<.05). The marginal gap values between different stages were not significantly different for all groups (P>.05). CONCLUSIONS The marginal accuracy of zirconia fixed dental prostheses is influenced by manufacturing technique.

Diagnosis of ulcerative colitis before onset of inflammation by multivariate modeling of genome-wide gene expression data

Background: Endoscopically obtained mucosal biopsies play an important role in the differential diagnosis between ulcerative colitis (UC) and Crohns disease (CD), but in some cases where neither macroscopic nor microscopic signs of inflammation are present the biopsies provide only inconclusive information. Previous studies indicate that CD cannot be diagnosed by molecular and histological diagnostic tools using colonic biopsies without microscopic signs of inflammation, but it is unknown if this is also the case for UC. Methods: The aim of the present study was to apply multivariate modeling of genome‐wide gene expression to investigate if a diagnosable preinflammatory state exists in biopsies of noninflamed UC colon, and to exploit such information to build a diagnostic tool. Results: Genome‐wide gene expression data were obtained from control subjects and UC and CD patients. In total, 89 biopsies from 78 patients were included. A diagnostic model was derived with the random forest method based on 71 biopsies from 60 patients. The model‐internal out‐of‐bag performance measure yielded perfect classification. Furthermore, the model was validated in independent 18 noninflamed biopsies from 18 patients (7 UC, 7 CD, 4 control) where the model achieved 100% sensitivity (95% confidence limits: 60.0–100) and 100% specificity (95% confidence limits: 71.5–100). Conclusions: The present study demonstrates a preinflammatory state in patients diagnosed with UC. In addition, we demonstrate the usefulness of random forest modeling of genome‐wide gene expression data for distinguishing quiescent and active UC colonic mucosa versus control and CD colonic mucosa.

Antidepressant Use and Risk of Out-of-Hospital Cardiac Arrest: A Nationwide Case–Time–Control Study

Treatment with some types of antidepressants has been associated with sudden cardiac death. It is unknown whether the increased risk is due to a class effect or related to specific antidepressants within drug classes. All patients in Denmark with an out‐of‐hospital cardiac arrest (OHCA) were identified (2001–2007). Association between treatment with specific antidepressants and OHCA was examined by conditional logistic regression in case–time–control models. We identified 19,110 patients with an OHCA; 2,913 (15.2%) were receiving antidepressant treatment at the time of OHCA, with citalopram being the most frequently used type of antidepressant (50.8%). Tricyclic antidepressants (TCAs; odds ratio (OR) = 1.69, confidence interval (CI): 1.14–2.50) and selective serotonin reuptake inhibitors (SSRIs; OR = 1.21, CI: 1.00–1.47) were both associated with comparable increases in risk of OHCA, whereas no association was found for serotonin–norepinephrine reuptake inhibitors/noradrenergic and specific serotonergic antidepressants (SNRIs/NaSSAs; OR = 1.06, CI: 0.81–1.39). The increased risks were primarily driven by: citalopram (OR = 1.29, CI: 1.02–1.63) and nortriptyline (OR = 5.14, CI: 2.17–12.2). An association between cardiac arrest and antidepressant use could be documented in both the SSRI and TCA classes of drugs.

ERG Protein Expression in Diagnostic Specimens Is Associated with Increased Risk of Progression During Active Surveillance for Prostate Cancer

BACKGROUND Compelling biomarkers identifying prostate cancer patients with a high risk of progression during active surveillance (AS) are needed. OBJECTIVE To examine the association between ERG expression at diagnosis and the risk of progression during AS. DESIGN, SETTING, AND PARTICIPANTS This study included 265 patients followed on AS with prostate-specific antigen (PSA) measurements, clinical examinations, and 10-12 core rebiopsies from 2002 to 2012 in a prospectively maintained database. ERG immunohistochemical staining was performed on diagnostic paraffin-embedded formalin-fixed sections with a ready-to-use kit (anti-ERG, EPR3864). Men were characterised as ERG positive if a minimum of one tumour focus demonstrated ERG expression. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS Overall AS progression was defined as clinical progression: increased clinical tumour category ≥cT2b by digital rectal examination and ultrasound, and/or histopathologic progression: upgrade of Gleason score, more than three positive cores or bilateral positive cores, and/or PSA progression: PSA doubling time <3 yr. Risk of progression was analysed using multiple cause-specific Cox regression and stratified cumulative incidences (Aalen-Johansen method). Curatively intended treatment, watchful waiting, and death without progression were treated as competing events. RESULTS AND LIMITATIONS A total of 121 of 142 ERG-negative and 96 of 123 ERG-positive patients had complete diagnostic information. In competing risk models, the ERG-positive group showed significantly higher incidences of overall AS progression (p<0.0001) and of the subgroups PSA progression (p<0.0001) and histopathologic progression (p<0.0001). The 2-yr cumulative incidence of overall AS progression was 21.7% (95% confidence interval [CI], 14.3-29.1) in the ERG-negative group compared with 58.6% (95% CI, 48.7-68.5) in the ERG-positive group. ERG positivity was a significant predictor of overall AS progression in multiple Cox regression (hazard ratio: 2.45; 95% CI, 1.62-3.72; p<0.0001). The main limitation of this study is its observational nature. CONCLUSIONS In our study, ERG positivity at diagnosis can be used to estimate the risk of progression during AS. If confirmed, ERG status can be used to individualise AS programmes. PATIENT SUMMARY The tissue biomarker ERG identifies active surveillance patients with an increased risk of disease progression.

Concordance for prognostic models with competing risks

The concordance probability is a widely used measure to assess discrimination of prognostic models with binary and survival endpoints. We formally define the concordance probability for a prognostic model of the absolute risk of an event of interest in the presence of competing risks and relate it to recently proposed time-dependent area under the receiver operating characteristic curve measures. For right-censored data, we investigate inverse probability of censoring weighted (IPCW) estimates of a truncated concordance index based on a working model for the censoring distribution. We demonstrate consistency and asymptotic normality of the IPCW estimate if the working model is correctly specified and derive an explicit formula for the asymptotic variance under independent censoring. The small sample properties of the estimator are assessed in a simulation study also against misspecification of the working model. We further illustrate the methods by computing the concordance probability for a prognostic model of coronary heart disease (CHD) events in the presence of the competing risk of non-CHD death.

On pseudo-values for regression analysis in competing risks models

For regression on state and transition probabilities in multi-state models Andersen et al. (Biometrika 90:15–27, 2003) propose a technique based on jackknife pseudo-values. In this article we analyze the pseudo-values suggested for competing risks models and prove some conjectures regarding their asymptotics (Klein and Andersen, Biometrics 61:223–229, 2005). The key is a second order von Mises expansion of the Aalen-Johansen estimator which yields an appropriate representation of the pseudo-values. The method is illustrated with data from a clinical study on total joint replacement. In the application we consider for comparison the estimates obtained with the Fine and Gray approach (J Am Stat Assoc 94:496–509, 1999) and also time-dependent solutions of pseudo-value regression equations.