Michael W. Kattan

Assessing the performance of prediction models: a framework for traditional and novel measures.

The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic [ROC] curve), and goodness-of-fit statistics for calibration. Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions. We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation). We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

CANCER CONTROL WITH RADICAL PROSTATECTOMY ALONE IN 1,000 CONSECUTIVE PATIENTS

PURPOSE We analyzed the long-term progression-free probability after radical retropubic prostatectomy in a consecutive series of patients with localized prostate cancer. MATERIALS AND METHODS From 1983 to 1998, 1,000 patients (median age 62.9 years, range 37.7 to 81.4) with clinical stage T1 to T2 prostate cancer were treated with radical retropubic prostatectomy and pelvic lymphadenectomy, without other cancer related therapy before recurrence. Mean followup was 53.2 months (median 46.9, range 1 to 170). RESULTS Ten years after radical retropubic prostatectomy the mean probability +/- 2 standard errors that patients remained free of progression and of any further treatment was 75.0% +/- 3.7% and of metastasis 84.2% +/- 4.4%. Mean actuarial cancer specific survival rate +/- 2 standard error was 97.6% +/- 1.7%. In a multivariate analysis pretreatment prostate specific antigen level (p <0.0001), biopsy Gleason sum (p <0.0001) and clinical stage (p=0.0071) were independent prognostic factors for progression. After prostatectomy independent risk factors were Gleason sum in the prostatectomy specimen (p=0.0008), extracapsular extension (p=0.0019), seminal vesical involvement (p <0.0001), lymph node metastasis (p <0.0001) and surgical margin status (p <0.0001). Margins were positive in 12.8% of cases. At 10 years postoperatively radical retropubic prostatectomy was effective for cancer confined to the prostate (92.2% progression-free probability) and also not confined (52.8%), including 71.4% progression-free probability for patients with only extracapsular extension and 37.4% with seminal vesicle invasion without lymph node metastasis. CONCLUSIONS Radical retropubic prostatectomy provided long-term cancer control in 75% of patients with clinically localized prostate cancer and was effective in the majority of those with high risk cancer, including T2c or biopsy Gleason sum 8 to 10, or PSA greater than 20 ng./ml. Further research should address identifying patients who can safely avoid aggressive therapy.

Predicting the Outcome of Salvage Radiation Therapy for Recurrent Prostate Cancer After Radical Prostatectomy

PURPOSE An increasing serum prostate-specific antigen (PSA) level is the initial sign of recurrent prostate cancer among patients treated with radical prostatectomy. Salvage radiation therapy (SRT) may eradicate locally recurrent cancer, but studies to distinguish local from systemic recurrence lack adequate sensitivity and specificity. We developed a nomogram to predict the probability of cancer control at 6 years after SRT for PSA-defined recurrence. PATIENTS AND METHODS Using multivariable Cox regression analysis, we constructed a model to predict the probability of disease progression after SRT in a multi-institutional cohort of 1,540 patients. RESULTS The 6-year progression-free probability was 32% (95% CI, 28% to 35%) overall. Forty-eight percent (95% CI, 40% to 56%) of patients treated with SRT alone at PSA levels of 0.50 ng/mL or lower were disease free at 6 years, including 41% (95% CI, 31% to 51%) who also had a PSA doubling time of 10 months or less or poorly differentiated (Gleason grade 8 to 10) cancer. Significant variables in the model were PSA level before SRT (P < .001), prostatectomy Gleason grade (P < .001), PSA doubling time (P < .001), surgical margins (P < .001), androgen-deprivation therapy before or during SRT (P < .001), and lymph node metastasis (P = .019). The resultant nomogram was internally validated and had a concordance index of 0.69. CONCLUSION Nearly half of patients with recurrent prostate cancer after radical prostatectomy have a long-term PSA response to SRT when treatment is administered at the earliest sign of recurrence. The nomogram we developed predicts the outcome of SRT and should prove valuable for medical decision making for patients with a rising PSA level.

A Nomogram for Predicting the Likelihood of Additional Nodal Metastases in Breast Cancer Patients With a Positive Sentinel Node Biopsy

AbstractBackground:The standard of care for breast cancer patients with sentinel lymph node (SLN) metastases includes complete axillary lymph node dissection (ALND). However, many question the need for complete ALND in every patient with detectable SLN metastases, particularly those perceived to have a low risk of non-SLN metastases. Accurate estimates of the likelihood of additional disease in the axilla could assist greatly in decision-making regarding further treatment. Methods:Pathological features of the primary tumor and SLN metastases of 702 patients who underwent complete ALND were assessed with multivariable logistic regression to predict the presence of additional disease in the non-SLNs of these patients. A nomogram was created using pathological size, tumor type and nuclear grade, lymphovascular invasion, multifocality, and estrogen-receptor status of the primary tumor; method of detection of SLN metastases; number of positive SLNs; and number of negative SLNs. The model was subsequently applied prospectively to 373 patients. Results:The nomogram for the retrospective population was accurate and discriminating, with an area under the receiver operating characteristic (ROC) curve of 0.76. When applied to the prospective group, the model accurately predicted likelihood of non-SLN disease (ROC, 0.77). Conclusions:We have developed a user-friendly nomogram that uses information commonly available to the surgeon to easily and accurately calculate the likelihood of having additional, non-SLN metastases for an individual patient.

A POSTOPERATIVE PROGNOSTIC NOMOGRAM FOR RENAL CELL CARCINOMA

PURPOSE Few published studies have combined prognostic factors to predict the likelihood of recurrence after surgery for renal cell carcinoma. We developed a nomogram for this purpose. MATERIALS AND METHODS With Cox proportional hazards regression analysis, we modeled pathological data and disease followup for 601 patients with renal cell carcinoma who were treated with nephrectomy. Predictor variables were patient symptoms, including incidental, local or systemic, histology, including chromophobe, papillary or conventional, tumor size, and pathological stage. Treatment failure was recorded when there was either clinical evidence of disease recurrence or death from disease. Validation was performed with a statistical (bootstrapping) technique. RESULTS Disease recurrence was noted in 66 of the 601 patients, and those in whom treatment was successful had a median and maximum followup of 40 and 123 months, respectively. The 5-year probability of freedom from failure for the patient cohort was 86% (95% confidence interval 82 to 89). With statistical validation, predictions by the nomogram appeared accurate and discriminating with an area under the receiver operating characteristic curve, that is a comparison of the predicted probability with the actual outcome of 0.74. CONCLUSIONS A nomogram has been developed that can be used to predict the 5-year probability of treatment failure among patients with newly diagnosed renal cell carcinoma. The nomogram may be useful for patient counseling, clinical trial design and patient followup planning.

Postoperative Nomogram Predicting the 10-Year Probability of Prostate Cancer Recurrence After Radical Prostatectomy

PURPOSE A postoperative nomogram for prostate cancer recurrence after radical prostatectomy (RP) has been independently validated as accurate and discriminating. We have updated the nomogram by extending the predictions to 10 years after RP and have enabled the nomogram predictions to be adjusted for the disease-free interval that a patient has maintained after RP. METHODS Cox regression analysis was used to model the clinical information for 1,881 patients who underwent RP for clinically-localized prostate cancer by two high-volume surgeons. The model was externally validated separately on two independent cohorts of 1,782 patients and 1,357 patients, respectively. Disease progression was defined as a rising prostate-specific antigen (PSA) level, clinical progression, radiotherapy more than 12 months postoperatively, or initiation of systemic therapy. RESULTS The 10-year progression-free probability for the modeling set was 79% (95% CI, 75% to 82%). Significant variables in the multivariable model included PSA (P = .002), primary (P < .0001) and secondary Gleason grade (P = .0006), extracapsular extension (P < .0001), positive surgical margins (P = .028), seminal vesicle invasion (P < .0001), lymph node involvement (P = .030), treatment year (P = .008), and adjuvant radiotherapy (P = .046). The concordance index of the nomogram when applied to the independent validation sets was 0.81 and 0.79. CONCLUSION We have developed and validated as a robust predictive model an enhanced postoperative nomogram for prostate cancer recurrence after RP. Unique to predictive models, the nomogram predictions can be adjusted for the disease-free interval that a patient has achieved after RP.

Prognostic Model for Predicting Survival in Men With Hormone-Refractory Metastatic Prostate Cancer

PURPOSE To develop and validate a model that can be used to predict the overall survival probability among metastatic hormone-refractory prostate cancer patients (HRPC). PATIENTS AND METHODS Data from six Cancer and Leukemia Group B protocols that enrolled 1,101 patients with metastatic hormone-refractory adenocarcinoma of the prostate during the study period from 1991 to 2001 were pooled. The proportional hazards model was used to develop a multivariable model on the basis of pretreatment factors and to construct a prognostic model. The area under the receiver operating characteristic curve (ROC) was calculated as a measure of predictive discrimination. Calibration of the model predictions was assessed by comparing the predicted probability with the actual survival probability. An independent data set was used to validate the fitted model. RESULTS The final model included the following factors: lactate dehydrogenase, prostate-specific antigen, alkaline phosphatase, Gleason sum, Eastern Cooperative Oncology Group performance status, hemoglobin, and the presence of visceral disease. The area under the ROC curve was 0.68. Patients were classified into one of four risk groups. We observed a good agreement between the observed and predicted survival probabilities for the four risk groups. The observed median survival durations were 7.5 (95% confidence interval [CI], 6.2 to 10.9), 13.4 (95% CI, 9.7 to 26.3), 18.9 (95% CI, 16.2 to 26.3), and 27.2 (95% CI, 21.9 to 42.8) months for the first, second, third, and fourth risk groups, respectively. The corresponding median predicted survival times were 8.8, 13.4, 17.4, and 22.80 for the four risk groups. CONCLUSION This model could be used to predict individual survival probabilities and to stratify metastatic HRPC patients in randomized phase III trials.

Prognostic Significance of Positive Surgical Margins in Radical Prostatectomy Specimens

PURPOSE The prognostic significance of positive surgical margins in radical prostatectomy specimens was assessed. MATERIALS AND METHODS The interval to progression (increasing prostate specific antigen level) was measured in 478 patients by status of the surgical margins. RESULTS At 5 years, the nonprogression rate was 64% for patients with and 83% for those without positive surgical margins. With a high grade cancer, seminal vesicle invasion or lymph node metastases, positive surgical margins had no effect on prognosis; with extracapsular extension and a Gleason score of 6 or less positive surgical margins were associated with a higher progression rate. CONCLUSIONS Prognosis was adversely affected by positive surgical margins only in moderately differentiated cancers with extracapsular extension alone. If the cancer is otherwise confined, positive surgical margins are associated with an excellent prognosis unlikely to be improved by adjuvant therapy.

Risk Factors for Urinary Incontinence after Radical Prostatectomy

PURPOSE We identified risk factors associated with urinary incontinence after radical retropubic prostatectomy. MATERIALS AND METHODS The time from operation until urinary continence was achieved was determined by chart review and questionnaire in 581 patients who were continent before undergoing radical retropubic prostatectomy between 1983 and 1994. Using univariate and multivariate analyses of data gathered prospectively, we examined risk factors associated with incontinence in these patients. RESULTS The actuarial rate of urinary continence at 24 months was 91% for the entire patient population and 95% for those treated after 1990. Many factors were associated with the risk of incontinence in univariate Cox proportional hazards regression analysis (patient age and weight, degree of obstructive voiding symptoms, prior transurethral resection of the prostate, clinical stage, intraoperative blood loss, resection of neurovascular bundles, postoperative anastomotic stricture and technique of vesicourethral anastomosis). However, in a multivariate analysis the factors that were independently associated with increased chance of regaining continence were decreasing age, a modification in the technique of anastomosis (introduced in 1990), preservation of both neurovascular bundles and absence of an anastomotic stricture. With introduction of the new surgical technique in 1990 the median time to continence decreased from 5.6 to 1.5 months and the rate of continence at 24 months increased from 82 to 95%. CONCLUSIONS While the risk of urinary incontinence after radical prostatectomy is related to the uncontrollable factor of patient age, it is also sensitive to the surgical technique used.

FACTORS PREDICTING RECOVERY OF ERECTIONS AFTER RADICAL PROSTATECTOMY

PURPOSE Because preservation of functioning penile erections is a major concern for many patients considering treatment for localized prostate cancer, we analyzed various factors determined before and after radical retropubic prostatectomy to identify those significantly associated with recovery of erectile function. MATERIALS AND METHODS Our prospective database of patients undergoing pelvic lymphadenectomy and radical retropubic prostatectomy was used to determine factors predictive of erection recovery after radical prostatectomy. The study included 314 consecutive men with prostate cancer treated with radical retropubic prostatectomy between November 1993 and December 1996. Preoperative potency satisfactory for intercourse and degree of neurovascular bundle preservation during the operation were documented. RESULTS Patient age, preoperative potency status and extent of neurovascular bundle preservation but not pathological stage were predictive of potency recovery after radical prostatectomy. At 3 years after the operation 76% of men younger than age 60 years with full erections preoperatively who had bilateral neurovascular bundle preservation would be expected to regain erections sufficient for intercourse. Compared to the younger men, those 60 to 65 years old were only 56% (95% confidence interval [CI] 37 to 84) and those older than 65 years were 47% (95% CI 30 to 73) as likely to recover potency. Patients with recently diminished erections were only 63% (95% CI 38 to 100) as likely to recover potency as men with full erections preoperatively, and those with partial erections were only 47% (95% CI 23 to 96) as likely to recover potency. Resection of 1 neurovascular bundle reduced the chance of recovery to 25% (95% CI 10 to 61) compared to preserving both nerves. CONCLUSIONS Knowledge of preoperative erectile function and patient age before the operation and the degree of neurovascular bundle preservation afterward may aid in patient counseling regarding potency recovery after radical prostatectomy.