J. Dik F. Habbema

Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Original articleInternal validation of predictive models: Efficiency of some procedures for logistic regression analysis

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Quantification of clinical morbidity associated with schistosome infection in sub-Saharan Africa

Health policy making in developing countries requires estimates of the (global) burden of disease. At present, most of the available data on schistosomiasis is limited to numbers of individuals harbouring the infection. We explored the relationship between the presence of schistosome infection and clinical morbidity, in order to estimate numbers of individuals with disease-specific morbidity for Schistosoma haematobium and Schistosoma mansoni infection in sub-Saharan Africa. We searched the literature for cross-sectional data from field studies reporting both schistosome infection and morbidity. This was used to derive a functional relationship between morbidity and infection. After standardisation for diagnostic method, the number of individuals with specific types of clinical morbidity or pathology was predicted. As only aggregated prevalences of infection were available for countries or areas, we adjusted for heterogeneity in infection levels within communities in those countries. In total, 70 million individuals out of 682 million (2000 estimate) in sub-Saharan Africa were estimated to experience haematuria in the last 2 weeks associated with S. haematobium infection, and 32 million dysuria. Ultrasound detected serious consequences of S. haematobium, major bladder wall pathology and major hydronephrosis, were predicted at 18 and 10 million, respectively. Infection with S. mansoni was estimated to cause diarrhoea in 0.78 million individuals, blood in stool in 4.4 million and hepatomegaly in 8.5 million. As the associations between prevalence of S. mansoni infection and prevalence of diarrhoea and blood in stool were not very clear, the resulting estimates may be underestimations. Using the very limited data available, we estimated the mortality rates due to non-functioning kidney (from S. haematobium) and haematemesis (from S. mansoni) at 150000 and 130000 per year. Given the overall high number of cases with schistosomiasis-related disease and associated death, we conclude that schistosomiasis remains an important public health problem in sub-Saharan Africa.

Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics

Background Traumatic brain injury (TBI) is a leading cause of death and disability. A reliable prediction of outcome on admission is of great clinical relevance. We aimed to develop prognostic models with readily available traditional and novel predictors. Methods and Findings Prospectively collected individual patient data were analyzed from 11 studies. We considered predictors available at admission in logistic regression models to predict mortality and unfavorable outcome according to the Glasgow Outcome Scale at 6 mo after injury. Prognostic models were developed in 8,509 patients with severe or moderate TBI, with cross-validation by omission of each of the 11 studies in turn. External validation was on 6,681 patients from the recent Medical Research Council Corticosteroid Randomisation after Significant Head Injury (MRC CRASH) trial. We found that the strongest predictors of outcome were age, motor score, pupillary reactivity, and CT characteristics, including the presence of traumatic subarachnoid hemorrhage. A prognostic model that combined age, motor score, and pupillary reactivity had an area under the receiver operating characteristic curve (AUC) between 0.66 and 0.84 at cross-validation. This performance could be improved (AUC increased by approximately 0.05) by considering CT characteristics, secondary insults (hypotension and hypoxia), and laboratory parameters (glucose and hemoglobin). External validation confirmed that the discriminative ability of the model was adequate (AUC 0.80). Outcomes were systematically worse than predicted, but less so in 1,588 patients who were from high-income countries in the CRASH trial. Conclusions Prognostic models using baseline characteristics provide adequate discrimination between patients with good and poor 6 mo outcomes after TBI, especially if CT and laboratory findings are considered in addition to traditional predictors. The model predictions may support clinical practice and research, including the design and analysis of randomized controlled trials.

Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets.

Logistic regression analysis may well be used to develop a prognostic model for a dichotomous outcome. Especially when limited data are available, it is difficult to determine an appropriate selection of covariables for inclusion in such models. Also, predictions may be improved by applying some sort of shrinkage in the estimation of regression coefficients. In this study we compare the performance of several selection and shrinkage methods in small data sets of patients with acute myocardial infarction, where we aim to predict 30-day mortality. Selection methods included backward stepwise selection with significance levels alpha of 0.01, 0.05, 0. 157 (the AIC criterion) or 0.50, and the use of qualitative external information on the sign of regression coefficients in the model. Estimation methods included standard maximum likelihood, the use of a linear shrinkage factor, penalized maximum likelihood, the Lasso, or quantitative external information on univariable regression coefficients. We found that stepwise selection with a low alpha (for example, 0.05) led to a relatively poor model performance, when evaluated on independent data. Substantially better performance was obtained with full models with a limited number of important predictors, where regression coefficients were reduced with any of the shrinkage methods. Incorporation of external information for selection and estimation improved the stability and quality of the prognostic models. We therefore recommend shrinkage methods in full models including prespecified predictors and incorporation of external information, when prognostic models are constructed in small data sets.

Predictors of poor ovarian response in in vitro fertilization: a prospective study comparing basal markers of ovarian reserve

OBJECTIVE To identify and quantify predictors of poor ovarian response in in vitro fertilization (IVF). DESIGN; Prospective study. SETTING; Tertiary fertility center. PATIENT(S) One hundred twenty women undergoing their first IVF cycle. INTERVENTION(S) Measurement of the number of antral follicles and the total ovarian volume by ultrasound, and of basal levels of FSH, E(2), and inhibin B on cycle day 3. MAIN OUTCOME MEASURE(S) Ovarian response, and clinical and ongoing pregnancy rates. RESULT(S); The antral follicle count was the best single predictor for poor ovarian response: area under the receiver operating characteristic curve = 0.87. Addition of basal FSH and inhibin B levels to a logistic model with the antral follicle count significantly improved the prediction of poor response; the addition of basal E(2) levels and total ovarian volume did not improve the prediction. To express the discriminative performance of this model toward poor response, a maximum area under the receiver operating characteristic curve of 0.92 was calculated. Poor responders had significantly lower clinical and ongoing pregnancy rates than did normal responders. CONCLUSION(S) Our data demonstrate that the antral follicle count provides better prognostic information on the occurrence of poor response during hormone stimulation for IVF than does the patients chronological age and the currently used endocrine markers. However, endocrine tests remain informative. Multivariate models can achieve more accurate predictions of outcomes of complex events like ovarian response in IVF.

Risk-based selection from the general population in a screening trial : Selection criteria, recruitment and power for the Dutch-Belgian randomised lung cancer multi-slice CT screening trial (NELSON)

A method to obtain the optimal selection criteria, taking into account available resources and capacity and the impact on power, is presented for the Dutch‐Belgian randomised lung cancer screening trial (NELSON). NELSON investigates whether 16‐detector multi‐slice computed tomography screening will decrease lung cancer mortality compared to no screening. A questionnaire was sent to 335,441 (mainly) men, aged 50–75. Smoking exposure (years smoked, cigarettes/day, years quit) was determined, and expected lung cancer mortality was estimated for different selection scenarios for the 106,931 respondents, using lung cancer mortality data by level of smoking exposure (US Cancer Prevention Study I and II). Selection criteria were chosen so that the required response among eligible subjects to reach sufficient sample size was minimised and the required sample size was within our capacity. Inviting current and former smokers (quit ≤ 10 years ago) who smoked >15 cigarettes/day during >25 years or >10 cigarettes/day during >30 years was most optimal. With a power of 80%, 17,300–27,900 participants are needed to show a 20–25% lung cancer mortality reduction 10 years after randomisation. Until October 18, 2005 11,103 (first recruitment round) and 4,325 (second recruitment round) (total = 15,428) participants have been randomised. Selecting participants for lung cancer screening trials based on risk estimates is feasible and helpful to minimize sample size and costs. When pooling with Danish trial data (n = ±4,000) NELSON is the only trial without screening in controls that is expected to have 80% power to show a lung cancer mortality reduction of at least 25% 10 years after randomisation.

Prognostic Modeling with Logistic Regression Analysis: In Search of a Sensible Strategy in Small Data Sets

Clinical decision making often requires estimates of the likelihood of a dichotomous outcome in individual patients. When empirical data are available, these estimates may well be obtained from a logistic regression model. Several strategies may be followed in the development of such a model. In this study, the authors compare alternative strategies in 23 small subsamples from a large data set of patients with an acute myocardial infarction, where they developed predictive models for 30-day mortality. Evaluations were performed in an independent part of the data set. Specifically, the authors studied the effect of coding of covariables and stepwise selection on discriminative ability of the resulting model, and the effect of statistical “shrinkage” techniques on calibration. As expected, dichotomization of continuous covariables implied a loss of information. Remarkably, stepwise selection resulted in less discriminating models compared to full models including all available covariables, even when more than half of these were randomly associated with the outcome. Using qualitative information on the sign of the effect of predictors slightly improved the predictive ability. Calibration improved when shrinkage was applied on the standard maximum likelihood estimates of the regression coefficients. In conclusion, a sensible strategy in small data sets is to apply shrinkage methods in full models that include well-coded predictors that are selected based on external information.

A Clinical Prediction Rule for Renal Artery Stenosis

Renal artery stenosis impairs blood flow to the kidney and can consequently cause renovascular hypertension and renal failure [1, 2]. Although the prevalence of this condition among patients with hypertension is low, therapeutic options for relieving renal artery stenosis, such as renal angioplasty and stenting, make the search for renal artery stenosis worthwhile [2-4]. Renal angiography is the gold standard for diagnosing renal artery stenosis, but it is a costly and invasive procedure that can involve serious complications [5, 6]. To diagnose renal artery stenosis efficiently, angiography should be used selectively. Most physicians rely on captopril renal scintigraphy as a selection criterion, but the diagnostic accuracy of this test is low (sensitivity, 65% to 77%; specificity, 90%) [7, 8]. As an alternative, clinical characteristics can be used to select hypertensive patients for angiography [9]. Patients with normal renal function whose blood pressure can be controlled with one or two drugs can be excluded from angiography [9, 10]. In the remaining patients (those with drug-resistant hypertension), such clinical characteristics as atherosclerotic vascular disease, smoking history, and presence of an abdominal bruit can be used to estimate a patients probability of renal artery stenosis [11-14]. This estimate can then be used in selection for angiography. We analyzed the clinical characteristics of 477 patients with drug-resistant hypertension or an increase in serum creatinine concentration during therapy with angiotensin-converting enzyme (ACE) inhibitors who participated in the Dutch Renal Artery Stenosis Intervention Cooperative (DRASTIC) study [9]. We developed a clinical prediction rule for quantifying the probability of renal artery stenosis [15] and demonstrated the potential consequences of this rule for clinical practice by applying it to our patients. Methods Patients The DRASTIC study is a prospective cohort study conducted at 26 departments of internal medicine with an interest in hypertension throughout the Netherlands [9]. The diagnostic phase of the study was designed to find an optimal strategy for diagnosing renal artery stenosis. In the DRASTIC study, 1133 hypertensive patients 18 to 75 years of age with preserved renal function (serum creatinine concentration 200 mol/L [2.26 mg/dL]) were enrolled. These patients were referred for analysis of hypertension by general practitioners (55%) or hospital specialists (45%), in most cases because their hypertension was difficult to treat with antihypertensive drugs. Sixty percent of patients were from four hospitals. After giving written informed consent, patients were randomly assigned to one of two standard protocols with antihypertensive drugs: amlodipine, 10 mg, plus atenolol, 50 mg, in patients older than 40 years of age or enalapril, 20 mg, plus hydrochlorothiazide, 25 mg, in patients older than 40 years of age. Blood pressure was measured with a standard sphygmomanometer at three consecutive visits at least 1 week apart. Measurements were taken three times per visit after a 5-minute rest with the patient in the sitting position. Patients were selected for diagnostic workup if they had drug-resistant hypertension, defined as a mean diastolic blood pressure per visit of 95 mm Hg or more while receiving the standard drug regimen during all three visits or prescription of an additional drug regardless of blood pressure response. Patients were also selected if the serum creatinine concentration increased 20 mol/L (0.23 mg/dL) or more during therapy with ACE inhibitors. In these patients, intra-arterial digital subtraction angiography and other, noninvasive tests were performed. In accordance with the study protocol, patients who responded well to standard treatment were not evaluated further. The diagnostic phase of the study was followed by a therapeutic phase in which patients with atherosclerotic stenosis were randomly assigned to receive medication or renal angioplasty. Definitions After performing a literature study, we selected 12 clinical characteristics indicative of renovascular disease (predictors) [10, 11, 16-26]: age, sex, ethnicity (black or other), signs and symptoms of atherosclerotic vascular disease (femoral or carotid bruit, angina pectoris, claudication, myocardial infarction, cerebrovascular accident, or vascular surgery), recent onset of hypertension (within the past 2 years), family history of hypertension (parents, siblings, or children with hypertension), smoking history (ever or never), obesity (body mass index 25 kg/m2), abdominal bruit, advanced hypertensive retinopathy (fundus grade III or IV), serum creatinine concentration, and hypercholesterolemia (serum cholesterol level > 6.5 mmol/L [251.35 mg/dL] or use of cholesterol-lowering agents). These characteristics were used to predict the presence of renal artery stenosis. A patient was considered to have renal artery stenosis when the angiogram showed at least one stenosis of 50% or more in a renal artery according to the local-radiologist. Model Development Data are presented as a proportion or as the mean SD. The univariable association between clinical characteristics and presence of renal artery stenosis was studied by computing the value and 95% CI of the odds ratio. In a multivariable analysis, clinical characteristics were combined as predictor variables in a logistic regression model predicting the presence of renal artery stenosis (outcome) [27]. For each patient in the multivariable analysis, the probability of renal artery stenosis was calculated from the regression model (predicted probability). The reliability, discriminative ability, and validity of the model were assessed. The Appendix gives details on model development and evaluation. To enable the use of the regression model in clinical practice, a prediction rule was constructed for predicting renal artery stenosis in future patients with drug-resistant hypertension or an increase in serum creatinine concentration during therapy with ACE inhibitors. For the presence or level of each clinical characteristic in the regression model, a score was calculated on the basis of the regression coefficients (Appendix). These scores were added into a sum score. All possible sum scores and their corresponding predicted probabilities of renal artery stenosis were combined in a graph with 95% CIs of the predicted probabilities. Role of the Funding Source Our funding source had no role in the collection, analysis, or interpretation of the data or in the decision to submit the manuscript for publication. Results Statistical Analyses Angiography was performed in 439 patients with drug-resistant hypertension and 39 patients with an increase in serum creatinine concentration during therapy with ACE inhibitors. The procedure failed in 1 patient. For the remaining 477 patients, angiography showed renal artery stenosis in 107 patients (22%), of whom 90 (84%) had atherosclerotic stenosis and 17 (16%) had fibromuscular dysplasia. Bilateral stenoses were found in 27 of 107 affected patients (25%). Renal scintigraphy was performed in 458 patients; it had a sensitivity of 72% and a specificity of 90% for the diagnosis of renal artery stenosis. Table 1 shows the univariable distribution of the clinical characteristics for patients with renal artery stenosis and those with essential hypertension. Most clinical characteristics were indicative of renal artery stenosis (P < 0.05 or borderline significant) except sex, recent onset of hypertension, and presence of advanced hypertensive retinopathy. More young women without signs of atherosclerotic disease were found among patients with fibromuscular dysplasia than among those with atherosclerotic stenosis, but abdominal bruits occurred with the same frequency in both groups (29% and 27%, respectively). Table 1. Associations of Clinical Characteristics with Renal Artery Stenosis The results of multivariable analysis are also shown in Table 1. Advanced hypertensive retinopathy was not studied any further because this clinical characteristic was missing for 43% of the patients. Data on 11 clinical characteristics of 460 patients were considered predictive of renal artery stenosis. Ethnicity and family history of hypertension were removed from the regression model because their contribution to predicting renal artery stenosis was small. Because renal artery stenosis is believed to be more prevalent in young women and old men, interaction between age and sex was tested; this interaction was not statistically significant (P = 0.09). We included an interaction term between age and smoking because this was the only biologically plausible interaction term that was statistically significant (P = 0.01). This interaction term accounts for the fact that the predictive value of increasing age was stronger for patients who never smoked than for current and former smokers. Finally, the type of standard treatment did not provide additional diagnostic information when it was included in the regression model (P > 0.2). The multivariable odds ratios in Table 1 reflect the predictive effect of the individual clinical characteristics while correcting for the other predictors in the multivariable model. For example, the multivariable odds ratio for atherosclerotic vascular disease was lower than the univariable odds ratio because the model also accounted for the effects of age and smoking history. Model Performance Figure 1 shows the agreement between the predicted and the observed probabilities. For 204 patients (44%), the predicted probability of stenosis was 0% to 10%. The predicted probabilities of stenosis obtained from the model agreed well with the observed frequency of stenosis (goodness-of-fit test, P > 0.2). The model discriminated well between patients with renal artery stenosis (predicted probability, 49% 29%) and patients with essential hypertension (predicted probability, 15% 16%); the are

A nomogram to predict the probability of live birth after clomiphene citrate induction of ovulation in normogonadotropic oligoamenorrheic infertility

OBJECTIVE To establish whether initial screening characteristics of normogonadotropic anovulatory infertile women can aid in predicting live birth after induction of ovulation with clomiphene citrate (CC). DESIGN Prospective longitudinal single-center study. SETTING Specialist academic fertility unit. PATIENT(S) Two hundred fifty-nine couples with a history of infertility, oligoamenorrhea, and normal follicle-stimulating hormone (FSH) concentrations who have not been previously treated with any ovulation-induction medication. INTERVENTION(S) 50, 100, or 150 mg of oral CC per day, for 5 subsequent days per cycle. MAIN OUTCOME MEASURE(S) Conception leading to live birth after CC administration. RESULT(S) After receiving CC, 98 (38%) women conceived, leading to live birth. The cumulative live birth rate within 12 months was 42% for the total study population and 56% for the ovulatory women who had received CC. Factors predicting the chances for live birth included free androgen index (testosterone/sex hormone-binding globulin ratio), body mass index, cycle history (oligomenorrhea versus amenorrhea), and the womans age. CONCLUSION(S) It is possible to predict the individual chances of live birth after CC administration using two distinct prediction models combined in a nomogram. Applying this nomogram in the clinic may be a step forward in optimizing the decision-making process in the treatment of normogonadotropic anovulatory infertility. Alternative first line of treatment options could be considered for some women who have limited chances for success.