Marinus J.C. Eijkemans

Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Original articleInternal validation of predictive models: Efficiency of some procedures for logistic regression analysis

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets.

Logistic regression analysis may well be used to develop a prognostic model for a dichotomous outcome. Especially when limited data are available, it is difficult to determine an appropriate selection of covariables for inclusion in such models. Also, predictions may be improved by applying some sort of shrinkage in the estimation of regression coefficients. In this study we compare the performance of several selection and shrinkage methods in small data sets of patients with acute myocardial infarction, where we aim to predict 30-day mortality. Selection methods included backward stepwise selection with significance levels alpha of 0.01, 0.05, 0. 157 (the AIC criterion) or 0.50, and the use of qualitative external information on the sign of regression coefficients in the model. Estimation methods included standard maximum likelihood, the use of a linear shrinkage factor, penalized maximum likelihood, the Lasso, or quantitative external information on univariable regression coefficients. We found that stepwise selection with a low alpha (for example, 0.05) led to a relatively poor model performance, when evaluated on independent data. Substantially better performance was obtained with full models with a limited number of important predictors, where regression coefficients were reduced with any of the shrinkage methods. Incorporation of external information for selection and estimation improved the stability and quality of the prognostic models. We therefore recommend shrinkage methods in full models including prespecified predictors and incorporation of external information, when prognostic models are constructed in small data sets.

Age at menopause as a risk factor for cardiovascular mortality

BACKGROUND Although an association of occurrence of menopause and subsequent oestrogen deficiency with increased cardiovascular disease has been postulated, studies on this association have not shown convincing results. We investigated whether age at menopause is associated with cardiovascular mortality risk. METHODS We studied a cohort of 12115 postmenopausal women living in Utrecht, Netherlands, aged 50-65 years at enrolment in a breast cancer screening project. During follow-up of up to 20 years the women attended screening rounds at which we asked questions on menopausal status, age of menopause, medication use, cardiovascular risk factors, and ovarian function. Deaths were ascertained from the patients family physicians. Life-table analysis and Cox regression analysis were used to investigate the association between aga at menopause and cardiovascular mortality. All analyses were adjusted for biological age. FINDINGS 824 women died of cardiovascular causes. 1459 women had left the study area. The risk of cardiovascular mortality was higher for women with early menopauses than for those with late menopauses. The age-adjusted hazard ratio of age at menopause was 0.982 (95% CI 0-968-0-996, p=0.01)_ie, for each years delay in the menopause the cardiovascular mortality risk decreased by 2%. The extra risk of early menopause seemed to decrease with biological age (p for interaction 0.07); at biological age 60 the reduction of the annual hazard was 3%, but at age 80 there was no reduction. Adjustment for known cardiovascular risk factors and indicators of ovarian function did not significantly alter the risk estimate. INTERPRETATION These results support the hypothesis that longer exposure to endogenous oestrogens protects against cardiovascular diseases. The effect of an early menopause may be more important at younger biological ages.

Common Carotid Intima-Media Thickness Measurements in Cardiovascular Risk Prediction: A Meta-analysis

CONTEXT The evidence that measurement of the common carotid intima-media thickness (CIMT) improves the risk scores in prediction of the absolute risk of cardiovascular events is inconsistent. OBJECTIVE To determine whether common CIMT has added value in 10-year risk prediction of first-time myocardial infarctions or strokes, above that of the Framingham Risk Score. DATA SOURCES Relevant studies were identified through literature searches of databases (PubMed from 1950 to June 2012 and EMBASE from 1980 to June 2012) and expert opinion. STUDY SELECTION Studies were included if participants were drawn from the general population, common CIMT was measured at baseline, and individuals were followed up for first-time myocardial infarction or stroke. DATA EXTRACTION Individual data were combined into 1 data set and an individual participant data meta-analysis was performed on individuals without existing cardiovascular disease. RESULTS We included 14 population-based cohorts contributing data for 45,828 individuals. During a median follow-up of 11 years, 4007 first-time myocardial infarctions or strokes occurred. We first refitted the risk factors of the Framingham Risk Score and then extended the model with common CIMT measurements to estimate the absolute 10-year risks to develop a first-time myocardial infarction or stroke in both models. The C statistic of both models was similar (0.757; 95% CI, 0.749-0.764; and 0.759; 95% CI, 0.752-0.766). The net reclassification improvement with the addition of common CIMT was small (0.8%; 95% CI, 0.1%-1.6%). In those at intermediate risk, the net reclassification improvement was 3.6% in all individuals (95% CI, 2.7%-4.6%) and no differences between men and women. CONCLUSION The addition of common CIMT measurements to the Framingham Risk Score was associated with small improvement in 10-year risk prediction of first-time myocardial infarction or stroke, but this improvement is unlikely to be of clinical importance.

Predictors of poor ovarian response in in vitro fertilization: a prospective study comparing basal markers of ovarian reserve

OBJECTIVE To identify and quantify predictors of poor ovarian response in in vitro fertilization (IVF). DESIGN; Prospective study. SETTING; Tertiary fertility center. PATIENT(S) One hundred twenty women undergoing their first IVF cycle. INTERVENTION(S) Measurement of the number of antral follicles and the total ovarian volume by ultrasound, and of basal levels of FSH, E(2), and inhibin B on cycle day 3. MAIN OUTCOME MEASURE(S) Ovarian response, and clinical and ongoing pregnancy rates. RESULT(S); The antral follicle count was the best single predictor for poor ovarian response: area under the receiver operating characteristic curve = 0.87. Addition of basal FSH and inhibin B levels to a logistic model with the antral follicle count significantly improved the prediction of poor response; the addition of basal E(2) levels and total ovarian volume did not improve the prediction. To express the discriminative performance of this model toward poor response, a maximum area under the receiver operating characteristic curve of 0.92 was calculated. Poor responders had significantly lower clinical and ongoing pregnancy rates than did normal responders. CONCLUSION(S) Our data demonstrate that the antral follicle count provides better prognostic information on the occurrence of poor response during hormone stimulation for IVF than does the patients chronological age and the currently used endocrine markers. However, endocrine tests remain informative. Multivariate models can achieve more accurate predictions of outcomes of complex events like ovarian response in IVF.

Prognostic Modeling with Logistic Regression Analysis: In Search of a Sensible Strategy in Small Data Sets

Clinical decision making often requires estimates of the likelihood of a dichotomous outcome in individual patients. When empirical data are available, these estimates may well be obtained from a logistic regression model. Several strategies may be followed in the development of such a model. In this study, the authors compare alternative strategies in 23 small subsamples from a large data set of patients with an acute myocardial infarction, where they developed predictive models for 30-day mortality. Evaluations were performed in an independent part of the data set. Specifically, the authors studied the effect of coding of covariables and stepwise selection on discriminative ability of the resulting model, and the effect of statistical “shrinkage” techniques on calibration. As expected, dichotomization of continuous covariables implied a loss of information. Remarkably, stepwise selection resulted in less discriminating models compared to full models including all available covariables, even when more than half of these were randomly associated with the outcome. Using qualitative information on the sign of the effect of predictors slightly improved the predictive ability. Calibration improved when shrinkage was applied on the standard maximum likelihood estimates of the regression coefficients. In conclusion, a sensible strategy in small data sets is to apply shrinkage methods in full models that include well-coded predictors that are selected based on external information.

Treatment of Vulvar Intraepithelial Neoplasia with Topical Imiquimod

BACKGROUND Alternatives to surgery are needed for the treatment of vulvar intraepithelial neoplasia. We investigated the effectiveness of imiquimod 5% cream, a topical immune-response modulator, for the treatment of this condition. METHODS Fifty-two patients with grade 2 or 3 vulvar intraepithelial neoplasia were randomly assigned to receive either imiquimod or placebo, applied twice weekly for 16 weeks. The primary outcome was a reduction of more than 25% in lesion size at 20 weeks. Secondary outcomes were histologic regression, clearance of human papillomavirus (HPV) from the lesion, changes in immune cells in the epidermis and dermis of the vulva, relief of symptoms, improvement of quality of life, and durability of response. Reduction in lesion size was classified as complete response (elimination), strong partial response (76 to 99% reduction), weak partial response (26 to 75% reduction), or no response (< or =25% reduction). The follow-up period was 12 months. RESULTS Lesion size was reduced by more than 25% at 20 weeks in 21 of the 26 patients (81%) treated with imiquimod and in none of those treated with placebo (P<0.001). Histologic regression was significantly greater in the imiquimod group than in the placebo group (P<0.001). At baseline, 50 patients (96%) tested positive for HPV DNA. HPV cleared from the lesion in 15 patients in the imiquimod group (58%), as compared with 2 in the placebo group (8%) (P<0.001). The number of immune epidermal cells increased significantly and the number of immune dermal cells decreased significantly with imiquimod as compared with placebo. Imiquimod reduced pruritus and pain at 20 weeks (P=0.008 and P=0.004, respectively) and at 12 months (P=0.04 and P=0.02, respectively). The lesion progressed to invasion (to a depth of <1 mm) in 3 of 49 patients (6%) followed for 12 months (2 in the placebo group and 1 in the imiquimod group). Nine patients, all treated with imiquimod, had a complete response at 20 weeks and remained free from disease at 12 months. CONCLUSIONS Imiquimod is effective in the treatment of vulvar intraepithelial neoplasia. (Current Controlled Trials number, ISRCTN11290871 [controlled-trials.com].).

PCOS according to the Rotterdam consensus criteria: change in prevalence among WHO‐II anovulation and association with metabolic factors

Objective  The current report aims to compare the prevalence of polycystic ovary syndrome (PCOS) diagnosed according to the new Rotterdam criteria (Rott‐PCOS) versus the previous criteria as formulated by the National Institutes of Health (NIH) (NIH‐PCOS) in women with normogonadotropic (WHO‐II) anovulation and assess the frequency of obesity and related factors determined in these women.

Anti-mullerian hormone predicts menopause: a long-term follow-up study in normoovulatory women

CONTEXT It has been hypothesized that a fixed interval exists between age at natural sterility and age at menopause. Both events show considerable individual variability, with a range of 20 yr. Correct prediction of age at menopause could open avenues of individualized prevention of age-related infertility and other menopause-related conditions, like cardiovascular disease and breast carcinoma. OBJECTIVE The aim of this study was to explore the ability of ovarian reserve tests to predict age at menopause. DESIGN AND SETTING We conducted a long-term follow-up study at an academic hospital. PARTICIPANTS A total of 257 normoovulatory women (age, 21-46 yr) were derived from three cohorts with highly comparable selection criteria. INTERVENTIONS Anti-Müllerian hormone (AMH), antral follicle count, and FSH were assessed at time 1 (T1). At time 2 (T2), approximately 11 yr later, cycle status (strictly regular, menopausal transition, or postmenopause) and age at menopause were inventoried. MAIN OUTCOME MEASURES Accuracy of the ovarian reserve tests in predicting time to menopause was assessed by Cox regression, and a nomogram was constructed for the relationship between age-specific AMH concentrations at T1 and age at menopause. RESULTS A total of 48 (19%) women had reached postmenopause at T2. Age, AMH, and antral follicle count at T1 were significantly related with time to menopause (P < 0.001) and showed a good percentage of correct predictions (C-statistic, 0.87, 0.86, and 0.84, respectively). After adjusting for age, only AMH added to this prediction (C-statistic, 0.90). From the constructed nomogram, it appeared that the normal distribution of age at menopause will shift considerably, depending on the individual age-specific AMH level. CONCLUSIONS AMH is highly predictive for timing of menopause. Using age and AMH, the age range in which menopause will subsequently occur can be individually calculated.