Gerard J. J. M. Borsboom

Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Original articleInternal validation of predictive models: Efficiency of some procedures for logistic regression analysis

The performance of a predictive model is overestimated when simply determined on the sample of subjects that was used to construct the model. Several internal validation methods are available that aim to provide a more accurate estimate of model performance in new subjects. We evaluated several variants of split-sample, cross-validation and bootstrapping methods with a logistic regression model that included eight predictors for 30-day mortality after an acute myocardial infarction. Random samples with a size between n = 572 and n = 9165 were drawn from a large data set (GUSTO-I; n = 40,830; 2851 deaths) to reflect modeling in data sets with between 5 and 80 events per variable. Independent performance was determined on the remaining subjects. Performance measures included discriminative ability, calibration and overall accuracy. We found that split-sample analyses gave overly pessimistic estimates of performance, with large variability. Cross-validation on 10% of the sample had low bias and low variability, but was not suitable for all performance measures. Internal validity could best be estimated with bootstrapping, which provided stable estimates with low bias. We conclude that split-sample validation is inefficient, and recommend bootstrapping for estimation of internal validity of a predictive logistic regression model.

Predictive testing for complex diseases using multiple genes: Fact or fiction?

Purpose: There is ongoing debate about whether testing low-risk genes at multiple loci will be useful in clinical care and public health. We investigated the usefulness of multiple genetic testing using simulated data.Methods: Usefulness was evaluated by the area under the receiver-operating characteristic curve (AUC), which indicates the accuracy of genetic profiling in discriminating between future patients and nonpatients. The AUC was investigated in relation to the number of genes assumed to be involved, the risk allele frequency, the odds ratio of the risk genotypes, and to the proportion of variance explained by genetic factors as an approximation of the heritability of the disease.Results: We demonstrated that a high (AUC > 0.80) to excellent discriminative accuracy (AUC > 0.95) can be obtained by simultaneously testing multiple susceptibility genes. A higher discriminative accuracy is obtained when genetic factors play a larger role in the disease, as indicated by the proportion of explained variance. The maximum discriminative accuracy of future genetic profiling can be estimated at present from the heritability and prevalence of disease.Conclusions: Genetic profiling may have the potential to identify individuals at higher risk of disease depending on the prevalence and heritability of the disease.

Five-year follow-up of health-related quality of life after primary treatment of localized prostate cancer

Although with earlier detection of prostate cancer more men face the long‐term consequences of primary treatment, studies on the impact of treatment on long‐term health‐related quality of life (HRQoL) are scarce. We followed 314 men with newly diagnosed localized prostate cancer from 1 month before until 5 years after radical prostatectomy (n = 127) or external beam radiotherapy (n = 187; median follow‐up = 52 months). Questionnaires addressing disease‐specific (UCLA PCI) and generic (SF‐36, EQ‐5D) HRQoL were sent 1 month before and 6, 12 and 52 months after treatment. Repeated‐measures modeling was used to study HRQoL over time. Regular urinary leakage was reported by 12% of prostatectomy patients before treatment and by 31% at the 52‐month assessment. Erectile dysfunction before treatment was reported by 31% of prostatectomy patients and by 40% of radiotherapy patients; at the 52‐month assessment, these percentages were 88% and 64%, respectively. Erectile dysfunction present at 1 year posttreatment can be considered permanent. Prostatectomy patients reported better generic functioning both before and after treatment than radiotherapy patients, who were on average 5.9 years older and had more comorbid conditions. General physical functioning of prostatectomy patients slightly improved over time, but declined in radiotherapy patients. The relation between age and physical scores was found to be nonlinear. The long‐term physical decline in radiotherapy patients partly resulted from aging and its nonlinear impact on health, although treatment effects cannot be excluded. Scores of both patient groups remained above those of norm populations. Innovative graphs describing disease‐specific and generic functions after treatment can help patients and physicians in their treatment choices.

Spatial and temporal variations of malaria epidemic risk in Ethiopia: factors involved and implications

The aim of this study was to describe spatial and temporal variations in malaria epidemic risk in Ethiopia and to examine factors involved in relation to their implications for early warning and interpretation of geographical risk models. Forty-eight epidemic episodes were identified in various areas between September 1986 and August 1993 and factors that might have led to the events investigated using health facility records and weather data. The study showed that epidemics in specific years were associated with specific geographical areas. A major epidemic in 1988 affected the highlands whereas epidemics in 1991 and 1992 affected highland-fringe areas on the escarpments of the Rift Valley and in southern and north-western parts of the country. Malaria epidemics were significantly more often preceded by a month of abnormally high minimum temperature in the preceding 3 months than based on random chance, whereas frequency of abnormally low minimum temperature prior to epidemics was significantly lower than expected. Abnormal increases of maximum temperature and rainfall had no positive association with the epidemics. A period of low incidence during previous transmission seasons might have aggravated the events, possibly due to low level of immunity in affected populations. Epidemic risk is a dynamic phenomenon with changing geographic pattern based on temporal variations in determinant factors including weather and other eco-epidemiological characteristics of areas at risk. Epidemic early warning systems should take account of non-uniform effects of these factors by space and time and thus temporal dimensions need to be considered in spatial models of epidemic risks.

Impact of ivermectin on onchocerciasis transmission: Assessing the empirical evidence that repeated ivermectin mass treatments may lead to elimination/eradication in West-Africa

Background The Onchocerciasis Control Program (OCP) in West Africa has been closed down at the end of 2002. All subsequent control will be transferred to the participating countries and will almost entirely be based on periodic mass treatment with ivermectin. This makes the question whether elimination of infection or eradication of onchocerciasis can be achieved using this strategy of critical importance. This study was undertaken to explore this issue. Methods An empirical approach was adopted in which a comprehensive analysis was undertaken of available data on the impact of more than a decade of ivermectin treatment on onchocerciasis infection and transmission. Relevant entomological and epidemiological data from 14 river basins in the OCP and one basin in Cameroon were reviewed. Areas were distinguished by frequency of treatment (6-monthly or annually), endemicity level and additional control measures such as vector control. Assessment of results were in terms of epidemiological and entomological parameters, and as a measure of inputs, therapeutic and geographical coverage rates were used. Results In all of the river basins studied, ivermectin treatment sharply reduced prevalence and intensity of infection. Significant transmission, however, is still ongoing in some basins after 10–12 years of ivermectin treatment. In other basins, transmission may have been interrupted, but this needs to be confirmed by in-depth evaluations. In one mesoendemic basin, where 20 rounds of four-monthly treatment reduced prevalence of infection to levels as low as 2–3%, there was significant recrudescence of infection within a few years after interruption of treatment. Conclusions Ivermectin treatment has been very successful in eliminating onchocerciasis as a public health problem. However, the results presented in this paper make it almost certain that repeated ivermectin mass treatment will not lead to the elimination of transmission of onchocerciasis from West Africa. Data on 6-monthly treatments are not sufficient to draw definitive conclusions.

A psychometric comparison of health-related quality of life measures in chronic liver disease.

Four generic [the Sickness Impact Profile (SIP-68), Short-Form Health Survey (SF-36), EuroQol instrument (EQ-5D), COOP/WONCA charts], two domain-specific health-related quality of life measures [the sexuality scale of the HIV Overview Problems Evaluating System (HOPES), Multi-dimensional Fatigue Index (MFI-20)], and a self-developed 12-item symptom index were compared in terms of feasibility, test-retest reliability, internal consistency reliability, construct validity, and known groups validity in patients with chronic liver disease. All instruments could be completed within 10 min and exhibited a good psychometric performance in patients with chronic liver disease. The SF-36 and the MFI-20 performed relatively best in terms of reliability, construct validity, and discriminative ability. The sexuality scale of the HOPES demonstrated a relatively poor performance, as the missing value rate was higher than 5%. Further research is needed into the sensitivity to important clinical changes of the instruments.

Short-term health-related quality of life consequences in a lung cancer CT screening trial (NELSON)

Background:In lung cancer CT screening, participants often have an indeterminate screening result at baseline requiring a follow-up CT. In subjects with either an indeterminate or a negative result after screening, we investigated whether health-related quality of life (HRQoL) changed over time and differed between groups in the short term.Methods:A total of 733 participants in the NELSON trial received four questionnaires: T0, before randomisation; T1, 1 week before the baseline screening; T2, 1 day after the screening; and T3, 2 months after the screening results but before the 3-month follow-up CT. HRQoL was measured as generic HRQoL (the 12-item Short Form, SF-12; the EuroQol questionnaire, EQ-5D), anxiety (the Spielberger State-Trait Anxiety Inventory, STAI-6), and lung-cancer-specific distress (the Impact of Event Scale, IES). For analyses, repeated-measures analysis of variance was used, adjusted for covariates.Results:Response to each questionnaire was 88% or higher. Scores on SF-12, EQ-5D, and STAI-6 showed no clinically relevant changes over time. At T3, IES scores that were clinically relevant increased after an indeterminate result, whereas these scores showed a significant decrease after a negative result. At T3, differences in IES scores between the two baseline result groups were both significant and clinically relevant (P<0.01).Conclusion:This longitudinal study among participants of a lung cancer screening programme showed that in the short term recipients of an indeterminate result experienced increased lung-cancer-specific distress, whereas the HRQoL changes after a negative baseline screening result may be interpreted as a relief.

Mobility and HIV in Tanzanian couples: Both mobile persons and their partners show increased risk

Objective:To investigate how mobility is related to sexual risk behavior and HIV infection, with special reference to the partners who stay behind in mobile couples. Methods:HIV status, sexual behavior and demographic data of 2800 couples were collected in a longitudinal study in Kisesa, rural Tanzania. People were considered short-term mobile if they had slept outside the household at least once on the night before one of the five demographic interviews, and long-term mobile if they were living elsewhere at least once at the time of a demographic round. Results:Overall, whereas long-term mobile men did not report more risk behavior than resident men, short-term mobile men reported having multiple sex partners in the last year significantly more often. In contrast, long-term mobile women reported having multiple sex partners more often than resident women (6.8 versus 2.4%; P = 0.001), and also had a higher HIV prevalence (7.7 versus 2.7%; P = 0.02). In couples, men and women who were resident and had a long-term mobile partner both reported more sexual risk behavior and also showed higher HIV prevalence than people with resident/short-term mobile partners. Remarkably, risk behavior of men increased more when their wives moved than when they were mobile themselves. Conclusions:More sexual risk behavior and an increased risk of HIV infection were seen not only in mobile persons, but also in partners staying behind. Interventions aiming at reducing risk behavior due to mobility should therefore include partners staying behind.

Determinants of levels and changes of physical functioning in chronically ill persons: results from the GLOBE Study

STUDY OBJECTIVE Declines in physical functioning are a common result of chronic illness, but relatively little is known about factors not directly related to severity of disease that influence the occurrence of disability among chronically ill persons. The aim of this study was to assess the effect of a large number of potential determinants (sociodemographic factors, health related behaviour, structural living conditions, and psychosocial factors). DESIGN Longitudinal study of levels and changes of physical functioning among persons suffering from four chronic diseases (asthma/chronic obstructive pulmonary disease (COPD), heart disease, diabetes, chronic low back pain). In 1991, persons suffering from one or more of these diseases were identified in a general population survey. Self reported disabilities, using a subset of the OECD disability indicator, were measured six times between 1991 and 1997. These data were analysed using generalised estimating equations, relating determinants measured in 1991 to disability between 1991 and 1997, and controlling for a number of potential confounders (age, gender, year of measurement, and type and severity of chronic disease). SETTING Region of Eindhoven (south eastern Netherlands). PARTICIPANTS 1784 persons with asthma/COPD, heart disease, diabetes mellitus and/or low back pain. MAIN RESULTS In a “repeated prevalence” model, statistically significant (p<0.05) and strong associations were found between most of the determinants and the prevalence of disabilities. In a “longitudinal change” model, statistically significant (p<0.05) predictors of unfavourable changes in physical functioning were low income and excessive alcohol consumption, while we also found indications for effects of marital status, degree of urbanisation, smoking, and external locus of control. CONCLUSIONS Other factors than characteristics of the underlying disease have an important influence on levels and changes of physical functioning among chronically ill persons. Reduction of the prevalence of disabilities in the population not only depends on medical interventions, but may also require social interventions, health education, and psychological interventions among chronically ill persons.