Rose Wharton

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

BackgroundBefore considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.MethodsWe conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.Results11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.ConclusionsThe vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Multicenter Randomized Controlled Trial of Conventional Versus Laparoscopic Surgery for Colorectal Cancer Within an Enhanced Recovery Programme: EnROL

PURPOSE Laparoscopic resection and a multimodal approach known as an enhanced recovery program (ERP) have been major changes in colorectal perioperative care that have improved clinical outcomes for colorectal cancer resection. EnROL (Enhanced Recovery Open Versus Laparoscopic) is a multicenter randomized controlled trial examining whether the benefits of laparoscopy still exist when open surgery is optimized within an ERP. PATIENTS AND METHODS Adults with colorectal cancer suitable for elective resection were randomly assigned at a ratio of 1:1 to laparoscopic or open surgery within an ERP, stratified by center, cancer site (colon v rectum), and age group (<66 v 66-75 v >75 years) using minimization. The primary outcome was physical fatigue at 1 month postsurgery. Secondary outcomes included hospital stay, complications, other patient-reported outcomes (PROs), and physical function. Patients and outcome assessors were blinded until 7 days postsurgery or discharge if earlier. Central independent and blinded pathologic assessment of surgical quality was undertaken. RESULTS A total of 204 patients (laparoscopy, n=103; open surgery, n=101) were recruited from 12 UK centers from July 2008 to April 2012. One-month physical fatigue scores were similar in both groups (mean: laparoscopy, 12.28; 95% CI, 11.37 to 13.19 v open surgery, 12.05; 95% CI, 11.14 to 12.96; adjusted mean difference, -0.23; 95% CI, -1.52 to 1.07). Median total hospital stay was significantly shorter after laparoscopic surgery (median: laparoscopy, 5; interquartile range [IQR], 4 to 9 v open surgery, 7; IQR, 5 to 11 days; P=.033). There were no differences in other secondary outcomes or in specimen quality after central pathologic review. CONCLUSION In patients treated by experienced surgeons within an ERP, physical fatigue and other PROs were similar in both groups, but laparoscopic surgery significantly reduced length of hospital stay.

Modafinil for the Treatment of Fatigue in Lung Cancer: Results of a Placebo-Controlled, Double-Blind, Randomized Trial

PURPOSE Fatigue is a distressing symptom occurring in more than 60% of patients with cancer. The CNS stimulants modafinil and methylphenidate are recommended for the treatment of cancer-related fatigue, despite a limited evidence base. We aimed to evaluate the efficacy and tolerability of modafinil in the management of fatigue in patients with non-small-cell lung cancer (NSCLC). PATIENTS AND METHODS Adults with advanced NSCLC and performance status of 0 to 2, who were not treated with chemotherapy or radiotherapy within the last 4 weeks, were randomly assigned to daily modafinil (100 mg on days 1 to 14; 200 mg on days 15 to 28) or matched placebo. The primary outcome was change in Functional Assessment of Chronic Illness Therapy (FACIT) -Fatigue score from baseline to 28 days, adjusted for baseline fatigue and performance status. Secondary outcomes included safety and patient-reported measures of depression, daytime sleepiness, and quality of life. RESULTS A total of 208 patients were randomly assigned, and 160 patients (modafinil, n = 75; placebo, n = 85) completed questionnaires at both baseline and day 28 and were included in the modified intention-to-treat analysis. FACIT-Fatigue scores improved from baseline to day 28 (mean score change: modafinil, 5.29; 95% CI, 2.57 to 8.02; placebo, 5.09; 95% CI, 2.54 to 7.65), but there was no difference between treatments (0.20; 95% CI, -3.56 to 3.97). There was also no difference between treatments for the secondary outcomes; 47% of the modafinil group and 23% of the placebo group stated that the intervention was not helpful. CONCLUSION Modafinil had no effect on cancer-related fatigue and should not be prescribed outside a clinical trial setting. Its use was associated with a clinically significant placebo effect.

Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study.

Objective To investigate the effectiveness of open peer review as a mechanism to improve the reporting of randomised trials published in biomedical journals. Design Retrospective before and after study. Setting BioMed Central series medical journals. Sample 93 primary reports of randomised trials published in BMC-series medical journals in 2012. Main outcome measures Changes to the reporting of methodological aspects of randomised trials in manuscripts after peer review, based on the CONSORT checklist, corresponding peer reviewer reports, the type of changes requested, and the extent to which authors adhered to these requests. Results Of the 93 trial reports, 38% (n=35) did not describe the method of random sequence generation, 54% (n=50) concealment of allocation sequence, 50% (n=46) whether the study was blinded, 34% (n=32) the sample size calculation, 35% (n=33) specification of primary and secondary outcomes, 55% (n=51) results for the primary outcome, and 90% (n=84) details of the trial protocol. The number of changes between manuscript versions was relatively small; most involved adding new information or altering existing information. Most changes requested by peer reviewers had a positive impact on the reporting of the final manuscript—for example, adding or clarifying randomisation and blinding (n=27), sample size (n=15), primary and secondary outcomes (n=16), results for primary or secondary outcomes (n=14), and toning down conclusions to reflect the results (n=27). Some changes requested by peer reviewers, however, had a negative impact, such as adding additional unplanned analyses (n=15). Conclusion Peer reviewers fail to detect important deficiencies in reporting of the methods and results of randomised trials. The number of these changes requested by peer reviewers was relatively small. Although most had a positive impact, some were inappropriate and could have a negative impact on reporting in the final publication.

Reply to C. Zhuang et al

We thank Zhuang et al for their interest in the EnROL (Enhanced Recovery Open Versus Laparoscopic) trial and appreciate the opportunity to clarify the points that they raised. Zhuang et al were concerned that compliance with early oral feeding, early mobilization, and optimized analgesia was low in the enhanced recovery program (ERP) in the trial. They suggested that low compliance with specific aspects of an ERP means that the trial is underpowered and that more work is still needed. Although it is important to consider which aspects of an ERP are most critical to patient recovery, the concerns of Zhuang et al are not well founded because EnROL aimed to compare laparoscopy with open surgery, and the application of and compliance within the ERP was similar in both groups of the trial; therefore, the treatment effect that was reported is likely to be robust. To date, there have been limited data reported on compliance with enhanced recovery interventions within randomized controlled trials (RCTs). The results of the EnROL trial are similar to those reported for the LAFA (Laparoscopy and/or Fast Track) trial, which is the only other similar, multicenter RCT. Maximum differences would be postulated to be seen within a single-center study, but the advantage of multicenter research is that it reflects clinical effectiveness and thus is more representative of practice generally. More than 30 items were included in the EnROL trial, although compliance measurement was limited to the 20 items that were recommended by the international expert collaborative advisory group on enhanced recovery care (Enhanced Recovery After Surgery Society [ERAS]) to make data collection manageable. This is, to our knowledge, more than has ever previously been reported. Although the administration of antiemetics was not measured, their use has been standard practice within the United Kingdom for many years. EnROL was a pragmatic trial and allowed a certain amount of variability between centers, as one would expect in real-world practice. As a result, the trial aimed to maximize the generalizability and translation of results while maintaining scientific rigor and integrity. Some researchers might concur with Zhuang et al that more well-designed and conducted RCTs are important to validate these findings and increase evidence-based practice in surgery worldwide. Others might take the view that recruitment for a similar trial is becoming increasingly difficult as patients become better informed, and surgeons would be better occupied in gaining competence in laparoscopic colorectal surgery so that their patients might benefit from its advantages.

Routine cognitive screening in older patients admitted to acute medicine: abbreviated mental test score (AMTS) and subjective memory complaint versus Montreal Cognitive Assessment and IQCODE

Introduction: routine cognitive screening for in-patients aged ≥75 years is recommended, but there is uncertainty around how this should be operationalised. We therefore determined the feasibility and reliability of the Abbreviated mental test score (AMTS/10) and its relationship to subjective memory complaint, Montreal Cognitive Assessment (MoCA/30) and informant report in unselected older admissions. Methods: consecutive acute general medicine patients aged ≥75 years admitted over 10 weeks (March–May 2013) had AMTS and a question regarding subjective memory complaint (if no known dementia/delirium). At ≥72 h, the 30-point Montreal Cognitive Assessment (MoCA) and Informant Questionnaire for Cognitive Decline in the Elderly (IQCODE) were done. Cognitive impairment was defined as AMTS < 9 or MoCA < 26 (mild impairment) and MoCA < 20 (moderate/severe impairment) or IQCODE ≥ 3.6. Results: among 264 patients (mean age/SD = 84.3/5.6 years, 117 (44%) male), 228 (86%) were testable with AMTS. 49/50 (98%) testable patients with dementia/delirium had low AMTS compared with 79/199 (44%) of those without (P < 0.001). Subjective memory complaint agreed poorly with objective cognitive deficit (39% denying a memory problem had AMTS < 9 (kappa = 0.134, P = 0.086)) as did informant report (kappa = 0.18, P = 0.15). In contrast, correlation between AMTS and MoCA was strong (R2 = 0.59, P < 0.001) with good agreement between AMTS < 9 and MoCA < 20 (kappa = 0.50, P < 0.01), although 85% of patients with normal AMTS had MoCA < 26. Conclusions: the AMTS was feasible and valid in older acute medicine patients agreeing well with the MoCA albeit with a ceiling effect. Objective cognitive deficits were prevalent in patients without known dementia or delirium but were not reliably identified by subjective cognitive complaint or informant report.

Methodological Factors in Determining Risk of Dementia After Transient Ischemic Attack and Stroke (II) Effect of Attrition on Follow-Up

Background and Purpose— Cognitive outcomes in cohorts and trials are often based only on face-to-face clinic assessment. However, cognitive impairment is strongly associated with increased morbidity and mortality, leading to substantial loss to clinic follow-up. In the absence of previous population-based data, we determined the effect of such attrition on measured risk of dementia after transient ischemic attack and stroke. Methods— Patients with transient ischemic attack or stroke prospectively recruited (2002–2007) into the Oxford Vascular (OXVASC) study had baseline clinical/cognitive assessment and follow-up to 2014. Dementia was diagnosed through face-to-face clinic interview, supplemented by home visits and telephone assessment in patients unable to attend clinic and by hand-searching of primary care records in uncontactable patients. Results— Of 1236 patients (mean age/SD, 75.2/12.1 years; 582 men), 527 (43%) died by 5-year follow-up. Follow-up assessment rates (study clinic, home visit, or telephone) of survivors were 947 in 1026 (92%), 857 in 958 (89%), 792 in 915 (87%), and 567 in 673 (84%) at 1, 6, 12 months and 5 years. Dementia developed in 260 patients, of whom 110 (42%; n=50 primary care records, n=49 home visit, and n=11 telephone follow-up) had not been available for face-to-face clinic follow-up at the time of diagnosis. The 5-year cumulative incidence of postevent dementia was 29% (26%–32%) overall but was only 17% (14% to 19%) in clinic assessed versus 45% (39%–51%) in nonclinic-assessed patients (P difference<0.001). Conclusions— Exclusion of patients unavailable for clinic follow-up reduces the measured risk of postevent dementia. Use of multiple follow-up methods, including home visits, telephone assessments, and consent, to access primary care records substantially increases ascertainment of longer-term dementia outcomes.

Methodological Factors in Determining Risk of Dementia After Transient Ischemic Attack and Stroke: (III) Applicability of Cognitive Tests.

Background and Purpose— Cognitive assessment is recommended after stroke but there are few data on the applicability of short cognitive tests to the full spectrum of patients. We therefore determined the rates, causes, and associates of untestability in a population-based study of all transient ischemic attack (TIA) and stroke. Methods— Patients with TIA or stroke prospectively recruited (2002–2007) into the Oxford Vascular Study had ≥1 short cognitive test (mini-mental state examination, telephone interview of cognitive status, Montreal cognitive assessment, and abbreviated mental test score) at baseline and on follow-up to 5 years. Results— Among 1097 consecutive assessed survivors (mean: age/SD, 74.8/12.1 years; 378 TIA), numbers testable with a short cognitive test at baseline, 1, 6, 12, and 60 months were 835/1097 (76%), 778/947 (82%), 756/857 (88%), 692/792 (87%), and 472/567 (83%). Eighty-eight percent (331/378) of assessed patients with TIA were testable at baseline compared with only 46% (133/290) of major stroke (P<0.001). Untestability was also associated with older age, premorbid dependency, death on follow-up, and with both pre- and postevent dementia (all P<0.01). Untestability (and problems with testing) were commonly caused by acute stroke effects at baseline (153/262 [58%]: dysphasia/anarthria/hemiparesis=84 [32%], drowsiness=58 [22%], and acute confusion=11 [4%]), whereas sensory deficits caused relatively more problems with testing at later time points (24/63 [38%] at 5 years). Conclusions— Substantial numbers of patients with TIA and stroke are untestable with short cognitive tests. Future studies should report data on untestable patients and those with problems with testing in whom the likelihood of dementia is high.

Delirium risk stratification in consecutive unselected admissions to acute medicine: validation of a susceptibility score based on factors identified externally in pooled data for use at entry to the acute care pathway.

Abstract Background recognition of prevalent delirium and prediction of incident delirium may be difficult at first assessment. We therefore aimed to validate a pragmatic delirium susceptibility (for any, prevalent and incident delirium) score for use in front-line clinical practice in a consecutive cohort of older acute medicine patients. Methods consecutive patients aged ≥65 years over two 8-week periods (2010–12) were screened prospectively for delirium using the Confusion Assessment Method (CAM), and delirium was diagnosed using the DSM IV criteria. The delirium susceptibility score was the sum of weighted risk factors derived using pooled data from UK-NICE guidelines: age >80 = 2, cognitive impairment (cognitive score below cut-off/dementia) = 2, severe illness (systemic inflammatory response syndrome) = 1, infection = 1, visual impairment = 1. Score reliability was determined by the area under the receiver operating curve (AUC). Results among 308 consecutive patients aged ≥65 years (mean age/SD = 81/8 years, 164 (54%) female), AUC was 0.78 (95% CI 0.71–0.84) for any delirium; 0.71 (0.64–0.79), for prevalent delirium; 0.81 (0.70–0.92), for incident delirium; odds ratios (ORs) for risk score 5–7 versus <2 were 17.9 (5.4–60.0), P < 0.0001 for any delirium, 8.1 (2.2–29.7), P = 0.002 for prevalent delirium, and 25.0 (3.0–208.9) P = 0.003 for incident delirium, with corresponding relative risks of 5.4, 4.7 and 13. Higher risk scores were associated with frailty markers, increased care needs and poor outcomes. Conclusions the externally derived delirium susceptibility score reliably identified prevalent and incident delirium using clinical data routinely available at initial patient assessment and might therefore aid recognition of vulnerability in acute medical admissions early in the acute care pathway.

Time Course of Evolution of Disability and Cause‐Specific Mortality After Ischemic Stroke: Implications for Trial Design

Background Outcome in stroke trials is often based on a 3‐month modified Rankin scale (mRS). How 3‐month mRS relates to longer‐term outcomes will depend on late recovery, delayed stroke‐related deaths, recurrent strokes, and nonstroke deaths. We evaluated 3‐month mRS and death/disability at 1 and 5 years in a population‐based cohort study. Methods and Results In 3‐month survivors of ischemic stroke (Oxford Vascular Study; 2002‐2014), we related 3‐month mRS to disability (defined as mRS >2) at 1 and 5 years and/or death rates (age/sex adjusted). Accrual of disability and index‐stroke‐related and nonstroke deaths in each poststroke year was categorized according to 3‐month mRS. Among 1606 patients with acute ischemic stroke, 181 died within 3 months, but 126 index‐stroke‐related deaths and 320 other deaths occurred during the subsequent 4866 patient‐years of follow‐up up to 5 years. Although 69/126 (54.8%) post‐3‐month index‐stroke‐related deaths occurred after 1 year, mRS>2 at 1 year strongly predicted these deaths (adjusted hazard ratio=21.94, 95%CI 7.88‐61.09, P<0.0001). Consequently, a 3‐month mRS >2 was a strong independent predictor of death at both 1 year (adjusted hazard ratio=6.67, 95%CI 4.16‐10.69, P<0.0001) and 5 years (adjusted hazard ratio=2.93, 95%CI 2.38‐3.60, P<0.0001). Although mRS improved by ≥1 point from 3 months to 1 year in 317/1266 (25.0%) patients with 3‐month mRS ≥1, improvement in mRS after 1 year was limited (improvement by ≥1 point: 91/858 [10.6%]; improvement to mRS ≤2: 13/353 [3.7%]). Conclusions Our results reaffirm use of the 3‐month mRS outcome in stroke trials. Although later recovery does occur, extending follow‐up to 1 year would capture most long‐term stroke‐related disability. However, administrative mortality follow‐up beyond 1 year has the potential to demonstrate translation of early disability gains into additional reductions in long‐term mortality without much erosion by non‐stroke‐related deaths.