Milensu Shanyinde

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

BackgroundBefore considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.MethodsWe conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.Results11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.ConclusionsThe vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods.

BACKGROUND Chronic kidney disease (CKD) is a global health concern that is increasing mainly as the result of increasing incidences of diabetes and hypertension. Furthermore, if left untreated, individuals with CKD may progress to end-stage kidney failure. Identifying individuals with undiagnosed CKD or those who are at an increased risk of developing CKD or progressing to end-stage kidney disease (ESKD) is therefore an important challenge. We sought to systematically review and critically assess the conduct and reporting of methods used to develop risk prediction models for predicting the risk of having undiagnosed (prevalent) or future risk of developing (incident) CKD or end-stage kidney failure in adults. METHODS We conducted a systematic search of PubMed database to identify studies published up until September 2011 that describe the development of models combining two or more variables to predict the risk of prevalent or incident CKD or ESKD. We extracted key information that describes aspects of developing a prediction model, including the study design, data quality, sample size and number of events, outcome definition, risk predictor selection and coding, missing data, model-building strategies, and aspects of performance. RESULTS Eleven studies describing the development of 14 prediction models were included. Eight studies reported the development of 11 models to predict incident CKD or ESKD, whereas 3 studies developed models for prevalent CKD. A total of 97 candidate risk predictors were considered, and 43 different risk predictors featured in the 14 prediction models. A method, not recommended to select risk predictors for inclusion in the multivariate model, using statistical significance from univariate screening was carried out in six studies. Missing data were frequently poorly handled and reported with no mention of missing data in four studies; 4 studies explicitly excluded individuals with missing data, and only 2 studies used multiple imputation to replace missing values. CONCLUSION We found that prediction models for chronic kidney were often developed using inappropriate methods and were generally poorly reported. Using poor methods can affect the predictive ability of the models, whereas inadequate reporting hinders an objective evaluation of the potential usefulness of the model.

Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study.

Objective To investigate the effectiveness of open peer review as a mechanism to improve the reporting of randomised trials published in biomedical journals. Design Retrospective before and after study. Setting BioMed Central series medical journals. Sample 93 primary reports of randomised trials published in BMC-series medical journals in 2012. Main outcome measures Changes to the reporting of methodological aspects of randomised trials in manuscripts after peer review, based on the CONSORT checklist, corresponding peer reviewer reports, the type of changes requested, and the extent to which authors adhered to these requests. Results Of the 93 trial reports, 38% (n=35) did not describe the method of random sequence generation, 54% (n=50) concealment of allocation sequence, 50% (n=46) whether the study was blinded, 34% (n=32) the sample size calculation, 35% (n=33) specification of primary and secondary outcomes, 55% (n=51) results for the primary outcome, and 90% (n=84) details of the trial protocol. The number of changes between manuscript versions was relatively small; most involved adding new information or altering existing information. Most changes requested by peer reviewers had a positive impact on the reporting of the final manuscript—for example, adding or clarifying randomisation and blinding (n=27), sample size (n=15), primary and secondary outcomes (n=16), results for primary or secondary outcomes (n=14), and toning down conclusions to reflect the results (n=27). Some changes requested by peer reviewers, however, had a negative impact, such as adding additional unplanned analyses (n=15). Conclusion Peer reviewers fail to detect important deficiencies in reporting of the methods and results of randomised trials. The number of these changes requested by peer reviewers was relatively small. Although most had a positive impact, some were inappropriate and could have a negative impact on reporting in the final publication.

Mobile Phone Text Messages to Support Treatment Adherence in Adults With High Blood Pressure (SMS-Text Adherence Support [StAR]) A Single-Blind, Randomized Trial

Background— We assessed the effect of automated treatment adherence support delivered via mobile phone short message system (SMS) text messages on blood pressure. Methods and Results— In this pragmatic, single-blind, 3-arm, randomized trial (SMS-Text Adherence Support [StAR]) undertaken in South Africa, patients treated for high blood pressure were randomly allocated in a 1:1:1 ratio to information only, interactive SMS text messaging, or usual care. The primary outcome was change in systolic blood pressure at 12 months from baseline measured with a validated oscillometric device. All trial staff were masked to treatment allocation. Analyses were intention to treat. Between June 26, 2012, and November 23, 2012, 1372 participants were randomized to receive information-only SMS text messages (n=457), interactive SMS text messages (n=458), or usual care (n=457). Primary outcome data were available for 1256 participants (92%). At 12 months, the mean adjusted change in systolic blood pressure compared with usual care was −2.2 mm Hg (95% confidence interval, −4.4 to −0.04) with information-only SMS and −1.6 mm Hg (95% confidence interval, −3.7 to 0.6) with interactive SMS. Odds ratios for the proportion of participants with a blood pressure <140/90 mm Hg were 1.42 (95% confidence interval, 1.03–1.95) for information-only messaging and 1.41 (95% confidence interval, 1.02–1.95) for interactive messaging compared with usual care. Conclusions— In this randomized trial of an automated adherence support program delivered by SMS text message in a general outpatient population of adults with high blood pressure, we found a small reduction in systolic blood pressure control compared with usual care at 12 months. There was no evidence that an interactive intervention increased this effect. Clinical Trial Registration— URL: http://www.clinicaltrials.gov. Unique identifier: NCT02019823. South African National Clinical Trials Register, number SANCTR DOH-27-1212-386; Pan Africa Trial Register, number PACTR201411000724141.

DOC-MEK: a double-blind randomized phase II trial of docetaxel with or without selumetinib in wild-type BRAF advanced melanoma

BACKGROUND Treatment options for wild-type BRAF melanoma patients remain limited. Selumetinib, a MEK 1/2 inhibitor, suppresses pERK levels independent of BRAF and NRAS mutation status, and combination with docetaxel has demonstrated synergy in xenograft models. The aim of this study was to assess the efficacy and safety of selumetinib plus docetaxel as first-line treatment in patients with wild-type BRAF advanced melanoma. PATIENTS AND METHODS In this double-blind multicentre phase II trial patients with wild-type BRAF melanoma were randomized (1:1) to docetaxel with selumetinib or placebo. Docetaxel 75 mg/m(2) was administered intravenously every 3 weeks up to six cycles. Selumetinib 75 mg or placebo was given orally twice daily until disease progression or unacceptable toxicity. The primary end point was progression-free survival (PFS). Tumour NRAS mutation status was analysed retrospectively and correlated with treatment outcomes. RESULTS Eighty-three patients were randomized to docetaxel plus selumetinib (n = 41) or docetaxel plus placebo (n = 42). The PFS hazard ratio (HR) (selumetinib:placebo) was 0.75 [90% confidence interval (CI) 0.50-1.14; P = 0.130], with a median PFS of 4.23 months (90% CI 3.63-6.90) for docetaxel plus selumetinib and 3.93 months (90% CI 2.07-4.16) for docetaxel alone. There was no significant difference in overall survival. The objective response rate was 32% with selumetinib versus 14% with placebo (P = 0.059). In a retrospective subset analysis, NRAS mutation status did not affect significantly upon clinical outcomes in either arm. The combination of docetaxel and selumetinib could be administered effectively to patients with metastatic melanoma, although the combination was less well tolerated than docetaxel alone. CONCLUSIONS The combination of docetaxel with selumetinib showed no significant improvement in PFS compared with docetaxel alone, although more patients showed a response to combination therapy. We found no evidence to support using tumour NRAS mutation as a basis for selecting patients for combined MEK inhibitor and chemotherapy. CLINICAL TRIAL DOC-MEK (EudraCT no: 2009-018153-23).

Outcome of central nervous system relapses in childhood acute lymphoblastic leukaemia--prospective open cohort analyses of the ALLR3 trial.

The outcomes of Central Nervous System (CNS) relapses in children with acute lymphoblastic leukaemia (ALL) treated in the ALL R3 trial, between January 2003 and March 2011 were analysed. Patients were risk stratified, to receive a matched donor allogeneic transplant or fractionated cranial irradiation with continued treatment for two years. A randomisation of Idarubicin with Mitoxantrone closed in December 2007 in favour of Mitoxantrone. The estimated 3-year progression free survival for combined and isolated CNS disease were 40.6% (25·1, 55·6) and 38.0% (26.2, 49.7) respectively. Univariate analysis showed a significantly better survival for age <10 years, progenitor-B cell disease, good-risk cytogenetics and those receiving Mitoxantrone. Adjusting for these variables (age, time to relapse, cytogenetics, treatment drug and gender) a multivariate analysis, showed a poorer outcome for those with combined CNS relapse (HR 2·64, 95% CI 1·32, 5·31, p = 0·006 for OS). ALL R3 showed an improvement in outcome for CNS relapses treated with Mitoxantrone compared to Idarubicin; a potential benefit for matched donor transplant for those with very early and early isolated-CNS relapses. Trial Registration Controlled-Trials.com ISRCTN45724312

A novel meningococcal outer membrane vesicle vaccine with constitutive expression of FetA: A phase I clinical trial

Summary Objectives Outer membrane vesicle (OMV) vaccines are used against outbreaks of capsular group B Neisseria meningitidis (MenB) caused by strains expressing particular PorA outer membrane proteins (OMPs). Ferric enterobactin receptor (FetA) is another variable OMP that induces type-specific bactericidal antibodies, and the combination of judiciously chosen PorA and FetA variants in vaccine formulations is a potential approach to broaden protection of such vaccines. Methods The OMV vaccine MenPF-1 was generated by genetically modifying N. meningitidis strain 44/76 to constitutively express FetA. Three doses of 25 μg or 50 μg of MenPF-1 were delivered intra-muscularly to 52 healthy adults. Results MenPF-1 was safe and well tolerated. Immunogenicity was measured by serum bactericidal assay (SBA) against wild-type and isogenic mutant strains. After 3 doses, the proportion of volunteers with SBA titres ≥1:4 (the putative protective titre) was 98% for the wild-type strain, and 77% for the strain 44/76 FetAonPorAoff compared to 51% in the strain 44/76 FetAoffPorAoff, demonstrating that vaccination with MenPF-1 simultaneously induced FetA and PorA bactericidal antibodies. Conclusion This study provides a proof-of-concept for generating bactericidal antibodies against FetA after OMV vaccination in humans. Prevalence-based choice of PorA and FetA types can be used to formulate a vaccine for broad protection against MenB disease.

Interferon-driven alterations of the host's amino acid metabolism in the pathogenesis of typhoid fever.

Work in humans and mice highlights the role of tryptophan metabolism in the immunopathogenesis of typhoid fever, offering novel insight into clinical disease.

Mobile Phone Text Messages to Support Treatment Adherence in Adults With High Blood Pressure (StAR): A Single-Blind, Randomized Trial

Background— We assessed the effect of automated treatment adherence support delivered via mobile phone short message system (SMS) text messages on blood pressure. Methods and Results— In this pragmatic, single-blind, 3-arm, randomized trial (SMS-Text Adherence Support [StAR]) undertaken in South Africa, patients treated for high blood pressure were randomly allocated in a 1:1:1 ratio to information only, interactive SMS text messaging, or usual care. The primary outcome was change in systolic blood pressure at 12 months from baseline measured with a validated oscillometric device. All trial staff were masked to treatment allocation. Analyses were intention to treat. Between June 26, 2012, and November 23, 2012, 1372 participants were randomized to receive information-only SMS text messages (n=457), interactive SMS text messages (n=458), or usual care (n=457). Primary outcome data were available for 1256 participants (92%). At 12 months, the mean adjusted change in systolic blood pressure compared with usual care was −2.2 mm Hg (95% confidence interval, −4.4 to −0.04) with information-only SMS and −1.6 mm Hg (95% confidence interval, −3.7 to 0.6) with interactive SMS. Odds ratios for the proportion of participants with a blood pressure <140/90 mm Hg were 1.42 (95% confidence interval, 1.03–1.95) for information-only messaging and 1.41 (95% confidence interval, 1.02–1.95) for interactive messaging compared with usual care. Conclusions— In this randomized trial of an automated adherence support program delivered by SMS text message in a general outpatient population of adults with high blood pressure, we found a small reduction in systolic blood pressure control compared with usual care at 12 months. There was no evidence that an interactive intervention increased this effect. Clinical Trial Registration— URL: http://www.clinicaltrials.gov. Unique identifier: NCT02019823. South African National Clinical Trials Register, number SANCTR DOH-27-1212-386; Pan Africa Trial Register, number PACTR201411000724141.

Mobile Phone Text Messages to Support Treatment Adherence in Adults With High Blood Pressure (SMS-Text Adherence Support [StAR])CLINICAL PERSPECTIVE

Background— We assessed the effect of automated treatment adherence support delivered via mobile phone short message system (SMS) text messages on blood pressure. Methods and Results— In this pragmatic, single-blind, 3-arm, randomized trial (SMS-Text Adherence Support [StAR]) undertaken in South Africa, patients treated for high blood pressure were randomly allocated in a 1:1:1 ratio to information only, interactive SMS text messaging, or usual care. The primary outcome was change in systolic blood pressure at 12 months from baseline measured with a validated oscillometric device. All trial staff were masked to treatment allocation. Analyses were intention to treat. Between June 26, 2012, and November 23, 2012, 1372 participants were randomized to receive information-only SMS text messages (n=457), interactive SMS text messages (n=458), or usual care (n=457). Primary outcome data were available for 1256 participants (92%). At 12 months, the mean adjusted change in systolic blood pressure compared with usual care was −2.2 mm Hg (95% confidence interval, −4.4 to −0.04) with information-only SMS and −1.6 mm Hg (95% confidence interval, −3.7 to 0.6) with interactive SMS. Odds ratios for the proportion of participants with a blood pressure <140/90 mm Hg were 1.42 (95% confidence interval, 1.03–1.95) for information-only messaging and 1.41 (95% confidence interval, 1.02–1.95) for interactive messaging compared with usual care. Conclusions— In this randomized trial of an automated adherence support program delivered by SMS text message in a general outpatient population of adults with high blood pressure, we found a small reduction in systolic blood pressure control compared with usual care at 12 months. There was no evidence that an interactive intervention increased this effect. Clinical Trial Registration— URL: http://www.clinicaltrials.gov. Unique identifier: NCT02019823. South African National Clinical Trials Register, number SANCTR DOH-27-1212-386; Pan Africa Trial Register, number PACTR201411000724141.