Anders Granholm

Development and internal validation of the Simplified Mortality Score for the Intensive Care Unit (SMS-ICU)

Intensive care unit (ICU) mortality prediction scores deteriorate over time, and their complexity decreases clinical applicability and commonly causes problems with missing data. We aimed to develop and internally validate a new and simple score that predicts 90‐day mortality in adults upon acute admission to the ICU: the Simplified Mortality Score for the Intensive Care Unit (SMS‐ICU).

Predictive Performance of the Simplified Acute Physiology Score (SAPS) II and the Initial Sequential Organ Failure Assessment (SOFA) Score in Acutely Ill Intensive Care Patients: Post-Hoc Analyses of the SUP-ICU Inception Cohort Study

Purpose Severity scores including the Simplified Acute Physiology Score (SAPS) II and the Sequential Organ Failure Assessment (SOFA) score are used in intensive care units (ICUs) to assess disease severity, predict mortality and in research. We aimed to assess the predictive performance of SAPS II and the initial SOFA score for in-hospital and 90-day mortality in a contemporary international cohort. Methods This was a post-hoc study of the Stress Ulcer Prophylaxis in the Intensive Care Unit (SUP-ICU) inception cohort study, which included acutely ill adults from ICUs across 11 countries (n = 1034). We compared the discrimination of SAPS II and initial SOFA scores, compared the discrimination of SAPS II in our cohort with the original cohort, assessed the calibration of SAPS II customised to our cohort, and compared the discrimination for 90-day mortality vs. in-hospital mortality for both scores. Discrimination was evaluated using areas under the receiver operating characteristics curves (AUROC). Calibration was evaluated using Hosmer-Lemeshow’s goodness-of-fit Ĉ-statistic. Results AUROC for in-hospital mortality was 0.80 (95% confidence interval (CI) 0.77–0.83) for SAPS II and 0.73 (95% CI 0.69–0.76) for initial SOFA score (P<0.001 for the comparison). Calibration of the customised SAPS II for predicting in-hospital mortality was adequate (P = 0.60). Discrimination of SAPS II was reduced compared with the original SAPS II validation sample (AUROC 0.80 vs. 0.86; P = 0.001). AUROC for 90-day mortality was 0.79 (95% CI 0.76–0.82; P = 0.74 for comparison with in-hospital mortality) for SAPS II and 0.71 (95% CI 0.68–0.75; P = 0.66 for comparison with in-hospital mortality) for the initial SOFA score. Conclusions The predictive performance of SAPS II was similar for in-hospital and 90-day mortality and superior to that of the initial SOFA score, but SAPS II’s performance has decreased over time. Use of a contemporary severity score with improved predictive performance may be of value.

Respiratory rates measured by a standardised clinical approach, ward staff, and a wireless device

Respiratory rate is among the first vital signs to change in deteriorating patients. The aim was to investigate the agreement between respiratory rate measurements by three different methods.

Simplified Mortality Score for the Intensive Care Unit (SMS-ICU): protocol for the development and validation of a bedside clinical prediction rule

Introduction Mortality prediction scores are widely used in intensive care units (ICUs) and in research, but their predictive value deteriorates as scores age. Existing mortality prediction scores are imprecise and complex, which increases the risk of missing data and decreases the applicability bedside in daily clinical practice. We propose the development and validation of a new, simple and updated clinical prediction rule: the Simplified Mortality Score for use in the Intensive Care Unit (SMS-ICU). Methods and analysis During the first phase of the study, we will develop and internally validate a clinical prediction rule that predicts 90-day mortality on ICU admission. The development sample will comprise 4247 adult critically ill patients acutely admitted to the ICU, enrolled in 5 contemporary high-quality ICU studies/trials. The score will be developed using binary logistic regression analysis with backward stepwise elimination of candidate variables, and subsequently be converted into a point-based clinical prediction rule. The general performance, discrimination and calibration of the score will be evaluated, and the score will be internally validated using bootstrapping. During the second phase of the study, the score will be externally validated in a fully independent sample consisting of 3350 patients included in the ongoing Stress Ulcer Prophylaxis in the Intensive Care Unit trial. We will compare the performance of the SMS-ICU to that of existing scores. Ethics and dissemination We will use data from patients enrolled in studies/trials already approved by the relevant ethical committees and this study requires no further permissions. The results will be reported in accordance with the Transparent Reporting of multivariate prediction models for Individual Prognosis Or Diagnosis (TRIPOD) statement, and submitted to a peer-reviewed journal.

Stress ulcer prophylaxis versus placebo or no prophylaxis in adult hospitalised acutely ill patients—protocol for a systematic review with meta-analysis and trial sequential analysis

BackgroundStress ulcer prophylaxis is considered standard of care in many critically ill patients in the intensive care unit (ICU). However, the quality of evidence supporting this has recently been questioned, and clinical equipoise exists. Whether there is overall benefit or harm of stress ulcer prophylaxis in adult hospitalised acutely ill patients is unknown. Accordingly, we aim to assess patient-important benefits and harms of stress ulcer prophylaxis versus placebo or no treatment in adult hospitalised acutely ill patients with high risk of gastrointestinal bleeding irrespective of hospital setting.Methods/designWe will conduct a systematic review of randomised clinical trials with meta-analysis and trial sequential analysis and assess use of proton pump inhibitors (PPIs) or histamine-2-receptor antagonists (H2RAs) in any dose, formulation and duration. We will accept placebo or no prophylaxis as control interventions. The participants will be adult hospitalised acutely ill patients with high risk of gastrointestinal bleeding.We will systematically search the Cochrane Library, MEDLINE, EMBASE, Science Citation Index, BIOSIS and Epistemonikos for relevant literature. We will follow the recommendations by the Cochrane Collaboration and the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement. The risk of systematic errors (bias) and random errors will be assessed, and the overall quality of evidence will be evaluated using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach.DiscussionThere is a need for a high-quality systematic review to summarise the benefits and harms of stress ulcer prophylaxis in hospitalised patients to inform practice and future research. Although stress ulcer prophylaxis is used worldwide, no firm evidence for benefit or harm as compared to placebo or no treatments has been established. Critical illness is a continuum not limited to the ICU setting, which is why it is important to assess the benefits and harms of stress ulcer prophylaxis in a wider perspective than exclusively in ICU patients.Systematic review registrationPROSPERO CRD42017055676

Reply to the letter regarding our manuscript ‘Development and internal validation of the Simplified Mortality Score for the Intensive Care Unit (SMS-ICU)’

We have read with interest the letter by Dr. Haniffa and colleagues regarding our recently published manuscript on the Simplified Mortality Score for the Intensive Care Unit (SMS-ICU) clinical prediction rule. In general, we fully support the opinions and perspectives presented. We completely agree that prognostic models must be developed and validated, using appropriate methodology and reported in accordance with the TRIPOD statement. Also, we agree that appropriate reporting and handling of missing data is key, and we advocate for more widespread use of multiple imputation instead of complete case analysis or ‘normal’ imputation when data are not missing completely at random. Variables with high missingness in model development samples are likely to be missing when models are used in daily clinical practice. While statistical methods can account for this to a large extent in research, this is not possible in the clinical setting, and thus excluding these variables seems sound. Another important point is that all prognostic models should be externally validated, and if necessary, recalibrated (and subsequently externally validated) if used in markedly different settings. This improves predictions across settings, as for the different geographical equations for SAPS 3. Accordingly, clinical prediction rules developed in high-income countries – including the SMS-ICU should be externally validated, and recalibrated if necessary, in low-and-middle income countries prior to routine use in these countries and vice versa. Real-time automatic data capture from electronic healthcare information systems may decrease problems with missing data, however, sampling frequency affects model performance. Thus, automatic data capture may provide more extreme predictions due to few extreme or erroneous measurements. Consequently, it may be argued that data collection and verification, and handling of outliers by researchers is preferred. Prognostic models are useful tools, but with inherent limitations that must be acknowledged by users. No score will ever provide perfect predictions, and the use of severity scores to guide individual patient decisions has been recommended against. We are of the opinion that simpler scores are more valuable to clinicians than more complex scores, which are time-consuming and will often suffer from missing data. For clinicians, severity scores provide an objective extra piece of information to the puzzle, but never a definite answer. When used in research and to compare intensive care unit performances, we also believe that simpler scores with sufficient calibration has advantages over more complex scores, largely due to simpler data collection and handling and less missingness. Measuring the clinical usefulness and impact of prediction models has been recommended, and (cluster) randomised clinical trials can assess whether models that recommend certain actions (decision rules) improves patient-important outcomes. However, for severity scores that do not provide any treatment recommendations, the approach is less straightforward and research in this area is warranted. We perfectly agree that more research into better understanding the priorities of clinicians, researchers and administrators, when developing or advocating any prognostic model, as well as facilitators and barriers to clinical uptake is highly warranted.

The effect of blinding on estimates of mortality in randomised clinical trials of intensive care interventions: protocol for a systematic review and meta-analysis

Introduction Evidence exists that unblinded randomised clinical trials (RCTs) overestimate intervention effects compared with blinded RCTs. It has been suggested that this is less pronounced for objective (ie, not subject to interpretation) outcome measures, including mortality. This may not apply in the intensive care unit (ICU), as most deaths are preceded by decisions to withhold or withdraw treatments. Lack of blinding of physicians in RCTs of ICU interventions may potentially influence the decision towards a higher threshold for discontinuing treatment in patients who receive the investigational treatment and/or a lower threshold for discontinuing treatment in patients who receive the comparator (control). This may have important implications for patients, caregivers, researchers and society. Accordingly, we aim to assess whether lack of blinding affects mortality effect estimates in RCTs of ICU interventions. Methods and analysis We will conduct a systematic review with meta-analyses and assess the effect of blinding versus no blinding on mortality effect estimates in RCTs of interventions used in adult ICU patients. We will systematically search the Cochrane Library for systematic reviews reporting mortality effect estimates of any intervention used in adult ICU patients which includes at least one RCT with ‘low risk of bias’ in the bias domains ‘blinding of participants and personnel’ and/or ‘blinding of outcome assessment’ and one RCT with ‘unclear’ or ‘high risk of bias’ in the same bias domain(s). For each intervention, we will compare summary mortality effect estimates in blinded versus unblinded trials. Ethics and dissemination This research does not require ethical approval as we will use summary data from trials already approved by relevant ethical institutions. We will report the results in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement and submit the final paper to an international peer-reviewed journal. Trial registration number PROSPERO, registration number: CRD42017056212.

Trustworthy or flawed clinical prediction rule

We read with interest the recently published paper by Hilder et al. [1], where the authors present the PRESET-Score, a new clinical prediction rule for patients with acute respiratory distress syndrome treated with extracorporeal membrane oxygenation (ECMO). While the topic is clinically relevant and interesting, we are worried that spurious findings, biased results, and overstated findings are presented. First, both the new and the four existing scores assessed are at high risk of being underpowered. Multivariable risk prediction models should be based on an effective sample size (lowest number of events/non-events) of at least 10, often more, per predictor variable assessed [2, 3]. Using 11 variables and 41 non-events (3.7 per predictor) results in overfitting of the development sample and inflated performance estimates [2]. This will be evident upon use of the score in other populations. Second, comparing the performance of the new score with four existing scores using the development dataset is against recommendations [2], as this is biased to favor the new score due to overfitting. For comparison with other scores, an independent cohort not used to develop any of the scores must be used [2]. Third, internal validation is performed to quantify overfitting, and should be done by bootstrap resampling of the development dataset [2]. The authors state that they used logistic regression analysis to “reassess” the score, which essentially is a recalibration resulting in a new model generating new predictions. This is neither internal nor external validation, which requires assessment of predictions made by the score without modifications in a new sample [2]. Fourth, it is recommended to assess calibration by graphical methods or regressions of the predicted versus observed outcomes [2, 4], not by the Hosmer-Lemeshow Ĉ-test, as P > 0.05 is more likely to indicate lack of power than proper model fit when used on small samples. While we agree that clinical prediction rules may be valuable for clinicians considering ECMO, it is a prerequisite that such scores are developed and validated using appropriate methodology [2] and sufficient sample sizes, and that all relevant features are transparently reported with adequate discussion of the limitations [5]. Developing and sufficiently validating a clinical prediction rule for this highly selected patient group likely requires a large, multicentre collaboration to ensure trustworthy predictions that will benefit patients and relatives, the healthcare system, researchers, and society.

Reply to the letter ‘A brief comment about predictive models for mortality in intensive care units’

We have with interest read the comments by Palaz on-Bru and colleagues regarding our recently published manuscript entitled ‘Development and internal validation of the Simplified Mortality Score for the Intensive Care Unit (SMS-ICU)’, including the response to our discussion of their intensive care unit (ICU) mortality risk score. As outlined, we chose to use data from observational studies and randomised clinical trials (RCTs) for the development of the SMS-ICU. Limitations of this approach, including clinical heterogeneity, have been discussed in our manuscript. Strengths of using data from large international observational studies and pragmatic RCTs include high external validity (generalisability), high data quality owing to the prospective data collection and monitoring, and limited risk of inflated estimates, as compared to single-centre studies, which need to be interpreted with caution. Palaz on-Bru et al. argue that treatment effects in the RCTs could affect our model. However, the intervention in two of the three RCTs had no effect on mortality, and for the last RCT, a pre-planned sensitivity analysis excluding the intervention group was conducted without affecting model performance. Consequently, the risk of treatment effect in the included RCTs is limited if present at all. Regarding the outcome of interest, we chose a long-term fixed-time mortality outcome as recommended. Very few patients stay in the ICU for more than 90 days, and in those who do, clinical prediction rules perform less well. Importantly, in-ICU and in-hospital mortality are considered inferior outcomes as they are affected by discharge practices. We decided a priori to use all available data to develop the model to ensure sufficient power, and to plan external validation in a sample that allows us to assess temporal generalisability as well. Regarding the events per variable (EPV), the rule-of-thumb recommending ≥ 10 EPV for sufficient power relates to the number of variables assessed during model derivation, and not on the number of variables included in the final model. With 12 variables assessed in Palaz onBru et al.’s model, the EPV is 7.5 (and not 15), whereas the corresponding EPV is 64 in SMSICU. Importantly, an EPV ≥ 10 is a simplified rule-of-thumb, and the risk of overfitting depends on the modelling strategy and the total number of degrees of freedom and models assessed, and a model with ≥ 10 EPV may still be substantially underpowered. We agree that it is up to clinicians, administrators and researchers to judge which model best fits their needs, however, methodological rigour, including adequate power, internal validation and high external validity are prerequisites when developing clinical prediction rules.

Bias and sample size in intensive care unit trials: Protocol for a meta-epidemiological study

Systematic errors (bias) and random errors result in inflated and imprecise intervention effect estimates in randomised clinical trials (RCT) and meta‐analyses. We aim to assess time trends in the Cochrane risk of bias domains and sample size in RCTs of intensive care unit (ICU) interventions.