Daniele Poole

External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units

ObjectiveTo evaluate the SAPS 3 score predictive ability of hospital mortality in a large external validation cohort.DesignProspective observational study.Setting and patientsA total of 28,357 patients from 147 Italian ICUs joining the Project Margherita national database of the Gruppo italiano per la Valutazione degli interventi in Terapia Intensiva (GiViTI).InterventionsNone.MeasurementEvaluation of discrimination through ROC analysis and of overall goodness-of-fit through the Cox calibration test.Main resultsAlthough discrimination was good, calibration turned out to be poor. The general and the South-Europe Mediterranean countries equations overestimated hospital mortality overall (SMR values 0.73 with 95% CI 0.72–0.75 for both equations) and homogeneously across risk classes. Overprediction was confirmed among important subgroups, with SMR values ranging between 0.47 and 0.82.ConclusionsThe result strictly supported by our data is that the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings.

Calibration Belt for Quality-of-Care Assessment Based on Dichotomous Outcomes

Prognostic models applied in medicine must be validated on independent samples, before their use can be recommended. The assessment of calibration, i.e., the models ability to provide reliable predictions, is crucial in external validation studies. Besides having several shortcomings, statistical techniques such as the computation of the standardized mortality ratio (SMR) and its confidence intervals, the Hosmer–Lemeshow statistics, and the Cox calibration test, are all non-informative with respect to calibration across risk classes. Accordingly, calibration plots reporting expected versus observed outcomes across risk subsets have been used for many years. Erroneously, the points in the plot (frequently representing deciles of risk) have been connected with lines, generating false calibration curves. Here we propose a methodology to create a confidence band for the calibration curve based on a function that relates expected to observed probabilities across classes of risk. The calibration belt allows the ranges of risk to be spotted where there is a significant deviation from the ideal calibration, and the direction of the deviation to be indicated. This method thus offers a more analytical view in the assessment of quality of care, compared to other approaches.

Errors in the approval process and post-marketing evaluation of drotrecogin alfa (activated) for the treatment of severe sepsis

Regulatory agencies take responsibility for the safety and efficacy of the drugs they license. Over the past few years, however, several serious failings in the approval procedure have raised widespread concern that the present process of drug regulation is inadequate to guarantee the defence of public health. We discuss the approval process of drotrecogin alfa (activated), a non-antibacterial drug for the treatment of severe sepsis. The US Food and Drug Administration (FDA) and the European Medicines Agency (EMEA) approved drotrecogin alfa following a phase III trial that showed efficacy of the drug. However, on the basis of subgroup analyses, the drug was licensed only for use in part of the study population. This methodology is contrary to guidelines established by the FDA and EMEA themselves. According to these guidelines, analyses of non-predefined subgroups do not provide sufficient evidence for drug approval. Although the results of several post-marketing trials raised doubts about the efficacy of drotrecogin alfa, both regulatory agencies passively accepted the reassuring interpretations of sponsored investigators and of the manufacturing company itself. The recent requirement of a confirmatory trial by the EMEA without recalling the drug from the market is the latest inconsistent step taken by the agency. The case of the approval and post-marketing evaluation of drotrecogin alfa, we believe, shows that the current drug regulation system needs reforming.

Overoptimism in the interpretation of statistics

A letter to the BMJ in 2000 unveiled macroscopic flaws regarding the calculations of means in a manuscript published by the journal [1]. Although such serious mistakes may be the exception, frequently statistical errors have been found in published articles, indicating reviewing system failures [2]. In our experience with intensive care medicine, however, we noticed that editors frequently involve statistical reviewers, taking full account of their revisions and requiring their final evaluation after authors have complied with reviewers’ recommendations. This allows for the filtering of poor quality articles with evident mistakes such as applying exclusion criteria after randomisation, running multiple regression analyses on very small samples without accounting for the event-to-variable ratio, performing infinite bivariate comparisons to rule out basic differences between two study groups, and calculating sensitivities without having all patients submitted to the diagnostic test or reporting areas under the receiver operating characteristic curves values less than 0.5. The high frequency of these errors and the insufficient reporting of statistics and study design make the reviewer assignment unduly difficult. At the same time, statistical revisions seem to be effective in improving the quality of published articles [3]. In some cases, however, although statistical analyses are correct, authors may overemphasize their interpretation, as reported in the examples below.

Withdrawal of ‘Xigris’ from the market: old and new lessons

Drotrecogin alfa (activated), Xigris, was licensed in 2001 by the Food and Drug Administration (FDA) and in 2002 by the European Medicine Agency (EMA) for the treatment of severe sepsis and septic shock. The FDA approval process was marked by conflicting opinions during the Anti-Infective Drugs Advisory Committee meeting, where many concerns with regard to the Protein C Worldwide Evaluation in Severe Sepsis (PROWESS) trial, the study on which the approval was based, were raised.1 In the attempt to follow some of the criticisms, the approval was restricted to a subgroup of the study population that was not predefined, so violating well-known methodological rules and guidelines established by the FDA itself.2 The EMA approved the drug following a similar biased approach. Both regulatory agencies recognised efficacy and safety issues the drug increased the number of severe bleedings, and required post-marketing studies to support the licensing. These studies failed in confirming the first positive results,3 4 but their findings were …

Fluid therapy in neurointensive care patients: ESICM consensus and clinical practice recommendations

ObjectiveTo report the ESICM consensus and clinical practice recommendations on fluid therapy in neurointensive care patients.DesignA consensus committee comprising 22 international experts met in October 2016 during ESICM LIVES2016. Teleconferences and electronic-based discussions between the members of the committee subsequently served to discuss and develop the consensus process.MethodsPopulation, intervention, comparison, and outcomes (PICO) questions were reviewed and updated as needed, and evidence profiles generated. The consensus focused on three main topics: (1) general fluid resuscitation and maintenance in neurointensive care patients, (2) hyperosmolar fluids for intracranial pressure control, (3) fluid management in delayed cerebral ischemia after subarachnoid haemorrhage. After an extensive literature search, the principles of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system were applied to assess the quality of evidence (from high to very low), to formulate treatment recommendations as strong or weak, and to issue best practice statements when applicable. A modified Delphi process based on the integration of evidence provided by the literature and expert opinions—using a sequential approach to avoid biases and misinterpretations—was used to generate the final consensus statement.ResultsThe final consensus comprises a total of 32 statements, including 13 strong recommendations and 17 weak recommendations. No recommendations were provided for two statements.ConclusionsWe present a consensus statement and clinical practice recommendations on fluid therapy for neurointensive care patients.

Blood Component Therapy and Coagulopathy in Trauma: A Systematic Review of the Literature from the Trauma Update Group

Background Traumatic coagulopathy is thought to increase mortality and its treatment to reduce preventable deaths. However, there is still uncertainty in this field, and available literature results may have been overestimated. Methods We searched the MEDLINE database using the PubMed platform. We formulated four queries investigating the prognostic weight of traumatic coagulopathy defined according to conventional laboratory testing, and the effectiveness in reducing mortality of three different treatments aimed at contrasting coagulopathy (high fresh frozen plasma/packed red blood cells ratios, fibrinogen, and tranexamic acid administration). Randomized controlled trials were selected along with observational studies that used a multivariable approach to adjust for confounding. Strict criteria were adopted for quality assessment based on a two-step approach. First, we rated quality of evidence according to the Grading of Recommendations Assessment, Development and Evaluation (GRADE) criteria. Then, this rating was downgraded if other three criteria were not met: high reporting quality according to shared standards, absence of internal methodological and statistical issues not detailed by the GRADE system, and absence of external validity issues. Results With few exceptions, the GRADE rating, reporting and methodological quality of observational studies was “very low”, with frequent external validity issues. The only two randomized trials retrieved were, instead, of high quality. Only weak evidence was found for a relation between coagulopathy and mortality. Very weak evidence was found supporting the use of fibrinogen administration to reduce mortality in trauma. On the other hand, we found high evidence that the use of 1:1 vs. 1:2 high fresh frozen plasma/packed red blood cells ratios failed to obtain a 12% mortality reduction. This does not exclude lower mortality rates, which have not been investigated. The use of tranexamic acid in trauma was supported by “high” quality evidence according to the GRADE classification but was downgraded to “moderate” for external validity issues. Conclusions Tranexamic acid is effective in reducing mortality in trauma. The other transfusion practices we investigated have been inadequately studied in the literature, as well as the independent association between mortality and coagulopathy measured with traditional laboratory testing. Overall, in this field of research literature quality is poor.

Intensive care medicine in 2050: statistical tools for development of prognostic models (why clinicians should not be ignored)

Predictive ability assessment of prognostic models When physicians admit a patient to the intensive care unit (ICU), they automatically grade the degree of severity and formulate an initial prognosis. This is a complex integration process of anamnestic information and physiological data with data from experience and culture. This approach, however, does not numerically quantify the patient’s risk and may be heavily influenced by subjectivity. Prognostic models, such as the severity scores of the APACHE and SAPS series, have been developed to answer this need more objectively. However, they have turned out to provide unreliable predictions when applied to contexts different from those from which they were developed [1, 2]. Prognostic models are developed on large cohorts of patients, representative of the population to which they will be applied. The prognostic weight of single clinical features (e.g. age, weight, presence of comorbidities and acute conditions on admission, physiologic parameters derangement) is measured in the development cohort with appropriate statistical procedures. After having developed a model, its internal validity should be tested by measuring discrimination and calibration. In mortality models, discrimination is the ability to distinguish between survivors and non-survivors, while calibration measures the degree of correspondence between observed deaths and those predicted by the model across the whole range of prognostic estimates. Calibration can be visually appreciated with the GiViTI (Gruppo Italiano per la Valutazione degli interventi in Terapia Intensiva) calibration belt (Fig. 1), which provides statistically rigorous information on deviation from the ideal perfect matching between predicted and observed death rates [3, 4]. Traditionally, calibration has been assessed with the Hosmer–Lemeshow statistics [5], which, although detecting overall miscalibration, does not indicate whether the model predicts more or less deaths than those observed. The combination with traditional calibration plots does not overcome this drawback. Indeed, not being formal statistical tools, these plots provide only very rough and potentially misleading information. The GiViTI calibration belt, instead, shows exactly in which direction and for which range of risk the model miscalibrates, as illustrated in Fig. 1 and explained below. The GiViTI calibration test integrates the calibration belt, providing a p value. It is, hence, a statistical test for overall calibration, as is the Hosmer–Lemeshow statistics, which, however, corresponds exactly to the calibration belts level of statistical significance [6]. Furthermore, in two extensive simulation analyses [4, 6], the GiViTI calibration test proved to perform well and, in many situations, better than the Hosmer–Lemeshow C test, which tends to reject the null hypothesis of good calibration more often than it should. The GiViTI calibration belt and test are available as the givitiR software package for R [7].

The role of the intensive care unit in real-time surveillance of emerging pandemics: the Italian GiViTI experience

The prompt availability of reliable epidemiological information on emerging pandemics is crucial for public health policy-makers. Early in 2013, a possible new H1N1 epidemic notified by an intensive care unit (ICU) to GiViTI, the Italian ICU network, prompted the re-activation of the real-time monitoring system developed during the 2009-2010 pandemic. Based on data from 216 ICUs, we were able to detect and monitor an outbreak of severe H1N1 infection, and to compare the situation with previous years. The timely and correct assessment of the severity of an epidemic can be obtained by investigating ICU admissions, especially when historical comparisons can be made.

Mid regional pro-adrenomedullin for the prediction of organ failure in infection. Results from a single centre study

Biomarkers are widely used to confirm the presence of infection. However, it would be of the greatest importance to predict in advance the occurrence or worsening of organ dysfunction in infected patients allowing timely antibiotic escalation. This study investigates the ability of procalcitonin (PCT) and MR-proADM to predict the transition to sepsis in infected patients. The study was conducted in a neurointensive care unit over a three-month period. We included both patients with and without infection to investigate the specificity of organ dysfunction prediction in infected patients. Daily measurement of PCT and MR-proADM, SOFA, Pitt, and CPIS were performed. To measure the correlation between each biomarker and each severity score, linear mixed-effects models were developed. For each biomarker-score combination we tested the correlation of the score with the biomarker measured one and two days before, the same day, and the day after. Sixty-four critically ill patients, 31 with infection, were enrolled. The statistically significant biomarker-score combinations were PCT-SOFA, MR-proADM-SOFA, MR-proADM-Pitt, and MR-proADM-CPIS. The MR-proADM models predicting Pitt and CPIS variations with 24-hour anticipation showed the best fit. The scores increased by 0.6 ± 0.3 and 0.4 ± 0.2 for each unitary biomarker increase, respectively. The MR-proADM-SOFA combinations were equivalent when the biomarker was measured the day before or the same day (score increases were 1.5 ± 0.4 and 1.9 ± 0.4, respectively). The PCT-SOFA model had the best fit when PCT was measured the same day of the score. There was no difference in the predictive ability of the biomarker in infected and non-infected patients. This was a pivotal study conducted in a single neurointensive centre on a limited number of patients, and as such it does not provide definitive conclusions. PR-proADM predicted occurrence and worsening of organ failure in critically ill patients with and without infection. The combination with infection diagnostic biomarkers such as PCT would allow predicting evolution to sepsis in infected patients.