Allison Meisner

Methodological issues in current practice may lead to bias in the development of biomarker combinations for predicting acute kidney injury

Individual biomarkers of renal injury are only modestly predictive of acute kidney injury (AKI). Using multiple biomarkers has the potential to improve predictive capacity. In this systematic review, statistical methods of articles developing biomarker combinations to predict acute kidney injury were assessed. We identified and described three potential sources of bias (resubstitution bias, model selection bias and bias due to center differences) that may compromise the development of biomarker combinations. Fifteen studies reported developing kidney injury biomarker combinations for the prediction of AKI after cardiac surgery (8 articles), in the intensive care unit (4 articles) or other settings (3 articles). All studies were susceptible to at least one source of bias and did not account for or acknowledge the bias. Inadequate reporting often hindered our assessment of the articles. We then evaluated, when possible (7 articles), the performance of published biomarker combinations in the TRIBE-AKI cardiac surgery cohort. Predictive performance was markedly attenuated in six out of seven cases. Thus, deficiencies in analysis and reporting are avoidable and care should be taken to provide accurate estimates of risk prediction model performance. Hence, rigorous design, analysis and reporting of biomarker combination studies are essential to realizing the promise of biomarkers in clinical practice.

RiGoR: reporting guidelines to address common sources of bias in risk model development

Reviewing the literature in many fields on proposed risk models reveals problems with the way many risk models are developed. Furthermore, papers reporting new risk models do not always provide sufficient information to allow readers to assess the merits of the model. In this review, we discuss sources of bias that can arise in risk model development. We focus on two biases that can be introduced during data analysis. These two sources of bias are sometimes conflated in the literature and we recommend the terms resubstitution bias and model-selection bias to delineate them. We also propose the RiGoR reporting standard to improve transparency and clarity of published papers proposing new risk models.

Validity of a neurological scoring system for canine X-linked myotubular myopathy.

A simple clinical neurological test was developed to evaluate response to gene therapy in a preclinical canine model of X-linked myotubular myopathy (XLMTM). This devastating congenital myopathy is caused by mutation in the myotubularin (MTM1) gene. Clinical signs include muscle weakness, early respiratory failure, and ventilator dependence. A spontaneously occurring canine model has a similar clinical picture and histological abnormalities on muscle biopsy compared with patients. We developed a neuromuscular assessment score, graded on a scale from 10 (normal) to 1 (unable to maintain sternal recumbency). We hypothesize that this neurological assessment score correlates with genotype and established measures of disease severity and is reliable when performed by an independent observer. At 17 weeks of age, there was strong correlation between neurological assessment scores and established methods of severity testing. The neurological severity score correctly differentiated between XLMTM and wild-type dogs with good interobserver reliability, on the basis of strong agreement between neurological scores assigned by independent observers. Together, these data indicate that the neurological scoring system developed for this canine congenital neuromuscular disorder is reliable and valid. This scoring system may be helpful in evaluating response to therapy in preclinical testing in this disease model, such as response to gene therapy.

Evaluating biomarkers for prognostic enrichment of clinical trials

Background/Aims: A potential use of biomarkers is to assist in prognostic enrichment of clinical trials, where only patients at relatively higher risk for an outcome of interest are eligible for the trial. We investigated methods for evaluating biomarkers for prognostic enrichment. Methods: We identified five key considerations when considering a biomarker and a screening threshold for prognostic enrichment: (1) clinical trial sample size, (2) calendar time to enroll the trial, (3) total patient screening costs and the total per-patient trial costs, (4) generalizability of trial results, and (5) ethical evaluation of trial eligibility criteria. Items (1)–(3) are amenable to quantitative analysis. We developed the Biomarker Prognostic Enrichment Tool for evaluating biomarkers for prognostic enrichment at varying levels of screening stringency. Results: We demonstrate that both modestly prognostic and strongly prognostic biomarkers can improve trial metrics using Biomarker Prognostic Enrichment Tool. Biomarker Prognostic Enrichment Tool is available as a webtool at http://prognosticenrichment.com and as a package for the R statistical computing platform. Conclusion: In some clinical settings, even biomarkers with modest prognostic performance can be useful for prognostic enrichment. In addition to the quantitative analysis provided by Biomarker Prognostic Enrichment Tool, investigators must consider the generalizability of trial results and evaluate the ethics of trial eligibility criteria.

Urinalysis findings and urinary kidney injury biomarker concentrations

IntroductionUrinary biomarkers of kidney injury are presumed to reflect renal tubular damage. However, their concentrations may be influenced by other factors, such as hematuria or pyuria. We sought to examine what non-injury related urinalysis factors are associated with urinary biomarker levels.MethodsWe examined 714 adults who underwent cardiac surgery in the TRIBE-AKI cohort that did not experience post-operative clinical AKI (patients with serum creatinine change of ≥ 20% were excluded). We examined the association between urinalysis findings and the pre- and first post-operative urinary concentrations of 4 urinary biomarkers: neutrophil gelatinase-associated lipocalin (NGAL), interleukin-18 (IL-18), kidney injury molecule-1 (KIM-1), and liver fatty acid binding protein (L-FABP).ResultsThe presence of leukocyte esterase and nitrites on urinalysis was associated with increased urinary NGAL (R2 0.16, p < 0.001 and R2 0.07, p < 0.001, respectively) in pre-operative samples. Hematuria was associated with increased levels of all 4 biomarkers, with a much stronger association seen in post-operative samples (R2 between 0.02 and 0.21). Dipstick proteinuria concentrations correlated with levels of all 4 urinary biomarkers in pre-operative and post-operative samples (R2 between 0.113 and 0.194 in pre-operative and between 0.122 and 0.322 in post-operative samples). Adjusting the AUC of post-operative AKI for dipstick proteinuria lowered the AUC for all 4 biomarkers at the pre-operative time point and for 2 of the 4 biomarkers at the post-operative time point.ConclusionsSeveral factors available through urine dipstick testing are associated with increased urinary biomarker concentrations that are independent of clinical kidney injury. Future studies should explore the impact of these factors on the prognostic and diagnostic performance of these AKI biomarkers.

Using ordinal outcomes to construct and select biomarker combinations for single-level prediction

BackgroundBiomarker studies may involve an ordinal outcome, such as no, mild, or severe disease. There is often interest in predicting one particular level of the outcome due to its clinical significance.MethodsA simple approach to constructing biomarker combinations in this context involves dichotomizing the outcome and using a binary logistic regression model. We assessed whether more sophisticated methods offer advantages over this simple approach. It is often necessary to select among several candidate biomarker combinations. One strategy involves selecting a combination based on its ability to predict the outcome level of interest. We propose an algorithm that leverages the ordinal outcome to inform combination selection. We apply this algorithm to data from a study of acute kidney injury after cardiac surgery, where kidney injury may be absent, mild, or severe.ResultsUsing more sophisticated modeling approaches to construct combinations provided gains over the simple binary logistic regression approach in specific settings. In the examples considered, the proposed algorithm for combination selection tended to reduce the impact of bias due to selection and to provide combinations with improved performance.ConclusionsMethods that utilize the ordinal nature of the outcome in the construction and/or selection of biomarker combinations have the potential to yield better combinations.

Biomarker Combinations for Diagnosis and Prognosis in Multicenter Studies: Principles and Methods

Many investigators are interested in combining biomarkers to predict a binary outcome or detect underlying disease. This endeavor is complicated by the fact that many biomarker studies involve data from multiple centers. Depending upon the relationship between center, the biomarkers, and the target of prediction, care must be taken when constructing and evaluating combinations of biomarkers. We introduce a taxonomy to describe the role of center and consider how a biomarker combination should be constructed and evaluated. We show that ignoring center, which is frequently done by clinical researchers, is often not appropriate. The limited statistical literature proposes using random intercept logistic regression models, an approach that we demonstrate is generally inadequate and may be misleading. We instead propose using fixed intercept logistic regression, which appropriately accounts for center without relying on untenable assumptions. After constructing the biomarker combination, we recommend using performance measures that account for the multicenter nature of the data, namely the center-adjusted area under the receiver operating characteristic curve. We apply these methods to data from a multicenter study of acute kidney injury after cardiac surgery. Appropriately accounting for center, both in construction and evaluation, may increase the likelihood of identifying clinically useful biomarker combinations.