Andrew J. Buckler

The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions

The development and implementation of quantitative imaging biomarkers has been hampered by the inconsistent and often incorrect use of terminology related to these markers. Sponsored by the Radiological Society of North America, an interdisciplinary group of radiologists, statisticians, physicists, and other researchers worked to develop a comprehensive terminology to serve as a foundation for quantitative imaging biomarker claims. Where possible, this working group adapted existing definitions derived from national or international standards bodies rather than invent new definitions for these terms. This terminology also serves as a foundation for the design of studies that evaluate the technical performance of quantitative imaging biomarkers and for studies of algorithms that generate the quantitative imaging biomarkers from clinical scans. This paper provides examples of research studies and quantitative imaging biomarker claims that use terminology consistent with these definitions as well as examples of the rampant confusion in this emerging field. We provide recommendations for appropriate use of quantitative imaging biomarker terminological concepts. It is hoped that this document will assist researchers and regulatory reviewers who examine quantitative imaging biomarkers and will also inform regulatory guidance. More consistent and correct use of terminology could advance regulatory science, improve clinical research, and provide better care for patients who undergo imaging studies.

A Collaborative Enterprise for Multi-Stakeholder Participation in the Advancement of Quantitative Imaging

Medical imaging has seen substantial and rapid technical advances during the past decade, including advances in image acquisition devices, processing and analysis software, and agents to enhance specificity. Traditionally, medical imaging has defined anatomy, but increasingly newer, more advanced, imaging technologies provide biochemical and physiologic information based on both static and dynamic modalities. These advanced technologies are important not only for detecting disease but for characterizing and assessing change of disease with time or therapy. Because of the rapidity of these advances, research to determine the utility of quantitative imaging in either clinical research or clinical practice has not had time to mature. Methods to appropriately develop, assess, regulate, and reimburse must be established for these advanced technologies. Efficient and methodical processes that meet the needs of stakeholders in the biomedical research community, therapeutics developers, and health care delivery enterprises will ultimately benefit individual patients. To help address this, the authors formed a collaborative program-the Quantitative Imaging Biomarker Alliance. This program draws from the very successful precedent set by the Integrating the Healthcare Enterprise effort but is adapted to the needs of imaging science. Strategic guidance supporting the development, qualification, and deployment of quantitative imaging biomarkers will lead to improved standardization of imaging tests, proof of imaging test performance, and greater use of imaging to predict the biologic behavior of tissue and monitor therapy response. These, in turn, confer value to corporate stakeholders, providing incentives to bring new and innovative products to market.

Quantitative imaging biomarkers: A review of statistical methods for computer algorithm comparisons

Quantitative biomarkers from medical images are becoming important tools for clinical diagnosis, staging, monitoring, treatment planning, and development of new therapies. While there is a rich history of the development of quantitative imaging biomarker (QIB) techniques, little attention has been paid to the validation and comparison of the computer algorithms that implement the QIB measurements. In this paper we provide a framework for QIB algorithm comparisons. We first review and compare various study designs, including designs with the true value (e.g. phantoms, digital reference images, and zero-change studies), designs with a reference standard (e.g. studies testing equivalence with a reference standard), and designs without a reference standard (e.g. agreement studies and studies of algorithm precision). The statistical methods for comparing QIB algorithms are then presented for various study types using both aggregate and disaggregate approaches. We propose a series of steps for establishing the performance of a QIB algorithm, identify limitations in the current statistical literature, and suggest future directions for research.

Quantitative Imaging Test Approval and Biomarker Qualification: Interrelated but Distinct Activities

UNLABELLED Quantitative imaging biomarkers could speed the development of new treatments for unmet medical needs and improve routine clinical care. However, it is not clear how the various regulatory and nonregulatory (eg, reimbursement) processes (often referred to as pathways) relate, nor is it clear which data need to be collected to support these different pathways most efficiently, given the time- and cost-intensive nature of doing so. The purpose of this article is to describe current thinking regarding these pathways emerging from diverse stakeholders interested and active in the definition, validation, and qualification of quantitative imaging biomarkers and to propose processes to facilitate the development and use of quantitative imaging biomarkers. A flexible framework is described that may be adapted for each imaging application, providing mechanisms that can be used to develop, assess, and evaluate relevant biomarkers. From this framework, processes can be mapped that would be applicable to both imaging product development and to quantitative imaging biomarker development aimed at increasing the effectiveness and availability of quantitative imaging. SUPPLEMENTAL MATERIAL http://radiology.rsna.org/lookup/suppl/doi:10.1148/radiol.10100800/-/DC1.

The Use of Volumetric CT as an Imaging Biomarker in Lung Cancer

RATIONALE AND OBJECTIVES Lung cancer is the leading cause of cancer death in the United States. Mortality outcomes have improved only modestly over the past 30 years. There is intense focus on the development of better treatments for lung cancer. Major issues include the cost and time duration of the clinical trials required to establish the utility of a drug so that it can be formally approved by regulatory agencies. In clinical settings, biomarkers that accelerate assessments of responses to treatment could benefit patients by providing earlier diagnoses of progressive disease, particularly when there are multiple options for treatment, and the effects of toxicity from one treatment tend to limit the ability to administer the next line of therapy. MATERIALS AND METHODS Quantifying longitudinal changes in tumor volumes using computed tomography could eventually become a more useful surrogate endpoint for assessing tumor responses or progression events than simple unidimensional measurements. RESULTS The authors review the historical development of response measurements in lung cancer, set out the medical context for specifying volumetric imaging requirements and goals, compare volumetric technique to conventional methods, and identify the imaging profiles being pursued. CONCLUSION The Quantitative Imaging Biomarkers Alliance is investigating volumetric computed tomographic acquisition and analytic methods to increase the analytic power per subject enrolled in clinical trials to reduce the number of total subjects needed or shorten the length of time an individual needs to be followed to reliably establish drug response.

Volumetric CT in Lung Cancer: An Example for the Qualification of Imaging as a Biomarker

RATIONALE AND OBJECTIVES New ways to understand biology as well as increasing interest in personalized treatments requires new capabilities for the assessment of therapy response. The lack of consensus methods and qualification evidence needed for large-scale multicenter trials, and in turn the standardization that allows them, are widely acknowledged to be the limiting factor in the deployment of qualified imaging biomarkers. MATERIALS AND METHODS The Quantitative Imaging Biomarker Alliance is organized to establish a methodology whereby multiple stakeholders collaborate. It has charged the Volumetric Computed Tomography (CT) Technical Subcommittee with investigating the technical feasibility and clinical value of quantifying changes over time in either volume or other parameters as biomarkers. The group selected solid tumors of the chest in subjects with lung cancer as its first case in point. Success is defined as sufficiently rigorous improvements in CT-based outcome measures to allow individual patients in clinical settings to switch treatments sooner if they are no longer responding to their current regimens, and reduce the costs of evaluating investigational new drugs to treat lung cancer. RESULTS The team has completed a systems engineering analysis, has begun a roadmap of experimental groundwork, documented profile claims and protocols, and documented a process for imaging biomarker qualification as a general paradigm for qualifying other imaging biomarkers as well. CONCLUSION This report addresses a procedural template for the qualification of quantitative imaging biomarkers. This mechanism is cost-effective for stakeholders while simultaneously advancing the public health by promoting the use of measures that prove effective.

Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example

Quantitative imaging biomarkers are being used increasingly in medicine to diagnose and monitor patients’ disease. The computer algorithms that measure quantitative imaging biomarkers have different technical performance characteristics. In this paper we illustrate the appropriate statistical methods for assessing and comparing the bias, precision, and agreement of computer algorithms. We use data from three studies of pulmonary nodules. The first study is a small phantom study used to illustrate metrics for assessing repeatability. The second study is a large phantom study allowing assessment of four algorithms’ bias and reproducibility for measuring tumor volume and the change in tumor volume. The third study is a small clinical study of patients whose tumors were measured on two occasions. This study allows a direct assessment of six algorithms’ performance for measuring tumor change. With these three examples we compare and contrast study designs and performance metrics, and we illustrate the advantages and limitations of various common statistical methods for quantitative imaging biomarker studies.

Meta-analysis of the technical performance of an imaging procedure: Guidelines and statistical methodology

Medical imaging serves many roles in patient care and the drug approval process, including assessing treatment response and guiding treatment decisions. These roles often involve a quantitative imaging biomarker, an objectively measured characteristic of the underlying anatomic structure or biochemical process derived from medical images. Before a quantitative imaging biomarker is accepted for use in such roles, the imaging procedure to acquire it must undergo evaluation of its technical performance, which entails assessment of performance metrics such as repeatability and reproducibility of the quantitative imaging biomarker. Ideally, this evaluation will involve quantitative summaries of results from multiple studies to overcome limitations due to the typically small sample sizes of technical performance studies and/or to include a broader range of clinical settings and patient populations. This paper is a review of meta-analysis procedures for such an evaluation, including identification of suitable studies, statistical methodology to evaluate and summarize the performance metrics, and complete and transparent reporting of the results. This review addresses challenges typical of meta-analyses of technical performance, particularly small study sizes, which often causes violations of assumptions underlying standard meta-analysis techniques. Alternative approaches to address these difficulties are also presented; simulation studies indicate that they outperform standard techniques when some studies are small. The meta-analysis procedures presented are also applied to actual [18F]-fluorodeoxyglucose positron emission tomography (FDG-PET) test–retest repeatability data for illustrative purposes.

Determining the Variability of Lesion Size Measurements from CT Patient Data Sets Acquired under "No Change" Conditions.

PURPOSE: To determine the variability of lesion size measurements in computed tomography data sets of patients imaged under a “no change” (“coffee break”) condition and to determine the impact of two reading paradigms on measurement variability. METHOD AND MATERIALS: Using data sets from 32 non-small cell lung cancer patients scanned twice within 15 minutes (“no change”), measurements were performed by five radiologists in two phases: (1) independent reading of each computed tomography dataset (timepoint): (2) a locked, sequential reading of datasets. Readers performed measurements using several sizing methods, including one-dimensional (1D) longest in-slice dimension and 3D semi-automated segmented volume. Change in size was estimated by comparing measurements performed on both timepoints for the same lesion, for each reader and each measurement method. For each reading paradigm, results were pooled across lesions, across readers, and across both readers and lesions, for each measurement method. RESULTS: The mean percent difference (± SD) when pooled across both readers and lesions for 1D and 3D measurements extracted from contours was 2.8 ± 22.2% and 23.4 ± 105.0%, respectively, for the independent reads. For the locked, sequential reads, the mean percent differences (± SD) reduced to 2.52 ± 14.2% and 7.4 ± 44.2% for the 1D and 3D measurements, respectively. CONCLUSION: Even under a “no change” condition between scans, there is variation in lesion size measurements due to repeat scans and variations in reader, lesion, and measurement method. This variation is reduced when using a locked, sequential reading paradigm compared to an independent reading paradigm.

Role of the Quantitative Imaging Biomarker Alliance in optimizing CT for the evaluation of lung cancer screen-detected nodules

The Quantitative Imaging Biomarker Alliance (QIBA) is a multidisciplinary consortium sponsored by the RSNA to define processes that enable the implementation and advancement of quantitative imaging methods described in a QIBA profile document that outlines the process to reliably and accurately measure imaging features. A QIBA profile includes factors such as technical (product-specific) standards, user activities, and relationship to a clinically meaningful metric, such as with nodule measurement in the course of CT screening for lung cancer. In this report, the authors describe how the QIBA approach is being applied to the measurement of small pulmonary nodules such as those found during low-dose CT-based lung cancer screening. All sources of variance with imaging measurement were defined for this process. Through a process of experimentation, literature review, and assembly of expert opinion, the strongest evidence was used to define how to best implement each step in the imaging acquisition and evaluation process. This systematic approach to implementing a quantitative imaging biomarker with standardized specifications for image acquisition and postprocessing for a specific quantitative measurement of a pulmonary nodule results in consistent performance characteristics of the measurement (eg, bias and variance). Implementation of the QIBA small nodule profile may allow more efficient and effective clinical management of the diagnostic workup of individuals found to have suspicious pulmonary nodules in the course of lung cancer screening evaluation.