Adam Wunderlich

Utility as a rationale for choosing observer performance assessment paradigms for detection tasks in medical imaging

PURPOSEnStudies of lesion detectability are often carried out to evaluate medical imaging technology. For such studies, several approaches have been proposed to measure observer performance, such as the receiver operating characteristic (ROC), the localization ROC (LROC), the free-response ROC (FROC), the alternative free-response ROC (AFROC), and the exponentially transformed FROC (EFROC) paradigms. Therefore, an experimenter seeking to carry out such a study is confronted with an array of choices. Traditionally, arguments for different approaches have been made on the basis of practical considerations (statistical power, etc.) or the gross level of analysis (case-level or lesion-level). This article contends that a careful consideration of utility should form the rationale for matching the assessment paradigm to the clinical task of interest.nnnMETHODSnIn utility theory, task performance is commonly evaluated with total expected utility, which integrates the various event utilities against the probability of each event. To formalize the relationship between expected utility and the summary curve associated with each assessment paradigm, the concept of a natural utility structure is proposed. A natural utility structure is defined for a summary curve when the variables associated with the summary curve axes are sufficient for computing total expected utility, assuming that the disease prevalence is known.nnnRESULTSnNatural utility structures for ROC, LROC, FROC, AFROC, and EFROC curves are introduced, clarifying how the utilities of correct and incorrect decisions are aggregated by summary curves. Further, conditions are given under which general utility structures for localization-based methodologies reduce to case-based assessment.nnnCONCLUSIONSnOverall, the findings reveal how summary curves correspond to natural utility structures of diagnostic tasks, suggesting utility as a motivating principle for choosing an assessment paradigm.

Seamless Insertion of Pulmonary Nodules in Chest CT Images

The availability of large medical image datasets is critical in many applications, such as training and testing of computer-aided diagnosis systems, evaluation of segmentation algorithms, and conducting perceptual studies. However, collection of data and establishment of ground truth for medical images are both costly and difficult. To address this problem, we are developing an image blending tool that allows users to modify or supplement existing datasets by seamlessly inserting a lesion extracted from a source image into a target image. In this study, we focus on the application of this tool to pulmonary nodules in chest CT exams. We minimize the impact of user skill on the perceived quality of the composite image by limiting user involvement to two simple steps: the user first draws a casual boundary around a nodule in the source, and, then, selects the center of desired insertion area in the target. We demonstrate the performance of our system on clinical samples, and report the results of a reader study evaluating the realism of inserted nodules compared to clinical nodules. We further evaluate our image blending techniques using phantoms simulated under different noise levels and reconstruction filters. Specifically, we compute the area under the ROC curve of the Hotelling observer (HO) and noise power spectrum of regions of interest enclosing native and inserted nodules, and compare the detectability, noise texture, and noise magnitude of inserted and native nodules. Our results indicate the viability of our approach for insertion of pulmonary nodules in clinical CT images.

Seamless insertion of real pulmonary nodules in chest CT exams

The availability of large medical image datasets is critical in many applications such as training and testing of computer aided diagnosis (CAD) systems, evaluation of segmentation algorithms, and conducting perceptual studies. However, collection of large repositories of clinical images is hindered by the high cost and difficulties associated with both the accumulation of data and establishment of the ground truth. To address this problem, we are developing an image blending tool that allows users to modify or supplement existing datasets by seamlessly inserting a real lesion extracted from a source image into a different location on a target image. In this study we focus on the application of this tool to pulmonary nodules in chest CT exams. We minimize the impact of user skill on the perceived quality of the blended image by limiting user involvement to two simple steps: the user first draws a casual boundary around the nodule of interest in the source, and then selects the center of desired insertion area in the target. We demonstrate examples of the performance of the proposed system on samples taken from the Lung Image Database Consortium (LIDC) dataset, and compare the noise power spectrum (NPS) of blended nodules versus that of native nodules in simulated phantoms.

Equivalent isotropic response as a surrogate for incident field strength

The strength of an electromagnetic plane wave incident in the free field can be characterized in terms of power output by an idealized isotropic antenna probe. We refer to the parameter as equivalent isotropic incident power (EIIP), though it lacks an accepted name. This parameter has begun to enter use in various industry standards, technical reports, and peer-reviewed papers. To our knowledge, however, it has not been defined or studied in detail by prior work. We start to address this gap here with a proposed a definition, physical interpretation, and comparison to field strength.

A utility/cost analysis of breast cancer risk prediction algorithms

Breast cancer risk prediction algorithms are used to identify subpopulations that are at increased risk for developing breast cancer. They can be based on many different sources of data such as demographics, relatives with cancer, gene expression, and various phenotypic features such as breast density. Women who are identified as high risk may undergo a more extensive (and expensive) screening process that includes MRI or ultrasound imaging in addition to the standard full-field digital mammography (FFDM) exam. Given that there are many ways that risk prediction may be accomplished, it is of interest to evaluate them in terms of expected cost, which includes the costs of diagnostic outcomes. In this work we perform an expected-cost analysis of risk prediction algorithms that is based on a published model that includes the costs associated with diagnostic outcomes (true-positive, false-positive, etc.). We assume the existence of a standard screening method and an enhanced screening method with higher scan cost, higher sensitivity, and lower specificity. We then assess expected cost of using a risk prediction algorithm to determine who gets the enhanced screening method under the strong assumption that risk and diagnostic performance are independent. We find that if risk prediction leads to a high enough positive predictive value, it will be cost-effective regardless of the size of the subpopulation. Furthermore, in terms of the hit-rate and false-alarm rate of the of the risk prediction algorithm, iso-cost contours are lines with slope determined by properties of the available diagnostic systems for screening.

Optimal Joint Detection and Estimation That Maximizes ROC-Type Curves.

Combined detection-estimation tasks are frequently encountered in medical imaging. Optimal methods for joint detection and estimation are of interest because they provide upper bounds on observer performance, and can potentially be utilized for imaging system optimization, evaluation of observer efficiency, and development of image formation algorithms. We present a unified Bayesian framework for decision rules that maximize receiver operating characteristic (ROC)-type summary curves, including ROC, localization ROC (LROC), estimation ROC (EROC), free-response ROC (FROC), alternative free-response ROC (AFROC), and exponentially-transformed FROC (EFROC) curves, succinctly summarizing previous results. The approach relies on an interpretation of ROC-type summary curves as plots of an expected utility versus an expected disutility (or penalty) for signal-present decisions. We propose a general utility structure that is flexible enough to encompass many ROC variants and yet sufficiently constrained to allow derivation of a linear expected utility equation that is similar to that for simple binary detection. We illustrate our theory with an example comparing decision strategies for joint detection-estimation of a known signal with unknown amplitude. In addition, building on insights from our utility framework, we propose new ROC-type summary curves and associated optimal decision rules for joint detection-estimation tasks with an unknown, potentially-multiple, number of signals in each observation.

Approximate maximum likelihood estimation of scanning observer templates

In localization tasks, an observer is asked to give the location of some target or feature of interest in an image. Scanning linear observer models incorporate the search implicit in this task through convolution of an observer template with the image being evaluated. Such models are becoming increasingly popular as predictors of human performance for validating medical imaging methodology. In addition to convolution, scanning models may utilize internal noise components to model inconsistencies in human observer responses. In this work, we build a probabilistic mathematical model of this process and show how it can, in principle, be used to obtain estimates of the observer template using maximum likelihood methods. The main difficulty of this approach is that a closed form probability distribution for a maximal location response is not generally available in the presence of internal noise. However, for a given image we can generate an empirical distribution of maximal locations using Monte-Carlo sampling. We show that this probability is well approximated by applying an exponential function to the scanning template output. We also evaluate log-likelihood functions on the basis of this approximate distribution. Using 1,000 trials of simulated data as a validation test set, we find that a plot of the approximate log-likelihood function along a single parameter related to the template profile achieves its maximum value near the true value used in the simulation. This finding holds regardless of whether the trials are correctly localized or not. In a second validation study evaluating a parameter related to the relative magnitude of internal noise, only the incorrect localization images produces a maximum in the approximate log-likelihood function that is near the true value of the parameter.

MRMC analysis of agreement studies

The purpose of this work is to present and evaluate methods based on U-statistics to compare intra- or inter-reader agreement across different imaging modalities. We apply these methods to multi-reader multi-case (MRMC) studies. We measure reader-averaged agreement and estimate its variance accounting for the variability from readers and cases (an MRMC analysis). In our application, pathologists (readers) evaluate patient tissue mounted on glass slides (cases) in two ways. They evaluate the slides on a microscope (reference modality) and they evaluate digital scans of the slides on a computer display (new modality). In the current work, we consider concordance as the agreement measure, but many of the concepts outlined here apply to other agreement measures. Concordance is the probability that two readers rank two cases in the same order. Concordance can be estimated with a U-statistic and thus it has some nice properties: it is unbiased, asymptotically normal, and its variance is given by an explicit formula. Another property of a U-statistic is that it is symmetric in its inputs; it doesnt matter which reader is listed first or which case is listed first, the result is the same. Using this property and a few tricks while building the U-statistic kernel for concordance, we get a mathematically tractable problem and efficient software. Simulations show that our variance and covariance estimates are unbiased.

Nonparametric EROC analysis for observer performance evaluation on joint detection and estimation tasks

The majority of the literature on task-based image quality assessment has focused on lesion detection tasks, using the receiver operating characteristic (ROC) curve, or related variants, to measure performance. However, since many clinical image evaluation tasks involve both detection and estimation (e.g., estimation of kidney stone composition, estimation of tumor size), there is a growing interest in performance evaluation for joint detection and estimation tasks. To evaluate observer performance on such tasks, Clarkson introduced the estimation ROC (EROC) curve, and the area under the EROC curve as a summary figure of merit. In the present work, we propose nonparametric estimators for practical EROC analysis from experimental data, including estimators for the area under the EROC curve and its variance. The estimators are illustrated with a practical example comparing MRI images reconstructed from different k-space sampling trajectories.

eeDAP: an evaluation environment for digital and analog pathology

Purpose: The purpose of this work is to present a platform for designing and executing studies that compare pathologists interpreting histopathology of whole slide images (WSI) on a computer display to pathologists interpreting glass slides on an optical microscope. Methods: Here we present eeDAP, an evaluation environment for digital and analog pathology. The key element in eeDAP is the registration of theWSI to the glass slide. Registration is accomplished through computer control of the microscope stage and a camera mounted on the microscope that acquires images of the real time microscope view. Registration allows for the evaluation of the same regions of interest (ROIs) in both domains. This can reduce or eliminate disagreements that arise from pathologists interpreting different areas and focuses the comparison on image quality. Results: We reduced the pathologist interpretation area from an entire glass slide (≈10-30 mm)2 to small ROIs <(50 um)2. We also made possible the evaluation of individual cells. Conclusions: We summarize eeDAP’s software and hardware and provide calculations and corresponding images of the microscope field of view and the ROIs extracted from the WSIs. These calculations help provide a sense of eeDAP’s functionality and operating principles, while the images provide a sense of the look and feel of studies that can be conducted in the digital and analog domains. The eeDAP software can be downloaded from code.google.com (project: eeDAP) as Matlab source or as a precompiled stand-alone license-free application.