Brandon D. Gallas

Channelized Hotelling observers for the assessment of volumetric imaging data sets

Current clinical practice is rapidly moving in the direction of volumetric imaging. For two-dimensional (2D) images, task-based medical image quality is often assessed using numerical model observers. For three-dimensional (3D) images, however, these models have been little explored so far. In this work, first, two novel designs of a multislice channelized Hotelling observer (CHO) are proposed for the task of detecting 3D signals in 3D images. The novel designs are then compared and evaluated in a simulation study with five different CHO designs: a single-slice model, three multislice models, and a volumetric model. Four different random background statistics are considered, both gaussian (noncorrelated and correlated gaussian noise) and non-gaussian (lumpy and clustered lumpy backgrounds). Overall, the results show that the volumetric model outperforms the others, while the disparity between the models decreases for greater complexity of the detection task. Among the multislice models, the second proposed CHO could most closely approach the volumetric model, whereas the first new CHO seems to be least affected by the number of training samples.

A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance Estimators

In this article, we analyze the three-way bootstrap estimate of the variance of the reader-averaged nonparametric area under the receiver operating characteristic (ROC) curve. The setting for this work is medical imaging, and the experimental design involves sampling from three distributions: a set of normal and diseased cases (patients), and a set of readers (doctors). The experiment we consider is fully crossed in that each reader reads each case. A reading generates a score that indicates the readers level of suspicion that the patient is diseased. The distribution of scores for the normal patients is compared to the distribution of scores for the diseased patients via an ROC curve, and the area under the ROC curve (AUC) summarizes the readers diagnostic ability to separate the normal patients from the diseased ones. We find that the bootstrap estimate of the variance of the reader-averaged AUC is biased, and we represent this bias in terms of moments of success outcomes. This representation helps unify and improve several current methods for multi-reader multi-case (MRMC) ROC analysis.

Effect of viewing angle on visual detection in liquid crystal displays

Display devices for medical diagnostic workstations should have a diffuse emission with apparent luminance independent of viewing angle. Such displays are called Lambertian, or they obey Lamberts law. Actual display devices are never truly Lambertian; the luminance of a pixel depends on the viewing angle. In active-matrix liquid crystal displays (AMLCD), the departure from the Lambertian profile depends on the gray level and complex pixel designs having multiple domains, in-plain switching or vertically-aligned technology. Our previous measurements established that the largest deviation from the desired Lambertian distribution occurs in the low luminance range for the diagonal viewing direction. Our purpose in this work is to determine the effect that non-uniform changes of the angular emission have on the detection of low-contrast signals in noisy backgrounds. We used a sequential two-alternative forced choice (2AFC) approach with test images displayed at the center of the screen. The observer location was fixed at different viewing angles: on-axis and off-axis. The results are expressed in terms of percent-correct for each observer and for each experimental condition (viewing angle and luminance). Our results show that for the test images used in this experiment with human observers, the changes in detectability between on-axis and off-axis viewing are smaller than the observer variability. Model observers are consistent with these results but also indicate that different background and signal levels can lead to meaningful performance differences between on-axis and off-axis viewing.

2008 Special Issue: Reader studies for validation of CAD systems

Evaluation of computational intelligence (CI) systems designed to improve the performance of a human operator is complicated by the need to include the effect of human variability. In this paper we consider human (reader) variability in the context of medical imaging computer-assisted diagnosis (CAD) systems, and we outline how to compare the detection performance of readers with and without the CAD. An effective and statistically powerful comparison can be accomplished with a receiver operating characteristic (ROC) experiment, summarized by the reader-averaged area under the ROC curve (AUC). The comparison requires sophisticated yet well-developed methods for multi-reader multi-case (MRMC) variance analysis. MRMC variance analysis accounts for random readers, random cases, and correlations in the experiment. In this paper, we extend the methods available for estimating this variability. Specifically, we present a method that can treat arbitrary study designs. Most methods treat only the fully-crossed study design, where every reader reads every case in two experimental conditions. We demonstrate our method with a computer simulation, and we assess the statistical power of a variety of study designs.

Three-Class ROC Analysis—Toward a General Decision Theoretic Solution

Multiclass receiver operating characteristic (ROC) analysis has remained an open theoretical problem since the introduction of binary ROC analysis in the 1950s. Previously, we have developed a paradigm for three-class ROC analysis that extends and unifies decision theoretic, linear discriminant analysis, and probabilistic foundations of binary ROC analysis in a three-class paradigm. One critical element in this paradigm is the equal error utility (EEU) assumption. This assumption allows us to reduce the intrinsic space of the three-class ROC analysis (5-D hypersurface in 6-D hyperspace) to a 2-D surface in the 3-D space of true positive fractions (sensitivity space). In this work, we show that this 2-D ROC surface fully and uniquely provides a complete descriptor for the optimal performance of a system for a three-class classification task, i.e., the triplet of likelihood ratio distributions, assuming such a triplet exists. To be specific, we consider two classifiers that utilize likelihood ratios, and we assumed each classifier has a continuous and differentiable 2-D sensitivity-space ROC surface. Under these conditions, we proved that the classifiers have the same triplet of likelihood ratio distributions if and only if they have the same 2-D sensitivity-space ROC surfaces. As a result, the 2-D sensitivity surface contains complete information on the optimal three-class task performance for the corresponding likelihood ratio classifier.

Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances

Abstract. Modeling and simulation are often used to understand and investigate random quantities and estimators. In 1997, Roe and Metz introduced a simulation model to validate analysis methods for the popular endpoint in reader studies to evaluate medical imaging devices, the reader-averaged area under the receiver operating characteristic (ROC) curve. Here, we generalize the notation of the model to allow more flexibility in recognition that variances of ROC ratings depend on modality and truth state. We also derive and validate equations for computing population variances and covariances for reader-averaged empirical AUC estimates under the generalized model. The equations are one-dimensional integrals that can be calculated using standard numerical integration techniques. This work provides the theoretical foundation and validation for a Java application called iRoeMetz that can simulate multireader multicase ROC studies and numerically calculate the corresponding variances and covariances of the empirical AUC. The iRoeMetz application and source code can be found at the “iMRMC” project on the google code project hosting site. These results and the application can be used by investigators to investigate ROC endpoints, validate analysis methods, and plan future studies.

Sensitivity of Time-Resolved Fluorescence Analysis Methods for Disease Detection

Time-resolved fluorescence (TRF) measurements of biological tissue provide chemical and structural information useful for detecting subtle disease processes, including cancers originating in mucosal tissues. A number of techniques for analyzing such TRF measurements exist, but they have not yet been compared to determine which of them can provide the greatest sensitivity to differences between a TRF measurement from nondiseased tissue and one from diseased tissue. We have evaluated four TRF analysis methods in this study: biexponential curve fitting, monoexponential curve fitting, Laguerre function representation, and computing the area under the decay curve (AUDC). We performed this study on a large dataset of computer-generated TRF decay curves based on colonic mucosa and typical measurement instrumentation. We statistically determined the minimum detectable change (MDC) in the relative contribution of each mucosal layer to the total TRF signal with each analysis method. We also determined the MDC in fluorescence lifetime of the upper mucosal layer. These two types of changes are due to the structural and biochemical changes expected in mucosa with onset of neoplasia. Under a wide range of baseline conditions, the monoexponential, AUDC, and Laguerre analysis methods all yield dramatically superior sensitivity and robustness over the standard biexponential method, with several caveats.

A method to estimate the point response function of digital x-ray detectors from edge measurements

Currently, the most accurate measurement of the detector point response can be performed with the pinhole method. The small size of the pinhole however, severely reduces the x-ray intensity output, requiring long exposures, something that can potentially reduce the x-ray tube life-cycle. Even though deriving the 1D Line Response Function (LRF)of the detector using the edge method is much more effcient, the measurement process introduces a convolution with a line, in addition to the common pixel sampling, effectively broadening the LRF. We propose a practical method to recover the detector point response function by removing the effects of the line and the pixel from a set of Edge Response Function (ERF) measurements. We use the imaging equation to study the effects of the edge,line and pixel measurements, and derive an analytical formula for the recovered detector point response function based on a gaussian mixture model. The method allows for limited recovery of asymmetries in the detector response function. We verify the method with pinhole and edge measurements of a digital flat panel detector. Monte Carlo simulations are also performed, using the MANTIS x-ray and optical photon and electron transport simulation package, for comparison. We show that the standard LRF underestimates the detector when compared with the recovered response. Our simulation results suggest that both hole methods for estimating the detector response have limitations in that they cannot completely capture rotational asymmetries or other morphological details smaller than the detector pixel size.

Evaluation environment for digital and analog pathology: a platform for validation studies

Abstract. We present a platform for designing and executing studies that compare pathologists interpreting histopathology of whole slide images (WSIs) on a computer display to pathologists interpreting glass slides on an optical microscope. eeDAP is an evaluation environment for digital and analog pathology. The key element in eeDAP is the registration of the WSI to the glass slide. Registration is accomplished through computer control of the microscope stage and a camera mounted on the microscope that acquires real-time images of the microscope field of view (FOV). Registration allows for the evaluation of the same regions of interest (ROIs) in both domains. This can reduce or eliminate disagreements that arise from pathologists interpreting different areas and focuses on the comparison of image quality. We reduced the pathologist interpretation area from an entire glass slide (10 to 30  mm2) to small ROIs (<50  μm2). We also made possible the evaluation of individual cells. We summarize eeDAP’s software and hardware and provide calculations and corresponding images of the microscope FOV and the ROIs extracted from the WSIs. The eeDAP software can be downloaded from the Google code website (project: eeDAP) as a MATLAB source or as a precompiled stand-alone license-free application.

Performance Studies for Validation of CAD Systems

Evaluation of computational intelligence (CI) systems designed to improve the performance of a human operator is complicated by the necessity of including the effect of human variability. In this paper we examine the methodology available for addressing this variability within the context of medical imaging computer-assisted diagnosis (CAD) systems. We present a review of currently available techniques and give an example using computer simulation. It is shown how advanced statistical techniques lead to more efficient measures of CAD performance.