Eric Clarkson

Objective assessment of image quality. III. ROC metrics, ideal observers, and likelihood-generating functions

We continue the theme of previous papers [J. Opt. Soc. Am. A 7, 1266 (1990); 12, 834 (1995)] on objective (task-based) assessment of image quality. We concentrate on signal-detection tasks and figures of merit related to the ROC (receiver operating characteristic) curve. Many different expressions for the area under an ROC curve (AUC) are derived for an arbitrary discriminant function, with different assumptions on what information about the discriminant function is available. In particular, it is shown that AUC can be expressed by a principal-value integral that involves the characteristic functions of the discriminant. Then the discussion is specialized to the ideal observer, defined as one who uses the likelihood ratio (or some monotonic transformation of it, such as its logarithm) as the discriminant function. The properties of the ideal observer are examined from first principles. Several strong constraints on the moments of the likelihood ratio or the log likelihood are derived, and it is shown that the probability density functions for these test statistics are intimately related. In particular, some surprising results are presented for the case in which the log likelihood is normally distributed under one hypothesis. To unify these considerations, a new quantity called the likelihood-generating function is defined. It is shown that all moments of both the likelihood and the log likelihood under both hypotheses can be derived from this one function. Moreover, the AUC can be expressed, to an excellent approximation, in terms of the likelihood-generating function evaluated at the origin. This expression is the leading term in an asymptotic expansion of the AUC; it is exact whenever the likelihood-generating function behaves linearly near the origin. It is also shown that the likelihood-generating function at the origin sets a lower bound on the AUC in all cases.

T2 mapping from highly undersampled data by reconstruction of principal component coefficient maps using compressed sensing

Recently, there has been an increased interest in quantitative MR parameters to improve diagnosis and treatment. Parameter mapping requires multiple images acquired with different timings usually resulting in long acquisition times. While acquisition time can be reduced by acquiring undersampled data, obtaining accurate estimates of parameters from undersampled data is a challenging problem, in particular for structures with high spatial frequency content. In this work, principal component analysis is combined with a model‐based algorithm to reconstruct maps of selected principal component coefficients from highly undersampled radial MRI data. This novel approach linearizes the cost function of the optimization problem yielding a more accurate and reliable estimation of MR parameter maps. The proposed algorithm—reconstruction of principal component coefficient maps using compressed sensing—is demonstrated in phantoms and in vivo and compared with two other algorithms previously developed for undersampled data. Magn Reson Med, 2012.

Objective Assessment of Image Quality

Small-animal imaging has shown great promise in the areas of oncology,cardiology, molecular biology,drug discovery and development, and genetics [Green, 2001]. The demand for small-animal imaging has increased greatly with the recent advances in biomolecular research. Examples include functional genomics, functional protenomics, and molecular targeting of tumor cells or cells with other abnormalities. SPECT and PET imaging are of particular interest because these systems intrinsically image function instead of anatomy. Traditional thought has precluded SPECT and PET imaging of small animals because of the lack of resolution of these systems.In recent years at CGRI and other research facilities, numerous fast, high-resolution and high sensitivity SPECT imaging systems designed specifically for imaging small animals have been developed. These systems produce three-dimensional images of the distribution of radiotracers within the animal and, because these systems have no moving parts, dynamic studies can be readily performed. However, there are many components of such imaging systems that need to be optimized in order to best perform small-animal imaging studies.

Efficiency of the human observer detecting random signals in random backgrounds

The efficiencies of the human observer and the channelized-Hotelling observer relative to the ideal observer for signal-detection tasks are discussed. Both signal-known-exactly (SKE) tasks and signal-known-statistically (SKS) tasks are considered. Signal location is uncertain for the SKS tasks, and lumpy backgrounds are used for background uncertainty in both cases. Markov chain Monte Carlo methods are employed to determine ideal-observer performance on the detection tasks. Psychophysical studies are conducted to compute human-observer performance on the same tasks. Efficiency is computed as the squared ratio of the detectabilities of the observer of interest to the ideal observer. Human efficiencies are approximately 2.1% and 24%, respectively, for the SKE and SKS tasks. The results imply that human observers are not affected as much as the ideal observer by signal-location uncertainty even though the ideal observer outperforms the human observer for both tasks. Three different simplified pinhole imaging systems are simulated, and the humans and the model observers rank the systems in the same order for both the SKE and the SKS tasks.

Objective comparison of quantitative imaging modalities without the use of a gold standard

Imaging is often used for the purpose of estimating the value of some parameter of interest. For example, a cardiologist may measure the ejection fraction (EF) of the heart in order to know how much blood is being pumped out of the heart on each stroke. In clinical practice, however, it is difficult to evaluate an estimation method because the gold standard is not known, e.g., a cardiologist does not know the true EF of a patient. Thus, researchers have often evaluated an estimation method by plotting its results against the results of another (more accepted) estimation method, which amounts to using one set of estimates as the pseudogold standard. In this paper, we present a maximum-likelihood approach for evaluating and comparing different estimation methods without the use of a gold standard with specific emphasis on the problem of evaluating EF estimation methods. Results of numerous simulation studies will be presented and indicate that the method can precisely and accurately estimate the parameters of a regression line without a gold standard, i.e., without the x axis.

Estimation receiver operating characteristic curve and ideal observers for combined detection/estimation tasks

The localization receiver operating characteristic (LROC) curve is a standard method to quantify performance for the task of detecting and locating a signal. This curve is generalized to arbitrary detection/estimation tasks to give the estimation ROC (EROC) curve. For a two-alternative forced-choice study, where the observer must decide which of a pair of images has the signal and then estimate parameters pertaining to the signal, it is shown that the average value of the utility on those image pairs where the observer chooses the correct image is an estimate of the area under the EROC curve (AEROC). The ideal LROC observer is generalized to the ideal EROC observer, whose EROC curve lies above those of all other observers for the given detection/estimation task. When the utility function is nonnegative, the ideal EROC observer is shown to share many mathematical properties with the ideal observer for the pure detection task. When the utility function is concave, the ideal EROC observer makes use of the posterior mean estimator. Other estimators that arise as special cases include maximum a posteriori estimators and maximum-likelihood estimators.

Estimating random signal parameters from noisy images with nuisance parameters: linear and scanning-linear methods

In a pure estimation task, an object of interest is known to be present, and we wish to determine numerical values for parameters that describe the object. This paper compares the theoretical framework, implementation method, and performance of two estimation procedures. We examined the performance of these estimators for tasks such as estimating signal location, signal volume, signal amplitude, or any combination of these parameters. The signal is embedded in a random background to simulate the effect of nuisance parameters. First, we explore the classical Wiener estimator, which operates linearly on the data and minimizes the ensemble mean-squared error. The results of our performance tests indicate that the Wiener estimator can estimate amplitude and shape once a signal has been located, but is fundamentally unable to locate a signal regardless of the quality of the image. Given these new results on the fundamental limitations of Wiener estimation, we extend our methods to include more complex data processing. We introduce and evaluate a scanning-linear estimator that performs impressively for location estimation. The scanning action of the estimator refers to seeking a solution that maximizes a linear metric, thereby requiring a global-extremum search. The linear metric to be optimized can be derived as a special case of maximum a posteriori (MAP) estimation when the likelihood is Gaussian and a slowly varying covariance approximation is made.

Processing of radial fast spin-echo data for obtaining T2 estimates from a single k-space data set.

Radially acquired fast spin‐echo data can be processed to obtain T2‐weighted images and a T2 map from a single k‐space data set. The general approach is to use data at a specific TE (or narrow TE range) in the center of k‐space and data at other TE values in the outer part of k‐space. With this method high‐resolution T2‐weighted images and T2 maps are obtained in a time efficient manner. The mixing of TE data, however, introduces errors in the T2‐weighted images and T2 maps that affect the accuracy of the T2 estimates. In this work, various k‐space data processing methods for reconstructing T2‐weighted images and T2 maps from a single radial fast spin‐echo k‐space data set are analyzed in terms of the accuracy of T2 estimates. The analysis is focused on the effect of image artifacts, object dependency, and noise on the T2 estimates. Results are presented in computer‐generated phantoms and in vivo. Magn Reson Med, 2005.

Channelized-ideal observer using Laguerre-Gauss channels in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal

We investigate a channelized-ideal observer (CIO) with Laguerre-Gauss (LG) channels to approximate ideal-observer performance in detection tasks involving non-Gaussian distributed lumpy backgrounds and a Gaussian signal. A Markov-chain Monte Carlo approach is employed to determine the performance of both the ideal observer and the CIO using a large number of LG channels. Our results indicate that the CIO with LG channels can approximate ideal-observer performance within error bars, depending on the imaging system, object, and channel parameters. The CIO also outperforms a channelized-Hotelling observer using the same channels. In addition, an alternative approach for estimating the CIO is investigated. This approach makes use of the characteristic functions of channelized data and employs an approximation method to the area under the receiver operating characteristic curve. The alternative approach provides good estimates of the performance of the CIO with five LG channels. However, for large channel cases, more efficient computational methods need to be developed for the CIO to become useful in practice.

Megalopinakophobia: its symptoms and cures

This paper addresses issues in the calculation of a detectability measure for the ideal linear (Hotelling) observer performing a detection task on a digital radiograph. The main computational problem is that the inverse of a very large covariance matrix is required. The conventional approach is to assume some form of stationarity and argue that the matrix is diagonalized by discrete Fourier transformation, but there are many reasons why this assumption is unrealistic. After a brief review of the underlying mathematics, we present several practical algorithms for computing the detectability and some hints as to when each is applicable. The main conclusion is that large matrices should not be feared.