Colin B. Begg

Assessment of Diagnostic Tests When Disease Verification is Subject to Selection Bias

In the assessment of the statistical properties of a diagnostic test, for example the sensitivity and specificity of the test, it is common to derive estimates from a sample limited to those cases for whom subsequent definitive disease verification is obtained. Omission of nonverified cases can seriously bias the estimates. In order to adjust the estimates it is necessary to make assumptions about the mechanism for selecting cases for verification. Methods for making the necessary adjustments can then be derived.

Clinical trials and drug toxicity in the elderly. The experience of the eastern cooperative oncology group

Nineteen studies of advanced cancer in 8 disease sites have been examined using data from the Eastern Cooperative Oncology Group. The purpose of the investigation was to determine susceptibility of elderly patients (<70 years of age) to cancer chemotherapy and to compare the results with corresponding figures in control patients (<70 years of age). The results indicate that in general, the elderly patients have identical rates of severe toxicity as their younger counterparts. The only exception is for hematologic reactions in a few of the sites studied. On closer examination, the agents that appear to be responsible for these especially adverse effects are methotrexate and methyl‐CCNU. It is demonstrated that the elderly patients have similar response rates and survival expectancy to the nonelderly patients. Consequently, it is concluded that the apparent discrimination in not treating elderly patients as aggressively as younger patients, and in excluding elderly patients from protocols, does not appear to be justified. Exclusions should be based on physiologic functional parameters, such as measures of renal, liver and marrow function, or performance status, rather than on an arbitrary age limit. Exceptions should only be made for agents which have a clearly demonstrated adverse effect on the elderly. Cancer 52:1986‐1992, 1983.

A General Regression Methodology for ROC Curve Estimation

A method for applying generalized ordinal regression models to categorical rating data to estimate and analyze receiver operating characteristic (ROC) curves is presented. These models permit parsimonious adjustment of ROC curve parameters for relevant covariates through two regression equations that correspond to location and scale. Particular shapes of ROC curves are interpreted in relation to the kind of covariates included in the two regressions. The model is shown to be flexible because it is not restricted to the assumption of binormality that is commonly employed in smoothed ROC curve estimation, although the binormal model is one particular form of the more general model. The new method provides a mechanism for pinpointing the effect that interobserver variability has on the ROC curve. It also allows for the adjustment of ROC curves for temporal variation and case mix, and provides a way to assess the incremental diagnostic value of a test. The new methodology is recommended because it substantially improves the ability to assess diagnostic tests using ROC curves. Key words: ROC curves; ordinal regression; technology assessment; diagnostic tests. (Med Decis Making 8:204-215, 1988)

High-dose ifosfamide with mesna uroprotection: a phase I study.

Phase II trials of ifosfamide have been performed with standard doses of 5 to 8 g/m2/course. In this phase I study, 29 patients were treated with a 4-day continuous infusion ifosfamide to determine the maximum-tolerated dose and the nonhematologic dose-limiting toxicity. Autologous bone marrow support was to have been used for the subsequent dose level if granulocytes were more than 500/microL for more than 14 days in two of two to five patients at a given dose level. Doses were escalated from 8 to 18 g/m2 ifosfamide. Mesna was given at an equivalent dose by continuous infusion for 5 days. At the 18 g/m2 dose level, dose-limiting renal insufficiency and a median of 11 days (range, 8 to 18 days) of granulocytopenia (less than 500/microL) were observed. Thus, autologous bone marrow reinfusion ws not used. The duration of myelosuppression, the frequency and severity of mucositis, and renal tubular acidosis were all dose-dependent. Mild to moderate CNS toxicity also appeared to be related to dose; however, severe CNS toxicity (transient confusion, hallucinations, and somnolence) was observed sporadically at both low- and high-dose levels. Transient hematuria (greater than 50 red blood cells [RBCs]/high power field) occurred once but did not affect treatment. There were nine responses (two complete) in 27 heavily pretreated assessable patients including seven responses in 20 patients with advanced refractory sarcoma. Ifosfamide with mesna uroprotection can undergo considerable dose escalation over the usual prescribed doses before nonhematologic dose-limiting toxicity is encountered. Ifosfamide has broad cytotoxicity against solid tumors and may prove to be an important addition to high-dose combination chemotherapy regimens.

Construction of Receiver Operating Characteristic Curves when Disease Verification Is Subject to Selection Bias

The estimates of ROC curves, which are frequently used in the assessment of diagnostic tests, may be biased if the sample is restricted to subjects whose disease status has been definitively verified. A method to provide an unbiased estimate of the ROC curve under these sample conditions is proposed. The new method requires information on the probability distribution of test results in the population from which the verified sample is drawn. It is illustrated using data from a study of computed tomography for fever of uncertain origin.

Assessment of diagnostic technologies. Methodology for unbiased estimation from samples of selectively verified patients

Concern with the efficacy of diagnostic technologies has stimulated numerous studies aimed at quantifying the discriminatory properties of various tests and procedures. These have focused principally on estimations of the result conditional probabilities, given disease status, eg, the sensitivity and specificity or the ROC curve. A source of bias in estimating these probabilities that is often unavoidable is created by the existence of a nonrandom selection mechanism for determining which patients initially tested will receive definitive verification of disease status. Correction for verification bias requires frequency data on the test results and any symptoms or other factors that influence selection for verification, both in the verified sample and in the source sample of patients tested.

Participation of community hospitals in clinical trials: analysis of five years of experience in the Eastern Cooperative Oncology Group.

The Eastern Cooperative Oncology Group (ECOG) initiated a program in 1976 to involve community hospitals in multi-institutional clinical trials. The community hospitals can be characterized as generally having no tradition of participating in clinical trials of cancer therapy, whereas the ECOG member institutions are university hospitals or major treatment centers. More than 100 community hospitals participated in 97 randomized trials involving 4506 patients from November 1976 through February 1981. Comparisons between the community hospitals and the ECOG member institutions indicate that the quality of participation was similar, as measured by rates of ineligibility, compliance with the protocol, and submission of data. Objective measures of outcome, such as survival, response, and toxicity, were also comparable. We conclude that under the mechanism adopted by the ECOG, it is possible to include community hospitals in clinical trials of cancer therapy without reducing the quality of the data or compromising the therapeutic outcomes.

Diverse prognosis in metastatic breast cancer: Who should be offered alternative initial therapies?

In an attempt to clarify appropriate treatment options for women with stage IV breast cancer, we studied the survival experience of a large dataset of patients treated on Cancer and Leukemia Group B (CALGB) protocols. The study, restricted to women who had had no prior chemotherapy for metastatic disease, demonstrated a surprisingly poor prognosis, with an estimated median survival of 1.6 years and only 26% alive at 3 years. Analysis of prognostic factors permitted the identification of subsets with even shorter survival, such as women with estrogen receptor negative tumor in more than one metastatic site and prior adjuvant chemotherapy. We feel that an evaluation of intensive investigational treatment approaches, such as trials using autologous bone marrow transplantation, is justified for most stage IV breast cancer patients, in view of their poor prognosis.

The influence of uninterpretability on the assessment of diagnostic tests.

A frequent problem faced by physicians utilizing diagnostic tests is the occurrence of uninterpretable test results. Such results, if they occur commonly, can seriously impair the diagnostic performance of the test. Moreover, in assessing the characteristics of the test, i.e. sensitivity, specificity, etc. failure to consider the impact of uninterpretability will artificially inflate the test characteristics. In this paper we explore the implications of this issue. We observe that a relevant factor is the potential repeatability of the test, i.e. whether the cause of uninterpretability is a transient phenomenon or an inherent property of the subject. We distinguish uninterpretable results, in which no result is obtained, from indeterminate results, in which the result is equivocal, or for which predisposing concomitant factors limit the interpretability of the result. Our results demonstrate that the naive approach of ignoring uninterpretable results in test assessments may indeed be unbiased in certain circumstances. However, if the cause of uninterpretability is related to disease status or to the potentially observable test result, then this approach will lead to bias. In either case, the frequency of uninterpretability is an important consideration in the cost-effectiveness of the test.

Methodology for the differential diagnosis of a complex data set. A case study using data from routine CT scan examinations.

Data from routine CT scan examinations are employed to illustrate the use of the polychotomous logistic regression model as a statistical diagnostic tool. The assumptions of the model, the interpretation of its parameters, and its capabilities are described in detail. In carrying out the analysis on the CT data, a large, relatively sparse data set, many technical difficulties were encountered. Modifications to the methodology that were necessary to permit its implementation are described, and it is demonstrated that an unbiased analysis of T + 1 diagnostic categories can be implemented by separately performing T individual simple logistic analyses. The limitations of the methodology are discussed. It is hoped that this paper may serve as a basis for the practical implementation of the polychotomous logistic model in similar diagnostic settings.