Vance W. Berger

Pros and cons of permutation tests in clinical trials.

Hypothesis testing, in which the null hypothesis specifies no difference between treatment groups, is an important tool in the assessment of new medical interventions. For randomized clinical trials, permutation tests that reflect the actual randomization are design-based analyses for such hypotheses. This means that only such design-based permutation tests can ensure internal validity, without which external validity is irrelevant. However, because of the conservatism of permutation tests, the virtues of permutation tests continue to be debated in the literature, and conclusions are generally of the type that permutation tests should always be used or permutation tests should never be used. A better conclusion might be that there are situations in which permutation tests should be used, and other situations in which permutation tests should not be used. This approach opens the door to broader agreement, but begs the obvious question of when to use permutation tests. We consider this issue from a variety of perspectives, and conclude that permutation tests are ideal to study efficacy in a randomized clinical trial which compares, in a heterogeneous patient population, two or more treatments, each of which may be most effective in some patients, when the primary analysis does not adjust for covariates. We propose the p-value interval as a novel measure of the conservatism of a permutation test that can be defined independently of the significance level. This p-value interval can be used to ensure that the permutation test have both good global power and an acceptable degree of conservatism.

Admissibility of exact conditional tests of stochastic order

Consider the problem of testing for independence against stochastic order in a 2 × J contingency table, with two treatments and J ordered categories. By conditioning on the margins, the null hypothesis becomes simple. Careful selection of the conditional alternative hypothesis then allows the problem to be formulated as one of a class of problems for which the minimal complete class of admissible tests is known. The exact versions of many common tests, such as t-tests and the Smirnov test, are shown to be inadmissible, and thus the non-randomized versions are overly conservative. The proportional hazards and proportional odds tests are shown to be admissible for a given data set and size. A new result allows a proof of the admissibility of convex hull and adaptive tests for all data sets and sizes.

Improving Tests for Superior Treatment in Contingency Tables

Abstract When comparing two treatments on the basis of ordinal data, a natural alternative is stochastic order, as it implies that more favorable outcomes receive greater probability. Results of Eaton can be used to provide a complete class of tests. We find that many tests in current use are not in this class and hence are inadmissible. More important, we present methods of improving such tests. Often, the increase in power is substantial although the size remains the same.

The bias of linear rank tests when testing for stochastic order in ordered categorical data

Abstract In many hypothesis testing problems, the alternative hypothesis is characterized by one or several restrictions arising from a natural ordering among the outcome levels. It is known that tests which ignore the ordering may lack adequate statistical power for the alternatives which are of the most interest. Considerably less, however, is known about developing tests which conform to a natural ordering. Stochastic order is an objective and compelling characterization of the superiority of one treatment over another. Consequently, testing for stochastic order is of considerable importance in applications involving the comparison of one treatment to another on the basis of ordered categorical data (e.g., in clinical trials). With few exceptions (that we identify), there is no optimal test for this problem, so it is reasonable to consider the class of tests that are simultaneously unbiased and admissible. There are known necessary and sufficient conditions for admissibility, but unbiasedness is more elusive. It is known that a necessary condition for unbiasedness is exactness conditionally on the margins. Further, one can construct a test which is both admissible and unbiased by repeatedly improving the trivially unbiased “ignore-the-data” test. This complicated approach, however, does not, in general, lead to a nested family of tests. We provide a new necessary condition for unbiasedness, and show that linear rank tests, which are widely used and locally most powerful, fail this condition for certain sets of margins. For such margins, linear rank tests are severely biased (with power as low as zero to detect certain alternatives of interest) and least stringent.

Can objective endpoints be manipulated in unmasked trials

Seldrup [1] wrote a very nice article discussing some of the complex issues involved in designing clinical trials to satisfy multiple regulators. I agree with almost the entire article, but letters must necessarily focus on the areas of disagreement, so that is what I must do. First, it seems clear that ‘intent to treat’ means all patients randomized. If there was no intent to treat them, then why were they randomized in the first place? Whether or not they actually were treated is an entirely different matter (see chapter 11 in [2]). Second, the importance of masking depends on whether we are looking for equivalence or superiority (see Section 15.2.8 in [2]). But my primary area of disagreement concerns the statement that masking was unnecessary because ‘the definition of a responder was sufficiently objective to avoid any bias in the evaluation’. Unfortunately, this is a widely held belief, and one that I have heard in many discussions, pretty much in every place I have been. The fact that it is demonstrably false has done little to reverse the popular thinking in this area. As a parallel, many researchers will argue (incorrectly) that randomized trials are beyond bias merely by virtue of being randomized. This parallel establishes that one always needs to be cautious in making such sweeping statements, so we could never rule out the possibility that unmasking could cause some bias, even if we cannot identify it, in the evaluation of even an objective endpoint such as mortality. But in this case, there are at least two such mechanisms that can be identified. First, consider the incentives to obtain certain results, and not others, coupled with the discretion that investigators enjoy. In an unmasked trial, the study treatments received by each patient would be known and could be used, subconsciously or otherwise, to pay more attention to the patients in one treatment group than in the other treatment group. This extra attention may lead to better medical decisions, plus a better frame of mind for the patient, and this may translate into stronger immune functions, which may then manifest in a difference even in an endpoint as objective as death, or objective response. Second, consider that unmasking, coupled with any restrictions on the randomization scheme, allows for prediction of future allocations and therefore selection bias [3]. This is forced confounding, which means that healthier patients (i.e., those destined to survive longer) will be disproportionately represented in one treatment group whereas sicker patients (i.e., those destined not to survive longer) will be disproportionately represented in the other treatment group. Clearly, this can bias the evaluation of any endpoint, subjective or not. One final point concerns the two-stage randomization design. Although in theory this is a very good idea, some colleagues and I encountered difficulties in coming up with valid analyses of the resulting data [4]. Clearly, this is not a criticism of the present paper but rather a call for more research into how to best evaluate the data coming out of such trials. Again, I congratulate Seldrup for a wonderful contribution to the clinical trials literature.