Joseph F. Heyse

Analyzing multiple endpoints in clinical trials of pain treatments: IMMPACT recommendations

Abstract The increasing complexity of randomized clinical trials and the practice of obtaining a wide variety of measurements from study participants have made the consideration of multiple endpoints a critically important issue in the design, analysis, and interpretation of clinical trials. Failure to consider important outcomes can limit the validity and utility of clinical trials; specifying multiple endpoints for the evaluation of treatment efficacy, however, can increase the rate of false positive conclusions about the efficacy of a treatment. We describe the use of multiple endpoints in the design, analysis, and interpretation of pain clinical trials, and review available strategies and methods for addressing multiplicity. To decrease the probability of a Type I error (i.e., the likelihood of obtaining statistically significant results by chance) in pain clinical trials, the use of gatekeeping procedures and other methods that correct for multiple analyses is recommended when a single primary endpoint does not adequately reflect the overall benefits of treatment. We emphasize the importance of specifying in advance the outcomes and clinical decision rule that will serve as the basis for determining that a treatment is efficacious and the methods that will be used to control the overall Type I error rate.

Inverse relationship between six week postvaccination varicella antibody response to vaccine and likelihood of long term breakthrough infection

Background. We used the large clinical database that supported the development of Oka/Merck varicella vaccine to study the relationship between the primary varicella antibody response, as determined by gpELISA, an enzyme-linked immunosorbent assay that detects antibodies to varicella-zoster virus (VZV) glycoprotein, and the subsequent risk of postvaccination breakthrough varicella. Methods. We vaccinated 1164 healthy children with a single dose of varicella vaccine containing 2900 to 9000 plaque-forming units/dose. The primary immune response to vaccination was determined by gpELISA 6 weeks after vaccination. Subjects were followed annually for 7 years to ascertain cases of breakthrough varicella. Results. The estimated vaccine efficacy among children with a 6-week postvaccination antibody titer of ≥5 gpELISA units was 95.5% (95% confidence interval, 94.2%, 96.8%) compared with 83.5% (95% confidence interval, 76.9%, 89.5%) for subjects with a titer of <5 gpELISA units. Children with a 6-week postvaccination antibody titer of <5 gpELISA units were 3.5 times more likely than those with a titer of ≥5 gpELISA units to develop breakthrough varicella. Conclusions. We identified a 6-week postvaccination antibody titer of ≥5 gpELISA units as an approximate correlate of protection. In addition we established an accelerated failure time model based on log normal hazard that predicted varicella breakthrough rates based on the distribution of 6-week postvaccination varicella antibody titers.

Use of the false discovery rate for evaluating clinical safety data

Clinical adverse experience (AE) data are routinely evaluated using between group P values for every AE encountered within each of several body systems. If the P values are reported and interpreted without multiplicity considerations, there is a potential for an excess of false positive findings. Procedures based on confidence interval estimates of treatment effects have the same potential for false positive findings as P value methods. Excess false positive findings can needlessly complicate the safety profile of a safe drug or vaccine. Accordingly, we propose a novel method for addressing multiplicity in the evaluation of adverse experience data arising in clinical trial settings. The method involves a two-step application of adjusted P values based on the Benjamini and Hochberg1 false discovery rate (FDR). Data from three moderate to large vaccine trials are used to illustrate our proposed ‘Double FDR’ approach, and to reinforce the potential impact of failing to account for multiplicity. This work was in collaboration with the late Professor John W. Tukey who coined the term ‘Double FDR’.

The Use of Strictly Standardized Mean Difference for Hit Selection in Primary RNA Interference High-Throughput Screening Experiments

RNA interference (RNAi) high-throughput screening (HTS) has been hailed as the 2nd genomics wave following the 1st genomics wave of gene expression microarrays and single-nucleotide polymorphism discovery platforms. Following an RNAi HTS, the authors are interested in identifying short interfering RNA (siRNA) hits with large inhibition/activation effects. For hit selection, the z-score method and its variants are commonly used in primary RNAi HTS experiments. Recently, strictly standardized mean difference (SSMD) has been proposed to measure the siRNA effect represented by the magnitude of difference between an siRNA and a negative reference group. The links between SSMD and d +-probability offer a clear interpretation of siRNA effects from a probability perspective. Hence, SSMD can be used as a ranking metric for hit selection. In this article, the authors investigated both the SSMD-based testing process and the use of SSMD as a ranking metric for hit selection in 2 primary siRNA HTS experiments. The analysis results showed that, as a ranking metric, SSMD was more stable and reliable than percentage inhibition and led to more robust hit selection results. Using the SSMD -based testing method, the false-negative rate can more readily be obtained. More important, the use of the SSMD-based method can result in a reduction in both the false-negative and false-positive rates. The applications presented in this article demonstrate that the SSMD method addresses scientific questions and fills scientific needs better than both percentage inhibition and the commonly used z-score method for hit selection. (Journal of Biomolecular Screening 2007:497-509)

Analyzing multiple endpoints in clinical trials of pain treatments: IMMPACT recommendations. Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials.

Abstract The increasing complexity of randomized clinical trials and the practice of obtaining a wide variety of measurements from study participants have made the consideration of multiple endpoints a critically important issue in the design, analysis, and interpretation of clinical trials. Failure to consider important outcomes can limit the validity and utility of clinical trials; specifying multiple endpoints for the evaluation of treatment efficacy, however, can increase the rate of false positive conclusions about the efficacy of a treatment. We describe the use of multiple endpoints in the design, analysis, and interpretation of pain clinical trials, and review available strategies and methods for addressing multiplicity. To decrease the probability of a Type I error (i.e., the likelihood of obtaining statistically significant results by chance) in pain clinical trials, the use of gatekeeping procedures and other methods that correct for multiple analyses is recommended when a single primary endpoint does not adequately reflect the overall benefits of treatment. We emphasize the importance of specifying in advance the outcomes and clinical decision rule that will serve as the basis for determining that a treatment is efficacious and the methods that will be used to control the overall Type I error rate.

Beneficial effects of milrinone and enalapril on long-term survival of rats with healed myocardial infarction

The long-term survival of rats with healed myocardial infarction and congestive heart failure treated with milrinone, enalapril and the combination of milrinone plus enalapril, was documented. Seven days after sham or coronary ligation, 200 rats (99 sham and 101 myocardial infarcted) were randomized based on electrocardiographic criteria to receive tap water, milrinone (20-40 mg/l drinking water), enalapril (17-25 mg/l) or the combination of milrinone plus enalapril (20-40 mg/17-25 mg per l). The date of spontaneous death was recorded and heart weights and myocardial infarct size (by planimetry) were determined. Long-term enalapril therapy prolonged survival with a median 50% survival (MS50) of 233 days compared to 203 days in the tap water group. Milrinone therapy also prolonged survival with a MS50 of 297 days. The combination therapy prolonged survival with a MS50 of 277 days. In general, there were three times as many rats alive in the treatment groups at the end of one year compared to untreated control groups. Cardiac hypertrophy was evident in all myocardial infarcted groups and heart weights were significantly reduced by all treatments. The average myocardial infarct sizes and the distribution of infarct sizes were not different between groups (36.8-43% of left ventricle). This study demonstrates that long-term therapy with enalapril and milrinone prolongs survival in rats with healed myocardial infarctions. The prolongation of survival was comparable in the milrinone plus enalapril groups, indicating that there was no synergy with these two agents with survival as the end point.

A model for evaluating the cost-effectiveness of cholesterol-lowering treatment.

We describe and illustrate the use of a generalizable model for evaluating the cost-effectiveness of alternative cholesterol-lowering treatments. We combine standard incidence-based techniques for measuring the cost of illness with logistic risk functions from the Framingham Heart Study to project, for persons with known coronary risk characteristics, the likelihood of developing coronary heart disease (CHD) over a lifetime as well as a number of related outcomes, including the expected loss of years of life due to CHD, the expected lifetime direct and indirect costs of CHD, and the changes in these outcomes that would result from cholesterol-lowering treatment.

An improved goodness of fit statistic for probability prediction models

We consider the general case of probability prediction models having two or more outcomes and propose an adjusted X 2 statistic which can be used to assess the goodness of fit of these models. We present a simulation study to show that our proposed statistic has an approximate X 2 distribution under the null hypothesis. Two applications are provided to illustrate the use of the new statistic. The first application examines the fit of a logistic regression model using both the proposed statistic and the popular Hosmer-Lemeshow statistic and we compare and contrast these two methods. The second application evaluates the goodness of fit of a polychotomous regression model.

Non‐parametric estimates of overlap

Kernel densities provide accurate non-parametric estimates of the overlapping coefficient or the proportion of similar responses (PSR) in two populations. Non-parametric estimates avoid strong assumptions on the shape of the populations, such as normality or equal variance, and possess sampling variation approaching that of parametric estimates. We obtain accurate standard error estimates by bootstrap resampling. We illustrate the practical use of these methods in two examples and use simulations to explore the properties of the estimators under various sampling situations.

Vaccination with measles, mumps and rubella vaccine and varicella vaccine: safety, tolerability, immunogenicity, persistence of antibody and duration of protection against varicella in healthy children

Background. Administration of M-M-R II (Measles, Mumps and Rubella Virus Vaccine, Live) and VARIVAX [Varicella Virus Vaccine Live (Oka/Merck)] given concomitantly at separate injection sites during the same office visit could increase vaccine compliance by reducing the number of health care visits for immunizations. We compared the safety and immunogenicity of M-M-R II and VARIVAX given concomitantly at separate sites (Group A) with administration of the two vaccines 6 weeks apart (Group B) as well as the persistence of varicella antibody and the duration of protection afforded by varicella vaccine. Methods. A total of 603 healthy children, ages 12 months to 6 years, with no history of measles, mumps, rubella, varicella and zoster or vaccination against these diseases, were randomized to either Group A or B and were followed for clinical reactions and serologic responses to all four viral components. Children were enrolled from August through December, 1993. Subjects were followed for 5 years to evaluate persistence of varicella antibody and breakthrough varicella rates. We compared breakthrough rates to expected attack rates in unvaccinated children to produce estimates of vaccine efficacy. Results. Both vaccine regimens were generally well-tolerated. There were no significant differences between the groups in the rates of fever, injection site reactions or rashes after vaccination. Seroconversion rates and geometric mean titers for measles, mumps and rubella were not significantly different between groups. The varicella seroconversion rate and percentage with glycoprotein-based ELISA titers ≥5.0 units were similar between the two groups (99.5 and 92.5%vs. 100 and 94.8% for Groups A and B, respectively), but the geometric mean titers were statistically significantly different (13.2 for Group A and 17.9 for Group B). Varicella antibody persistence rates were >98 to 100% during 6 years of follow-up in both groups. Compared with historical rates, varicella vaccine efficacy during 5 years was estimated to be 90.5% (95% confidence interval, 86.2%, 95.0%) and 88.9% (95% confidence interval, 83.7%, 93.7%) in Groups A and B, respectively. Conclusions. Administration of M-M-R II and VARIVAX concomitantly at separate injection sites or 6 weeks apart was generally well-tolerated and immunogenic in healthy children 12 months to 6 years of age. VARIVAX administered with M-M-R II induced persistent immunity and long-term protection against breakthrough varicella infection.