Despina G. Contopoulos-Ioannidis

Replication validity of genetic association studies

The rapid growth of human genetics creates countless opportunities for studies of disease association. Given the number of potentially identifiable genetic markers and the multitude of clinical outcomes to which these may be linked, the testing and validation of statistical hypotheses in genetic epidemiology is a task of unprecedented scale. Meta-analysis provides a quantitative approach for combining the results of various studies on the same topic, and for estimating and explaining their diversity. Here, we have evaluated by meta-analysis 370 studies addressing 36 genetic associations for various outcomes of disease. We show that significant between-study heterogeneity (diversity) is frequent, and that the results of the first study correlate only modestly with subsequent research on the same association. The first study often suggests a stronger genetic effect than is found by subsequent studies. Both bias and genuine population diversity might explain why early association studies tend to overestimate the disease protection or predisposition conferred by a genetic polymorphism. We conclude that a systematic meta-analytic approach may assist in estimating population-wide effects of genetic risk factors in human disease.

Genetic associations in large versus small studies: an empirical assessment

BACKGROUND Advances in human genetics could help us to assess prognosis on an individual basis and to optimise the management of complex diseases. However, different studies on the same genetic association sometimes have discrepant results. Our aim was to assess how often large studies arrive at different conclusions than smaller studies, and whether this situation arises more frequently when findings of first published studies disagree with those of subsequent research. METHODS We examined the results of 55 meta-analyses (579 study comparisons) of genetic associations and tested whether the magnitude of the genetic effect differs in large versus smaller studies. FINDINGS We noted significant between-study heterogeneity in 26 (47%) meta-analyses. The magnitude of the genetic effect differed significantly in large versus smaller studies in ten (18%), 20 (36%), and 21 (38%) meta-analyses with tests of rank correlation, regression on SE, and regression on inverse of variance, respectively. The largest studies generally yielded more conservative results than the complete meta-analyses, which included all studies (p=0.005). In 14 (26%) meta-analyses the proposed association was significantly stronger in the first studies than in subsequent research. Only in nine (16%) meta-analyses was the genetic association significant and replicated without hints of heterogeneity or bias. There was little concordance in first versus subsequent discrepancies, and large versus small discrepancies. INTERPRETATION Genuine heterogeneity and bias could affect the results of genetic association studies. Genetic risk factors for complex diseases should be assessed cautiously and, if possible, using large scale evidence.

Reporting and interpretation of SF-36 outcomes in randomised trials: systematic review

Objective To determine how often health surveys and quality of life evaluations reach different conclusions from those of primary efficacy outcomes and whether discordant results make a difference in the interpretation of trial findings. Design Systematic review. Data sources PubMed, contact with authors for missing information, and author survey for unpublished SF-36 data. Study selection Randomised trials with SF-36 outcomes (the most extensively validated and used health survey instrument for appraising quality of life) that were published in 2005 in 22 journals with a high impact factor. Data extraction Analyses on the two composite and eight subdomain SF-36 scores that corresponded to the time and mode of analysis of the primary efficacy outcome. Results Of 1057 screened trials, 52 were identified as randomised trials with SF-36 results (66 separate comparisons). Only eight trials reported all 10 SF-36 scores in the published articles. For 21 of the 66 comparisons, SF-36 results were discordant for statistical significance compared with the results for primary efficacy outcomes. Of 17 statistically significant SF-36 scores where primary outcomes were not also statistically significant in the same direction, the magnitude of effect was small in six, moderate in six, large in three, and not reported in two. Authors modified the interpretation of study findings based on SF-36 results in only two of the 21 discordant cases. Among 100 additional randomly selected trials not reporting any SF-36 information, at least five had collected SF-36 data but only one had analysed it. Conclusions SF-36 measurements sometimes produce different results from those of the primary efficacy outcomes but rarely modify the overall interpretation of randomised trials. Quality of life and health related survey information should be utilised more systematically in randomised trials.

Life Cycle of Translational Research for Medical Interventions

From the initial discovery of a medical intervention to a highly cited article is a long road, and even this is not the end of the journey.

Clinical Efficacy of High-Dose Acyclovir in Patients with Human Immunodeficiency Virus Infection: A Meta-Analysis of Randomized Individual Patient Data

A meta-analysis of 8 randomized trials (1792 patients, 2947 patient-years of follow-up) showed that acyclovir (> or = 3200 mg/day) offered a significant survival benefit (P = .006 by log-rank test) in human immunodeficiency virus (HIV) infection. The treatment effect did not vary significantly in patient subgroups of different CD4 cell counts, hemoglobin levels, age, race, and sex, and with or without AIDS diagnosis. Acyclovir treatment (hazard ratio, 0.78; 95% confidence interval [CI], 0.65-0.93), higher CD4 cell count (P < .001), higher hemoglobin level (P < .001), and younger age (P < .001) reduced the hazard of mortality. Acyclovir decreased herpes simplex virus infections (odds ratio [OR], 0.28; 95% CI, 0.21-0.37) and varicella-zoster virus infections (OR, 0.29; 95% CI, 0.13-0.63) but not cytomegalovirus disease or mortality from lymphoma or Kaposis sarcoma. A survival advantage was seen specifically in studies with high incidence of clinical herpesvirus infections (> or = 25% per year). Given the wide confidence intervals, the small effect in low-risk patients, and recent changes in HIV therapeutics, the results should be interpreted cautiously, but the meta-analysis supports the importance of pathogenetic interactions between herpesviruses and HIV.

Standard 6: Age Groups for Pediatric Trials

* Abbreviations: RCT — : randomized controlled trial SSRI — : selective serotonin reuptake inhibitor It has long been an axiom in clinical pediatrics that “children are not just little adults.” It has also been recognized that there are many changes from birth through childhood and the adolescent years. However, the full implications of pediatric age groupings for health care and research are still not adequately understood. There is still much to be discovered about children’s biological and psychological development and how these processes affect the effectiveness and efficacy of interventions. Trial design that accounts for age differences and promotes consistency in reporting of age-related data is essential to ensure the validity and clinical usefulness of pediatric trial data. A recent study highlighted variable treatment efficacy in children versus adults. In this study, 128 meta-analyses from Cochrane reviews, containing data on at least 1 adult and 1 pediatric randomized controlled trial (RCT) with a binary primary efficacy outcome, were reviewed.1 The authors found that in all except 1 case, the 95% confidence intervals could not exclude a relative difference in treatment efficacy between adults and children of >20%; in two-thirds of these cases, the relative difference in observed point estimates was >50%. The study also highlighted the paucity of RCTs in pediatrics; the median number of children per meta-analysis was 2.5 times smaller than the number of adults. Children and adults seem to have distinctive responses to treatments. For example, administration of phenobarbitones is useful for adults with cerebral malaria and is associated with decreased convulsions. However, in children, this drug is associated with increased 6-month mortality. Similarly, corticosteroids may offer survival benefit for adults with bacterial meningitis but not for children with the same condition. In acute traumatic brain injury, corticosteroids did not decrease mortality in adults, but there was a trend for increased mortality in children.1 In asthma, long-acting β2-agonists decreased … Address correspondence to Martin Offringa, MD, PhD, Senior Scientist and Program Head, Child Health Evaluative Sciences, Research Institute, The Hospital for Sick Children, 555 University Ave, Toronto, Ontario, Canada M5G 1X8. E-mail: martin.offringa{at}sickkids.ca

Establishment of genetic associations for complex diseases is independent of early study findings

Numerous genetic association studies for complex diseases are performed. Investigators place emphasis on formal statistical significance (P-values <0.05), but the predictive ability of early statistically significant (‘positive’) findings is unclear. We scrutinized 55 cumulative meta-analyses of genetic associations (579 studies), in order to assess whether having statistical significance in the earliest (first) published study or in at least half among several (⩾3) early-published studies, or high statistical significance in early studies had any predictive ability for establishing or refuting the presence of the genetic association in subsequent research. In 35 associations, a first study was ‘positive’ and in 15 associations more than half of the early-published reports were ‘positive’. The average publication rate of subsequent studies increased 1.71-fold with a ‘positive’ first report. When compared against the summary results of subsequent research, sensitivity and specificity were 0.65 and 0.38 for the first reports, and 0.40 and 0.73, respectively, when at least three early studies were considered. First studies also had poor predictive ability, when we considered the estimated attributable fraction and coverage of the 95% confidence interval thereof or higher levels of statistical significance. We conclude that although ‘positive’ findings in the very first reports provide strong incentive for conducting more studies on a putative genetic epidemiological association, the statistical significance or even the magnitude of the effect of early studies cannot adequately predict eventual establishment of an association. Conversely, many genuine epidemiological associations would be missed, if research were abandoned after early underpowered ‘negative’ studies.

Recursive cumulative meta-analysis : a diagnostic for the evolution of total randomized evidence from group and individual patient data

Meta-analyses of randomized evidence may include published, unpublished, and updated data in an ongoing estimation process that continuously accommodates more data. Synthesis may be performed either with group data or with meta-analysis of individual patient data (MIPD). Although MIPD with updated data is considered the gold standard of evidence, there is a need for a careful study of the impact different sources of data have on a meta-analysis and of the change in the treatment effect estimates over sequential information steps. Unpublished data and late-appearing data may be different from early-appearing data. Updated information after the end of the main study follow-up may be affected by cross-overs, missing information, and unblinding. The estimated treatment effect may thus depend on the completeness and updating of the available evidence. To address these issues, we present recursive cumulative meta-analysis (RCM) as an extension of cumulative metaanalysis. Recursive cumulative meta-analysis is based on the principle of recalculating the results of a cumulative meta-analysis with each new or updated piece of information and focuses on the evolution of the treatment effect as a more complete and updated picture of the evidence becomes available. An examination of the perturbations of the cumulative treatment effect over sequential information steps may signal the presence of bias or heterogeneity in a meta-analysis. Recursive cumulative meta-analysis may suggest whether there is a true underlying treatment effect to which the meta-analysis is converging and how treatment effects are sequentially altered by new or modified evidence. The method is illustrated with an example from the conduct of an MIPD on acyclovir in human immunodeficiency virus infection. The relative strengths and limitations of both metaanalysis of group data and MIPD are discussed through the RCM perspective.

Reporting of safety data from randomised trials

Randomised trials estimate the efficacy and toxicity of interventions. While the quality of efficacy reporting in such trials has drawn a lot of attention, the reporting of safety data has not. We evaluated systematically the quality and extent of information on drug toxicity in reports of randomised trials. We selected treatments for HIV-1 infection, where drug toxicity is common and drugs are mostly new, and so safety should be a priority. All published randomised trials, 1987–97, with sample size of 100 patients or more (n=60; enrolment 36 280) were assessed with prespecified quality measures. Although most trials (81·7%) reported how many patients discontinued study treatment, reasons were given in only 38·3%. The severity of adverse events and abnormal laboratory tests was adequately defined only in 33·3% and 61·7% of reports, and partially defined in a further 20% and 13·3%. Estimates were closely replicated by a second independent rater (kappa coefficients 0·72 and 0·85 for clinical and laboratory measures, respectively). Adequate definition was specified as detailed description of severity; or reference to known toxicity scales with reporting of severe and higher-grade event frequencies per arm for at least two side-effects. Partial definition meant that the report did not separate moderate from severe or higher toxicity; or, severe and higher grade event frequencies were given only for one of many reported side-effects. Protocols with inadequate definitions lumped numbers for all grades; did not split numbers per arm; gave no numbers for any specific events; provided only generic statements (“the medication was overall quite well tolerated”); or did not report on safety. Nine trials gave no frequencies for any clinical adverse events and 11 trials gave no frequencies for any laboratory toxicity. No table was used for safety data in 19 reports (31·3%), while there was no figure with such data in 52 (86·3%). Considering text, tables, and figures together, the median percentage of space allocated to safety information in the results section was 13% (interquartile range, 7–19%). The median space allocated to safety results was 0·5 page (mean 0·54 page), marginally smaller (p=0·056) than the space (45 breaths/min), tachycardic (>140 beats/min), and cyanosed; his jugular venous pressure was not raised. He was able to talk in full sentences and gave a history of having taken two Ecstasy tablets with little effect, so he took a further half tablet, after which he began to feel shaky (estimated total dose 180 mg MDMA, calculated from the MDMA content of the remaining half tablet). Within 25 min of the first assessment he had an apparent tonic-clonic convulsion, but was able to respond to questions. He became increasingly tachypnoeic, and his carotid pulse rate was about 200 per min. A few minutes later he vomited and had a cardiorespiratory arrest. Attempts at resuscitation were unsuccessful. At necropsy, the lungs were strikingly oedematous and congested, the liver showed some pallor and had a slightly fatty appearance, but there were no other significantly abnormal features. No tissue samples were retained for possible pharmacogenetic studies. Toxicology showed blood concentrations of MDMA 4·56 mg/L, 3,4methylenedioxyamphetamine 0·36 mg/L, and ethanol of 0·24 g/L. No other illicit drugs were detected. Ritonavir concentrations were not measured. MDMA has been previously associated with death, most commonly through excessive exertion leading to hyperthermia. Cases of overdose have been described, with ingestion of 42 tablets leading to an MDMA concentration of 7·72 mg/L, which caused mild systemic effects; in another case, overdose of 18 tablets led to an MDMA concentration of 4·05 mg/L and life-threatening illness, with symptoms similar to our patient. The protease inhibitor, ritonavir, is a potential inhibitor of CYP2D6, an isozyme responsible for demethylenation, the principal pathway by which MDMA is metabolised. Thus, ingestion of MDMA in recreational amounts by a person taking ritonavir could lead to toxic effects due to a high plasma concentration of MDMA, which the results suggested had happened in our patient. A plasma concentration of about 0·5 mg/L of MDMA would be expected after ingestion of 180 mg, but the actual concentration was almost ten times that anticipated. In addition to a possible drug interaction, the patient’s metabolism of MDMA may have been poor; 3–10% of white people are deficient in CYP2D6. Impaired hepatic function may have decreased the biotransformation of MDMA. Death was consistent with a severe serotoninergic reaction to MDMA. Severe reactions to MDMA need, therefore, to be identified immediately and urgent transfer to hospital arranged. Samples should be taken for possible pharmacogenetic analysis. Ritonavir could interact with many drugs that are metabolised by CYP2D6, including amphetamine derivatives. The potential for interactions of prescribed medications with MDMA should be highlighted. Manufacturers’ instructions should contain the appropriate warnings of the danger of interactions with illicit drugs.

Patient Safety Strategies Targeted at Diagnostic Errors: A Systematic Review

Missed, delayed, or incorrect diagnosis can lead to inappropriate patient care, poor patient outcomes, and increased cost. This systematic review analyzed evaluations of interventions to prevent diagnostic errors. Searches used MEDLINE (1966 to October 2012), the Agency for Healthcare Research and Qualitys Patient Safety Network, bibliographies, and prior systematic reviews. Studies that evaluated any intervention to decrease diagnostic errors in any clinical setting and with any study design were eligible, provided that they addressed a patient-related outcome. Two independent reviewers extracted study data and rated study quality. There were 109 studies that addressed 1 or more intervention categories: personnel changes (n = 6), educational interventions (n = 11), technique (n = 23), structured process changes (n = 27), technology-based systems interventions (n = 32), and review methods (n = 38). Of 14 randomized trials, which were rated as having mostly low to moderate risk of bias, 11 reported interventions that reduced diagnostic errors. Evidence seemed strongest for technology-based systems (for example, text message alerting) and specific techniques (for example, testing equipment adaptations). Studies provided no information on harms, cost, or contextual application of interventions. Overall, the review showed a growing field of diagnostic error research and categorized and identified promising interventions that warrant evaluation in large studies across diverse settings.