Russell Localio

Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change

Key Summary Points The decision to calculate a summary estimate in a meta-analysis should be based on clinical judgment, the number of studies, and the degree of variation among studies. A random-effects model is a meta-analytic approach that incorporates study-to-study variability beyond what would be expected by chance. The DerSimonianLaird (DL) method, the earliest and most commonly used random-effects model, is the default method in many software packages. The DL method produces confidence bounds that are too narrow (and P values that are typically too small) when the number of studies is small or when there are substantive differences among study estimates. Alternative random-effects estimates based on small-sample adjustments, the profile likelihood, or hierarchical Bayesian models that perform better than the DL method are readily available in software packages. When it is appropriate to pool studies whose estimates vary widely, meta-analytic methods that provide a better accounting of uncertainty than the DL estimator should be used. The basic premise of meta-analysis is that the average of estimates provided by a group of studies is closer to the truth than the estimate provided by an individual study. This premise rests on the assumption that each study is a near-replication of a single experiment and that differences among study results are due only to chance. The technical jargon for this fundamental assumption is that each of the studies is estimating the same fixed effect, and the corresponding meta-analytic approach is dubbed the fixed-effects model. When studies are statistically heterogeneous and differences among their results cannot be explained by chance alone, the meta-analyst faces a conundrum. Qualitative heterogeneity among study designs, patient characteristics, and treatment and comparator regimens may be so great that it does not make sense to combine studies to derive a single summary estimate. However, when the qualitative and quantitative heterogeneity is not so great that a single number summarizing the evidence would be misleading, statistical models that incorporate the extra variability across studies not believed to be due to chance may be used to summarize the data. These models assume that the observed treatment effect for a study is a combination of a treatment effect common to all studies plus a component specific to that study alone. This extra, study-specific component is assumed to be random, hence the jargon that it is a random effect, with accompanying mathematical models dubbed random-effects models. The most widely used random-effects model is based on an estimator developed by DerSimonian and Laird in the mid-1980s and is known as the DerSimonianLaird (DL) estimator (1). Supplement. Alternative Random Effects Estimators An Example The Figure depicts a statistically heterogeneous set of studies followed by several methods of estimating their average effect. The example is from a 1985 meta-analysis by Collins and colleagues on the effect of administering a diuretic to women at risk for preeclampsia (11), and it is frequently used to illustrate different methods for estimating a common treatment effect when the body of evidence is heterogeneous (12, 13). The effect estimates from the individual studies range from a more than 4-fold statistically significant decrease in the odds of eclampsia with diuretics observed in the study by Fallis and colleagues (5) to an almost 3-fold nonsignificant increase in the study by Tervil and Vartiainen (9). A visual clue that these studies are statistically heterogeneous is that the confidence limits of several pairs of studies do not overlap. Figure. Heterogeneous evidence from Collins and colleagues meta-analysis of the effects of diuretics on preeclampsia (11). * The metafor package in R was used to compute the fixed-effects estimate and the DerSimonianLaird random-effects estimate. The metafor package in R was used to compute the KnappHartung small-sample adjustments, based on the DerSimonianLaird estimate. The small-sample (Skovgaard) estimate from the metaLik package in R was used to compute the profile likelihood estimate. The large-sample profile likelihood estimate produced a narrower CI that indicates a statistically significant effect (95% CI, 0.37 to 0.95). The hierarchical Bayesian estimate was computed using WinBugs and assumed a vague uniform (10, 10) prior distribution for . A sensitivity analysis assuming a vague (0.001, 0.001) on precision (1/2) produced a slightly smaller but statistically significant 95% CI (0.36 to 0.98). The Figure shows that different statistical approaches to combining data can produce results leading to different conclusions. The fixed-effects model, which is not appropriate for these data, shows a summary effect of 0.67, with 95% confidence limits (0.56 and 0.80) that are 19% less than and greater than that value. The DL random-effects estimate shows a slightly larger effect (odds ratio, 0.60), but the confidence limits are substantially wider33% less than (0.40) and greater than (0.89) the summary effect, albeit still highly statistically significant. Use of any of the other 3 random-effects estimators depicted in the Figure shows identical point estimates for the odds ratio of 0.60 but dramatically wider confidence limits that are 73% less than and greater than 0.60, with the upper limits all exceeding 1.00. The corresponding P values range from less than 0.001 for the fixed-effects model to 0.011 for the DL estimator and 0.070 or greater for the other random-effects models. Statistical Heterogeneity and Uncertainty The differences noted in the example are due to the ways that the models handle statistical heterogeneity. Statistical heterogeneity refers to variation in the true effects being estimated by each study. We characterize this variation by its SD, a statistic called . Assuming normality, we expect 95% of true effects to fall within2 of the central estimate. When odds ratios or relative risks are used, the normality is on a log scale, so that true study odds ratios or relative risks fall within a range of the estimate multiplied by e2. In the example, equals 0.48, so the true study effects are estimated to fall within 0.60e0.96, or 2.6 times greater than or less than 0.60 (0.23 to 1.56). This range should be smaller than the actual smallest and largest study estimates, as is the case in this example, with the remainder of the variation assumed to be due to chance. The models vary in their assumption of how certain we are about ; this uncertainty is included in the meta-analytic CIs. The DL method assumes that our guess about is exactly correct, with no uncertainty; thus, confidence limits are too narrow and the P values are too small. In Collins and colleagues meta-analysis, which pooled a modest number of studies (n= 9) with statistically heterogeneous effects, the DL estimator provided the narrowest confidence limits among the random-effects options. In addition to , meta-analysts commonly use statistical tests, such as the Cochran Q test, or indices, such as the I 2 index, to help gauge heterogeneity of effects. Both the Cochran Q test and the I 2 index are dimensionless measures of statistical heterogeneity. Neither conveys information about actual variation in effect size, and both have low power to detect heterogeneity in situations involving 10 or fewer studies (14). The DL Estimator and Alternative Approaches The DL method appeared in the literature just as meta-analytic methods were being adopted to help reviewers quantitatively summarize evidence about medical interventions. It was relatively simple to compute and is still the standard estimator programmed into many meta-analysis software packages, including the RevMan software developed by the Cochrane Collaboration (15). As statisticians began in the 1990s to recognize the problems with the DL approach, theyincluding DerSimonian and Kacker (16)proposed a wide range of alternatives that better capture the uncertainty associated with statistical heterogeneity. These included random-effects estimators based on small-sample adjustments, such as the KnappHartung approach (17), likelihood-based methods (13, 18, 19), and hierarchical Bayesian models (20). The KnappHartung approach, one of the more recent methods, assumes that variances are estimated from small samples, makes small-sample adjustments to the variance estimates, and constructs confidence limits based on the t distribution with k 1 degrees of freedom. This estimator produces a wider confidence limit than the DL estimate. It may slightly overestimate the amount of uncertainty in some cases, particularly when dealing with 5 or fewer studies. It is available in some specialized meta-analysis programs and packages, such as the metareg program (21) in Stata (StataCorp, College Station, Texas) and the metafor package (22) in R (R Foundation for Statistical Computing, Vienna, Austria). Likelihood estimates, which are readily available in such commonly used statistical packages as SAS (SAS Institute, Cary, North Carolina), are computed using standard mixed-effects linear models (18, 19). The profile likelihood is a good method for computing confidence bounds. Unlike estimators based on maximum likelihood or restricted maximum likelihood methods, the profile likelihood allows for asymmetrical intervals and uncertainty in estimation of the between-study variance (2). Simulation studies show that it provides a substantially better accounting of uncertainty than the DL estimator (13, 23). The profile likelihood estimates are available in the metaan package (24) in Stata and the metaLik package (25) in R. The latter provides a more accurate but possibly conservative small-sample profile likelihood estimate of uncertainty (26). Bayesian random-effects models, which are based on an exact binomial distribution, perform well in many situations where others do poorly, particularl

Incidence, complications, and risk factors for prolonged stay in children hospitalized with community-acquired influenza

OBJECTIVES. Few studies have examined the characteristics and clinical course of children hospitalized with laboratory-confirmed influenza. We sought to (1) estimate the age-specific incidence of influenza-related hospitalizations, (2) describe the characteristics and clinical course of children hospitalized with influenza, and (3) identify risk factors for prolonged hospitalization. PATIENTS AND METHODS. Children ≤21 years of age hospitalized with community-acquired laboratory-confirmed influenza at a large urban childrens hospital were identified through review of laboratory records and administrative data sources. A neighborhood cohort embedded within our study population was used to estimate the incidence of community-acquired laboratory-confirmed influenza hospitalizations among children <18 years old. Risk factors for prolonged hospitalization (>6 days) were determined by using logistic regression. RESULTS. We identified 745 children hospitalized with community-acquired laboratory-confirmed influenza during the 4-year study period. In this urban cohort, the incidence of community-acquired laboratory-confirmed influenza hospitalization was 7 per 10000 child-years of observation. The median age was 1.8 years; 25% were infants <6 months old, and 77% were children <5 years old. Many children (49%) had a medical condition associated with an increased risk of influenza-related complications. The incidence of influenza-related complications was higher among children with a preexisting high-risk condition than for previously healthy children (29% vs 21%). However, only cardiac and neurologic/neuromuscular diseases were found to be independent risk factors for prolonged hospitalization. CONCLUSIONS. Influenza is a common cause of hospitalization among both healthy and chronically ill children. Children with cardiac or neurologic/neuromuscular disease are at increased risk of prolonged hospitalization; therefore, children with these conditions and their contacts should be a high priority to receive vaccine. The impact on pediatric hospitalization of the new recommendation to vaccinate all children 6 months to <5 years old should be assessed.

Impact of Kinship Care on Behavioral Well-being for Children in Out-of-Home Care

OBJECTIVE To examine the influence of kinship care on behavioral problems after 18 and 36 months in out-of-home care. Growth in placement of children with kin has occurred despite conflicting evidence regarding its benefits compared with foster care. DESIGN Prospective cohort study. SETTING National Survey of Child and Adolescent Well-Being, October 1999 to March 2004. PARTICIPANTS One thousand three hundred nine children entering out-of-home care following a maltreatment report. MAIN EXPOSURE Kinship vs general foster care. MAIN OUTCOME MEASURES Predicted probabilities of behavioral problems derived from Child Behavior Checklist scores. RESULTS Fifty percent of children started in kinship care and 17% of children who started in foster care later moved to kinship care. Children in kinship care were at lower risk at baseline and less likely to have unstable placements than children in foster care. Controlling for a childs baseline risk, placement stability, and attempted reunification to birth family, the estimate of behavioral problems at 36 months was 32% (95% confidence interval, 25%-38%) if children in the cohort were assigned to early kinship care and 46% (95% confidence interval, 41%-52%) if children were assigned to foster care only (P = .003). Children who moved to kinship care after a significant time in foster care were more likely to have behavioral problems than children in kinship care from the outset. CONCLUSIONS Children placed into kinship care had fewer behavioral problems 3 years after placement than children who were placed into foster care. This finding supports efforts to maximize placement of children with willing and available kin when they enter out-of-home care.

Prioritization of Comparative Effectiveness Research Topics in Hospital Pediatrics

OBJECTIVE To use information about prevalence, cost, and variation in resource utilization to prioritize comparative effectiveness research topics in hospital pediatrics. DESIGN Retrospective analysis of administrative and billing data for hospital encounters. SETTING Thirty-eight freestanding US childrens hospitals from January 1, 2004, through December 31, 2009. PARTICIPANTS Children hospitalized with conditions that accounted for either 80% of all encounters or 80% of all charges. MAIN OUTCOME MEASURES Condition-specific prevalence, total standardized cost, and interhospital variation in mean standardized cost per encounter, measured in 2 ways: (1) intraclass correlation coefficient, which represents the fraction of total variation in standardized costs per encounter due to variation between hospitals; and (2) number of outlier hospitals, defined as having more than 30% of encounters with standardized costs in either the lowest or highest quintile across all encounters. RESULTS Among 495 conditions accounting for 80% of all charges, the 10 most expensive conditions accounted for 36% of all standardized costs. Among the 50 most prevalent and 50 most costly conditions (77 in total), 26 had intraclass correlation coefficients higher than 0.10 and 5 had intraclass correlation coefficients higher than 0.30. For 10 conditions, more than half of the hospitals met outlier hospital criteria. Surgical procedures for hypertrophy of tonsils and adenoids, otitis media, and acute appendicitis without peritonitis were high cost, were high prevalence, and displayed significant variation in interhospital cost per encounter. CONCLUSIONS Detailed administrative and billing data can be used to standardize hospital costs and identify high-priority conditions for comparative effectiveness research--those that are high cost, are high prevalence, and demonstrate high variation in resource utilization.

Oral versus intravenous rehydration of moderately dehydrated children: a randomized, controlled trial.

Background. Dehydration from viral gastroenteritis is a significant pediatric health problem. Oral rehydration therapy (ORT) is recommended as first-line therapy for both mildly and moderately dehydrated children; however, three quarters of pediatric emergency medicine physicians who are very familiar with the American Academy of Pediatrics recommendations for ORT still use intravenous fluid therapy (IVF) for moderately dehydrated children. Objective. To test the hypothesis that the failure rate of ORT would not be >5% greater than the failure rate of IVF. Secondary hypotheses were that patients in the ORT group will (1) require less time initiating therapy, (2) show more improvement after 2 hours of therapy, (3) have fewer hospitalizations, and (4) prefer ORT for future episodes of dehydration. Methods. A randomized, controlled clinical trial (noninferiority study design) was performed in the emergency department of an urban children’s hospital from December 2001 to April 2003. Children 8 weeks to 3 years old were eligible if they were moderately dehydrated, based on a validated 10-point score, from viral gastroenteritis. Patients were randomized to receive either ORT or IVF during the 4-hour study. Treating physicians were masked and assessed all patients before randomization at 2 and 4 hours of therapy. Successful rehydration at 4 hours was defined as resolution of moderate dehydration, production of urine, weight gain, and the absence severe emesis (≥5 mL/kg). Results. Seventy-three patients were enrolled in the study: 36 were randomized to ORT and 37 were randomized to IVF. Baseline dehydration scores and the number of prior episodes of emesis and diarrhea were similar in the 2 groups. ORT demonstrated noninferiority for the main outcome measure and was found to be favorable with secondary outcomes. Half of both the ORT and IVF groups were rehydrated successfully at 4 hours (difference: −1.2%; 95% confidence interval [CI]: −24.0% to 21.6%). The time required to initiate therapy was less in the ORT group at 19.9 minutes from randomization, compared with 41.2 minutes for the IVF group (difference: −21.2 minutes; 95% CI: −10.3 to −32.1 minutes). There was no difference in the improvement of the dehydration score at 2 hours between the 2 groups (78.8% ORT vs 80% IVF; difference: −1.2%; 95% CI: −20.5% to 18%). Less than one third of the ORT group required hospitalization, whereas almost half of the IVF group was hospitalized (30.6% vs 48.7%, respectively; difference: −18.1%; 95% CI: −40.1% to 4.0%). Patients who received ORT were as likely as those who received IVF to prefer the same therapy for the next episode of gastroenteritis (61.3% vs 51.4%, respectively; difference: 9.9%; 95% CI: −14% to 33.7%). Conclusions. This trial demonstrated that ORT is as effective as IVF for rehydration of moderately dehydrated children due to gastroenteritis in the emergency department. ORT demonstrated noninferiority for successful rehydration at 4 hours and hospitalization rate. Additionally, therapy was initiated more quickly for ORT patients. ORT seems to be a preferred treatment option for patients with moderate dehydration from gastroenteritis.

Effectiveness of Developmental Screening in an Urban Setting

OBJECTIVE: To determine the effectiveness of developmental screening on the identification of developmental delays, early intervention (EI) referrals, and EI eligibility. METHODS: This randomized controlled, parallel-group trial was conducted from December 2008 to June 2010 in 4 urban pediatric practices. Children were eligible if they were <30 months old, term, without congenital malformations or genetic syndromes, not in foster care, and not enrolled in EI. Children were randomized to receive 1 of the following: (1) developmental screening using Ages and Stages Questionnaire-II (ASQ-II and Modified Checklist for Autism in Toddlers (M-CHAT) with office staff assistance, (2) developmental screening using ASQ-II and M-CHAT without office staff assistance, or (3) developmental surveillance using age-appropriate milestones at well visits. Outcomes were assessed using an intention-to-treat analysis. RESULTS: A total of 2103 children were enrolled. Most were African-American with family incomes less than

Electronic health record-based decision support to improve asthma care: a cluster-randomized trial.

30 000. Children in either screening arm were more likely to be identified with delays (23.0% and 26.8% vs 13.0%; P < .001), referred to EI (19.9% and 17.5% vs 10.2%; P < .001), and eligible for EI services (7.0% and 5.3% vs 3.0%; P < .001) than children in the surveillance arm. Children in the screening arms incurred a shorter time to identification, EI referral, and EI evaluation than children in the surveillance arm. CONCLUSIONS: Children who participated in a developmental screening program were more likely to be identified with developmental delays, referred to EI, and eligible for EI services in a timelier fashion than children who received surveillance alone. These results support policies endorsing developmental screening.

Outcomes Attributable to Neonatal Candidiasis

OBJECTIVE: Asthma continues to be 1 of the most common chronic diseases of childhood and affects ∼6 million US children. Although National Asthma Education Prevention Program guidelines exist and are widely accepted, previous studies have demonstrated poor clinician adherence across a variety of populations. We sought to determine if clinical decision support (CDS) embedded in an electronic health record (EHR) would improve clinician adherence to national asthma guidelines in the primary care setting. METHODS: We conducted a prospective cluster-randomized trial in 12 primary care sites over a 1-year period. Practices were stratified for analysis according to whether the site was urban or suburban. Children aged 0 to 18 years with persistent asthma were identified by International Classification of Diseases, Ninth Revision codes for asthma. The 6 intervention-practice sites had CDS alerts imbedded in the EHR. Outcomes of interest were the proportion of children with at least 1 prescription for controller medication, an up-to-date asthma care plan, and the performance of office-based spirometry. RESULTS: Increases in the number of prescriptions for controller medications, over time, was 6% greater (P = .006) and 3% greater for spirometry (P = .04) in the intervention urban practices. Filing an up-to-date asthma care plan improved 14% (P = .03) and spirometry improved 6% (P = .003) in the suburban practices with the intervention. CONCLUSION: In our study, using a cluster-randomized trial design, CDS in the EHR, at the point of care, improved clinician compliance with National Asthma Education Prevention Program guidelines.

Clostridium difficile Infection is associated with Increased Risk of Death and Prolonged Hospitalization in Children

BACKGROUND The incidence of candidiasis has increased in neonatal intensive care units, and invasive candidiasis is associated with significant morbidity and mortality. However, few data exist on outcomes directly attributable to neonatal candidiasis. METHODS We estimated the incidence of systemic candidiasis in hospitalized neonates within the United States and determined the attributable mortality, length of hospital stay, and associated costs. We used the 2003 Kids Inpatient Database from the Healthcare Cost and Utilization Project. Systemic candidiasis and comorbidities were defined by International Classification of Diseases, Ninth Revision, Clinical Modification codes. Neonates with uncomplicated births and neonates who died within the first 3 days of life were excluded. We used propensity score methods to balance covariates between the neonates with and neonates without candidiasis. Attributable outcomes were calculated between propensity score-matched neonates with and neonates without candidiasis. Because of the known confounding effect of birth weight, we performed separate propensity score analyses for extremely low birth weight (ELBW) neonates (i.e., neonates weighing < 1000 g). RESULTS The overall incidence of invasive candidiasis in neonates is 15 cases per 10,000 neonatal admissions (95% confidence interval [CI], 13-16 cases per 10,000 neonatal admissions). ELBW neonates with invasive candidiasis were 2 times more likely to die (odds ratio, 2.2; 95% CI, 1.4-3.5) than propensity-matched ELBW neonates without candidiasis. The propensity score-adjusted mortality rate attributable to candidiasis among ELBW neonates was 11.9%. Candidiasis in ELBW infants was not associated with an increase in length of hospital stay but was associated with a mean increase in total charges of

Prevalence and risk factors for significant liver fibrosis among HIV-monoinfected patients.

39,045 (95% CI,