Eric Hedberg

Intraclass Correlation Values for Planning Group-Randomized Trials in Education

Experiments that assign intact groups to treatment conditions are increasingly common in social research. In educational research, the groups assigned are often schools. The design of group-randomized experiments requires knowledge of the intraclass correlation structure to compute statistical power and sample sizes required to achieve adequate power. This article provides a compilation of intraclass correlation values of academic achievement and related covariate effects that could be used for planning group-randomized experiments in education. It also provides variance component information that is useful in planning experiments involving covariates. The use of these values to compute the statistical power of group-randomized experiments is illustrated.

A Longitudinal Assessment of the Impact of Foreclosure on Neighborhood Crime

Objectives: To examine possible effects of housing foreclosure on neighborhood levels of crime and to assess temporal lags in the impact of foreclosure on neighborhood levels of crime. Methods: Longitudinal data from Glendale, Arizona, a city at the epicenter of the nation’s foreclosure problem. The authors rely on four data sources: (1) foreclosure data, (2) computer-aided dispatch (CAD)/police records management system (RMS) data, (3) U.S. census and census estimate data, and (4) land use data. Results: Foreclosure has a short-term impact, typically no more than 3 months, on total crime, property crime, and violent crime, and no more than 4 months for drug crime. Conclusions: Foreclosures do not have a long-term effect on crime in general, and have different, though modest effects on different types of crime. The relationship between foreclosure and crime is not linear in nature but rather is characterized by a temporal, short-term flux in crime.

The use of patient satisfaction surveys and alternative coding procedures to predict malpractice risk.

Purpose:Because costs associated with malpractice litigation draw substantial resources away from patient care, many health care organizations are seeking efficient methods to manage these risks. The purpose of this study was to identify methods by which commonly available patient satisfaction indicators could be used to identify potential malpractice litigation risks. Subject and Methods:Using data from the risk management department of a large academic medical center, we combined yearly administrative records from 1998 to 2006 of malpractice-related litigation activity, with patient satisfaction scores related to attending physicians. We then applied 3 approaches to code patient satisfaction for each year: (1) calculating the overall mean, (2) assigning tertiles, and (3) identifying the minimum satisfaction response to any question. We then estimated 3 versions of random-effect logit models to examine which estimators predicted whether an attending physician was named in a lawsuit in a given year. Results:Minimum satisfaction score was significantly associated with malpractice activity; the other analytic approaches did not yield significant associations. Although patient satisfaction explained little variation in an individual physicians contribution to malpractice risk, accounting for the minimum score explained more than a quarter of a departments contribution. Conclusions:Findings suggest that minimum satisfaction score may provide a useful metric for identifying and prioritizing malpractice risks.

Using and Interpreting Mental Health Measures in the National Social Life, Health, and Aging Project

INTRODUCTION National Social Life, Health, and Aging Project (NSHAP) included five unique mental health measures in Waves 1 and 2 that researchers can use to measure the overall emotional health of participants: depressive symptoms, happiness-unhappiness, anxiety symptoms, perceived stress, and felt loneliness. For each, we detail the rationale for its development and explain how to score, analyze, and interpret results. METHOD NSHAP developed its measures by modifying traditional short-form scales to improve response efficiency and reduce respondent burden. Scoring protocols and interpretations were developed for each measure. U.S. population estimates for older adults born between 1920 and 1947 were generated using age-eligible samples from Waves 1 and 2. RESULTS NSHAPs protocols yielded U.S. prevalence rates similar to other nationally representative studies of older adults and comparable between waves. Higher estimates of anxiety symptoms and perceived stress in Wave 2 compared with Wave 1 were explained by age, administration mode, and time period. Analytic strategies for longitudinal analyses are provided. In Wave 2, mental health generally was worse at older ages, with women having more symptoms at younger ages than men. Women had fewer anxiety symptoms at the oldest ages. DISCUSSION NSHAPs mental health measures were successfully integrated into the projects survey and showed acceptable external reliability as well as moderately stable individual characteristics across the 5 years between Waves 1 and 2. Depressive symptoms and unhappiness may form a mental health cluster in the elderly, distinct from anxiety symptoms, perceived stress, and felt loneliness. Gender differences in age-specific patterns of mental health were evident using the exact age of participants rather than the traditional decade groupings. Administration mode and time period (between 2005-2006 and 2010-2011) were determined to be potential confounds that need to be accommodated in longitudinal analyses of aging, whereas sample composition was not an issue for interpreting mental health measures.

Intraclass Correlations and Covariate Outcome Correlations for Planning Two-and Three-Level Cluster-Randomized Experiments in Education.

Background: Cluster-randomized experiments that assign intact groups such as schools or school districts to treatment conditions are increasingly common in educational research. Such experiments are inherently multilevel designs whose sensitivity (statistical power and precision of estimates) depends on the variance decomposition across levels. This variance decomposition is usually summarized by the intraclass correlation (ICC) structure and, if covariates are used, the effectiveness of the covariates in explaining variation at each level of the design. Objectives: This article provides a compilation of school- and district-level ICC values of academic achievement and related covariate effectiveness based on state longitudinal data systems. These values are designed to be used for planning group-randomized experiments in education. The use of these values to compute statistical power and plan two- and three-level group-randomized experiments is illustrated. Research Design: We fit several hierarchical linear models to state data by grade and subject to estimate ICCs and covariate effectiveness. The total sample size is over 4.8 million students. We then compare our average of state estimates with the national work by Hedges and Hedberg.

Body-Worn Cameras and Citizen Interactions with Police Officers: Estimating Plausible Effects Given Varying Compliance Levels

Recent citizen deaths involving police use of force have increased discussion surrounding police accountability and community relations. One piece of this discussion is the use of body worn cameras (BWCs) by officers. Unfortunately, little rigorous research has been conducted to estimate the effectiveness of BWCs in reducing problematic police-citizen interactions. In this paper, we estimate two measures of effectiveness of BWCs by comparing incidents that occur in a squad assigned cameras to incidents that occur in a squad assigned control. First, we estimate the effect of being assigned a BWC (but not necessarily using the camera) on reducing complaints and resistance associated with incidents. Second, we employ data on BWC use to estimate the effect of cameras if they were used with full compliance. Together, these two estimates provide a plausible range of effectiveness that policymakers can expect from BWCs. We find that BWCs have no effect on the rate of arrest or resistance, but can substantially reduce complaints.

Prevalence of Bacterial Vaginosis and Candida among Postmenopausal Women in the United States

OBJECTIVES To describe the prevalence of bacterial vaginosis (BV) and Candida among community-dwelling postmenopausal women in the United States and determine their change with age, using estimates based on Waves 1 and 2 of the National Social Life, Health and Aging Project (NSHAP). METHOD Self-administered vaginal swabs were collected in-home from women aged 57-85 (n = 1,016) in Wave 1 and again 5 years later in Wave 2 (n = 883). Gram-stained specimens were evaluated for BV using the Nugent score as well as presence of Candida. RESULTS BV was prevalent in 23% and 38% of postmenopausal women in Waves 1 and 2 and increased with age. Women initially categorized with BV in Wave 1 were more than 10 times as likely to be categorized with BV in Wave 2, relative risk ratio (RRR) = 10.5; 95% confidence interval (CI) (4.45-24.7); p < .001, whereas women initially categorized as intermediate in Wave 1 were five times more likely to have a BV categorization, RRR = 5.0; 95% CI (2.56-9.75); p < .001. Although the presence of Candida was similar in both waves (6% and 5%), its relationship with age only became evident in Wave 2, with odds of detecting Candida decreasing by 7% with each year of age, OR = 0.93, 95% CI (0.88, 0.98); p = .010. DISCUSSION In Wave 2, the prevalence of BV was higher and increased with age while the prevalence of Candida was low and declined with age. A 5-year age increase contributed to the prevalence change across waves. Methods refinements in Wave 2 improved the detection of BV and Candida and clarified their relationship with age.

The Variance of Intraclass Correlations in Three- and Four-Level Models

Intraclass correlations are used to summarize the variance decomposition in populations with multilevel hierarchical structure. There has recently been considerable interest in estimating intraclass correlations from surveys or designed experiments to provide design parameters for planning future large-scale randomized experiments. The large sample distribution of estimates of the intraclass correlation in two-level situations is well known. Contemporary educational and social experiments often involve sampling designs with three or four levels of nesting, leading to three- and four-level intraclass correlation structures. The present article provides expressions for the large sample variances and covariances of estimates of three- and four-level intraclass correlation structures.

Reference Values of Within-District Intraclass Correlations of Academic Achievement by District Characteristics Results From a Meta-Analysis of District-Specific Values

Background: Randomized experiments are often considered the strongest designs to study the impact of educational interventions. Perhaps the most prevalent class of designs used in large-scale education experiments is the cluster randomized design in which entire schools are assigned to treatments. In cluster randomized trials that assign schools to treatments within a set of school districts, the statistical power of the test for treatment effects depends on the within-district school-level intraclass correlation (ICC). Hedges and Hedberg (2014) recently computed within-district ICC values in 11 states using three-level models (students in schools in districts) that pooled results across all the districts within each state. Although values from these analyses are useful when working with a representative sample of districts, they may be misleading for other samples of districts because the magnitude of ICCs appears to be related to district size. To plan studies with small or nonrepresentative samples of districts, better information are needed about the relation of within-district school-level ICCs to district size. Objective: Our objective is to explore the relation between district size and within-district ICCs to provide reference values for math and reading achievement for Grades 3–8 by district size, poverty level, and urbanicity level. These values are not derived from pooling across all districts within a state as in previous work but are based on the direct calculation of within-district school-level ICCs for each school district. Research Design: We use mixed models to estimate over 7,000 district-specific ICCs for math and reading achievement in 11 states and for Grades 3–8. We then perform a random effects meta-analysis on the estimated within-district ICCs. Our analysis is performed by grade and subject for different strata designated by district size (number of schools), urbanicity, and poverty rates.

Evaluation of the Phoenix TRUCE Project: A Replication of Chicago CeaseFire

The Phoenix TRUCE Project was modeled after the Chicago CeaseFire program. There have been relatively few process and impact evaluations on the model compared to the level of funding and attention the program has rendered. This paper presents findings related to the evaluation of the TRUCE project. We found that the program engaged in a strong media campaign, conducted conflict mediations, and identified high-risk individuals for case management. The program did not, however, establish a coordinated and collaborative relationship with the faith-based community or other community groups. Time-series analysis showed that program implementation corresponded to a significant decrease in overall levels of violence by more than 16 incidents on average per month, a decrease of 16 assaults on average per month, and resulted in a significant increase of 3.2 shootings on average per month, controlling for the comparison areas and the trends in the data.