Thomas D. Cook

Qualitative and quantitative methods in evaluation research

Evaluation researchers, traditionally considered to be users of quantitative methods, are now actively exploring the qualitative aspects of the performance of the programmes they are evaluating. Rather than argue the validity of either the quantitative or the qualitative approach, most of the noted contributors to this volume conclude that both are required for comprehensive evaluation.

The Importance of Covariate Selection in Controlling for Selection Bias in Observational Studies

The assumption of strongly ignorable treatment assignment is required for eliminating selection bias in observational studies. To meet this assumption, researchers often rely on a strategy of selecting covariates that they think will control for selection bias. Theory indicates that the most important covariates are those highly correlated with both the real selection process and the potential outcomes. However, when planning a study, it is rarely possible to identify such covariates with certainty. In this article, we report on an extensive reanalysis of a within-study comparison that contrasts a randomized experiment and a quasi-experiment. Various covariate sets were used to adjust for initial group differences in the quasi-experiment that was characterized by self-selection into treatment. The adjusted effect sizes were then compared with the experimental ones to identify which individual covariates, and which conceptually grouped sets of covariates, were responsible for the high degree of bias reduction achieved in the adjusted quasi-experiment. Such results provide strong clues about preferred strategies for identifying the covariates most likely to reduce bias when planning a study and when the true selection process is not known.

Some Ways in Which Neighborhoods, Nuclear Families, Friendship Groups, and Schools Jointly Affect Changes in Early Adolescent Development.

This study assessed some ways in which schools, neighborhoods, nuclear families, and friendship groups jointly contribute to positive change during early adolescence. For each context, existing theory was used to develop a multiattribute index that should promote successful development. Descriptive analyses showed that the four resulting context indices were only modestly intercorrelated at the individual student level (N = 12,398), but clustered more tightly at the school and neighborhood levels (N = 23 and 151 respectively). Only for aggregated units did knowing the developmental capacity of any one context strongly predict the corresponding capacity of the other contexts. Analyses also revealed that each context facilitated individual change in a success index that tapped into student academic performance, mental health, and social behavior. However, individual context effects were only modest in size over the 19 months studied and did not vary much by context. The joint influence of all four contexts was cumulatively large, however, and because it was generally additive in form, no constellation of contexts was identified whose total effect reliably surpassed the sum of its individual context main effects. These results suggest that achieving significant population changes in multidimensional student growth during early adolescence most likely requires both theory and interventions that are explicitly pan-contextual.

Randomized Experiments in Educational Policy Research: A Critical Examination of the Reasons the Educational Evaluation Community has Offered for not Doing Them

This article notes the paucity with which reform efforts in education have been evaluated experimentally, despite well-nigh universal acknowledgement that experiments provide the best justification for causal conclusions. And of the few experiments that have been completed, most were done by groups outside of the community of educational evaluators working in schools and colleges of education. The reasons educational evaluators cite for not doing experiments are critically appraised. Some are shown to be wrong or over-generalized; and others point to the need for future experiments that depend more heavily on program theory and the assessment of implementation and other intervening processes. The alternatives to experiments that educational evaluators prefer are briefly appraised and found wanting whenever a high standard is needed for justifying causal conclusions.

Causes of death and rehospitalization in patients hospitalized with worsening heart failure and reduced left ventricular ejection fraction: Results from efficacy of vasopressin antagonism in heart failure outcome study with tolvaptan (EVEREST) program

BACKGROUND The postdischarge rehospitalization and death rates are high in patients with acute heart failure (HF) syndromes despite optimization of standard therapy for chronic HF. To the best of our knowledge, there has been no systematic analysis of the causes of death and rehospitalization in this patient population. METHODS This was a prespecified analysis of adjudicated cause-specific all-cause mortality and cardiovascular (CV) hospitalization in the Efficacy of Vasopressin Antagonism in Heart Failure Outcome Study with Tolvaptan (EVEREST) trial, a randomized, double-blind, placebo-controlled study in patients hospitalized with worsening HF and left ventricular ejection fraction < or =40% comparing tolvaptan, an oral vasopressin receptor antagonist to placebo, in addition to standard care. RESULTS Of the 4,133 randomized, there were 5,239 rehospitalizations and 1,080 deaths during a median of 9.9 months. Of all deaths, 41.0% were due to HF, 26.0% due to sudden cardiac death (SCD), 2.6% due to acute myocardial infarction (MI), 2.2% due to stroke, and 13.2% due to non-CV causes. Of all hospitalizations, 39.2% were non-CV, whereas 46.3% were for HF, and a minority of hospitalizations was due to stroke, MI, arrhythmia, or other CV causes. CONCLUSIONS Despite close follow-up and evidence-based therapy within a clinical trial, rehospitalization and death remain high. Although most deaths were from HF, one quarter of patients had SCD. In addition, there were almost as many non-CV hospitalizations as HF hospitalizations. Knowledge of the causes of death and rehospitalization may be essential for proper management and early initiation of therapy.

Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation

A decade ago, the Society of Prevention Research (SPR) endorsed a set of standards for evidence related to research on prevention interventions. These standards (Flay et al., Prevention Science 6:151–175, 2005) were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness. In 2013, SPR’s Board of Directors decided that the field has progressed sufficiently to warrant a review and, if necessary, publication of “the next generation” of standards of evidence. The Board convened a committee to review and update the standards. This article reports on the results of this committee’s deliberations, summarizing changes made to the earlier standards and explaining the rationale for each change. The SPR Board of Directors endorses “The Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation.”

The renaissance of field experimentation in evaluating interventions.

Most experiments are done in laboratories. However, there is also a theory and practice of field experimentation. It has had its successes and failures over the past four decades but is now increasingly used for answering causal questions. This is true for both randomized and-perhaps more surprisingly-nonrandomized experiments. In this article, we review the history of the use of field experiments, discuss some of the reasons for their current renaissance, and focus the bulk of the article on the particular technical developments that have made this renaissance possible across four kinds of widely used experimental and quasi-experimental designs-randomized experiments, regression discontinuity designs in which those units above a cutoff get one treatment and those below get another, short interrupted time series, and nonrandomized experiments using a nonequivalent comparison group. We focus this review on some of the key technical developments addressing problems that previously stymied accurate effect estimation, the solution of which opens the way for accurate estimation of effects under the often difficult conditions of field implementation-the estimation of treatment effects under partial treatment implementation, the prevention and analysis of attrition, analysis of nested designs, new analytic developments for both regression discontinuity designs and short interrupted time series, and propensity score analysis. We also cover the key empirical evidence showing the conditions under which some nonrandomized experiments may be able to approximate results from randomized experiments.

Redefine Statistical Significance

We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries.

Comer's school development program in prince george's county, maryland: A theory-based evaluation

A randomized experiment of Comers School Development Program was conducted in 23 middle schools in Prince Georges County, Maryland. The school population is predominantly African American, with considerable internal variation in household socioeconomic standing. This study involved repeated measurement with more than 12,000 students and 2,000 staff a survey of more than 1,000 parents, and extensive access to student records. It showed that Comer schools implemented some of the programs central elements better than control schools, but not all or even most of them. This shortfall in program implementation may have been responsible for students in the experimental schools not changing any more than controls. Quasi-experimental analyses showed that the program theory may be correct in many of its predictions about student changes in psychological and social outcomes, but not achievement. However, achievement gains were found in schools with a more explicit academic focus, suggesting that improving this focus should be as central to Comers program theory as improving a schools social climate. Even more needed, though, are ways to improve program implementability, the sine qua non for student change.

The Persistence of Experimentally Induced Attitude Change

Publisher Summary This chapter presents the persistence of experimentally induced change. Several theoretical benefits are expected from a review of the persistence literature that has slowly and unsystematically accumulated in experimental social psychology. Just as theories of long-term memory stress different concepts from theories of short-term memory, so theories of persistence may employ different concepts from theories of initial attitude change. It is also useful to highlight the constructs that are unique to persistence theories and are not part of current analyses of initial attitude change. The chapter indicates how well current procedures leading to initial attitude change succeed in producing persistent attitude change. The concern for predicting persistence can be expressed at several theoretical levels. The chapter explores that, at the level of specific theories, it is important to compare the level of persistence. Reviewing the persistence literature also helps in help in determining whether “attitude’’ really changes in laboratory experiments on attitude change.