Tammi Vacha-Haase

How to Estimate and Interpret Various Effect Sizes

The present article presents a tutorial on how to estimate and interpret various effect sizes. The 5th edition of the Publication Manual of the American Psychological Association (2001) described the failure to report effect sizes as a “defect” (p. 5), and 23 journals have published author guidelines requiring effect size reporting. Although dozens of effect size statistics have been available for some time, many researchers were trained at a time when effect sizes were not emphasized, or perhaps even taught. Consequently, some readers may appreciate a review of how to estimate and interpret various effect sizes. In addition to the tutorial, the authors recommend effect size interpretations that emphasize direct and explicit comparisons of effects in a new study with those reported in the prior related literature, with a focus on evaluating result replicability. For decades, statistical significance has been the norm for evaluating results. In fact, little change has occurred since Carver (1993) noted: “A quick perusal of research journals, educational and psychological statistic textbooks, and doctoral dissertations will confirm that tests of statistical significance continue to dominate the interpretation of quantitative data in social science research” (p. 294). Although statistical significance “evaluates the probability or likelihood of the sample results, given the sample size, and assuming that the sample came from a population in which the null hypothesis is exactly true” (Thompson, 2003, p. 7), statistical

Psychometrics is datametrics: The test is not reliable.

The present article responds to selected criticisms of some EPM editorial policies and Vacha-Haase’s “reliability generalization” meta-analytic methods. However, the treatment is more broadly a manifesto regarding the nature of score reliability and what are reasonable expectations for psychometric reporting practices in substantive inquiries. The consequences of misunderstandings of score reliability are explored. It is suggested that paradigmatic misconceptions regarding psychometric issues feed into a spiral of presumptions that measurement training is unnecessary for doctoral students, which then in turn further reinforces misunderstandings of score integrity issues.

A Reliability Generalization Study of the Teacher Efficacy Scale and Related Instruments.

Teacher efficacy has proven to be an important variable in teacher effectiveness. It is consistently related to positive teaching behaviors and student outcomes. However, the measurement of this construct is the subject of current debate, which includes critical examination of predominant instruments used to assess teacher efficacy. The present study extends this critical evaluation and examines sources of measurement error variance in the Teacher Efficacy Scale (TES), historically the most frequently used instrument in the area. Reliability generalization was used to characterize the typical score reliability for the TES and potential sources of measurement error variance across studies. Other related instruments were also examined as regards measurement integrity.

Sample Compositions and Variabilities in Published Studies versus Those in Test Manuals: Validity of Score Reliability Inductions

As measurement specialists, we have done a disservice to both ourselves and our profession by habitually referring to “the reliability of the test,” or saying that “the test is reliable.” This has created a mind-set implying that reliability, once proven, is immutable. More important, practitioners and scholars need not know measurement theories if they may simply rely on the reliability purportedly intrinsic within all uses of established measures. The present study investigated empirically exactly how dissimilar in both composition and variability samples inducting reliability coefficients from prior studies were from the cited prior samples from which coefficients were generalized.

Trainee Impairment A Review of Identification, Remediation, Dismissal, and Legal Issues

This article reviews the professional literature on the topic of evaluating the competence of trainees in professional psychology training programs including program policies, procedures, and actual practice for identifying, remediating, and, in extreme cases, dismissing trainees who are judged unable to provide competent, professional care. This review covers the literature on the following major issues related to trainee performance: (a) problems with definitions of impairment, (b) established professional standards for supervision and evaluation of trainees based on accreditation guidelines and ethical standards, (c) methodological critiques of empirical studies on trainee impairment, (d) issues related to evaluation and identification of trainees who are making inadequate progress toward professional competence, (e) issues related to remediation, (f) dismissal and due process, and (g) relevant legal cases and considerations. The review of these topics provides the platform for an extensive list of recommendations directed toward faculty and supervisors responsible for professional psychology training programs and internships.

Reliability generalization: Moving toward improved understanding and use of score reliability

Reliability generalization (RG) is a measurement meta-analytic method used to explore the variability in score reliability estimates and to characterize the possible sources of this variance. This article briefly summarizes some RG considerations. Included is a description of how reliability confidence intervals might be portrayed graphically. The article includes tabulations across various RG studies, including how frequently authors (a) report score reliabilities for their own data, (b) conduct reliability induction, or (c) do not even mention reliability.

Practices Regarding Reporting of Reliability Coefficients: A Review of Three Journals.

Practices regarding the reporting of reliability coefficients in 3 journals from 1990 to 1997 were examined. Given that scores, not tests, are reliable or unreliable, particular attention was paid to the provision of reliability coefficients computed for the data actually being analyzed in substantive studies. One third of the articles reviewed made no mention of reliability. Almost 36% of the articles provided reliability coefficients for the data being analyzed. Examples of good reporting practices are provided. In 2 of the 3 journals reviewed, there was little change in the frequency and style of reliability reporting in the period covered. The authors suggest a modification in editorial journal policies to bring about a change in reliability-coefficient reporting practices.

Problematic students: Gatekeeping practices of academic professional psychology programs

When the topic of problematic students arises, there are often more questions than answers. Professional psychology programs may serve as gatekeepers for the profession, yet there is little guidance on intervening with problematic students. This study surveyed training directors (TDs) of American Psychological Association-accredited academic psychology programs regarding problematic student behaviors. Over half of the programs that responded had terminated at least 1 student during a 3-year period, with TDs citing inadequate clinical skills as the major reason. Despite accreditation policies, 54% of the programs did not have written guidelines for intervening with problematic students. Recommendations to promote consistent evaluation and intervention at the faculty, student, program, and national-policy level are discussed.

Statistical Significance should not be Considered one of Life’s Guarantees: Effect Sizes are Needed

It has been said that there are two guarantees in life—death and taxes. Unfortunately, in social science research a third certainty appears to have emerged, that of the statistical significance test. As Shaver (1993) noted, “A quick perusal of research journals, educational and psychological statistic textbooks, and doctoral dissertations will confirm that tests of statistical significance continue to dominate the interpretation of quantitative data in social science research” (p. 294). Although social science research continues to be ruled by a tradition of statistical significance testing (Carver, 1993; Shaver, 1993; Steiger & Fouladi, 1997), probably few methodological issues have generated as much controversy as statistical significance testing (Nickerson, 2000). Indeed, across decades and disciplines, the frequencies of publication of criticisms have grown appreciably (Anderson, Burnham, & Thompson, 2000). In the ongoing decades of debate regarding the value of statistical testing, some (e.g., Carver, 1978, 1993; Schmidt, 1996) have argued that statistical significance testing is an overused and abused means of evaluating research results and should be banned. Others (e.g., Cohen, 1990, 1994; Kirk, 1996; Thompson, 1996, 1999b) have argued that these tests should be used and interpreted correctly, and that other statistics—especially effect sizes— should receive greater attention. Still others (e.g., Cortina & Dunlap, 1997; Frick, 1996) have argued that there is little or nothing wrong with contemporary analytic practices, though some of these arguments have themselves been seen to be flawed (e.g., Hagen, 1997; Thompson, 1998). Perhaps statistical significant testing in itself is not inherently evil, but problems instead lie in the way these tests are utilized. Dar (1987) explained,

Score Reliability: A Retrospective Look Back at 12 Years of Reliability Generalization Studies

The present study was conducted to characterize (a) the features of the thousands of primary reports synthesized in 47 reliability generalization (RG) measurement meta-analysis studies and (b) typical methodological practice within the RG literature to date. With respect to the treatment of score reliability in the literature, in an astounding 54.6% of the 12,994 primary reports authors did not even mention reliability! Furthermore, in 15.7% of the primary reports authors did mention score reliability, but merely inducted previously reported values as if they applied to their data. Clearly, the admonitions of Wilkinson and the APA Task Force (1999) have yet to have their desired impacts with respect to reporting reliability estimates for one’s own data.