Ariel M. Aloe

Life After NHST: How to Describe Your Data Without “p-ing” Everywhere

In this article we provide concrete guidance to researchers on ways that they can explore and communicate the results of their studies. Although we believe the methods we outline are important for any study, they are particularly useful for researchers who wish to avoid the null hypothesis significance testing paradigm. We articulate three basic principles of data presentation: (a) use graphic displays to facilitate understanding of descriptive statistics, (b) provide measures of variability with measures of central tendency for continuous outcomes, and (c) compute and thoughtfully interpret effect sizes and effect size translations. We then put these principles into action using data drawn from two real social psychological experiments and provide tools (including software code and a new effect size translation) that will help researchers to quickly and efficiently adopt the recommendations that they find sensible.

Measurement of the bystander intervention model for bullying and sexual harassment

Although peer bystanders can exacerbate or prevent bullying and sexual harassment, research has been hindered by the absence of a validated assessment tool to measure the process and sequential steps of the bystander intervention model. A measure was developed based on the five steps of Latané and Darleys (1970) bystander intervention model applied to bullying and sexual harassment. Confirmatory factor analysis with a sample of 562 secondary school students confirmed the five-factor structure of the measure. Structural equation modeling revealed that all the steps were influenced by the previous step in the model, as the theory proposed. In addition, the bystander intervention measure was positively correlated with empathy, attitudes toward bullying and sexual harassment, and awareness of bullying and sexual harassment facts. This measure can be used for future research and to inform intervention efforts related to the process of bystander intervention for bullying and sexual harassment.

An Effect Size for Regression Predictors in Meta-Analysis

A new effect size representing the predictive power of an independent variable from a multiple regression model is presented. The index, denoted as rsp, is the semipartial correlation of the predictor with the outcome of interest. This effect size can be computed when multiple predictor variables are included in the regression model and represents a partial effect size in the correlation family. The derivations presented in this article provide the effect size and its variance. Standard errors and confidence intervals can be computed for individual rsp values. Also, meta-analysis of the semipartial correlations can proceed in a similar fashion to typical meta-analyses, where weighted analyses can be used to explore heterogeneity and to estimate central tendency and variation in the effects. The authors provide an example from a meta-analysis of studies of the relationship of teacher verbal ability to school outcomes.

Teacher Verbal Ability and School Outcomes Where Is the Evidence

Many have claimed that teachers’ verbal ability is among the most important predictors of school outcomes. Teachers’ verbal ability has been thought to predict student achievement ever since the relationship was found in the classic Equality of Educational Opportunity (EEO) study by Coleman et al. By way of meta-analysis the authors examine the evidence on teachers’ verbal ability as a predictor of school outcomes. They find that the evidence is not as extensive as might be inferred from prior reports. Results of 19 studies indicate that teachers’ verbal ability is at best very weakly related to school outcomes, and the main evidence for this weak relationship arises from the EEO study. Other studies find that verbal ability is unrelated to school outcomes.

The Door-in-the-Face Persuasive Message Strategy: A Meta-Analysis of the First 35 Years

A random-effects meta-analysis was undertaken to examine the effectiveness of the Door-in-the-Face (DITF) persuasive message strategy on compliance. Results indicate an overall significant effect of the DITF strategy on verbal compliance (k=78, r=.126), but an insignificant effect for behavioral compliance (k=39, r=.052). In terms of verbal compliance, the DITF strategy works significantly better than controls for different samples, across varied communication media, and for prosocial causes. Additionally, the DITF technique is more successful than controls for volunteering/research than other target behaviors (e.g., monetary donation). For both verbal and behavioral compliance outcomes, the toughness (measured as amount of baseline compliance) of the donation context negatively predicted the magnitude of the DITF effect. It is argued social responsibility theory best accounts for observed moderator factors.

An alternative to R2 for assessing linear models of effect size

Reviewers often use regression models in meta-analysis (‘meta-regressions’) to examine the relationships between effect sizes and study characteristics. In this paper, we propose and illustrate the use of an index (R) that expresses the amount of variance in the outcome that is explained by the meta-regression model. The values of R2 obtained from the standard computer output for linear models of effect size in the meta-analysis context are typically too small, because the typical R2 considers sampling variance to be unexplained whereas in meta-analysis it can be quantified. Although the idea of removing the unexplainable variance from the estimator of variance accounted for in meta-analysis is not new (Cook et al., 1992; Raudenbush, 1994) we explicitly define four estimators of variance explained, and illustrate via two examples that the typical R2 obtained in a linear model of effect size is always lower than our indices. Thus, the typical R2 underestimates the explanatory power of linear models of effect sizes. Our four estimators improve upon typical weighted R2 values. Copyright

Inaccuracy of regression results in replacing bivariate correlations

This manuscript considers discrepancies between the bivariate correlation and several indices of association estimated from regression results. These indices can be estimated from results typically reported in primary studies. In recent years, many researchers conducting meta-analyses have used these indices in place of, or together with, the bivariate correlation. I illustrate the differences among these indices and the bivariate correlation. I demonstrate the inaccuracy of these indices as replacements for bivariate effects. Thus, I recommend discontinuing the use of these indices and partial effect sizes as replacement for the bivariate correlation.

An Empirical Investigation of Partial Effect Sizes in Meta-Analysis of Correlational Data

ABSTRACT. The partial correlation and the semi-partial correlation can be seen as measures of partial effect sizes for the correlational family. Thus, both indices have been used in the meta-analysis literature to represent the relationship between an outcome and a predictor of interest, controlling for the effect of other variables in the model. This article evaluates the accuracy of synthesizing these two indices under different situations. Both partial correlation and the semi-partial correlation appear to behave as expected with respect to bias and root mean squared error (RMSE). However, the partial correlation seems to outperform the semi-partial correlation regarding Type I error of the homogeneity test (Q statistic). Although further investigation is needed to fully understand the impact of meta-analyzing partial effect sizes, the current study demonstrates the accuracy of both indices.

The Synthesis of Partial Effect Sizes

In this article we focus on three partial effect sizes for the correlation (r) family of effects: the standardized slope (b), the partial correlation (rp), and the semi-partial correlation (rsp). These partial effect sizes are useful for meta-analyses in two common situations: when primary studies reporting regression models do not report bivariate correlations, and when it is of specific interest to partial out the effects of other variables. We clarify the use of these three indices in the context of meta-analysis and describe how the indices can be estimated and analyzed. We provide examples of syntheses of these partial effect sizes using a published social work meta-analysis. Finally, we share practical recommendations for meta-analysts wanting to use such indices.

Quasi-experimental study designs series –Paper 9: Collecting Data from Quasi-Experimental Studies

OBJECTIVE To identify variables that must be coded when synthesizing primary studies that use quasi-experimental designs. STUDY DESIGN AND SETTING All quasi-experimental (QE) designs. RESULTS When designing a systematic review of QE studies, potential sources of heterogeneity-both theory-based and methodological-must be identified. We outline key components of inclusion criteria for syntheses of quasi-experimental studies. We provide recommendations for coding content-relevant and methodological variables and outlined the distinction between bivariate effect sizes and partial (i.e., adjusted) effect sizes. Designs used and controls used are viewed as of greatest importance. Potential sources of bias and confounding are also addressed. CONCLUSION Careful consideration must be given to inclusion criteria and the coding of theoretical and methodological variables during the design phase of a synthesis of quasi-experimental studies. The success of the meta-regression analysis relies on the data available to the meta-analyst. Omission of critical moderator variables (i.e., effect modifiers) will undermine the conclusions of a meta-analysis.