Thomas Scherndl

Publication bias in psychology: a diagnosis based on the correlation between effect size and sample size.

Background The p value obtained from a significance test provides no information about the magnitude or importance of the underlying phenomenon. Therefore, additional reporting of effect size is often recommended. Effect sizes are theoretically independent from sample size. Yet this may not hold true empirically: non-independence could indicate publication bias. Methods We investigate whether effect size is independent from sample size in psychological research. We randomly sampled 1,000 psychological articles from all areas of psychological research. We extracted p values, effect sizes, and sample sizes of all empirical papers, and calculated the correlation between effect size and sample size, and investigated the distribution of p values. Results We found a negative correlation of r = −.45 [95% CI: −.53; −.35] between effect size and sample size. In addition, we found an inordinately high number of p values just passing the boundary of significance. Additional data showed that neither implicit nor explicit power analysis could account for this pattern of findings. Conclusion The negative correlation between effect size and samples size, and the biased distribution of p values indicate pervasive publication bias in the entire field of psychology.

A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough?

Over-reliance on significance testing has been heavily criticized in psychology. Therefore the American Psychological Association recommended supplementing the p value with additional elements such as effect sizes, confidence intervals, and considering statistical power seriously. This article elaborates the conclusions that can be drawn when these measures accompany the p value. An analysis of over 30 summary papers (including over 6,000 articles) reveals that, if at all, only effect sizes are reported in addition to p’s (38%). Only every 10th article provides a confidence interval and statistical power is reported in only 3% of articles. An increase in reporting frequency of the supplements to p’s over time owing to stricter guidelines was found for effect sizes only. Given these practices, research faces a serious problem in the context of dichotomous statistical decision making: since significant results have a higher probability of being published (publication bias), effect sizes reported in articles may be seriously overestimated.

The significance fallacy in inferential statistics.

BackgroundStatistical significance is an important concept in empirical science. However the meaning of the term varies widely. We investigate into the intuitive understanding of the notion of significance.MethodsWe described the results of two different experiments published in a major psychological journal to a sample of students of psychology, labeling the findings as ‘significant’ versus ‘non-significant.’ Participants were asked to estimate the effect sizes and sample sizes of the original studies.ResultsLabeling the results of a study as significant was associated with estimations of a big effect, but was largely unrelated to sample size. Similarly, non-significant results were estimated as near zero in effect size.ConclusionsAfter considerable training in statistics, students largely equate statistical significance with medium to large effect sizes, rather than with large sample sizes. The data show that students assume that statistical significance is due to real effects, rather than to ‘statistical tricks’ (e.g., increasing sample size).

On the correlation between effect size and sample size: A reply

In a comment on our paper, Bradley and Brand (2013) argue that effect sizes are exaggerated owing to low power and publication bias. They propose to correct these exaggerations by application of a specific formula leading to a better estimate of the “true” effect size. In a simulation we test the effect of this formula and find this “corrective” approach unsatisfactory. We agree with Bradley and Brand on the points that effect sizes are important in primary and secondary research, and that exaggerated effect sizes are a serious problem in research. However, we disagree on the appropriate reaction: A diagnostic approach may be more appropriate than a corrective approach.

Evaluating the ICRA 2008 HRI challenge

This paper reports on the evaluation of the ICRA 2008 Human-Robot Interaction (HRI) Challenge. Five research groups demonstrated state-of-the-art work on HRI with a special focus on social and learning abilities. The demonstrations were rated by expert evaluators, in charge of awarding the prize, and 269 participants, i.e. 20 percent of the conference attendees through a standardized questionnaire (semantic differential). The data was analyzed with respect to six independent variables: expert evaluators vs. attendees, nationality of participants, origin region of the demo, age, gender and knowledge level of the attendees. Conference attendees tended to give higher scores for Social Skills, General Impression, and Overall Score than the expert evaluators. Irrespectively of the level of knowledge, age, and gender, conference attendees rated all demos relatively homogeneously. However, a comparative analysis of the conference attendeess ratings nationality-wise showed that demonstrations were rated differently depending on the region of origin. Conference attendees for the USA and Asian countries tended to rate demos from the same country of origin more frequently and more positively.

Global-local processing relates to spatial and verbal processing: implications for sex differences in cognition

Sex differences have been reported for a variety of cognitive tasks and related to the use of different cognitive processing styles in men and women. It was recently argued that these processing styles share some characteristics across tasks, i.e. male approaches are oriented towards holistic stimulus aspects and female approaches are oriented towards stimulus details. In that respect, sex-dependent cognitive processing styles share similarities with attentional global-local processing. A direct relationship between cognitive processing and global-local processing has however not been previously established. In the present study, 49 men and 44 women completed a Navon paradigm and a Kimchi Palmer task as well as a navigation task and a verbal fluency task with the goal to relate the global advantage (GA) effect as a measure of global processing to holistic processing styles in both tasks. Indeed participants with larger GA effects displayed more holistic processing during spatial navigation and phonemic fluency. However, the relationship to cognitive processing styles was modulated by the specific condition of the Navon paradigm, as well as the sex of participants. Thus, different types of global-local processing play different roles for cognitive processing in men and women.

Differentiating Self-Projection from Simulation during Mentalizing: Evidence from fMRI

We asked participants to predict which of two colors a similar other (student) and a dissimilar other (retiree) likes better. We manipulated if color-pairs were two hues from the same color-category (e.g. green) or two conceptually different colors (e.g. green versus blue). In the former case, the mental state that has to be represented (i.e., the percept of two different hues of green) is predominantly non-conceptual or phenomenal in nature, which should promote mental simulation as a strategy for mentalizing. In the latter case, the mental state (i.e. the percept of green versus blue) can be captured in thought by concepts, which facilitates the use of theories for mentalizing. In line with the self-projection hypothesis, we found that cortical midline areas including vmPFC / orbitofrontal cortex and precuneus were preferentially activated for mentalizing about a similar other. However, activation was not affected by the nature of the color-difference, suggesting that self-projection subsumes simulation-like processes but is not limited to them. This indicates that self-projection is a universal strategy applied in different contexts—irrespective of the availability of theories for mentalizing. Along with midline activations linked to self-projection, we also observed activation in right lateral frontal and dorsal parietal areas showing a theory-like pattern. Taken together, this shows that mentalizing does not operate based on simulation or theory, but that both strategies are used concurrently to predict the choices of others.

Components of Mathematics Anxiety: Factor Modeling of the MARS30-Brief

Mathematics anxiety involves feelings of tension, discomfort, high arousal, and physiological reactivity interfering with number manipulation and mathematical problem solving. Several factor analytic models indicate that mathematics anxiety is rather a multidimensional than unique construct. However, the factor structure of mathematics anxiety has not been fully clarified by now. This issue shall be addressed in the current study. The Mathematics Anxiety Rating Scale (MARS) is a reliable measure of mathematics anxiety (Richardson and Suinn, 1972), for which several reduced forms have been developed. Most recently, a shortened version of the MARS (MARS30-brief) with comparable reliability was published. Different studies suggest that mathematics anxiety involves up to seven different factors. Here we examined the factor structure of the MARS30-brief by means of confirmatory factor analysis. The best model fit was obtained by a six-factor model, dismembering the known two general factors “Mathematical Test Anxiety” (MTA) and “Numerical Anxiety” (NA) in three factors each. However, a more parsimonious 5-factor model with two sub-factors for MTA and three for NA fitted the data comparably well. Factors were differentially susceptible to sex differences and differences between majors. Measurement invariance for sex was established.

#Psychology: a bibliometric analysis of psychological literature in the online media

Online media and especially social media are becoming more and more relevant to our everyday life. Reflecting this tendency in the scientific community, alternative metrics for measuring scholarly impact on the web are increasingly proposed, extending (or even replacing) traditional metrics (e.g., citations, journal impact factor, etc.). This paper explores the relationship between traditional metrics and alternative metrics for psychological research in the years from 2010 to 2012. Traditional publication metrics (e.g., number of citations, impact factor) and alternative metrics (collected from Altmetric, a website that collects and counts references as they appear in Wikipedia, public policy documents, research blogs, mainstream media, or social networks) were extracted and compared, using a dataset of over 245,000 publications from the Web of Science. Results show positive, small to medium, correlations on the level of individual publications, and frequently medium to high correlations on the level of research fields of Psychology. The more accumulated the level of analysis, the higher the correlations. These findings are fairly robust over time and comparable to findings from research areas other than Psychology. Additionally, a new metric, the Score Factor, is proposed as a useful alternative metric to assess a journal’s impact in the online media.