The need for intellectual diversity in psychological science: Our own studies of actively open-minded thinking as a case study

Abstract

Actively open-minded thinking (AOT) is measured by questionnaire items that tap the willingness to consider alternative opinions, the sensitivity to evidence contradictory to current beliefs, the willingness to postpone closure, and reflective thought. AOT has been found to be a strong predictor of performance on heuristics and biases tasks and of the avoidance of reasoning traps such as superstitious thinking and belief in conspiracy theories. Recently, several studies that have employed short forms of the AOT scale have shown startlingly high negative correlations with religiosity (in the range of -0.50 to -0.70). In a re-analysis of a large dataset, we demonstrate that it was a particular type of AOT item (termed a belief revision item, BR) that accounts for these large correlations. To our consternation, we realized that it was our research team that had introduced these items into the literature two decades ago, but we had heretofore never realized the potential for these items to skew correlations. In a new experiment, we demonstrate how BR items of this type disadvantage religious-minded subjects, and we show that it is possible to construct BR items with parallel content that are not so demographically biased. We also show that unbiased BR items do not sacrifice the predictive power that has previously been shown by AOT scales. We believe this lesson in item construction resulted from the lack of intellectual diversity in our own laboratory (specifically, the overwhelmingly secular composition of our lab personnel). We believe this case study shows the importance of intellectual diversity in psychology, especially when studying such topics as religiosity and political attitudes.