John V. McDonnell

Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenters control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

Measuring category intuitiveness in unconstrained categorization tasks

What makes a category seem natural or intuitive? In this paper, an unsupervised categorization task was employed to examine observer agreement concerning the categorization of nine different stimulus sets. The stimulus sets were designed to capture different intuitions about classification structure. The main empirical index of category intuitiveness was the frequency of the preferred classification, for different stimulus sets. With 169 participants, and a within participants design, with some stimulus sets the most frequent classification was produced over 50 times and with others not more than two or three times. The main empirical finding was that cluster tightness was more important in determining category intuitiveness, than cluster separation. The results were considered in relation to the following models of unsupervised categorization: DIVA, the rational model, the simplicity model, SUSTAIN, an Unsupervised version of the Generalized Context Model (UGCM), and a simple geometric model based on similarity. DIVA, the geometric approach, SUSTAIN, and the UGCM provided good, though not perfect, fits. Overall, the present work highlights several theoretical and practical issues regarding unsupervised categorization and reveals weaknesses in some of the corresponding formal models.

psiTurk: An open-source framework for conducting replicable behavioral experiments online

Online data collection has begun to revolutionize the behavioral sciences. However, conducting carefully controlled behavioral experiments online introduces a number of new of technical and scientific challenges. The project described in this paper, psiTurk, is an open-source platform which helps researchers develop experiment designs which can be conducted over the Internet. The tool primarily interfaces with Amazon’s Mechanical Turk, a popular crowd-sourcing labor market. This paper describes the basic architecture of the system and introduces new users to the overall goals. psiTurk aims to reduce the technical hurdles for researchers developing online experiments while improving the transparency and collaborative nature of the behavioral sciences.

Formal Approaches in Categorization: Adaptive clustering models of categorization

Summary Numerous proposals have been put forward concerning the nature of human category representations, ranging from rules to exemplars to prototypes. However, it is unlikely that a single, fixed form of representation is sufficient to account for the flexibility of human categories. In this chapter, we describe an alternative to these fixed-representation accounts based on the principle of adaptive clustering. The specific model we consider, SUSTAIN, represents categories in terms of feature bundles called clusters which are adaptively recruited in response to task demands. In some cases, SUSTAIN acts like an exemplar model, storing each category instance as a separate memory trace, while in others it appears more like a prototype model, extracting only the central tendency of a number of items. In addition, selective attention in the model allows it to mimic many of the behaviours associated with rule-based systems. We review a variety of evidence in support of the clustering principle, including studies of the relationship between categorization and recognition memory, changes in unsupervised category learning abilities across development, and the influence of category learning on perceptual discrimination. In each case, we show how the nature of human category representations is best accounted for using an adaptive clustering scheme. SUSTAIN is just one example of a system that casts category learning in terms of adaptive clustering, and future directions for the approach are discussed.