Todd M. Gureckis

Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenters control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

SUSTAIN: a network model of category learning.

SUSTAIN (Supervised and Unsupervised STratified Adaptive Incremental Network) is a model of how humans learn categories from examples. SUSTAIN initially assumes a simple category structure. If simple solutions prove inadequate and SUSTAIN is confronted with a surprising event (e.g., it is told that a bat is a mammal instead of a bird), SUSTAIN recruits an additional cluster to represent the surprising event. Newly recruited clusters are available to explain future events and can themselves evolve into prototypes-attractors-rules. SUSTAINs discovery of category substructure is affected not only by the structure of the world but by the nature of the learning task and the learners goals. SUSTAIN successfully extends category learning models to studies of inference learning, unsupervised learning, category construction, and contexts in which identification learning is faster than classification learning.

Self-Directed Learning: A Cognitive and Computational Perspective

A widely advocated idea in education is that people learn better when the flow of experience is under their control (i.e., learning is self-directed). However, the reasons why volitional control might result in superior acquisition and the limits to such advantages remain poorly understood. In this article, we review the issue from both a cognitive and computational perspective. On the cognitive side, self-directed learning allows individuals to focus effort on useful information they do not yet possess, can expose information that is inaccessible via passive observation, and may enhance the encoding and retention of materials. On the computational side, the development of efficient “active learning” algorithms that can select their own training data is an emerging research topic in machine learning. This review argues that recent advances in these related fields may offer a fresh theoretical perspective on how people gather information to support their own learning.

Emergent Processes in Group Behavior

Just as neurons interconnect in networks that create structured thoughts beyond the ken of any individual neuron, so people spontaneously organize themselves into groups to create emergent organizations that no individual may intend, comprehend, or even perceive. Recent technological advances have provided us with unprecedented opportunities for conducting controlled laboratory experiments on human collective behavior. We describe two experimental paradigms in which we attempt to build predictive bridges between the beliefs, goals, and cognitive capacities of individuals and patterns of behavior at the group level, showing how the members of a group dynamically allocate themselves to resources and how innovations diffuse through a social network. Agent-based computational models have provided useful explanatory and predictive accounts. Together, the models and experiments point to tradeoffs between exploration and exploitation—that is, compromises between individuals using their own innovations and using innovations obtained from their peers—and the emergence of group-level organizations such as population waves, bandwagon effects, and spontaneous specialization.

Regulatory Fit and Systematic Exploration in a Dynamic Decision-Making Environment

This work explores the influence of motivation on choice behavior in a dynamic decision-making environment, where the payoffs from each choice depend on ones recent choice history. Previous research reveals that participants in a regulatory fit exhibit increased levels of exploratory choice and flexible use of multiple strategies over the course of an experiment. The present study placed promotion and prevention-focused participants in a dynamic environment for which optimal performance is facilitated by systematic exploration of the decision space. These participants either gained or lost points with each choice. Our experiment revealed that participants in a regulatory fit were more likely to engage in systematic exploration of the task environment than were participants in a regulatory mismatch and performed more optimally as a result. Implications for contemporary models of human reinforcement learning are discussed.

psiTurk: An open-source framework for conducting replicable behavioral experiments online

Online data collection has begun to revolutionize the behavioral sciences. However, conducting carefully controlled behavioral experiments online introduces a number of new of technical and scientific challenges. The project described in this paper, psiTurk, is an open-source platform which helps researchers develop experiment designs which can be conducted over the Internet. The tool primarily interfaces with Amazon’s Mechanical Turk, a popular crowd-sourcing labor market. This paper describes the basic architecture of the system and introduces new users to the overall goals. psiTurk aims to reduce the technical hurdles for researchers developing online experiments while improving the transparency and collaborative nature of the behavioral sciences.

Re-evaluating dissociations between implicit and explicit category learning: An event-related fmri study

Recent fMRI studies have found that distinct neural systems may mediate perceptual category learning under implicit and explicit learning conditions. In these previous studies, however, different stimulus-encoding processes may have been associated with implicit versus explicit learning. The present design was aimed at decoupling the influence of these factors on the recruitment of alternate neural systems. Consistent with previous reports, following incidental learning in a dot-pattern classification task, participants showed decreased neural activity in occipital visual cortex (extrastriate region V3, BA 19) in response to novel exemplars of a studied category compared to members of a foil category, but did not show this decreased neural activity following explicit learning. Crucially, however, our results show that this pattern was primarily modulated by aspects of the stimulus-encoding instructions provided at the time of study. In particular, when participants in an implicit learning condition were encouraged to evaluate the overall shape and configuration of the stimuli during study, we failed to find the pattern of brain activity that has been taken to be a signature of implicit learning, suggesting that activity in this area does not uniquely reflect implicit memory for perceptual categories but instead may reflect aspects of processing or perceptual encoding strategies.

Navigating through abstract decision spaces: Evaluating the role of state generalization in a dynamic decision-making task

Research on dynamic decision-making tasks, in which the payoffs associated with each choice vary with participants’ recent choice history, shows that humans have difficulty making long-term optimal choices in the presence of attractive immediate rewards. However, a number of recent studies have shown that simple cues providing information about the underlying state of the task environment may facilitate optimal responding. In this study, we examined the mechanism by which this state knowledge influences choice behavior. We examined the possibility that participants use state information in conjunction with changing payoffs to extrapolate payoffs in future states. We found support for this hypothesis in an experiment in which generalizations based on this state information worked to the benefit or detriment of task performance, depending on the task’s payoff structure.

Strategies to intervene on causal systems are adaptively selected.

How do people choose interventions to learn about causal systems? Here, we considered two possibilities. First, we test an information sampling model, information gain, which values interventions that can discriminate between a learners hypotheses (i.e. possible causal structures). We compare this discriminatory model to a positive testing strategy that instead aims to confirm individual hypotheses. Experiment 1 shows that individual behavior is described best by a mixture of these two alternatives. In Experiment 2 we find that people are able to adaptively alter their behavior and adopt the discriminatory model more often after experiencing that the confirmatory strategy leads to a subjective performance decrement. In Experiment 3, time pressure leads to the opposite effect of inducing a change towards the simpler positive testing strategy. These findings suggest that there is no single strategy that describes how intervention decisions are made. Instead, people select strategies in an adaptive fashion that trades off their expected performance and cognitive effort.

Towards a unified account of supervised and unsupervised category learning

(Supervised and Unsupervised STratified Adaptive IncrementalNetwork) is a network model of human category learning. SUSTAIN initially assumes a simple category structure. If simple solutions prove inadequate and SUSTAIN is confronted with a surprising event (e.g. it is told that a bat is a mammal instead of a bird), SUSTAIN recruits an additional cluster to represent the surprising event. Newly recruited clusters are available to explain future events and can themselves evolve into prototypes/attractors/rules. SUSTAIN has expanded the scope of findings that models of human category learning can address. This paper extends SUSTAIN so that it can be used to account for both supervised and unsupervised learning data through a common mechanism. A modified recruitment rule is introduced that creates new conceptual clusters in response to surprising events during learning. The new formulation of the model is called uSUSTAIN for ‘unified SUSTAIN.’ The implications of using a unified recruitment method for both supervised and unsupervised learning are discussed.