Matthew J. C. Crump

Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenters control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

The context-specific proportion congruent Stroop effect: Location as a contextual cue.

The Stroop effect has been shown to depend on the relative proportion of congruent and incongruent trials. This effect is commonly attributed to experiment-wide word-reading strategies that change as a function of proportion congruent. Recently, Jacoby, Lindsay, and Hessels (2003) reported an itemspecific proportion congruent effect that cannot be due to these strategies and instead may reflect rapid, stimulus driven control over word-reading processes. However, an item-specific proportion congruent effect may also reflect learned associations between color word identities and responses. In two experiments, we demonstrate a context-specific proportion congruent effect that cannot be explained by such word—response associations. Our results suggest that processes other than learning of word—response associations can produce contextual control over Stroop interference.

In Support of a Distinction between Voluntary and Stimulus-Driven Control: A Review of the Literature on Proportion Congruent Effects

Cognitive control is by now a large umbrella term referring collectively to multiple processes that plan and coordinate actions to meet task goals. A common feature of paradigms that engage cognitive control is the task requirement to select relevant information despite a habitual tendency (or bias) to select goal-irrelevant information. At least since the 1970s, researchers have employed proportion congruent (PC) manipulations to experimentally establish selection biases and evaluate the mechanisms used to control attention. PC manipulations vary the frequency with which irrelevant information conflicts (i.e., is incongruent) with relevant information. The purpose of this review is to summarize the growing body of literature on PC effects across selective attention paradigms, beginning first with Stroop, and then describing parallel effects in flanker and task-switching paradigms. The review chronologically tracks the expansion of the PC manipulation from its initial implementation at the list-wide level, to more recent implementations at the item-specific and context-specific levels. An important theoretical aim is demonstrating that PC effects at different levels (e.g., list-wide vs. item or context-specific) support a distinction between voluntary forms of cognitive control, which operate based on anticipatory information, and relatively automatic or reflexive forms of cognitive control, which are rapidly triggered by the processing of particular stimuli or stimulus features. A further aim is to highlight those PC manipulations that allow researchers to dissociate stimulus-driven control from other stimulus-driven processes (e.g., S-R responding; episodic retrieval). We conclude by discussing the utility of PC manipulations for exploring the distinction between voluntary control and stimulus-driven control in other relevant paradigms.

Contingency learning without awareness : Evidence for implicit control

The results of four experiments provide evidence for controlled processing in the absence of awareness. Participants identified the colour of a neutral distracter word. Each of four words (e.g., MOVE) was presented in one of the four colours 75% of the time (Experiments 1 and 4) or 50% of the time (Experiments 2 and 3). Colour identification was faster when the words appeared in the colour they were most often presented in relative to when they appeared in another colour, even for participants who were subjectively unaware of any contingencies between the words and the colours. An analysis of sequence effects showed that participants who were unaware of the relation between distracter words and colours nonetheless controlled the impact of the word on performance depending on the nature of the previous trial. A block analysis of contingency-unaware participants revealed that contingencies were learned rapidly in the first block of trials. Experiment 3 showed that the contingency effect does not depend on the level of awareness, thus ruling out explicit strategy accounts. Finally, Experiment 4 showed that the contingency effect results from behavioural control and not from semantic association or stimulus familiarity. These results thus provide evidence for implicit control.

Cognitive Illusions of Authorship Reveal Hierarchical Error Detection in Skilled Typists

Touchy Typing Even the most able typist makes errors, and Logan and Crump (p. 683) have used this real-world task to probe for the existence of two error-detection mechanisms. They inserted errors into words that had been typed correctly by the subjects, and they corrected errors that had been made. By measuring implicit error detection as the slowing of movement just after an error had been committed and by eliciting explicit monitoring of errors by the output shown on the screen, they uncovered a double dissociation. Inserted errors did not lengthen the interval until the next letter was typed, but they were reported by the typist as errors; on the other hand, corrected errors did increase the interval, but were nevertheless claimed by the subjects as having been typed correctly. One error-detection mechanism monitors the correctness of one’s action, whereas a second mechanism monitors the output. The ability to detect errors is an essential component of cognitive control. Studies of error detection in humans typically use simple tasks and propose single-process theories of detection. We examined error detection by skilled typists and found illusions of authorship that provide evidence for two error-detection processes. We corrected errors that typists made and inserted errors in correct responses. When asked to report errors, typists took credit for corrected errors and accepted blame for inserted errors, claiming authorship for the appearance of the screen. However, their typing rate showed no evidence of these illusions, slowing down after corrected errors but not after inserted errors. This dissociation suggests two error-detection processes: one sensitive to the appearance of the screen and the other sensitive to keystrokes.

The flexibility of context-specific control: Evidence for context-driven generalization of item-specific control settings

In two experiments we address an ongoing debate concerning the processes driving context-driven modulations to the Stroop effect (Crump, Gong, & Milliken, 2006). In particular, we demonstrate that context-driven processes can modulate the size of the Stroop effect for frequency-unbiased item types. We also clarify the role of item frequency in producing context-driven modulations to the Stroop effect. Taken together, our results provide unambiguous support for the claim that contextual processing can impart fast and flexible control over the operation of selective attention processes during online performance.

Context-specific learning and control: The roles of awareness, task relevance, and relative salience

The processes mediating dynamic and flexible responding to rapidly changing task-environments are not well understood. In the present research we employ a Stroop procedure to clarify the contribution of context-sensitive control processes to online performance. In prior work Stroop interference varied as a function of probe location context, with larger Stroop interference occurring for contexts associated with a high proportion of congruent items [Crump, M. J., Gong, Z., & Milliken, B. (2006). The context-specific proportion congruent stroop effect: location as a contextual cue. Psychonomic Bulletin &Review, 13, 316-321.] Here, we demonstrate that this effect does not depend on awareness of the context manipulation, but that it can depend on attention to the predictive context dimension, and on the relative salience of the target and predictive context dimensions. We discuss the implications of our results for current theories of cognitive control.

The Left Hand Doesn't Know What the Right Hand Is Doing The Disruptive Effects of Attention to the Hands in Skilled Typewriting

Everyone knows that attention to the details disrupts skilled performance, but little empirical evidence documents this fact. We show that attention to the hands disrupts skilled typewriting. We had skilled typists type words preceded by cues that told them to type only the letters assigned to one hand or to type all of the letters. Cuing the hands disrupted performance markedly, slowing typing and increasing the error rate (Experiment 1); these deleterious effects were observed even when no keystrokes were actually inhibited (Experiment 3). However, cuing the same letters with colors was not disruptive (Experiment 2). We account for the disruption with a hierarchical control model, in which an inner loop controls the hands and an outer loop controls what is typed. Typing letters using only one hand requires the outer loop to monitor the inner loops output; the outer loop slows inner-loop cycle time to increase the likelihood of inhibiting responses with the unwanted hand. This produces the disruption.

Hierarchical Control of Cognitive Processes: The Case for Skilled Typewriting

The idea that cognition is controlled hierarchically is appealing to many but is difficult to demonstrate empirically. Often, nonhierarchical theories can account for the data as well as hierarchical ones do. The purpose of this chapter is to document the case for hierarchical control in skilled typing and present it as an example of a strategy for demonstrating hierarchical control in other cognitive acts. We propose that typing is controlled by two nested feedback loops that can be distinguished in terms of the factors that affect them, that communicate through intermediate representations (words), that know little about how each other work, and rely on different kinds of feedback. We discuss hierarchical control in other skills; the relation between hierarchical control and familiar concepts like automaticity, procedural memory, and implicit knowledge; and the development of hierarchical skills. We end with speculations about the role of hierarchical control in everyday cognition and the search for a meaningful life.Abstract The idea that cognition is controlled hierarchically is appealing to many but is difficult to demonstrate empirically. Often, nonhierarchical theories can account for the data as well as hierarchical ones do. The purpose of this chapter is to document the case for hierarchical control in skilled typing and present it as an example of a strategy for demonstrating hierarchical control in other cognitive acts. We propose that typing is controlled by two nested feedback loops that can be distinguished in terms of the factors that affect them, that communicate through intermediate representations (words), that know little about how each other work, and rely on different kinds of feedback. We discuss hierarchical control in other skills; the relation between hierarchical control and familiar concepts like automaticity, procedural memory, and implicit knowledge; and the development of hierarchical skills. We end with speculations about the role of hierarchical control in everyday cognition and the search for a meaningful life.

Hierarchical Control and Skilled Typing: Evidence for Word-Level Control over the Execution of Individual Keystrokes.

Routine actions are commonly assumed to be controlled by hierarchically organized processes and representations. In the domain of typing theories, word-level information is assumed to activate the constituent keystrokes required to type each letter in a word. We tested this assumption directly using a novel single-letter probe technique. Subjects were primed with a visual or auditory word or a visually presented random consonant string and then probed to type a single letter from the prime or another randomly selected letter. Relative to randomly selected letters, probe responses were speeded for first, middle, and last letters contained in visual and auditory word primes but not for middle and last letters contained in random consonant primes. This suggests that word-level information causes parallel activation of constituent keystrokes, consistent with hierarchical processing. The role of hierarchical processing in typing and routine action is discussed.