Pam Mueller

Using Mechanical Turk to Study Clinical Populations

Although participants with psychiatric symptoms, specific risk factors, or rare demographic characteristics can be difficult to identify and recruit for participation in research, participants with these characteristics are crucial for research in the social, behavioral, and clinical sciences. Online research in general and crowdsourcing software in particular may offer a solution. However, no research to date has examined the utility of crowdsourcing software for conducting research on psychopathology. In the current study, we examined the prevalence of several psychiatric disorders and related problems, as well as the reliability and validity of participant reports on these domains, among users of Amazon’s Mechanical Turk. Findings suggest that crowdsourcing software offers several advantages for clinical research while providing insight into potential problems, such as misrepresentation, that researchers should address when collecting data online.

Nonnaïveté among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers.

Crowdsourcing services—particularly Amazon Mechanical Turk—have made it easy for behavioral scientists to recruit research participants. However, researchers have overlooked crucial differences between crowdsourcing and traditional recruitment methods that provide unique opportunities and challenges. We show that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants. We describe how both of these problems can be avoided using advanced interface features that also allow prescreening and longitudinal data collection. Using these techniques can minimize the effects of previously ignored drawbacks and expand the scope of crowdsourcing as a tool for psychological research.

The Pen Is Mightier Than the Keyboard Advantages of Longhand Over Laptop Note Taking

Taking notes on laptops rather than in longhand is increasingly common. Many researchers have suggested that laptop note taking is less effective than longhand note taking for learning. Prior studies have primarily focused on students’ capacity for multitasking and distraction when using laptops. The present research suggests that even when laptops are used solely to take notes, they may still be impairing learning because their use results in shallower processing. In three studies, we found that students who took notes on laptops performed worse on conceptual questions than students who took notes longhand. We show that whereas taking more notes can be beneficial, laptop note takers’ tendency to transcribe lectures verbatim rather than processing information and reframing it in their own words is detrimental to learning.

Risks and Rewards of Crowdsourcing Marketplaces

Crowdsourcing has become an increasingly popular means of flexibly deploying large amounts of human computational power. The present chapter investigates the role of microtask labor marketplaces in managing human and hybrid human machine computing. Labor marketplaces offer many advantages that in combination allow human intelligence to be allocated across projects rapidly and efficiently and information to be transmitted effectively between market participants. Human computation comes with a set of challenges that are distinct from machine computation, including increased unsystematic error (e.g. mistakes) and systematic error (e.g. cognitive biases), both of which can be exacerbated when motivation is low, incentives are misaligned, and task requirements are poorly communicated. We provide specific guidance about how to ameliorate these issues through task design, workforce selection, data cleaning and aggregation.

When Does Knowledge Become Intent? Perceiving the Minds of Wrongdoers

In a series of experimental studies, we asked people to assign appropriate civil and/or criminal liability to individuals who cause harm with various culpable states of mind and kinds of knowledge. The studies are principally aimed at two related issues. First, do people actually separate the various states of mind conceptually? How much knowledge, and what kind of knowledge, regarding something that may go wrong (understanding risk) is sufficient to count as knowing that something will go wrong (having knowledge legally equivalent to intent)? Second, to the extent that people distinguish among the states of mind that help define normative behavior, how much do those distinctions contribute to peoples judgments of civil liability? Our studies show that people are able to make explicit distinctions about the states of mind of others that more or less correspond to legally relevant categories. Yet, when asked to assign consequences, their “hot” moral judgments play a larger role than do their “cold” cognitive categorizations.