Gabriele Paolacci

Inside the Turk: Understanding Mechanical Turk as a Participant Pool

Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this market have been assessed on dimensions that are universally relevant to understanding whether, why, and when they should be recruited as research participants. We discuss the characteristics of MTurk as a participant pool for psychology and other social sciences, highlighting the traits of the MTurk samples, why people become MTurk workers and research participants, and how data quality on MTurk compares to that from other pools and depends on controllable and uncontrollable factors.

Nonnaïveté among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers.

Crowdsourcing services—particularly Amazon Mechanical Turk—have made it easy for behavioral scientists to recruit research participants. However, researchers have overlooked crucial differences between crowdsourcing and traditional recruitment methods that provide unique opportunities and challenges. We show that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants. We describe how both of these problems can be avoided using advanced interface features that also allow prescreening and longitudinal data collection. Using these techniques can minimize the effects of previously ignored drawbacks and expand the scope of crowdsourcing as a tool for psychological research.

Risks and Rewards of Crowdsourcing Marketplaces

Crowdsourcing has become an increasingly popular means of flexibly deploying large amounts of human computational power. The present chapter investigates the role of microtask labor marketplaces in managing human and hybrid human machine computing. Labor marketplaces offer many advantages that in combination allow human intelligence to be allocated across projects rapidly and efficiently and information to be transmitted effectively between market participants. Human computation comes with a set of challenges that are distinct from machine computation, including increased unsystematic error (e.g. mistakes) and systematic error (e.g. cognitive biases), both of which can be exacerbated when motivation is low, incentives are misaligned, and task requirements are poorly communicated. We provide specific guidance about how to ameliorate these issues through task design, workforce selection, data cleaning and aggregation.

Lie for a Dime: When Most Prescreening Responses Are Honest but Most Study Participants Are Impostors

The Internet has enabled recruitment of large samples with specific characteristics. However, when researchers rely on participant self-report to determine eligibility, data quality depends on participant honesty. Across four studies on Amazon Mechanical Turk, we show that a substantial number of participants misrepresent theoretically relevant characteristics (e.g., demographics, product ownership) to meet eligibility criteria explicit in the studies, inferred by a previous exclusion from the study or inferred in previous experiences with similar studies. When recruiting rare populations, a large proportion of responses can be impostors. We provide recommendations about how to ensure that ineligible participants are excluded that are applicable to a wide variety of data collection efforts, which rely on self-report.

Coordination Under Threshold Uncertainty in a Public Goods Game

We explored experimentally how threshold uncertainty affects coordination success in a threshold public goods game. Whereas all groups succeeded in providing the public good when the exact value of the threshold was known, uncertainty was generally detrimental for the public good provision. The negative effect of threshold uncertainty was particularly severe when it took the form of ambiguity, i.e. when players were not only unaware of the value of the threshold but also of its probability distribution. Early signaling of willingness to contribute and share the burden equitably helped groups in coping with threshold uncertainty.

Crowdsourcing Samples in Cognitive Science

Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science. We review who is in the crowd and who can be reached by the average laboratory. We discuss reproducibility and review some recent methodological innovations for online experiments. We consider the design of research studies and arising ethical issues. We review how to code experiments for the web, what is known about video and audio presentation, and the measurement of reaction times. We close with comments about the high levels of experience of many participants and an emerging tragedy of the commons.

Participant Nonnaiveté and the reproducibility of cognitive psychology

Many argue that there is a reproducibility crisis in psychology. We investigated nine well-known effects from the cognitive psychology literature—three each from the domains of perception/action, memory, and language, respectively—and found that they are highly reproducible. Not only can they be reproduced in online environments, but they also can be reproduced with nonnaïve participants with no reduction of effect size. Apparently, some cognitive tasks are so constraining that they encapsulate behavior from external influences, such as testing situation and prior recent experience with the experiment to yield highly robust effects.

Inherent, constructed, revealed preferences: Guidelines for marketers

A theory of preferences was recently proposed on how consumers’ choices depend on both stable “inherent” likings and sensitivity to framing, task, and context effects (Simonson, 2008). Managers can employ this approach to overcome the contradictions exhibited so far in their treatment of preferences, and ultimately improve the outcome of the marketing process. The article proposes some practical methods to discover chronic dispositions and preference constructability, and suggests how to strategize over the inherent and the constructed components of consumer preferences.

Income Tax and the Motivation to Work

Does income tax influence the motivation to work? We propose that the degree of effort exertion in the presence of income tax depends on people’s attitudes toward two key components of taxation: redistribution and government intervention. For people favorable toward both, working while taxed is aligned with personal identity and may actually enhance motivation. All others, however, may find taxes demotivating. In two incentive-compatible labor experiments, framing wages as subject to an income tax reduced participants’ productivity unless they were chronically favorable toward both redistribution and government intervention. This latter group was significantly more productive when taxed. An objectively equivalent intervention that did not redistribute a portion of participants’ wages (framed as a wage “match�? rather than a “tax�?) did not motivate anyone to work harder. Our findings suggest that the net effect of income tax on productivity partly depends on the distribution of attitudes toward redistribution and government intervention.

Estensioni di marca e accettazione da parte dei consumatori

This article presents the results of a survey dealing with consumer acceptance of a brand extension. A questionnaire was submitted in order to gain perceptions about sixteen hypothetical brand extensions. By using some regression models, it was determined which factors affect consumers attitude toward a brand extension. The original contribution to brand extension research is the introduction of an instructive instrument called consumption relations matrix: this tool allowed analysis to be declined into different segments of consumers with reference to brands and products tested, corresponding to different interpretations of the results.