Aaron Shaw

Designing incentives for inexpert human raters

The emergence of online labor markets makes it far easier to use individual human raters to evaluate materials for data collection and analysis in the social sciences. In this paper, we report the results of an experiment - conducted in an online labor market - that measured the effectiveness of a collection of social and financial incentive schemes for motivating workers to conduct a qualitative, content analysis task. Overall, workers performed better than chance, but results varied considerably depending on task difficulty. We find that treatment conditions which asked workers to prospectively think about the responses of their peers - when combined with financial incentives - produced more accurate performance. Other treatments generally had weak effects on quality. Workers in India performed significantly worse than US workers, regardless of treatment group.

Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India

In this study we extend research on online collaboration by examining motivation to do work on the crowdsoucing service Amazon Mechanical Turk (MTurk). We address a challenge to many existing studies of motivation in online contexts: they are based on survey self-reports, which are susceptible to effects such as social desirability bias. In addition we investigate a second challenge to the extant research on motivation in the context of MTurk: a failure to examine potential differences between MTurk workers (Turkers) from different parts of the world, especially those from the US and India, MTurks two largest worker groups. Using a survey technique called the list experiment, we observe distinct profiles of motivation and patterns of social desirability effects among Turkers in the US and India. Among US Turkers, we find that social desirability encourages over-reporting of each of four motivating factors we examined. The over-reporting was particularly large in the case of money as a motivator. In contrast, among Turkers in India we find a more complex pattern of social desirability effects, with workers under-reporting killing time and fun as motivations, and drastically over-reporting sense of purpose. We conclude by discussing these results and proposing implications for future research and design.

A Tale of Two Blogospheres Discursive Practices on the Left and Right

In this article, the authors compare the practices of discursive production among top U.S. political blogs on the left and right during summer 2008. An examination of the top 155 political blogs reveals significant cross-ideological variations along several dimensions. Notably, the authors find evidence of an association between ideological affiliation and the technologies, institutions, and practices of participation. Blogs on the left adopt different, and more participatory, technical platforms, comprise significantly fewer sole-authored sites, include user blogs, maintain more fluid boundaries between secondary and primary content, include longer narrative and discussion posts, and (among the top half of the blogs in the sample) more often use blogs as platforms for mobilization. The findings suggest that the attenuation of the news producer-consumer dichotomy is more pronounced on the left wing of the political blogosphere than on the right. The practices of the left are more consistent with the prediction that the networked public sphere offers new pathways for discursive participation by a wider array of individuals, whereas the practices of the right suggest that a small group of elites may retain more exclusive agenda-setting authority online. The cross-ideological divergence in the findings illustrates that the Internet can be adopted equally to undermine or to replicate the traditional distinction between the production and consumption of political information. The authors conclude that these findings have significant implications for the study of prosumption and for the mechanisms by which the networked public sphere may or may not alter democratic participation relative to the mass mediated public sphere.

The Wikipedia Gender Gap Revisited: Characterizing Survey Response Bias with Propensity Score Estimation

Opt-in surveys are the most widespread method used to study participation in online communities, but produce biased results in the absence of adjustments for non-response. A 2008 survey conducted by the Wikimedia Foundation and United Nations University at Maastricht is the source of a frequently cited statistic that less than 13% of Wikipedia contributors are female. However, the same study suggested that only 39.9% of Wikipedia readers in the US were female – a finding contradicted by a representative survey of American adults by the Pew Research Center conducted less than two months later. Combining these two datasets through an application and extension of a propensity score estimation technique used to model survey non-response bias, we construct revised estimates, contingent on explicit assumptions, for several of the Wikimedia Foundation and United Nations University at Maastricht claims about Wikipedia editors. We estimate that the proportion of female US adult editors was 27.5% higher than the original study reported (22.7%, versus 17.8%), and that the total proportion of female editors was 26.8% higher (16.1%, versus 12.7%).

Digitally Savvy Citizenship: The Role of Internet Skills and Engagement in Young Adults' Political Participation around the 2008 Presidential Election

Popular narratives assume that digital media play a central role mobilizing voters and especially young adults. Based on unique survey data of a diverse group or young adults from Spring, 2009, we consider the relationship between differentiated internet uses, and online and offline political engagement around the time of the 2008 U.S. presidential election. Thanks to our rich data set, we are able to consider both online and offline activities while taking into consideration more traditional measures. Our findings suggest that online forms of political engagement complement offline engagement. The pathways to young adults political participation remain relatively stable. We also find an association between Internet skills, social network site usage and greater levels of engagement. These findings imply that although Internet usage alone is unlikely to transform existing patterns in political participation radically, it may facilitate the creation of new pathways for engagement.

Scalable crisis relief: Crowdsourced SMS translation and categorization with Mission 4636

Crowdsourced crisis response harnesses distributed networks of humans in combination with information and communication technology (ICT) to create scalable, flexible and rapid communication systems that promote well-being, survival, and recovery during the acute phase of an emergency. In this paper, we analyze a recent experience in which CrowdFlower conducted crowdsourced translation, categorization and geo-tagging for SMS-based reporting as part of Mission 4636 after a 7.0 magnitude earthquake struck Haiti on January 12, 2010. We discuss CrowdFlowers approach to this task, lessons learned from the experience, and opportunities to generalize the techniques and technologies involved for other ICT for development (ICTD) applications. We find that CrowdFlowers most significant contribution to Mission 4636 and to the broader field of crowdsourced crisis relief lies in the flexible, scalable nature of the pool of earthquake survivors, volunteers, workers, and machines that the organization engaged during the emergency response efforts.

Insurgent Expertise: The Politics of Free/Livre and Open Source Software in Brazil

ABSTRACT Under the administration of President Luiz Inácio Lula da Silva, the Brazilian state has advocated the use of Free/Livre and Open Source Software (FLOSS) throughout the public sector. How did FLOSS adoption gain traction as a developmental strategy across a large federal bureaucracy that had embraced information technology policies supporting export-oriented growth and market liberalization during the 1990s? In a historical case study, I argue that the FLOSS agenda emerged as a result of the actions of a network of insurgent experts working within elite political, technical, and educational institutions. I trace the history of this mobilization and show how a dedicated network of experts brought about conditions for institutional transformation that contradicted prevailing neoliberal policy proscriptions. The Brazilian FLOSS insurgency offers insights into the means by which a group of elites endeavored to reframe debates about technology-driven economic growth around questions of state-led access to source code and knowledge.

Apples to oranges?: comparing across studies of open collaboration/peer production

This panel seeks to begin a discussion of how we can meaningfully compare and contrast between the diverse instances of open collaboration and peer production employed on the Internet today. Current research on the topic have tended to be too platform - (e.g. Wikipedia) or domain - (e.g. Open source) specific. The panelists will be tasked with addressing this problem using their own expertise and research projects to bear on the issue. Ultimately, the panel will seek to lay the foundations for the development of theoretical frameworks and principles for the design and application of open collaboration and CBPP based systems.

Fail whaling: designing from deviance and failures in social computing

Social computing technologies are pervasive in our work, relationships, and culture. Despite their promise for transforming the structure of communication and human interaction, the complex social dimensions of these technological systems often reproduce offline social ills or create entirely novel forms of conflict and deviance. This panel brings together scholars who study deviance and failure in diverse social computing systems to examine four design-related themes that contribute to and support these problematic uses: theft, anonymity, deviance, and polarization.