Maja Vukovic

An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets

Crowdsourced labor markets represent a powerful new paradigm for accomplishing work. Understanding the motivating factors that lead to high quality work could have significant benefits. However, researchers have so far found that motivating factors such as increased monetary reward generally increase workers’ willingness to accept a task or the speed at which a task is completed, but do not improve the quality of the work. We hypothesize that factors that increase the intrinsic motivation of a task – such as framing a task as helping others – may succeed in improving output quality where extrinsic motivators such as increased pay do not. In this paper we present an experiment testing this hypothesis along with a novel experimental design that enables controlled experimentation with intrinsic and extrinsic motivators in Amazon’s Mechanical Turk, a popular crowdsourcing task market. Results suggest that intrinsic motivation can indeed improve the quality of workers’ output, confirming our hypothesis. Furthermore, we find a synergistic interaction between intrinsic and extrinsic motivators that runs contrary to previous literature suggesting “crowding out” effects. Our results have significant practical and theoretical implications for crowd work.

CrisisTracker: crowdsourced social media curation for disaster awareness

Victims, volunteers, and relief organizations are increasingly using social media to report and act on large-scale events, as witnessed in the extensive coverage of the 2010-2012 Arab Spring uprisings and 2011 Japanese tsunami and nuclear disasters. Twitter® feeds consist of short messages, often in a nonstandard local language, requiring novel techniques to extract relevant situation awareness data. Existing approaches to mining social media are aimed at searching for specific information, or identifying aggregate trends, rather than providing narratives. We present CrisisTracker, an online system that in real time efficiently captures distributed situation awareness reports based on social media activity during large-scale events, such as natural disasters. CrisisTracker automatically tracks sets of keywords on Twitter and constructs stories by clustering related tweets on the basis of their lexical similarity. It integrates crowdsourcing techniques, enabling users to verify and analyze stories. We report our experiences from an 8-day CrisisTracker pilot deployment during 2012 focused on the Syrian civil war, which processed, on average, 446,000 tweets daily and reduced them to consumable stories through analytics and crowdsourcing. We discuss the effectiveness of CrisisTracker based on the usage and feedback from 48 domain experts and volunteer curators.

Towards a research agenda for enterprise crowdsourcing

Over the past few years the crowdsourcing paradigm has evolved from its humble beginnings as isolated purpose-built initiatives, such as Wikipedia and Elance and Mechanical Turk to a growth industry employing over 2 million knowledge workers, contributing over half a billion dollars to the digital economy. Web 2.0 provides the technological foundations upon which the crowdsourcing paradigm evolves and operates, enabling networked experts to work collaboratively to complete a specific task. Enterprise crowdsourcing poses interesting challenges for both academic and industrial research along the social, legal, and technological dimensions. In this paper we describe the challenges that researchers and practitioners face when thinking about various aspects of enterprise crowdsourcing. First, to establish technological foundations, what are the interaction models and protocols between the Enterprise and the crowd. Secondly, how is crowdsourcing going to face the challenges in quality assurance, enabling Enterprises to optimally leverage the scalable workforce. Thirdly, what are the novel (Web) applications enabled by Enterprise crowdsourcing.

Ubiquitous crowdsourcing

Web 2.0 provides the technological foundations upon which the crowdsourcing paradigm evolves and operates, enabling networked experts to work on various problem solving and data-intensive tasks. During the past decade crowdsourcing grew from a number of purpose-built initiatives, such as Wikipedia and Mechanical Turk, to a technique that today attracts and engages over 2 million people worldwide. As the computing systems are becoming more intimately embedded in physical and social contexts, promising truly ubiquitous computing, crowdsourcing takes new forms. Increasingly, crowds are engaged through mobile devices, to capture, share and validate sheer amount data (e.g. reporting security threats or capturing social events). This workshop challenges researchers and practitioners to think about three key aspects of ubiquitous crowdsourcing. Firstly, to establish technological foundations, what are the interaction models and protocols between the ubiquitous computing systems and the crowd? Secondly, how is crowdsourcing going to face the challenges in quality assurance, while providing valuable incentive frameworks that enable honest contributions? Finally, what are the novel applications of crowdsourcing enabled by ubiquitous computing systems?

Peoplecloud for the globally integrated enterprise

Crowdsourcing has emerged as the new on-line distributed production model in which people collaborate and may be awarded to complete a task. While many existing services enable enterprises to employ the wisdom of crowd, there is no existing practice defined for integration of crowdsourcing with the business processes. We propose PeopleCloud, as the (1) mechanism to enable access to scalable workforce on-line, connecting it to the enterprise and (2) an interface to services required for crowdsourcing tasks. We define requirements for PeopleCloud, based on our experiences in employing wisdom of crowd to source business and IT information within the enterprise.

GoalMorph: partial goal satisfaction for flexible service composition

AI planning has proven to be a valuable tool for service composition. However, it can fail to satisfy a composite service request if not all the goal states can be reached, due to the context changes or missing service descriptions. Goals themselves are complex structures, conjuctions of goals states, and each representing partial solution. Therefore satisfying some goal states instead of all can be better than satisfying none of the goal states at all. In this paper, we present GoalMorph, a framework for context aware goal transformation that (a) constructs context aware goals and (b) reformulates failed goals into problems that can be solved by the AI planner. We discuss initial evaluation results, and demonstrate that our implementation provides a practical and scalable solution.

Challenges and experiences in deploying enterprise crowdsourcing service

The value of crowdsourcing, arising from an instant access to a scalable expert network on-line, has been demonstrated by many success stories, such as GoldCorp, Netflix, and TopCoder. For enterprises, crowdsourcing promises significant cost-savings, quicker task completion times, and formation of expert communities (both within and outside the enterprise). Many aspects of the vision of enterprise crowdsourcing are under vigorous refinement. The reasons for this lack of progress, beyond the isolated and purpose-specific crowdsourcing efforts, are manifold. In this paper, we present our experience in deploying an enterprise crowdsourcing service in the IT Inventory Management domain. We focus on the technical and sociological challenges of creating enterprise crowdsourcing service that are generalpurpose, and that extend beyond mere specific-purpose, run-once prototypes. Such systems are deployed to the extent that they become an integrated part of business processes. Only when such degree of integration is achieved, the enteprises can fully adopt crowdsourcing and reap its benefits. We discuss the challenges in creating and deploying the enterprise crowdsourcing platform, and articulate current technical, governance and sociological issues towards defining a research agenda.

PeopleCloud Service for Enterprise Crowdsourcing

Web 2.0 provides the technological foundations upon which the crowdsourcing paradigm evolves and operates, enabling enterprises, universities and eGovernments, to access scalable networks of knowledge experts on-line. However, there is no existing practice allowing for coordination of crowdsourcing tasks, and their integration with existing business processes and embedding these services into the Web fabric. In this paper, we examine two applications of enterprise crowdsourcing service in the domain of IT Service Delivery: 1) IT Inventory Management and 2) End-User Support. We illustrate how a) expert discovery mechanism, b) virtual team building capabilities, c) task management and d) provisioning of task-based services, enable enterprises to effectively build knowledge networks, which are able to execute complex and transformative knowledge-intensive tasks. Finally, based on the application analysis, we propose PeopleCloud, an on-demand service system, which spawns and manages scalable virtual teams of knowledge workers by either (1) building on the wisdom of crowds within an enterprise or across a value chain or (2) creating a marketplace for accessing specialists on-line.

An architecture for rapid, on-demand service composition

Legacy application design models, which are still widely used for developing context-aware applications, incur important limitations. Firstly, embedding contextual dependencies in the form of if–then rules specifying how applications should react to context changes is impractical to accommodate the large variety of possibly even unanticipated context types and their values. Additionally, application development is complicated and challenging, as programmers have to manually determine and encode the associations of all possible combinations of context parameters with application behaviour. In this paper we propose a framework for building context aware applications on-demand, as dynamically composed sequences of calls to services. We present the design and implementation of our system, which employs goal-oriented inferencing for assembling composite services, dynamically monitors their execution, and adapts applications to deal with contex- tual changes. We describe the failure recovery mechanisms we have implemented, allowing the deployment of the system in a non-perfect environment, and avoiding the delays inherent in re-discovering a suitable service instance. By means of experimental evaluation in a realistic infotainment application, we demonstrate the potential of the proposed solution an effective, efficient, and scalable approach.

Towards Real-time Emergency Response using Crowd Supported Analysis of Social Media

This position paper outlines an ongoing research project that aims to incorporate crowdsourcing as part of an emergency response system. The proposed systems novelty is that it integrates crowdsourcing into its architecture to analyze and structure social media content posted by microbloggers and service users, including emergency response coordinators and victims, during the event or disaster. An important challenge in this approach is identifying appropriate tasks to crowdsource, and adopting effective motivation strategies. Author Keywords Emergency response, social media, crowdsourcing, text mining.