Tobias Sturn

Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game

Volunteered geographic information (VGI) is the assembly of spatial information based on public input. While VGI has proliferated in recent years, assessing the quality of volunteer-contributed data has proven challenging, leading some to question the efficiency of such programs. In this paper, we compare several quality metrics for individual volunteers’ contributions. The data were the product of the ‘Cropland Capture’ game, in which several thousand volunteers assessed 165,000 images for the presence of cropland over the course of 6 months. We compared agreement between volunteer ratings and an images majority classification with volunteer self-agreement on repeated images and expert evaluations. We also examined the impact of experience and learning on performance. Volunteer self-agreement was nearly always higher than agreement with majority classifications, and much greater than agreement with expert validations although these metrics were all positively correlated. Volunteer quality showed a broad trend toward improvement with experience, but the highest accuracies were achieved by a handful of moderately active contributors, not the most active volunteers. Our results emphasize the importance of a universal set of expert-validated tasks as a gold standard for evaluating VGI quality.

A global dataset of crowdsourced land cover and land use reference data

Global land cover is an essential climate variable and a key biophysical driver for earth system models. While remote sensing technology, particularly satellites, have played a key role in providing land cover datasets, large discrepancies have been noted among the available products. Global land use is typically more difficult to map and in many cases cannot be remotely sensed. In-situ or ground-based data and high resolution imagery are thus an important requirement for producing accurate land cover and land use datasets and this is precisely what is lacking. Here we describe the global land cover and land use reference data derived from the Geo-Wiki crowdsourcing platform via four campaigns. These global datasets provide information on human impact, land cover disagreement, wilderness and land cover and land use. Hence, they are relevant for the scientific community that requires reference data for global satellite-derived products, as well as those interested in monitoring global terrestrial ecosystems in general.

Crowdsourcing In-Situ Data on Land Cover and Land Use Using Gamification and Mobile Technology

Citizens are increasingly becoming involved in data collection, whether for scientific purposes, to carry out micro-tasks, or as part of a gamified, competitive application. In some cases, volunteered data collection overlaps with that of mapping agencies, e.g., the citizen-based mapping of features in OpenStreetMap. LUCAS (Land Use Cover Area frame Sample) is one source of authoritative in-situ data that are collected every three years across EU member countries by trained personnel at a considerable cost to taxpayers. This paper presents a mobile application called FotoQuest Austria, which involves citizens in the crowdsourcing of in-situ land cover and land use data, including at locations of LUCAS sample points in Austria. The results from a campaign run during the summer of 2015 suggest that land cover and land use can be crowdsourced using a simple protocol based on LUCAS. This has implications for remote sensing as this data stream represents a new source of potentially valuable information for the training and validation of land cover maps as well as for area estimation purposes. Although the most detailed and challenging classes were more difficult for untrained citizens to recognize, the agreement between the crowdsourced data and the LUCAS data for basic high level land cover and land use classes in homogeneous areas (ca. 80%) shows clear potential. Recommendations for how to further improve the quality of the crowdsourced data in the context of LUCAS are provided so that this source of data might one day be accurate enough for land cover mapping purposes.

Limitations of Majority Agreement in Crowdsourced Image Interpretation

Crowdsourcing can efficiently complete tasks that are difficult to automate, but the quality of crowdsourced data is tricky to evaluate. Algorithms to grade volunteer work often assume that all tasks are similarly difficult, an assumption that is frequently false. We use a cropland identification game with over 2,600 participants and 165,000 unique tasks to investigate how best to evaluate the difficulty of crowdsourced tasks and to what extent this is possible based on volunteer responses alone. Inter-volunteer agreement exceeded 90% for about 80% of the images and was negatively correlated with volunteer-expressed uncertainty about image classification. A total of 343 relatively difficult images were independently classified as cropland, non-cropland or impossible by two experts. The experts disagreed weakly (one said impossible while the other rated as cropland or non-cropland) on 27% of the images, but disagreed strongly (cropland vs. non-cropland) on only 7%. Inter-volunteer disagreement increased significantly with inter-expert disagreement. While volunteers agreed with expert classifications for most images, over 20% would have been mis-categorized if only the volunteers’ majority vote was used. We end with a series of recommendations for managing the challenges posed by heterogeneous tasks in crowdsourcing campaigns.

Local knowledge and professional background have a minimal impact on volunteer citizen science performance in a land-cover classification task

The idea that closer things are more related than distant things, known as ‘Tobler’s first law of geography’, is fundamental to understanding many spatial processes. If this concept applies to volunteered geographic information (VGI), it could help to efficiently allocate tasks in citizen science campaigns and help to improve the overall quality of collected data. In this paper, we use classifications of satellite imagery by volunteers from around the world to test whether local familiarity with landscapes helps their performance. Our results show that volunteers identify cropland slightly better within their home country, and do slightly worse as a function of linear distance between their home and the location represented in an image. Volunteers with a professional background in remote sensing or land cover did no better than the general population at this task, but they did not show the decline with distance that was seen among other participants. Even in a landscape where pasture is easily confused for cropland, regional residents demonstrated no advantage. Where we did find evidence for local knowledge aiding classification performance, the realized impact of this effect was tiny. Rather, the inherent difficulty of a task is a much more important predictor of volunteer performance. These findings suggest that, at least for simple tasks, the geographical origin of VGI volunteers has little impact on their ability to complete image classifications.

Using Citizen Science to Help Monitor Urban Landscape Changes and Drive Improvements

Citizen Science has become a vital source for data collection when the spatial and temporal extent of a project makes it too expensive to send experts into the field. However, involving citizens can go further than that – participatory projects focusing on subjective parameters can fill in the gap between local community needs and stakeholder approaches to tackle key social and environmental issues. LandSense, a Horizon 2020 project that is deeply rooted in environmental challenges and solutions, aims to establish a citizen observatory that will provide data to stakeholders, from researchers to businesses. Within this project, a mobile application has been developed that aims not only to stimulate civic engagement to monitor changes within the urban environment, but also to enable users to drive improvements by providing city planners with information about the public perception of urban spaces. The launch of a public version of such an app requires preparation and testing by focus groups. Recently, a prototype of the app was used by both staff and students from Vienna University of Technology, who contributed valuable insights to help enhance this citizen science tool for engaging and empowering the inhabitants of the city.