Davide Ceolin

Semi-automated assessment of annotation trustworthiness

Cultural heritage institutions and multimedia archives often delegate the task of annotating their collections of artifacts to Web users. The use of crowdsourced annotations from the Web gives rise to trust issues. We propose an algorithm that, by making use of a combination of subjective logic, semantic relatedness measures and clustering, automates the process of evaluation for annotations represented by means of the Open Annotation ontology. The algorithm is evaluated over two different datasets coming from the cultural heritage domain.

Automated Evaluation of Annotators for Museum Collections Using Subjective Logic

Museums are rapidly digitizing their collections, and face a huge challenge to annotate every digitized artifact in store. Therefore they are opening up their archives for receiving annotations from experts world-wide. This paper presents an architecture for choosing the most eligible set of annotators for a given artifact, based on semantic relatedness measures between the subject matter of the artifact and topics of expertise of the annotators. We also employ mechanisms for evaluating the quality of provided annotations, and constantly manage and update the trust, reputation and expertise information of registered annotators.

Efficient semi-automated assessment of annotations trustworthiness

AbstractCrowdsourcing provides a valuable means for accomplishing large amounts of work which may require a high level of expertise. We present an algorithm for computing the trustworthiness of user-contributed tags of artifacts, based on the reputation of the user, represented as a probability distribution, and on provenance of the tag. The algorithm only requires a small number of manually assessed tags, and computes two trust values for each tag, based on reputation and provenance. We moreover present a computationally cheaper adaptation of the algorithm, which clusters semantically similar tags in the training set, and builds an opinion on a new tag based on its semantic relatedness with respect to the medoids of the clusters. Also, we introduce an adaptation of the algorithm based on the use of provenance stereotypes as an alternative basis for the estimation. Two case studies from the cultural heritage domain show that the algorithms produce satisfactory results.

A Semantic Pattern-Based Recommender

This paper presents a novel approach for Linked Data-based recommender systems through the use of semantic patterns - generalized paths in a graph described through the types of the nodes and links involved. We apply this novel approach to the book dataset from the ESWC2014 recommender systems challenge. User profiles are built by aggregating ratings on patterns with respect to each book in provided user training set. Ratings are aggregated by estimating the expected value of a Beta distribution describing the rating given to each individual book. Our approach allows the determination of a rating for a book, even if the book is poorly connected with user profile. It allows for a “prudent” estimation thanks to smoothing. However, if many patterns are available, it considers all the contributions. Additionally, it allows for a lightweight computation of ratings as it exploits the knowledge encoded in the patterns. Our approach achieved a precision of 0.60 and an overall F-measure of about 0.52 on the ESWC2014 challenge.

Georeferencing Animal Specimen Datasets

For biodiversity research, the field of study that is concerned with the richness of species of our planet, it is of the utmost importance that the location of an animal specimen find is known with high precision. Due to specimens often having been collected over the course of many years, their accompanying geographical data is often ambiguous or may be very imprecise. In this article, we detail an approach that utilizes reasoning and external sources to improve the geographical information of animal finds. Our main contribution is to show that adding external domain knowledge improves the ability to georeference locations over traditional methods that focus solely on analyzing geographical information. Additionally, our system is able to output the confidence it has in its decisions through a confidence measure based on the difficulty of the instance and the steps undertaken to disambiguate it. Our results show that adding domain knowledge to the georeferencing process increases the accuracy @5km from 38.9% to 61.7% and from 47.0% to 74.5% @25km. Furthermore, we reduce the mean distance by more than half, from 251.1km to 114.5km, and decrease the number of records for which no reference can be found from 26.2% to 7.4%.

The Simple Event Model

This chapter introduces the Simple Event Model and shows how it can be used for modeling events and their related concepts like actors, places, times, and their types. The event modeling discussed in this chapter is motivated from the need to abstract over historical situations to analyze what happened in the past. We show how the Simple Event Model can be used to model events by means of two example cases from different domains related to maritime situation awareness, one having to do with AIS ship observations in a harbor area and one having to do with maritime piracy reports. To show how the Simple Event Model can be used in practice for adaptive information access we demonstrate how an analysis of an open data set from the Semantic Web can be done using the Simple Event Model and the SPARQL query language.

Combining User Reputation and Provenance Analysis for Trust Assessment

Trust is a broad concept that in many systems is often reduced to user reputation alone. However, user reputation is just one way to determine trust. The estimation of trust can be tackled from other perspectives as well, including by looking at provenance. Here, we present a complete pipeline for estimating the trustworthiness of artifacts given their provenance and a set of sample evaluations. The pipeline is composed of a series of algorithms for (1) extracting relevant provenance features, (2) generating stereotypes of user behavior from provenance features, (3) estimating the reputation of both stereotypes and users, (4) using a combination of user and stereotype reputations to estimate the trustworthiness of artifacts, and (5) selecting sets of artifacts to trust. These algorithms rely on the W3C PROV recommendations for provenance and on evidential reasoning by means of subjective logic. We evaluate the pipeline over two tagging datasets: tags and evaluations from the Netherlands Institute for Sound and Vision’s Waisda? video tagging platform, as well as crowdsourced annotations from the Steve.Museum project. The approach achieves up to 85% precision when predicting tag trustworthiness. Perhaps more importantly, the pipeline provides satisfactory results using relatively little evidence through the use of provenance.

Predicting Quality of Crowdsourced Annotations Using Graph Kernels

Annotations obtained by Cultural Heritage institutions from the crowd need to be automatically assessed for their quality. Machine learning using graph kernels is an effective technique to use structural information in datasets to make predictions. We employ the Weisfeiler-Lehman graph kernel for RDF to make predictions about the quality of crowdsourced annotations in Steve.museum dataset, which is modelled and enriched as RDF. Our results indicate that we could predict quality of crowdsourced annotations with an accuracy of 75 %. We also employ the kernel to understand which features from the RDF graph are relevant to make predictions about different categories of quality.

Two Procedures for Analyzing the Reliability of Open Government Data

Open Government Data often contain information that, in more or less detail, regard private citizens. For this reason, before publishing them, public authorities manipulate data to remove any sensitive information while trying to preserve their reliability. This paper addresses the lack of tools aimed at measuring the reliability of these data. We present two procedures for the assessment of the Open Government Data reliability, one based on a comparison between open and closed data, and the other based on analysis of open data only. We evaluate the procedures over data from the data.police.uk website and from the Hampshire Police Constabulary in the United Kingdom. The procedures effectively allow estimating the reliability of open data and, actually, their reliability is high even though they are aggregated and smoothed.

Assessing Trust for Determining the Reliability of Information

This chapter explores methods for determining the reliability of Automated Identification System (AIS) messages. The primary use of AIS messages in the naval domain is to avoid collisions, therefore they contain kinematic information about ships. Moreover, AIS messages contain information like the ship name and its identifiers, so AIS messages can be used to identify ships. However, since the information contained in these messages is not necessarily correct (because, for instance, a malicious sender might want to declare a different identity than its own), in order to properly use them, we should assess their trust level. In general, trust is an important concept that helps to take decisions when the available information is limited or contradicting. In the case of AIS messages, this might occur when only few messages about a given ship are available or when messages conflict either against themselves or against other sources like Web sites reporting ship information. We describe ongoing work about the quantification of trust assessments in AIS messages, by means of statistical and logical analysis and by enriching AIS messages with information obtained from the Web.