Samantha Bail

The cognitive complexity of OWL justifications

In this paper, we present an approach to determining the cognitive complexity of justifications for entailments of OWL ontologies. We introduce a simple cognitive complexity model and present the results of validating that model via experiments involving OWL users. The validation is based on test data derived from a large and diverse corpus of naturally occurring justifications. Our contributions include validation for the cognitive complexity model, new insights into justification complexity, a significant corpus with novel analyses of justifications suitable for experimentation, and an experimental protocol suitable for model validation and refinement.

A Snapshot of the OWL Web

Tool development for and empirical experimentation in OWL ontology engineering require a wide variety of suitable ontologies as input for testing and evaluation purposes and detailed characterisations of real ontologies. Empirical activities often resort to (somewhat arbitrarily) hand curated corpora available on the web, such as the NCBO BioPortal and the TONES Repository, or manually selected sets of well-known ontologies. Findings of surveys and results of benchmarking activities may be biased, even heavily, towards these datasets. Sampling from a large corpus of ontologies, on the other hand, may lead to more representative results. Current large scale repositories and web crawls are mostly uncurated and suffer from duplication, small and (for many purposes) uninteresting ontology files, and contain large numbers of ontology versions, variants, and facets, and therefore do not lend themselves to random sampling. In this paper, we survey ontologies as they exist on the web and describe the creation of a corpus of OWL DL ontologies using strategies such as web crawling, various forms of de-duplications and manual cleaning, which allows random sampling of ontologies for a variety of empirical applications.

JustBench: a framework for OWL benchmarking

Analysing the performance of OWL reasoners on expressive OWL ontologies is an ongoing challenge. In this paper, we present a new approach to performance analysis based on justifications for entailments of OWL ontologies. Justifications are minimal subsets of an ontology that are sufficient for an entailment to hold, and are commonly used to debug OWL ontologies. In JustBench, justifications form the key unit of test, which means that individual justifications are tested for correctness and reasoner performance instead of entire ontologies or random subsets. Justifications are generally small and relatively easy to analyse, which makes them very suitable for transparent analytic micro-benchmarks. Furthermore, the JustBench approach also allows us to isolate reasoner errors and inconsistent behaviour. We present the results of initial experiments using JustBench with FaCT++, HermiT, and Pellet. Finally, we show how JustBench can be used by reasoner developers and ontology engineers seeking to understand and improve the performance characteristics of reasoners and ontologies.

The justificatory structure of the NCBO bioportal ontologies

Current ontology development tools offer debugging support by presenting justifications for entailments of OWL ontologies. While these minimal subsets have been shown to support debugging and understanding tasks, the occurrence of multiple justifications presents a significant cognitive challenge to users. In many cases even a single entailment may have many distinct justifications, and justifications for distinct entailments may be critically related. However, it is currently unknown how prevalent significant numbers of multiple justifications per entailment are in the field. To address this lack, we examine the justifications from an independently motivated corpus of actively used biomedical ontologies from the NCBO BioPortal. We find that the majority of ontologies contain multiple justifications, while also exhibiting structural features (such as patterns) which can be exploited in order to reduce user effort in the ontology engineering process.

Overcoming the pitfalls of ontology authoring: Strategies and implications for tool design

The process of authoring ontologies appears to be fragmented across several tools and workarounds, and there exists no well accepted framework for common authoring tasks such as exploring ontologies, comparing versions, debugging, and testing. This lack of an adequate and seamless tool chain potentially hinders the broad uptake of ontologies, especially OWL, as a knowledge representation formalism. We start to address this situation by presenting insights from an interview-based study with 15 ontology experts. We uncover the tensions that may emerge between ontology authors including antagonistic ontology building styles (definition-driven vs. manually crafted hierarchies). We identify the problems reported by the ontology authors and the strategies they employ to solve them. These data are mapped to a set of key design recommendations, which should inform and guide future efforts for improving ontology authoring tool support, thus opening up ontology authoring to a new generation of users. We discuss future research avenues in light of these results.

Toward cognitive support for OWL justifications

Justifications are the dominant form of explanation for entailments of OWL ontologies, with popular OWL ontology editors, such as Protege 4, providing justification-based explanation facilities. A justification is a minimal subset of an ontology which is sufficient for an entailment to hold; they correspond to the premises of a proof. Unlike proofs, however, justifications do not articulate how their axioms support the entailment. We frequently observe that ontology developers find certain justifications difficult to work with; and while in some cases the sources of difficulty are obvious (such as a large number of axioms), we do not have a good general understanding of what makes justifications easy or difficult for ontology users. In this paper, we present an approach to determining the cognitive complexity of justifications for entailments of OWL ontologies. We describe an exploratory study which forms the basis for a cognitive complexity model that predicts the complexity of OWL justifications, and present the results of validating that model via experiments involving OWL users. This is concluded by an investigation into strategies OWL users apply to support them in understanding justifications. Our contributions include an evaluation of the cognitive complexity model, new insights into the complexity of justifications for entailments of OWL ontologies, a significant corpus with novel analyses of justifications suitable for experimentation, and an experimental protocol suitable for model validation and refinement.

The logical diversity of explanations in OWL ontologies

Given the high expressivity of the Web Ontology Language OWL 2, there is a potential for great diversity in the logical content of OWL ontologies. The fact that many naturally occurring entailments of such ontologies have multiple justifications indicates that ontologies often overdetermine their consequences, suggesting a diversity in supporting reasons. On closer inspection, however, we often find that justifications---even for multiple entailments---appear to be structurally similar, suggesting that their multiplicity might be due to diverse material, not formal grounds for an entailment. In this paper, we introduce and explore several equivalence relations over justifications for entailments of OWL ontologies which partition a set of justifications into structurally similar subsets. These equivalence relations range from strict isomorphism to looser notions of similarity, covering justifications which contain different class expressions, or even different numbers of axioms. We present the results of a survey of 78 ontologies from the biomedical domain which shows that OWL ontologies used in practice often contain large numbers of structurally similar justifications. We find that a large justification corpus can be reduced by 97% of its original size to a small core of frequently occurring justification templates.