OObservations on Annotations ∗ Georg Rehm † Abstract
The annotation of textual information is a fundamental activity in Linguistics and ComputationalLinguistics. This article presents various observations on annotations. It approaches the topicfrom several angles including Hypertext, Computational Linguistics and Language Technology,Artificial Intelligence and Open Science. Annotations can be examined along different dimensions.In terms of complexity, they can range from trivial to highly sophisticated, in terms of maturityfrom experimental to standardised. Annotations can be annotated themselves using more abstractannotations. Primary research data such as, e. g., text documents can be annotated on different layersconcurrently, which are independent but can be exploited using multi-layer querying. Standardsguarantee interoperability and reusability of data sets. The chapter concludes with four finalobservations, formulated as research questions or rather provocative remarks on the current state ofannotation research.
The annotation of textual information is one of the most fundamental activities in Linguistics and Com-putational Linguistics including neighbouring fields such as, among others, Literary Studies, LibraryScience and Digital Humanities (Ide and Pustejovsky, 2017, Bludau et al., 2020). Horizontally, dataannotation plays an increasingly important role in Open Science, in the development of NLP/NLU proto-types (Natural Language Processing/Understanding), more application- and solution-oriented LanguageTechnologies (LT) and systems based on neural technologies in the area of Artificial Intelligence (AI).This article reflects on more than two decades of research in the wider area of annotation includingmulti-layer annotations (Witt et al., 2007a,b), the modelling of linguistic data structures (W¨orneret al., 2006, Rehm et al., 2007b, Witt et al., 2009) including hypertext and web genres (Rehm, 2002,2007, 2010a), the production and distribution of annotated corpora (Piperidis et al., 2014, Rehm, 2016,Rehm et al., 2020a) and the use of metadata, annotation schemes and markup languages (Rehm et al.,2008a,b, 2009, Rehm, 2010b). After an initial approximation of a definition (Section 2), the chapterprovides lessons learned, future research directions as well as observations on the scientific and technicalprocess of annotating textual data from several angles including Hypertext, Markup and the World WideWeb (Section 3), Computational Linguistics (Section 4), Artificial Intelligence (Section 5), LanguageTechnology (Section 6) and Open Science (Section 7). The article concludes with an overview of themain conceptual dimensions involved in the annotation of textual information (Section 8) and a summary(Section 9). ∗ To be published in:
Annotations in Scholarly Editions and Research: Functions, Differentiation, Systematization (2020),Julia Nantke and Frederik Schlupkothen (editors). De Gruyter. In print. † Georg Rehm, DFKI GmbH, Alt-Moabit 91c, 10559 Berlin, Germany – [email protected] a r X i v : . [ c s . C L ] A p r Definition
Definitions of the term “annotation” typically focus on either procedural (i. e., process-related), technical(i. e., markup-related) or conceptual (i. e., semantics-related) aspects, sometimes also combinations ofthe different layers (Goecke et al., 2010, Ide and Pustejovsky, 2017). The notion we follow in this articleis loosely based on the concept of Annotation Graphs (Bird and Liberman, 2001), which can be used torepresent an unlimited number of annotation layers, while links between the text and annotations can beestablished in an unrestricted way (Witt et al., 2007b, Ide and Suderman, 2007). Specifically, we viewannotations as secondary research data added to primary research data . Annotations are, therefore,part of the metadata that also include general information on the primary data (author/creator, modality,creation date etc.).
Linguistic annotations, then, cover “any descriptive or analytic notations appliedto raw language data. The basic data may be in the form of [. . . ] audio, video and/or physiologicalrecordings [. . . ] or it may be textual. The added notations may include transcriptions of all sorts (fromphonetic features to discourse structures), part-of-speech and sense tagging, syntactic analysis, ‘namedentity’ identification, co-reference annotation, and so on.” (Bird and Liberman, 2001). The procedure ofannotating data can include, among several other variants, highlighting and labelling specific segments,commenting upon certain aspects, and selecting as well as inserting markup elements (tags) into a textdocument. The design of a concrete annotation scheme typically follows at least two consecutive phases:based on linguistic theory or insights, an annotation model is created (Pustejovsky et al., 2017) forwhich, then, a technical representation is developed (Ide et al., 2017b). Finlayson and Erjavec (2017)provide an overview of the processes and tools involved in the creation of annotations.
Annotations have always been an integral concept of hypertext (Nelson, 1987) itself as well as the WorldWide Web. In his seminal piece, “As we may think”, Bush (1945) described his vision of the Memex,explaining that the user of the Memory Extender “can add marginal notes and comments [. . . ] by astylus scheme”. And Berners-Lee (1989) described, in the original concept note that laid the groundworkfor what later became the World Wide Web, that one “must be able to add one’s own private links to andfrom public information. One must also be able to annotate links, as well as nodes, privately.” WhileBerners-Lee had this specific idea in mind already back in 1989, it took more than 20 years of work forWeb Annotations to become a web standard proper (see below).Linguistic annotations are, procedurally, conceptually, and technically, closely linked to markupand markup languages, especially the ones based on XML (Extensible Markup Language, Bray et al.,2008), enriched, processed, presented and queried with related formalisms such as, among others, XMLSchema, XSLT, XPath, XQuery, CSS, RDF and OWL. Through their unambiguous, syntactic separationof annotations from the primary data, markup languages are a natural candidate for linguistic annotations,especially those based on XML, the most widely used meta-language for the definition of concretemarkup languages using approaches such as XML Schema or Document Type Definitions (DTD). Oneof the most widely used annotation systems in Linguistics and Digital Humanities are the TEI guidelines(TEI Consortium, 2019), initially developed in the late 1980s. The formalisms mentioned above weredeveloped and standardised by the World Wide Web Consortium (W3C), an international non-profitorganisation founded by Tim Berners-Lee in 1994 to lead the further development of the World WideWeb’s technical building blocks. Just like XML, the W3C’s effort to move from a static, document-centric to a
Semantic
Web also lead to a number of highly influential and innovative developments inLinguistics and Computational Linguistics, especially with regard to modelling and querying annotations2Rehm et al., 2007a, Farrar and Langendoen, 2010, Chiarcos and Sukhareva, 2015). The interfacebetween technical markup and linguistic annotations is examined by Metzing and Witt (2010) includingthe interface between HTML and linguistic markup (Rehm, 2010a).Most stand-alone tools for the annotation of linguistic data, often implemented in Java, have bynow vanished or, if they are still in use, target a specific niche for which a browser-based solutionhas not been developed yet. Nowadays, actual annotation work is typically carried out in the webenvironment, i. e., in the browser, using one of the web-based annotation tools such as, among others,Brat (Stenetorp et al., 2012), WebAnno (Eckart de Castilho et al., 2016), INCEpTION (Klie et al., 2018)or CATMA (Meister et al., 2019). Crucially, the textual data that is annotated this way may be web data(i. e., HTML documents) that was downloaded or crawled, but it is typically not live web data becauseanchoring annotations to live web documents that can change, in a subtle or substantial way, any minuteis technically challenging.The fairly recent W3C standard Web Annotation was developed for exactly this purpose, i. e., toenable the annotation of live web data. The standard consists of three W3C recommendations. The WebAnnotation Data Model (Sanderson et al., 2017a) describes the underlying annotation data model as wellas a JSON-LD serialisation. The Web Annotation Vocabulary (Sanderson et al., 2017b) underpins theData Model, and the Web Annotation Protocol (Sanderson, 2017) defines an HTTP API for publishing,syndicating and distributing Web Annotations. The standard enables users to annotate arbitrary piecesof web content in the browser, essentially creating an additional, independent layer on top of the regularWorld Wide Web. Web Annotations are the natural mechanism to enable web users and readers, on ageneral level, interactively to work with content, to include notes, feedback and assessments, to askthe author or their peers for references or to provide criticism. However, there are still limitations. Asof now, none of the larger browsers implement Web Annotations natively, i. e., content providers needto enable Web Annotations by integrating a corresponding JavaScript library. Another barrier for thewidespread adoption of Web Annotations are proprietary commenting systems, as used, among others,by all major social networks who are keen on keeping all annotations (i. e., comments and other types ofuser-generated content) in their own respective silos and, thus, under their own control.Nevertheless, services such as the popular Hypothes.is tool (see below) enable Web Annotations onany web page, but native browser support, ideally across all platforms, is still lacking. In addition to the(still somewhat limited) ability of handling live web data, the Web Annotation standard has multipleadvantages that make it perfectly suited for linguistic annotations. The Web Annotation Data Modelis very general and can be conceptualised as a multi-layer Annotation Graph. Annotations are sets ofconnected resources, typically an annotation body and the target of the annotation. If and when theWeb Annotation standard is finally available natively in all browsers, conversations between users andcontent creators can take place anywhere on the web in a standards-compliant way, where, and this iscrucial, the annotations are under the control of the users because annotations can live separately fromthe documents they are pointing to – they are reunited and re-anchored in real time.The annotation tool developed by the non-profit organisation Hypothes.is is by the far the most popularone. It enables taking private notes or publishing public annotations. It can be used in collaborativegroups, it provides Linked Data connections and works with different formats including HTML, PDFand EPUB. It is used in scholarly publishing and as a technical tool for open peer review, in research,education and investigative journalism. It can also be used for automated annotations, e. g., to tagResearch Resource Identifiers (RRIDs). See, for example, the projects presented in the various events of the “I Annotate” conference series, which started in 2013:http://iannotate.org. The annotation landscape, which consists, generally speaking, of tools and formats, has had severaldecades to grow and to mature into an area that is impossible to characterise in the context of a shortbook chapter alone. Many colleagues provided general or specific overviews, including, among others,Bird and Liberman (2001), Dipper et al. (2004), Metzing and Witt (2010), St¨uhrenberg (2012), Ide andPustejovsky (2017), Biemann et al. (2017), Stede (2018), Neves and ˇSeva (2019). In addition to a largenumber of all-purpose and specialised formats (Ide et al., 2017a) such as, among many others, TEI, NIF,NAF, LAF, GRAF, TIGER, STTS, FoLIA, there is a plethora of editors and tools to chose from, such asBrat, WebAnno, Exmaralda, Praat, ELAN, ANNIS, CATMA, INCEpTION and Prodigy as well as manyothers including crowd-sourced approaches.Both annotation tools and also annotation formats can be described along a number of dimensionsand continuums. Annotation schemes range from trivial (e. g., marking up single tokens) to complex (enabling semantically deep and nuanced annotations). These often correlate with their annotation task,from easy , straightforward and well understood (e. g., annotating named entities) to hard , challenging and novel (e. g., the annotation of actors and events in storylines). Accordingly, simple annotationtasks, the goals of which can be summarised and specified in concise annotation guidelines effectively,typically result in very high inter-annotator agreement scores while hard, ambitious and challenging tasksthat may require a certain level of expertise or training, rather result in low inter-annotator agreement(Gut and Bayerl, 2004, Bayerl and Paul, 2007, 2011, Snow et al., 2008, Artstein, 2017). Finally, simpleannotation tasks are typically carried out using general all-purpose tools while complex annotation tasksusually require specialised or customised tools. Artificial Intelligence (AI) as an academic discipline was founded in the 1950s. While it consists ofvarious subfields, by now, it is ubiquituous first and foremost due to the recent breakthroughs madein the area of Machine Learning (ML) using Deep Neural Networks (DNNs). These have been madepossible due to powerful supervised but also unsupervised machine learning algorithms, fast hardwareand, crucially, large amounts of data. This is why the relevance of annotations and annotated data setsfor AI at large, including Language-Centric AI (Rehm et al., 2020d), i. e., Computational Linguisticsand Natural Language Understanding, has increased dramatically in recent years.Modern AI methods are data-driven. Supervised learning methods rely on very large annotateddata sets, many of which consist of primary (language) data and secondary annotations, as defined inSection 2. In fact, data curation and annotation has become so important that new business models https://markupdeclaration.org In Natural Language Understanding, DNNs are also used for language modelling, i. e., for generating statistical modelsout of enormous amounts of unannotated language data. These can be used for various classification and prediction tasks(Ostendorff et al., 2019). Key aspects of any datageneration process include the annotation speed, the quality and relevance of the annotations, and howmeaningful, reliable and representative the annotations are.With regard to the context of AI-based applications, the line between the construction of structureddata sets on the one hand and the collection of – typically user-generated – data points on the other, isblurry, as both can be conceptualised as annotations. In the former, language data is annotated withregard to, for example, word senses or intents. In the latter, actual live content is “annotated”, forexample, by liking a tweet, leaving a five-star rating for a restaurant or commenting on a news article.All of these activities are annotations that add metadata to existing data. Clicking a headline to go toan article or even turning the page in an ebook can also be and, in fact, are interpreted as annotationswith regard to the underlying primary data in question. Increasingly slower page turns in an ebook, forexample, could be interpreted by the user modelling algorithm as “boredom” with the current chapterand may, later on, result in automatically adjusted book recommendations. Even the non-action of nolonger reading an ebook can be seen as an “implicit” annotation. In the future, for certain non-fictiongenres it will be possible to identify the chapters in which readers lose interest and then to generateslightly different versions or paraphrases of those chapters with the intent of not losing any readers bykeeping their engagement high. In these cases, the original human author will compete with the machinein an A/B test, i. e., both variants are presented to users in a short experimental phase, while only thestatistically more effective variant will be used in the long-term. In today’s digital age, users of largeonline applications must be aware of the fact that every single action or click they perform, i. e., everysingle annotation, is recorded, associated with their profile, and made use of by user modelling andrecommender algorithms, including advertisements.
The applied field of Language Technology (LT) transfers theoretical results from language-orientedresearch into technologies and applications that are ready for production use. Linguistics, ComputationalLinguistics, Psycholinguistics, Computer Science, AI and Cognitive Science are among the relevantfields made use of in LT-solutions. Spell checkers, dictation systems, translation software, searchengines, report generators, expert systems, text summarisation tools and conversational agents are typicalLT-applications.This Section takes a brief look at potential ways how LT as well as AI can interface with the WebAnnotation technology stack (Section 3). LT can be embedded in various phases and places of theWeb Annotation workflow to address and eventually solve a number of common challenges (Rehmet al., 2016). First, the web content to be enriched with annotations can be created automatically orsemi-automatically using Natural Language Generation (NLG) approaches; in fact, this is already thecase for vast amounts of online content, including online shops, weather reports, and articles aboutsport events. Second, the web content can be automatically analysed and then annotated using LT, forexample, for the purpose of generating an abstract of a longer article using automated text summarisationand then presenting the article to users in the form of an annotation. Third, the content of the actualannotations, potentially made by many different users, can be analysed using LT, for example, for thepurpose of mining the feedback of the users or readers for sentiments and opinions towards the primary For example, Appen’s current slogan is “Data with a human touch: High-quality data for machine learning, enhanced byhuman interaction” (https://appen.com). we use a similarapproach, the NLP Interchange Format (NIF, see Hellmann et al., 2013). NIF was developed especiallyfor LT applications and is based on the Linked Data paradigm, i. e., RDF and OWL.Between the development phase and the deployment phase of an LT-based solution, annotation formatscan also be mixed. For example, in LYNX, all processing solutions make use of NIF (Rehm et al., 2019)but during the development and training phase of the German Legal NER model we used the CONLLformat which is a simple, tab-seperated value, i. e., non-XML-based inline annotation format (Leitneret al., 2019, 2020). The umbrella term Open Science denotes the movement to make scientific research, data and dissemina-tion accessible to interested stakeholders. It includes a multitude of different aspects, e. g., publishingopen research, pushing for Open Access (instead of closed) and encouraging researchers of all fields topublish not only their results but also their data for easier verification and reproducibility. Open Scienceis becoming more and more popular and is, crucially, relevant to the broader topic of annotations. Ifwe examine the taxonomy produced by the EU project FOSTER to describe the different aspects ofOpen Science, these connections become immediately apparent: Open Science advocates for Open Data,which should not only be open but also annotated using standards, made available using platforms thatare accessible (e. g., Linked Data) and described with metadata and semantics including well definedcategories and taxonomies.One of the key goals of promoting Open Research Data is to enable data re-use and, thus, OpenReproducible Research that also includes Open Science Workflows, often made possible by distributingOpen Source software and specifying the workflows used to arrive at the results published in a scientificarticle. Annotations, the meaning and semantics of which are clearly documented, ideally usinginternational standards, are the glue between the software components that produce the annotations,annotated open research data, annotation guidelines, research data repositories, query mechanisms andscientific publications. DKT (http://digitale-kuratierung.de) (Bourgonje et al., 2016), QURATOR (https://qurator.ai) (Rehm et al., 2020b) andLYNX (http://lynx-project.eu) (Rehm et al., 2019). Mostof the FAIR principles refer to metadata, which can, especially if they relate to primary data, also beconceptualised as annotations. The relevant principles are the following ones: F2 Data are described with rich metadata. F3 Metadata clearly and explicitly include the identifier of the data they describe. A1 (Meta)data are retrievable by their identifier using a standardised communications protocol. A2 Metadata are accessible, even when the data are no longer available. I1 (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. I2 (Meta)data use vocabularies that follow FAIR principles. I3 (Meta)data include qualified references to other (meta)data. R1 (Meta)data are richly described with a plurality of accurate and relevant attributes. R1.2 (Meta)data are associated with detailed provenance.
R1.3 (Meta)data meet domain-relevant community standards.
As can be seen, the FAIR principles – and also Open Science in general – recommend, at their core,the use of standards for the purpose of enabling or enhancing, as much as possible, the findability,accessibility, interoperability and reusability of research data (see Labropoulou et al., 2020, for a practicalexample). While these recommendations are important and, thus, to be supported, it is also worth notingthat especially basic research is about trying and inventing new things, i. e., things that have, almost bydefinition, not been standardised yet. This contradicts, on a fundamental level, with the recommendationof using standards as the consensus reached within a specific research community to represent, forexample, temporal expressions in natural language text. The contradiction can be resolved, though,if the recommendation is relaxed to the use of established tools and best practice approaches as wellas the modification and extension of standards. The crucial aspect is to document the semantics ofthe annotation scheme used in a corpus or data set. If an established, standardised approach does notwork for an emerging piece of research, a new approach needs to be created or an established approachmodified.It is safe to predict that Open Science will be transforming research in the next years, making itmore sustainable, more visible and more transparent. Several disciplines have already been followingOpen Science-like approaches for quite a while. On a larger scale, though, Open Science will only befully possible with substantially improved digital infrastructures. Notable initiatives are the EuropeanOpen Science Cloud (EOSC) and the Nationale Forschungsdateninfrastruktur (NFDI) in Germany.Additionally, we can predict that, soon, robust and large-scale services for the annotation of documentswill be provided, starting with scientific publications, for which it will be possible to annotate and,thus, explicitly represent, using standardised metadata schemas and ontologies, their methods used orexpanded upon, evaluation approaches, data sets as well as findings and contributions – this structured setof semantic information associated with one research article, as the atomic unit of scientific publication,will be contextualised in larger knowledge graphs which will capture the research output of entirescientific fields, including annotations. Several larger scientific publishing houses are already nowdeveloping corresponding digital infrastructures to capture the results they publish. At the same time, https://ec.europa.eu/research/openscience/index.cfm?pg=open-science-cloud document-based to a knowledge-based approachby, first, automatically identifying and extracting and, second, representing and expressing scientificknowledge through semantically rich, interlinked graphs (Jaradeh et al., 2019). In a third step, theknowledge contained in the ORKG can be used, for example, to compare the approaches followed indifferent scientific papers on the same research question.
The process of adding annotations to a set of primary research data can be conceptualised as the insertionof secondary research data (see Section 2). The secondary data added to the primary data typically refersto one or more (often interconnected) properties of the primary data that are explicitly marked usingsyntactically identifiable methods. Figure 1 shows the general aspects and dimensions involved in anannotation in more detail; Ide and Romary (2001) provide a similar but more technical view focusedupon syntactic annotations. … Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod temporincididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrudexercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. …
PropertyLabel ofproperty
Value ofproperty Pointer to annotation schema
Annotation schema (possibly external) may constrainor restrict
Examples: lemma,part of speech,instance-of etc.
Important questions related to the label: • What is the conceptual or epistemological nature of this property? • Is a specific property label best practice in research or can it be entirely made up? • How many colleagues in the scientific community agree on the label? • Is the label adequate and self-explanatory?
Primary research data Annotation:
Property:
Examples: adjective,JJ, object, “some freetext comment” etc.
Important questions related to the value: • The actual annotation payload • Is the value free text or taken from a fixed and shared vocabulary? • Is the shared vocabulary prescribed by an annotation schema or ontology? • How many colleagues in the scientific community agree on the value? • How many colleagues in the scientific community agree on the whole shared vocabulary?
Important questions related to the property: • Is there some type of inherent structure among the set of different properties? • Can this inherent structure perhaps be modelled using markup languages or document grammars? • Difference between syntactic and semantic structure: – Syntactic structure: • Example: “HVBXJ” à “AHXB”, “HKVZ” – Semantic, i.e., logical structure: • Example: “NP” à “DET”, “N” Figure 1: General aspects and dimensions of annotationsAn annotation explicitly describes a property of a piece of primary data using a tuple that consists ofthe label of the property in question (e. g., “part of speech”) and a corresponding value (e. g., “adjective”).An annotation can also include a pointer to an abstract, internally or externally represented annotationscheme that, typically, specifies the semantics of all possible annotations. This annotation scheme, inturn, can be used to constrain or to restrict specific annotations, i. e., the ¡label, value¿ pair that makesup an annotation.Especially when designing a new or modifying an existing annotation scheme to address a specificresearch experiment, several relevant questions need to be taken into account, some of which are included
8n Figure 1. These question pertain, among others, to the conceptual or epistemological nature of thespecific label of an annotation: on the one hand, this label can denote a concept that has been establishedin a scientific field for decades or it can refer to a fairly recent aspect, phenomenon or finding, for whichan established term in the respective scientific community does not exist yet. Another aspect relates tothe set of properties that are being described with the help of an annotation scheme: are these isolatedproperties without any inherent structure that governs the sequence or distribution of their instantiations(e. g., different types of named entities) or does some kind of linguistic or syntactic structure existon top of the different annotations? If the latter is the case, can this structure be explicitly modelled,for example, using mechanisms built into XML DTD or XML Schema-based document grammars(Maler and Andaloussi, 1996, Megginson, 1998)? Can, maybe as an additional mechanism on top of thedocument grammar, an ontology be used to describe higher-level semantic concepts?The various notions hinted at in Figure 1 lead us to a more abstract aspect of annotations: just likeprimary research data, annotations have various properties themselves. Depending on the researchquestion and overall use case, it may be important or even necessary to explicitly represent theseproperties, i. e., to annotate annotations. Among this set of properties are the following: annotator of theannotation (i. e., was it created by a human expert or by an automatic process?), annotation layer (i. e.,does the annotation refer to the “document structure”, “layout”, “syntax”, “semantics”, “informationstructure” etc.?), confidence value (i. e., how confident is the human annotator or automated processthat an annotation is correct?), timestamp (i. e., when the annotation was added), style (i. e., how anannotation is rendered in a certain system) and application scenario (i. e., is the annotation primarilymeant for human or machine consumption?). It is important to note that more structure can be explicitlyadded even on top of these annotations, especially with regard to the relationship and interdependenceof the various annotation layers.Instantiated sets of annotations can be described along various axes and dimensions, some of whichare rather vague while others are more concrete. • Annotator:
The actual source or origin of annotations included in a data set, for example, oneor more automated components, human experts, human laypersons, crowd workers etc. Thisdimension also refers to the methodology followed for including the annotations into the primarydata. • Semantics:
The semantics of the annotations, i. e., the nature of the properties explicitly and for-mally described through the annotations, e. g., linguistic concepts or aspects relating to documentstructure, rhetorical structure, genre, style, terminology etc. This dimension is connected to the annotation scheme used, which could be an experimental scheme developed, e. g., in a researchproject for a novel purpose, or one of the well known annotation schemes and standards that havebeen in use for decades, e. g., TEI. • Layers:
The nature and interconnectedness of the different annotation layers if an annotated dataset contains multiple layers. • Guidelines:
A crucial question with regard to annotation projects primarily carried out by humans,relates to the presence of annotation guidelines, especially with regard to the specification ofconcrete examples and exceptions, i. e., which concepts to annotate how in a specific context. • Research question or application use case:
An annotated data set is typically associated eitherwith an underlying research question that has motivated the construction of a data set or with aconcrete annotation pipeline (i. e., application use case) that was used to annotate the primarydata. 9
Complexity:
This dimension refers to the notion that some annotations are more complex thanothers, it is closely related to several other dimensions. • Evaluation:
Most annotated data sets have been evaluated in some way, e. g., with regard to theinter-annotator agreement (if the primary data was annotated by multiple annotators).Space restrictions prevent us from describing all dimensions in more detail, which is why weconcentrate on
Complexity (Section 8.1) and
Evaluation (Section 8.2).
In Computational Linguistics and also in the wider Digital Humanities area, several fairly detailedannotation schemes and markup languages have been developed for the annotation of textual data in thelast 30 years. The TEI guidelines are probably the most extensive ones – the PDF version of the TEIP5 guidelines (TEI Consortium, 2019) has a length of almost 2000 pages, in which hundreds of XMLelements and attributes, grouped into various modules, are described. In stark contrast, the annotationschemes used in many current data sets, especially for large-scale, data-driven AI approaches that relyon vast amounts of training data, are quite shallow and highly generalised. Machine learning approachesperform best with large amounts of training data; it is beneficial for the performance of the resultingmodels and classifiers if the number of unique class labels is rather small and the number of differentexamples per class label rather high. Especially for environments in which such AI-based classifiers areused in production, the corresponding data sets are often created by professional annotation teams orcompanies (see Section 5). In these scenarios and use cases it is not feasible to annotate data sets withcomplex annotation schemes.It is an interesting question for future research if the difference in complexity or the “level ofsophistication” of different annotation schemes – from a simple set of a few labels to highly complexmarkup languages like TEI P5 – can be measured or formally described. To the best of the author’sknowledge, there has not been any work on this topic so far. Many different data points and statisticsabout an annotation scheme could be exploited for this purpose, e. g., the number of property labels (i. e.,XML tags), the number of meta properties (e. g., XML attributes), the number of free text and predefinedvalues, the presence of inherent structure including nesting levels etc. These, and other, statistics couldbe included in a formula that captures the complexity of an annotation scheme; it could also be used,together with data such as token/annotation ratio, to model the complexity of the annotations containedin a concrete data set.
The evaluation of annotations is a crucial dimension of formally describing a data set or corpus, especiallywhen it was created for the purpose of training a practical tool and also when an emerging annotationscheme was used. In that regard, two different aspects can be evaluated that are intricately interrelated:the annotation scheme itself and concrete annotations.The evaluation of the validity of an abstract, possibly emerging, annotation scheme is typically aniterative process (Dickinson and Tufis¸, 2017, Artstein, 2017): first, an initial version of the annotationscheme is applied to a small and, ideally, representative data set to examine if it is practical and balancedconcerning its ability to annotate all the characteristics and phenomena it is supposed to be able tomark up explicitly. An overarching aspect that should be taken into account when developing anditeratively evaluating an annotation scheme relates to the question if it models scientific consensus.These initial tests are, later on, repeated with more mature versions of the annotation scheme until all10equirements, prescribed by the respective research question, are met. As the two go hand in hand, theseinitial evaluations typically concern not only the annotation scheme but also the annotation guidelines aswell as their applicability using a specific annotation tool. Important questions regarding the annotationguidelines relate to their length, coverage, examples, and exceptions as well as how long it usually takesto train annotators so that they can perform an annotation task.The result of an annotation task or process can also be evaluated, both qualitatively and quantitatively.In the context of this chapter, the typical approach is to compare multiple annotations of the sameprimary data, created by multiple annotators, and to compare their inter-annotator agreement, i. e., howwell do the various annotators agree when comparing their respective annotations. Multiple approachesto calculate inter-annotator agreement exist (Gut and Bayerl, 2004, Bayerl and Paul, 2007, 2011). Thisanalysis is crucial for data and experiment-related aspects such as replicability and reproducibilityand for measuring the consensus among the annotators, especially for complex annotation tasks oremerging annotation formats. A variation of measuring inter-annotator agreement can be describedas “intra-annotator agreement”, i. e., the same annotator is asked to perform the same annotation taskmultiple times but under different conditions or several days or weeks apart. This approach can also beused to identify weaknesses in emerging annotation schemes or guidelines.
This article presents various observations on annotations. It approaches the topic from multiple anglesincluding Hypertext, Computational Linguistics and Language Technology, Artificial Intelligence andOpen Science. Annotations can be examined along different dimensions. In terms of complexity, theycan range from trivial to highly sophisticated, in terms of maturity from experimental to standardised.Annotations can be annotated themselves using more abstract annotations. Primary research data suchas, e. g., text documents can be annotated on different layers concurrently (e. g., general segmentationincluding text structure, coherence relations, syntax), which are independent but can be exploitedusing multi-layer querying. Standards guarantee interoperability and reusability of data sets, which isespecially crucial in terms of Open Science.The chapter concludes with four final observations, formulated as research questions or ratherprovocative remarks on the current state of the field.
Do standards hold back innovative annotation research?
Standard annotation schemes representthe condensed consensus gathered within a wider research community regarding certain phenomena.This class of standardised formats is crucial for interoperability and reproducibility. However, one aspectthat is often neglected concerns the fundamental nature of research itself, which is about finding, creatingand inventing new things, new pieces of knowledge, new insights, including new ways of annotatinglanguage data. Especially taking into account those annotation schemes that are, both conceptually andalso technically, highly similar, it is worth emphasising that new breakthroughs require new approaches.Focusing on standards too much may hold back research.
Can we concentrate on annotating live web data instead of dead web data?
Primary researchdata is nowadays typically annotated within a web-based environment, i. e., using a dynamic webapplication that visualises both the primary and the secondary research data in a browser. Very often,said primary data is, in fact, web data, i. e., text or multimedia data that was either crawled or collectedusing other means from the World Wide Web. Crawling and archiving live web data decouples thedocuments from their natural habitat, which essentially results in frozen snapshots of these documents.While this approach has been best practice in Computational Linguistics almost since the beginning ofthe World Wide Web, it would be much more interesting to treat the live
World Wide Web as a corpus.11iven that the web technology stack even includes its own annotation approach (Web Annotation, seeSection 3), we should attempt to treat the whole, live World Wide Web as a giant corpus by parsingthe whole web and by adding linguistic information using the Web Annotation approach, which canthen be queried for linguistic analyses or for training machine learning models (Rehm, 2018, Rehmet al., 2018a). To that end, larger collections of web-native Language Technology services (Rehm et al.,2020a,b) could be used in high-performance infrastructures (Rehm et al., 2020c).
Is it possible to design a machine-readable packaging format for describing annotations?
An-notations have different dimensions along which they can be described (Section 8). It would be ahighly interesting question to examine if it is possible to design a compact, machine-readable packagingformat for describing annotation projects including the annotations themselves as well as the overallapproach, main formal aspects of the annotation scheme (including its complexity) and the concreteannotations. This is a relevant and important question from the point of view of Open Science (and moretransparent as well as reproducible and interoperable science). The question also relates to machinelearning, language resources and emerging AI and LT platforms. Soon, these will be able to import adata set and use a machine learning toolkit automatically to train a new model (Rehm et al., 2020c). Inorder for this to work fully automatically, we need metadata schemes to describe annotated data setsincluding formal aspects such as their annotation schemes and involved dimensions.
Is the field ignoring decades of valuable annotation science research?
Since the emergence ofthe first large corpora and the statistical turn in the early 1990s, Computational Linguistics has produceda plethora of results and insights regarding the annotation of language resources – so much so that Ide(2007) even speaks of “annotation science”. In the last five years, neural approaches have turned outto be very popular in Language Technology, outperforming essentially all of the previous methods.Generally speaking, neural technologies require very large data sets for training models. Correspondingapplications are often generalised as classification tasks that are based on large data sets that wereannotated with only few labels. In many cases, both the classification tasks and also the sets of labels orannotations must be described as rather simplistic, often focusing upon incremental research challenges.At the same time, many of the recent language resources were annotated on a rather shallow level,with only a few highly generalised and abstract labels, often using crowd-workers who are only able toproduce large amounts of consistent and high quality annotations if the annotation task is rather simpleand does not require expert linguistic knowledge or in-depth training (Poesio et al., 2017, call these“microtasks”). In short, since the neural turn in approx. 2014/2015 we can observe a trend towards simplymore and more annotations with increasing quantity while ignoring complexity and structure, and also atrend towards more and more simple annotations that are cheaper to produce and easier to generalisefrom. Has annotation science perhaps become obsolete? Have the lessons and insights learned in thelast 30 years become irrelevant, given today’s popularity and power of neural approaches for processingand, perhaps, finally, understanding language?
Acknowledgement
This chapter is based on a presentation given at the conference
Annotation in Scholarly Editions andResearch: Function – Differentiation – Systematization , held at the University of Wuppertal, Germany,on 20-22 February 2019. The author would like to thank the organisers, Julia Nantke and FrederikSchlupkothen, for the invitation and especially for their patience. Peter Bourgonje and Karolina VictoriaZaczynska provided comments on an early draft of this article for which the author is grateful. Work onthis chapter was partially supported by the projects ELG (EU Horizon 2020, no. 825627), LYNX (EUHorizon 2020, no. 780602) and QURATOR (BMBF, no. 03WKDA1A).12 eferences
Ron Artstein. Inter-annotator Agreement. In Ide and Pustejovsky (2017), pages 297–313.Petra Saskia Bayerl and Karsten Ingmar Paul. Squibs and Discussions: Identifying Sources of Disagree-ment: Generalizability Theory in Manual Annotation Studies.
Computational Linguistics
Computational Linguistics
Speech Communication ,33:23–60, January 2001.Mark-Jan Bludau, Marian D¨ork, Heiner Fangerau, Thorsten Halling, Elena Leitner, Sina Menzel, Ger-hard M¨uller, Vivien Petras, Georg Rehm, Clemens Neudecker, David Zellh¨ofer, and Juli´an MorenoSchneider. SoNAR (IDH): Datenschnittstellen f¨ur historische Netzwerkanalyse. In Christof Sch¨och,editor,
DHd 2020 Spielr¨aume: Digital Humanities zwischen Modellierung und Interpretation. Kon-ferenzabstracts. , pages 360–362, Paderborn, Germany, 03 2020. doi: 10.5281/zenodo.3666690. 02-06March 2020.Peter Bourgonje, Juli´an Moreno-Schneider, Jan Nehring, Georg Rehm, Felix Sasaki, and Ankit Srivastava.Towards a Platform for Curation Technologies: Enriching Text Collections with a Semantic-WebLayer. In Harald Sack, Giuseppe Rizzo, Nadine Steinmetz, Dunja Mladeni´c, S¨oren Auer, andChristoph Lange, editors,
The Semantic Web
Atlantic Monthly , 176:101–108, 1945.Christian Chiarcos and Maria Sukhareva. OLiA – Ontologies of Linguistic Annotation.
Semantic WebJournal , 6(4):379–386, 2015.Markus Dickinson and Dan Tufis¸. Iterative Enhancement. In Ide and Pustejovsky (2017), pages 257–276.Stefanie Dipper, Michael G¨otze, and Manfred Stede. Simple annotation tools for complex annotationtasks: an evaluation. In
Proceedings of the LREC Workshop on XML-based richly annotated corpora ,pages 54–62, May 2004. 13ichard Eckart de Castilho, ´Eva M´ujdricza-Maydt, Seid Muhie Yimam, Silvana Hartmann, IrynaGurevych, Anette Frank, and Chris Biemann. A Web-based Tool for the Integrated Annotationof Semantic and Syntactic Structures. In
Proceedings of the Workshop on Language TechnologyResources and Tools for Digital Humanities (LT4DH) , pages 76–84, Osaka, Japan, December 2016.The COLING 2016 Organizing Committee.Scott Farrar and D. Terrence Langendoen. An OWL-DL Implementation of Gold: An Ontology for theSemantic Web. In Metzing and Witt (2010), pages 45–66.Mark A. Finlayson and Tomaˇz Erjavec. Overview of Annotation Creation: Processes and Tools. In Ideand Pustejovsky (2017), pages 167–191.Daniela Goecke, Harald L¨ungen, Dieter Metzing, Maik St¨uhrenberg, and Andreas Witt. Different Viewson Markup – Distinguishing Levels and Layers. In Metzing and Witt (2010), pages 1–21.Ulrike Gut and Petra Saskia Bayerl. Measuring the reliability of manual annotations of speech corpora.In
Proceedings of the 2nd International Conference on Speech Prosody , pages 565–568, Nara, Japan,January 2004.Sebastian Hellmann, Jens Lehmann, S¨oren Auer, and Martin Br¨ummer. Integrating nlp using linkeddata. In Harith Alani, Lalana Kagal, Achille Fokoue, Paul Groth, Chris Biemann, Josiane XavierParreira, Lora Aroyo, Natasha Noy, Chris Welty, and Krzysztof Janowicz, editors,
The Semantic Web –Proceedings of ISWC 2013 , pages 98–113, Sydney, Australia, 2013. 21-25 October.Nancy Ide. Annotation Science: From Theory to Practice and Use. In Georg Rehm, Andreas Witt,and Lothar Lemnitzer, editors,
Datenstrukturen f¨ur linguistische Ressourcen und ihre Anwendungen– Data Structures for Linguistic Resources and Applications: Proceedings of the Biennial GLDVConference 2007 , pages 3–7. Gunter Narr, T¨ubingen, 2007.Nancy Ide and James Pustejovsky, editors.
Handbook of Linguistic Annotation . Springer, Dordrecht,2017.Nancy Ide and Laurent Romary. A Common Framework for Syntactic Annotation. In
Proceedings ofthe 39th Annual Meeting on Association for Computational Linguistics , ACL’01, pages 306–313,Stroudsburg, PA, USA, July 2001. Association for Computational Linguistics.Nancy Ide and Keith Suderman. GrAF: A graph-based format for linguistic annotations. In proceedingsof the Linguistic Annotation Workshop , pages 1–8. Association for Computational Linguistics, June2007.Nancy Ide, Nicoletta Calzolari, Judith Eckle-Kohler, Dafydd Gibbon, Sebastian Hellmann, Kiyong Lee,Joakim Nivre, and Laurent Romary. Community Standards for Linguistically-Annotated Resources.In Ide and Pustejovsky (2017), pages 113–165.Nancy Ide, Christian Chiarcos, Manfred Stede, and Steve Cassidy. Designing Annotation Schemes:From Model to Representation. In Ide and Pustejovsky (2017), pages 73–111.Mohamad Yaser Jaradeh, Allard Oelen, Kheir Eddine Farfar, Manuel Prinz, Jennifer D’Souza, G´aborKismih´ok, Markus Stocker, and S¨oren Auer. Open Research Knowledge Graph: Next GenerationInfrastructure for Semantic Scholarly Knowledge. In
Proceedings of the 10th International Conferenceon Knowledge Capture , K-CAP ’19, pages 243–246, New York, NY, USA, 2019. ACM. doi:10.1145/3360901.3364435. URL https://doi.org/10.1145/3360901.3364435.14an-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych.The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In
Proceedings of the 27th International Conference on Computational Linguistics: System Demonstra-tions , pages 5–9. Association for Computational Linguistics, Juni 2018.Penny Labropoulou, Katerina Gkirtzou, Maria Gavriilidou, Miltos Deligiannis, Dimitris Galanis, SteliosPiperidis, Georg Rehm, Maria Berger, Val´erie Mapelli, Michael Rigault, Victoria Arranz, KhalidChoukri, Gerhard Backfried, Jos´e Manuel G´omez P´erez, and Andres Garcia-Silva. Making MetadataFit for Next Generation Language Technology Platforms: The Metadata Schema of the EuropeanLanguage Grid. In Nicoletta Calzolari, Fr´ed´eric B´echet, Philippe Blache, Christopher Cieri, KhalidChoukri, Thierry Declerck, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Asuncion Moreno, JanOdijk, and Stelios Piperidis, editors,
Proceedings of the 12th Language Resources and EvaluationConference (LREC 2020) , Marseille, France, 5 2020. European Language Resources Association(ELRA). Accepted for publication.Elena Leitner, Georg Rehm, and Juli´an Moreno-Schneider. Fine-grained Named Entity Recognition inLegal Documents. In Maribel Acosta, Philippe Cudr´e-Mauroux, Maria Maleshkova, Tassilo Pellegrini,Harald Sack, and York Sure-Vetter, editors,
Semantic Systems. The Power of AI and KnowledgeGraphs. Proceedings of the 15th International Conference (SEMANTiCS 2019) , number 11702 inLecture Notes in Computer Science, pages 272–287, Karlsruhe, Germany, 9 2019. Springer. 10/11September 2019.Elena Leitner, Georg Rehm, and Juli´an Moreno-Schneider. A Dataset of German Legal Documents forNamed Entity Recognition. In Nicoletta Calzolari, Fr´ed´eric B´echet, Philippe Blache, ChristopherCieri, Khalid Choukri, Thierry Declerck, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, AsuncionMoreno, Jan Odijk, and Stelios Piperidis, editors,
Proceedings of the 12th Language Resources andEvaluation Conference (LREC 2020) , Marseille, France, 5 2020. European Language ResourcesAssociation (ELRA). Accepted for publication.Eve Maler and Jeanne El Andaloussi.
Developing SGML DTDs – From Text to Model to Markup .Prentice Hall, Upper Saddle River, New Jersey, 1996.Dave Megginson.
Structuring XML Documents . Charles F. Goldfarb Series on Open InformationManagement. Prentice Hall, Upper Saddle River, New Jersey, 1998.Jan Christoph Meister, Jan Horstmann, Marco Petris, Janina Jacke, Christian Bruck, Mareike Schu-macher, and Marie Fl¨uh. CATMA – Computer Assisted Text Markup and Analysis, 2019. URLhttps://doi.org/10.5281/zenodo.3523228.Dieter Metzing and Andreas Witt, editors.
Linguistic Modelling of Information and Markup Languages.Contributions to Language Technology . Springer, Dordrecht, Heidelberg, London, New York, 2010.Juli´an Moreno-Schneider, Ankit Srivastava, Peter Bourgonje, David Wabnitz, and Georg Rehm. SemanticStorytelling, Cross-lingual Event Detection and other Semantic Services for a Newsroom ContentCuration Dashboard. In Octavian Popescu and Carlo Strapparava, editors,
Proceedings of theSecond Workshop on Natural Language Processing meets Journalism – EMNLP 2017 Workshop(NLPMJ 2017) , pages 68–73, Copenhagen, Denmark, 9 2017. 7 September.Theodor Holm Nelson. Literary Machines. Mindful Press, 1987. Edition 87.1.15ariana Neves and Jurica ˇSeva. An extensive review of tools for manual annotation of documents.
Briefings in Bioinformatics , 12 2019. ISSN 1477-4054. doi: 10.1093/bib/bbz130. URL https://doi.org/10.1093/bib/bbz130.Malte Ostendorff, Peter Bourgonje, Maria Berger, Juli´an Moreno-Schneider, and Georg Rehm. EnrichingBERT with Knowledge Graph Embeddings for Document Classification. In Steffen Remus, RamiAly, and Chris Biemann, editors,
Proceedings of the GermEval Workshop 2019 – Shared Task on theHierarchical Classification of Blurbs , Erlangen, Germany, 10 2019. 8 October 2019.Stelios Piperidis, Harris Papageorgiou, Christian Spurk, Georg Rehm, Khalid Choukri, Olivier Hamon,Nicoletta Calzolari, Riccardo del Gratta, Bernardo Magnini, and Christian Girardi. META-SHARE:One year after. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, HrafnLoftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis,editors,
Proceedings of the 9th Language Resources and Evaluation Conference (LREC 2014) , pages1532–1538, Reykjavik, Iceland, 5 2014. European Language Resources Association (ELRA).Massimo Poesio, Jon Chamberlain, and Udo Kruschwitz. Crowdsourcing. In Ide and Pustejovsky(2017), pages 277–295.James Pustejovsky, Harry Bunt, and Annie Zaenen. Designing Annotation Schemes: From Theory toModel. In Ide and Pustejovsky (2017), pages 21–72.Georg Rehm. Towards Automatic Web Genre Identification – A Corpus-Based Approach in theDomain of Academia by Example of the Academic’s Personal Homepage. In Ralph Sprague, editor,
Proceedings of the 35th Hawaii International Conference on System Sciences (HICSS-35) , pages1143–1152, Big Island, Hawaii, 1 2002. IEEE Computer Society.Georg Rehm.
Hypertextsorten: Definition – Struktur – Klassifikation . Books on Demand, Norderstedt,2007. PhD thesis in Applied and Computational Linguistics, Justus-Liebig-Universit¨at Gießen, 2005.Georg Rehm. Hypertext Types and Markup Languages – The Relationship Between HTML and WebGenres. In Metzing and Witt (2010), pages 143–164.Georg Rehm. Texttechnologische Grundlagen. In Kai-Uwe Carstensen, Christian Ebert, Cornelia Endriss,Susanne Jekat, Ralf Klabunde, and Hagen Langer, editors,
Computerlinguistik und Sprachtechnologie– Eine Einf¨uhrung , pages 159–168. Spektrum, Heidelberg, 3 edition, 2010b.Georg Rehm. The Language Resource Life Cycle: Towards a Generic Model for Creating, Maintaining,Using and Distributing Language Resources. In Nicoletta Calzolari (Conference Chair), KhalidChoukri, Thierry Declerck, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Asuncion Moreno,Jan Odijk, and Stelios Piperidis, editors,
Proceedings of the 10th Language Resources and Evalua-tion Conference (LREC 2016) , pages 2450–2454, Portoroˇz, Slovenia, 5 2016. European LanguageResources Association (ELRA).Georg Rehm. An Infrastructure for Empowering Internet Users to handle Fake News and other OnlineMedia Phenomena. In Georg Rehm and Thierry Declerck, editors,
Language Technologies for theChallenges of the Digital Age: 27th International Conference, GSCL 2017, Berlin, Germany, Septem-ber 13-14, 2017, Proceedings , number 10713 in Lecture Notes in Artificial Intelligence (LNAI), pages216–231, Cham, Switzerland, 1 2018. Gesellschaft f¨ur Sprachtechnologie und Computerlinguistike.V., Springer. 13/14 September 2017. 16eorg Rehm, Richard Eckart, and Christian Chiarcos. An OWL- and XQuery-Based Mechanism for theRetrieval of Linguistic Patterns from XML-Corpora. In Galia Angelova, Kalina Bontcheva, RuslanMitkov, Nicolas Nicolov, and Nicolai Nikolov, editors,
International Conference Recent Advances inNatural Language Processing (RANLP 2007) , pages 510–514, Borovets, Bulgaria, 9 2007a. Incoma.Georg Rehm, Andreas Witt, and Lothar Lemnitzer, editors.
Datenstrukturen f¨ur linguistische Ressourcenund ihre Anwendungen – Data Structures for Linguistic Resources and Applications: Proceedings ofthe Biennial GLDV Conference 2007 . Gunter Narr, T¨ubingen, 2007b. This book at amazon.de.Georg Rehm, Richard Eckart, Christian Chiarcos, and Johannes Dellert. Ontology-Based XQuery’ingof XML-Encoded Language Resources on Multiple Annotation Layers. In Nicoletta Calzolari (Con-ference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, andDaniel Tapias, editors,
Proceedings of the 6th Language Resources and Evaluation Conference(LREC 2008) , pages 525–532, Marrakesh, Morocco, 5 2008a.Georg Rehm, Oliver Schonefeld, Andreas Witt, Timm Lehmberg, Christian Chiarcos, Hanan Bechara,Florian Eishold, Kilian Evang, Magdalena Leshtanska, Aleksandar Savkov, and Matthias Stark.The Metadata-Database of a Next Generation Sustainability Web-Platform for Language Resources.In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, JanOdijk, Stelios Piperidis, and Daniel Tapias, editors,
Proceedings of the 6th Language Resources andEvaluation Conference (LREC 2008) , pages 371–378, Marrakesh, Morocco, 5 2008b.Georg Rehm, Oliver Schonefeld, Andreas Witt, Erhard Hinrichs, and Marga Reis. Sustainability ofAnnotated Resources in Linguistics: A Web-Platform for Exploring, Querying and DistributingLinguistic Corpora and Other Resources.
Literary and Linguistic Computing
Proceedings of the 11th Language Resources and Evaluation Confer-ence (LREC 2018) , pages 2416–2422, Miyazaki, Japan, 5 2018a. European Language ResourcesAssociation (ELRA).Georg Rehm, Juli´an Moreno Schneider, Peter Bourgonje, Ankit Srivastava, Rolf Fricke, Jan Thomsen,Jing He, Joachim Quantz, Armin Berger, Luca K¨onig, S¨oren R¨auchle, Jens Gerth, and David Wabnitz.Different Types of Automated and Semi-Automated Semantic Storytelling: Curation Technologiesfor Different Sectors. In Georg Rehm and Thierry Declerck, editors,
Language Technologies forthe Challenges of the Digital Age: 27th International Conference, GSCL 2017, Berlin, Germany,September 13-14, 2017, Proceedings , number 10713 in Lecture Notes in Artificial Intelligence(LNAI), pages 232–247, Cham, Switzerland, 1 2018b. Gesellschaft f¨ur Sprachtechnologie undComputerlinguistik e.V., Springer. 13/14 September 2017.17eorg Rehm, Juli´an Moreno-Schneider, Jorge Gracia, Artem Revenko, Victor Mireles, Maria Khvalchik,Ilan Kernerman, Andis Lagzdins, Marcis Pinnis, Artus Vasilevskis, Elena Leitner, Jan Milde, andPia Weißenhorn. Developing and Orchestrating a Portfolio of Natural Legal Language Processingand Document Curation Services. In Nikolaos Aletras, Elliott Ash, Leslie Barrett, Daniel Chen,Adam Meyers, Daniel Preotiuc-Pietro, David Rosenberg, and Amanda Stent, editors,
Proceedings ofWorkshop on Natural Legal Language Processing (NLLP 2019) , pages 55–66, Minneapolis, USA, 62019. Co-located with NAACL 2019. 7 June 2019.Georg Rehm, Maria Berger, Ela Elsholz, Stefanie Hegele, Florian Kintzel, Katrin Marheinecke, SteliosPiperidis, Miltos Deligiannis, Dimitris Galanis, Katerina Gkirtzou, Penny Labropoulou, KalinaBontcheva, David Jones, Ian Roberts, Jan Hajic, Jana Hamrlov´a, Luk´aˇs Kaˇcena, Khalid Choukri,Victoria Arranz, Andrejs Vasil¸jevs, Orians Anvari, Andis Lagzdin¸ ˇs, J¯ulija Mel¸n¸ika, Gerhard Backfried,Erinc¸ Dikici, Miroslav Janosik, Katja Prinz, Christoph Prinz, Severin Stampler, Dorothea Thomas-Aniola, Jos´e Manuel G´omez P´erez, Andres Garcia Silva, Christian Berr´ıo, Ulrich Germann, SteveRenals, and Ondrej Klejch. European Language Grid: An Overview. In Nicoletta Calzolari, Fr´ed´ericB´echet, Philippe Blache, Christopher Cieri, Khalid Choukri, Thierry Declerck, Hitoshi Isahara, BenteMaegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors,
Proceedingsof the 12th Language Resources and Evaluation Conference (LREC 2020) , Marseille, France, 5 2020a.European Language Resources Association (ELRA). Accepted for publication.Georg Rehm, Peter Bourgonje, Stefanie Hegele, Florian Kintzel, Juli´an Moreno Schneider, MalteOstendorff, Karolina Zaczynska, Armin Berger, Stefan Grill, S¨oren R¨auchle, Jens Rauenbusch, LisaRutenburg, Andr´e Schmidt, Mikka Wild, Henry Hoffmann, Julian Fink, Sarah Schulz, Jurica Seva,Joachim Quantz, Joachim B¨ottger, Josefine Matthey, Rolf Fricke, Jan Thomsen, Adrian Paschke,Jamal Al Qundus, Thomas Hoppe, Naouel Karam, Frauke Weichhardt, Christian Fillies, ClemensNeudecker, Mike Gerber, Kai Labusch, Vahid Rezanezhad, Robin Schaefer, David Zellh¨ofer, DanielSiewert, Patrick Bunk, Lydia Pintscher, Elena Aleynikova, and Franziska Heine. QURATOR:Innovative Technologies for Content and Data Curation. In Adrian Paschke, Clemens Neudecker,Georg Rehm, Jamal Al Qundus, and Lydia Pintscher, editors,
Proceedings of QURATOR 2020 – Theconference for intelligent content solutions , Berin, Germany, 02 2020b. CEUR Workshop Proceedings,Volume 2535. 20/21 January 2020.Georg Rehm, Dimitrios Galanis, Penny Labropoulou, Stelios Piperidis, Martin Welß, Ricardo Usbeck,Joachim K¨ohler, Miltos Deligiannis, Katerina Gkirtzou, Johannes Fischer, Christian Chiarcos, NilsFeldhus, Juli´an Moreno-Schneider, Florian Kintzel, Elena Montiel, V´ıctor Rodr´ıguez Doncel, John P.McCrae, David Laqua, Irina Patricia Theile, Christian Dittmar, Kalina Bontcheva, Ian Roberts,Andrejs Vasiljevs, and Andis Lagzdin¸ ˇs. Towards an Interoperable Ecosystem of AI and LT Platforms:A Roadmap for the Implementation of Different Levels of Interoperability. In Georg Rehm, KalinaBontcheva, Khalid Choukri, Jan Hajic, Stelios Piperidis, and Andrejs Vasiljevs, editors,
Proceedingsof the 1st International Workshop on Language Technology Platforms (IWLTP 2020, co-located withLREC 2020) , pages 96–107, Marseille, France, 5 2020c. 16 May 2020.Georg Rehm, Katrin Marheinecke, Stefanie Hegele, Stelios Piperidis, Kalina Bontcheva, Jan Hajic,Khalid Choukri, Andrejs Vasil¸jevs, Gerhard Backfried, Christoph Prinz, Jos´e Manuel G´omez P´erez,Luc Meertens, Paul Lukowicz, Josef van Genabith, Andrea L¨osch, Philipp Slusallek, Morten Irgens,Patrick Gatellier, Joachim K¨ohler, Laure Le Bars, Dimitra Anastasiou, Albina Auksori¯ut˙e, N´uriaBel, Ant´onio Branco, Gerhard Budin, Walter Daelemans, Koenraad De Smedt, Radovan Garab´ık,Maria Gavriilidou, Dagmar Gromann, Svetla Koeva, Simon Krek, Cvetana Krstev, Krister Lind´en,18ernardo Magnini, Jan Odijk, Maciej Ogrodniczuk, Eir´ıkur R¨ognvaldsson, Mike Rosner, BolettePedersen, Inguna Skadina, Marko Tadi´c, Dan Tufis , , Tam´as V´aradi, Kadri Vider, Andy Way, andFranc¸ois Yvon. The European Language Technology Landscape in 2020: Language-Centric andHuman-Centric AI for Cross-Cultural Communication in Multilingual Europe. In Nicoletta Calzolari,Fr´ed´eric B´echet, Philippe Blache, Christopher Cieri, Khalid Choukri, Thierry Declerck, HitoshiIsahara, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020)
Proceedings of the conference onempirical methods in natural language processing , pages 254–263. Association for ComputationalLinguistics, October 2008.Manfred Stede.
Korpusgest¨utzte Textanalyse: Grundz¨uge der Ebenen-orientierten Textlinguistik . NarrFrancke Attempto, T¨ubingen, 2018. 2. ¨uberarbeitete Auflage.Pontus Stenetorp, Sampo Pyysalo, Goran Topi´c, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii.brat: a Web-based Tool for NLP-Assisted Text Annotation. In
Proceedings of the Demonstrations atthe 13th Conference of the European Chapter of the Association for Computational Linguistics , pages102–107, Avignon, France, April 2012. Association for Computational Linguistics.Maik St¨uhrenberg. The TEI and current standards for structuring linguistic data. An overview.
Journalof the Text Encoding Initiative , 3, November 2012.TEI Consortium, editor.
TEI: P5 Guidelines for Electronic Text Encoding and Interchange . TEIConsortium, 2019. Version 3.6.0. Last updated on 16th July 2019. Originally edited by C.M. Sperberg-McQueen and Lou Burnard for the ACH-ALLC-ACL Text Encoding Initiative, now entirely revisedand expanded under the supervision of the Technical Council of the TEI Consortium.Norman Walsh and Tovey Bethan. The Markup Declaration. In B. Tommie Usdin, editor,
Proceedings ofBalisage: The Markup Conference 2018 , Washington, DC, USA, 8 2018. Balisage Series on MarkupTechnologies, Vol. 21.Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton,Arie Baak, Niklas Blomberg, Jan Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne,Jildau Bouwman, Anthony J. Brookes, Tim Clark, Merc`e Crosas, Ingrid Dillo, Olivier Dumon,Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J.G. Gray,Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A C ’t Hoen, Rob Hooft, Tobias19uhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E Martone, Albert Mons, Abel L. Packer,Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna Assunta Sansone, ErikSchultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, JohanVan Der Lei, Erik Van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, KatherineWolstencroft, Jun Zhao, and Barend Mons. The fair guiding principles for scientific data managementand stewardship.
Scientific Data , 3, 2016. ISSN 2052-4463. doi: 10.1038/sdata.2016.18.Andreas Witt, Georg Rehm, Timm Lehmberg, and Erhard Hinrichs. Mapping Multi-Rooted Trees froma Sustainable Exchange Format to TEI Feature Structures. In
TEI@20: 20 Years of Supporting theDigital Humanities. The 20th Anniversary Text Encoding Initiative Consortium Members’ Meeting ,University of Maryland, College Park, 10 2007a.Andreas Witt, Oliver Schonefeld, Georg Rehm, Jonathan Khoo, and Kilian Evang. On the LosslessTransformation of Single-File, Multi-Layer Annotations into Multi-Rooted Trees. In B. TommieUsdin, editor,
Proceedings of Extreme Markup Languages 2007 , Montr´eal, Canada, 8 2007b.Andreas Witt, Georg Rehm, Erhard Hinrichs, Timm Lehmberg, and Jens Stegmann. SusTEInability ofLinguistic Resources through Feature Structures.
Literary and Linguistic Computing , 24(3):363–372,2009. URL http://llc.oxfordjournals.org/content/24/3/363.short.Kai W¨orner, Andreas Witt, Georg Rehm, and Stefanie Dipper. Modelling Linguistic Data Structures. InB. Tommie Usdin, editor,