Ágnes Sándor

Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study

We propose the concept of Contested Collective Intelligence (CCI) as a distinctive subset of the broader Collective Intelligence design space. CCI is relevant to the many organizational contexts in which it is important to work with contested knowledge, for instance, due to different intellectual traditions, competing organizational objectives, information overload or ambiguous environmental signals. The CCI challenge is to design sociotechnical infrastructures to augment such organizational capability. Since documents are often the starting points for contested discourse, and discourse markers provide a powerful cue to the presence of claims, contrasting ideas and argumentation, discourse and rhetoric provide an annotation focus in our approach to CCI. Research in sensemaking, computer-supported discourse and rhetorical text analysis motivate a conceptual framework for the combined human and machine annotation of texts with this specific focus. This conception is explored through two tools: a social-semantic web application for human annotation and knowledge mapping (Cohere), plus the discourse analysis component in a textual analysis software tool (Xerox Incremental Parser: XIP). As a step towards an integrated platform, we report a case study in which a document corpus underwent independent human and machine analysis, providing quantitative and qualitative insight into their respective contributions. A promising finding is that significant contributions were signalled by authors via explicit rhetorical moves, which both human analysts and XIP could readily identify. Since working with contested knowledge is at the heart of CCI, the evidence that automatic detection of contrasting ideas in texts is possible through rhetorical discourse analysis is progress towards the effective use of automatic discourse analysis in the CCI framework.

Detecting key sentences for automatic assistance in peer reviewing research articles in educational sciences

The evaluation of scientific performance is gaining importance in all research disciplines. The basic process of the evaluation is peer reviewing, which is a time-consuming activity. In order to facilitate and speed up peer reviewing processes we have developed an exploratory NLP system in the field of educational sciences. The system highlights key sentences, which are supposed to reflect the most important threads of the article The highlighted sentences offer guidance on the content-level while structural elements -- the title, abstract, keywords, section headings -- give an orientation about the design of the argumentation in the article. The system is implemented using a discourse analysis module called concept matching applied on top of the Xerox Incremental Parser, a rule-based dependency parser. The first results are promising and indicate the directions for the future development of the system.

A Risk Assessment System with Automatic Extraction of Event Types

In this article we describe the joint effort of experts in linguistics, information extraction and risk assessment to integrate EventSpotter, an automatic event extraction engine, into ADAC, an automated early warning system. By detecting as early as possible weak signals of emerging risks ADAC provides a dynamic synthetic picture of situations involving risk. The ADAC system calculates risk on the basis of fuzzy logic rules operated on a template graph whose leaves are event types. EventSpotter is based on a general purpose natural language dependency parser, XIP, enhanced with domain-specific lexical resources (Lexicon-Grammar). Its role is to automatically feed the leaves with input data.

Reflective writing analytics for actionable feedback

Reflective writing can provide a powerful way for students to integrate professional experience and academic learning. However, writing reflectively requires high quality actionable feedback, which is time-consuming to provide at scale. This paper reports progress on the design, implementation, and validation of a Reflective Writing Analytics platform to provide actionable feedback within a tertiary authentic assessment context. The contributions are: (1) a new conceptual framework for reflective writing; (2) a computational approach to modelling reflective writing, deriving analytics, and providing feedback; (3) the pedagogical and user experience rationale for platform design decisions; and (4) a pilot in a student learning context, with preliminary data on educator and student acceptance, and the extent to which we can evidence that the software provided actionable feedback for reflective writing.

Collective intelligence as community discourse and action

Collective Intelligence (CI) research investigates the design of infrastructures to enable collectives to think and act intelligently, and intriguingly, more intelligently than individuals. Technologies such as idea management or argumentation tools, blogs, wikis, chats, forums, Q&A sites, and social networks provide unprecedented opportunities for entire communities or organizations to express a discourse and act at a massive scale. This workshop seeks to understand the forms of CI that can be constructed through discourse and action, which enables advanced forms of collective sensemaking such as idea generation and prioritization, argumentation, and deliberation. When does effective discourse help a collective outperform individuals? What functions should the next generation of social platforms support? How can we allow communities to efficiently manage many diverse ideas, argument, and deliberate? What patterns in discourse and action can be modeled computationally?

DCLA14: second international workshop on discourse-centric learning analytics

This workshop anticipates that an important class of learning analytic will emerge at the intersection of research into learning dynamics, online discussion platforms, and computational linguistics. Written discourse is arguably the primary class of data that can give us insights into deeper learning and higher order qualities such as critical thinking, argumentation, mastery of complex ideas, empathy, collaboration and interpersonal skills. Moreover, the ability to write in a scholarly manner is a core competence, often taking the form of discourse with oneself and the literature. Computational linguistics research has developed a rich array of tools for machine interpretation of human discourse, but work to develop these tools in the context of learning is at a relatively early stage. Moreover, there is a significant difference between designing tools to assist researchers in discourse analysis, and their deployment on platforms to provide meaningful analytics for the learners and educators who are conducting that discourse. This workshop aims to catalyse ideas and build community connections among those who want to shape this field.

Cohere and XIP: human annotation harnessing machine annotation power

In this video we demonstrate the practical application of research on human and machine annotation of online documents to support reflective reading and collective sensemaking of online documents. We present an innovative research prototype which integrate a discourse analysis software (XIP) with an open source Web Annotation and Knowledge-Mapping tool (Cohere). We visualize an interactive scenario of use of the two integrated technologies in a unique user experience. This dynamic scenario will give an inspiring vision of future CSCW systems, which combine human annotation to harness machine analysis and reasoning power.

Finding Quality: A Multilingual Search Engine for Educational Research

To develop a field specific and multilingual search-engine, numerous algorithms are needed in addition to a general-purpose search engine. Here we describe the focal areas of development done in EERQI: Automatic classification for educational research, multilingual retrieval, query extension and relevance ranking. The classification algorithms, developed in EERQI enable a crawler to identify relevant objects with respect to a scientific field; the multilingual algorithms allow the retrieval of documents in several languages; query extension proposes related query terms to the user; relevance ranking is enhanced by semantic analysis.

Enriching How-to Guides by Linking Actionable Phrases

The World Wide Web contains a large number of community created knowledge of instructional nature. Similarly, in a commercial setting, databases of instructions are used by customer-care providers to guide clients in the resolution of issues. Most of these instructions are expressed in natural language. Knowledge Bases including such information are valuable through the sum of their single entries. However, as each entry is created mostly independently, users (e.g. other community members) cannot take advantage of the accumulated knowledge that can be developed via the aggregation of related entries. In this paper we consider the problem of inter-linking Knowledge Base entries, in order to get relevant information from other parts of the Knowledge Base. To achieve this, we propose to detect \textit{actionable phrases} -- text fragments that describe how to perform a certain action -- and link them to other entries. The extraction method that we implement achieves an F-score of 67.35\%. We also show that using actionable phrases results in better linking quality than using coarser-grained spans of text, as proposed in the literature. Besides the evaluation of both steps, we also include a detailed error analysis and release our annotation to the community.

Identifying User Issues and Request Types in Forum Question Posts Based on Discourse Analysis

In this paper we propose the detection of user issues and request types in technical forum question posts with a twofold purpose: supporting up-to-date knowledge generation in organizations that provide (semi-) automated customer-care services, and enriching forum metadata in order to enhance the effectiveness of search. We present a categorization system for detecting the proposed question post types based on discourse analysis, and show the advantage of using discourse patterns compared to a baseline relying on standard linguistic features. Besides the detailed description of our method, we also release our annotated corpus to the community.