Iraide Zipitria

From Human to Automatic Summary Evaluation

One of the goals remaining in Intelligent Tutoring Systems is to create applications to evaluate open-ended text in a human-like manner. The aim of this study is to produce the design for a fully automatic summary evaluation system that could stand for human-like summarisation assessment. In order to gain this goal, an empirical study has been carried out to identify underlying cognitive processes. The studied sample is compound by 15 expert raters on summary evaluation with different professional backgrounds in education. Pearson’s correlation has been calculated to see inter-rater agreement level and stepwise linear regression to observe predicting variables and weights. In addition, interviews with subjects provided qualitative information that could not be acquired numerically. Based on this research, a design of a fully automatic summary evaluation environment has been described.

Observing lemmatization effect in LSA coherence and comprehension grading of learner summaries

Current work in learner evaluation of Intelligent Tutoring Systems (ITSs), is moving towards open-ended educational content diagnosis. One of the main difficulties of this approach is to be able to automatically understand natural language. Our work is directed to produce automatic evaluation of learner summaries in Basque. Therefore, in addition to language comprehension, difficulties emerge from Basque morphology itself. In this work, Latent Semantic Analysis (LSA) is used to model comprehension in a language in which lemmatization has shown to be highly significant. This paper tests the influence of corpus lemmatization while performing automatic comprehension and coherence grading. Summaries graded by human judges in coherence and comprehension, have been tested against LSA based measures from source lemmatized and non-lemmatized corpora. After lemmatization, the amount of LSA known single terms was reduced in a 56% of its original number. As a result, LSA grades almost match human measures, producing no significant differences between the lemmatized and non-lemmatized approaches.

What is behind a summary-evaluation decision?

Research in psychology has reported that, among the variety of possibilities for assessment methodologies, summary evaluation offers a particularly adequate context for inferring text comprehension and topic understanding. However, grades obtained in this methodology are hard to quantify objectively. Therefore, we carried out an empirical study to analyze the decisions underlying human summary-grading behavior. The task consisted of expert evaluation of summaries produced in critically relevant contexts of summarization development, and the resulting data were modeled by means of Bayesian networks using an application called Elvira, which allows for graphically observing the predictive power (if any) of the resultant variables. Thus, in this article, we analyzed summary-evaluation decision making in a computational framework.

Discourse Measures for Basque Summary Grading.

In the context of Learning Technologies, the need to be able to assess the learning and domain comprehension in open-ended learner responses has been present in artificial intelligence and education since its beginnings. The advantage of using summaries is that they allow teachers to diagnose comprehension and the amount of information remembered from text in the learning process. This study addresses the issue of automatically obtaining overall discourse scores from surface discourse measures for Basque language. Global measures have been studied for cohesion, adequacy and use of language. The approach taken was to estimate the presence of the automatically gathered surface discourse measures in expert grading decisions in cohesion, adequacy and use of language. As a consequence, three grading decision-making regression models were obtained to estimate overall grades from text written in Basque. Next, the obtained regression models were tested in corpus-containing summaries written by learners with different degrees of summarisation maturity. The results show that the obtained grading frameworks significantly reflect human decisions and are able to discriminate summarisation maturity differences.

Corpus-based performance analysis for automatically grading use of language in essays

From its early beginning a big issue in Computer Supported Learning Systems research has been directed to automatically evaluating freely written text. Previous work in use of language grading of summaries showed to be successful identifying critical differences in summary writing maturity. This work, describes further testing discriminating course-to-course improvements of second language learners. Automatic grades are tested on an essay corpus.

LEA: A Summarization Web Environment Based on Human Instructors' Behaviour

This paper presents LEA a Web application for summarization exercise development. Appropriate for learners on different summarization ability levels as well as for various learning domains. The summarization environment provides edition facilities and aid tools to support learning. It assists during the summarization task and performs automatic assessment. The automatic summary grading process is based on expert performance, to emulate the evaluation behaviour underneath. Expert criterion is represented by a grading decision making model. In addition, discourse analysis and comprehension measures are modelled by means of natural language processing (NLP) techniques and latent semantic analysis (LSA).