Inmaculada Escudero

Latent Semantic Analysis Parameters for Essay Evaluation using Small-Scale Corpora*

Abstract Some previous studies (e.g. that carried out by Van Bruggen et al. in 2004) have pointed to a need for additional research in order to firmly establish the usefulness of LSA (latent semantic analysis) parameters for automatic evaluation of academic essays. The extreme variability in approaches to this technique makes it difficult to identify the most efficient parameters and the optimum combination. With this goal in mind, we conducted a high spectrum study to investigate the efficiency of some of the major LSA parameters in small-scale corpora. We used two specific domain corpora that differed in the structure of the text (one containing only technical terms and the other with more tangential information). Using these corpora we tested different semantic spaces, formed by applying different parameters and different methods of comparing the texts. Parameters varied included weighting functions (Log-IDF or Log-Entropy), dimensionality reduction (truncating the matrices after SVD to a set percentage of dimensions), methods of forming pseudo-documents (vector sum and folding-in) and measures of similarity (cosine or Euclidean distances). We also included two groups of essays to be graded, one written by experts and other by non-experts. Both groups were evaluated by three human graders and also by LSA. We extracted the correlations of each LSA condition with human graders, and conducted an ANOVA to analyse which parameter combination correlates best. Results suggest that distances are more efficient in academic essay evaluation than cosines. We found no clear evidence that the classical LSA protocol works systematically better than some simpler version (the classical protocol achieves the best performance only for some combinations of parameters in a few cases), and found that the benefits of reducing dimensionality arise only when the essays are introduced into semantic spaces using the folding-in method.

Procesos inferenciales en la comprensión del discurso escrito: Infuencia de la estructura del texto en los procesos de comprensión

A traves de este trabajo se examinan diversas cuestiones relacionadas con los procesos de comprension y los tipos de texto o discurso escrito. El objetivo fundamental de este articulo es tratar de desvelar si el procesamiento que realiza el lector durante la comprension de un discurso determinado requiere de una actividad cognitiva diferente a cuando se trata de comprender otro de distinta naturaleza. Esta actividad cognitiva se identifica con la generacion de inferencias. Para ello, se analizan las diferencias existentes entre diversos tipos de texto, centrandonos fundamentalmente en el narrativo y el expositivo, analizando sus repercusiones sobre la cognicion humana. Se proponen diferencias en funcion del analisis de su naturaleza causal y de las inferencias que se generan cuando se comprenden

New algorithms assessing short summaries in expository texts using latent semantic analysis.

In this study, we compared four expert graders with latent semantic analysis (LSA) to assess short summaries of an expository text. As is well known, there are technical difficulties for LSA to establish a good semantic representation when analyzing short texts. In order to improve the reliability of LSA relative to human graders, we analyzed three new algorithms by two holistic methods used in previous research (León, Olmos, Escudero, Cañas, & Salmerón, 2006). The three new algorithms were (1) the semantic common network algorithm, an adaptation of an algorithm proposed by W. Kintsch (2001, 2002) with respect to LSA as a dynamic model of semantic representation; (2) a best-dimension reduction measure of the latent semantic space, selecting those dimensions that best contribute to improving the LSA assessment of summaries (Hu, Cai, Wiemer-Hastings, Graesser, & McNamara, 2007); and (3) the Euclidean distance measure, used by Rehder et al. (1998), which incorporates at the same time vector length and the cosine measures. A total of 192 Spanish middle-grade students and 6 experts took part in this study. They read an expository text and produced a short summary. Results showed significantly higher reliability of LSA as a computerized assessment tool for expository text when it used a best-dimension algorithm rather than a standard LSA algorithm. The semantic common network algorithm also showed promising results.

Transforming Selected Concepts Into Dimensions in Latent Semantic Analysis

This study presents a new approach for transforming the latent representation derived from a Latent Semantic Analysis (LSA) space into one where dimensions have nonlatent meanings. These meanings are based on lexical descriptors, which are selected by the LSA user. The authors present three analyses that provide examples of the utility of this methodology. The first analysis demonstrates how document terms can be projected into meaningful new dimensions. The second demonstrates how to use the modified space to perform multidimensional document labeling to obtain a high and substantive reliability between LSA experts. Finally, the internal validity of the method is assessed by comparing an original semantic space with a modified space. The results show high consistency between the two spaces, supporting the conclusion that the nonlatent coordinates generated using this methodology preserve the semantic relationships within the original LSA space.

Using latent semantic analysis to grade brief summaries: some proposals

In this paper, we present several proposals in order to improve the LSA tools to evaluate brief summaries (less than 50 words) of narrative and expository texts. First, we analyse the quality of six different methods assessing essays that have been widely employed before (Foltz et al., 2000). The second objective is to analyse how new algorithms inspired by some authors (Denhiere et al., 2007) that try to emulate human behaviour to improve the reliability of LSA with human graders when assessing short summaries, compared with standard LSA use in expository text. Finally, we present an assessment method to combine LSA as a semantic computational linguistic model with ROUGE-N as a lexical model, to show how combining different automatic evaluation systems (LSA and ROUGE) can improve the quality of assessments in different academic levels.

Understanding Causality in Science Discourse for Middle and High School Students. Summary Task as a Strategy for Improving Comprehension

Reading comprehension involves a reader developing a mental representation of a text through the establishment of causal relations based on the ideas and events in the text. This is especially relevant to scientific text comprehension. Causal relations are fundamental to the process of comprehension as they provide a framework or scaffolding to order information in a logical way that is consistent with the argument. The most common method of assessing comprehension is based on the reader answering a series of multiple choice questions. It is unusual for comprehension measures to use an open task such as a summary. However, summaries require the reader to use writing skills as well as those of comprehension, thus revealing wide individual differences among students. This gives rise to two questions: (a) up to what point is a summary a reflection of the causal structure of a text, and (b) what—if any—is the influence of the causal relations on the comprehension of more competent and less competent readers? In this chapter we analyze the causal structure of scientific texts, as opposed to that of narratives, and explore how high school students process and comprehend these causal relations. We also examine how students’ comprehension of causal relations can be evaluated by multiple choice tasks or open tasks such as summaries. Finally, we discuss some educational implications for improving comprehension in science.

Do Causal and Concessive Connectives Guide Emotional Expectancies in Comprehension? A Double-Task Paradigm Using Emotional Icons

Continuity and discontinuity are sometimes marked in discourse by means of connectives. This study tested for the first time whether causal and concessive connectives induce expectations of emotional continuity and discontinuity, respectively. Using a novel double-task paradigm, participants first listened to an antecedent clause with a causal or concessive connective (“Because/Although the pupil studied a lot…”), followed by an emotional icon (emoticon), which could match or mismatch the emotional valence of the antecedent. In Experiment 1 participants had to choose the best continuation for the antecedent clause (“he passed” vs. “he failed” the exam) and then identify the emoticon previously shown. In Experiment 2 they had to judge a perceptual feature of the emoticon before performing the consequent choice task. For causal connectives the congruence between the antecedent and the emoticon valence facilitates the consequent choice task (Experiment 1 and Experiment 2) and the emoticon recognition task (Experiment 1) but not the early perceptual judgment task (Experiment 2). This means that causal connectives promote emotional valence continuity at the stage of meaning integration processes. By contrast, concessive connectives do not induce emotional continuity expectancies. In addition, performance in causal positive antecedents and in concessive negative antecedents was more efficient than in the contrasting conditions, suggesting strong emotional biases for these connectives.

The representation of polysemy through vectors: some building blocks for constructing models and applications with LSA

The problem of the multiplicity of word meanings has preoccupied so many researches from the linguistics, psychology or computational linguistic. In this paper, we revised how LSA represents the polysemous words and we explain some bias related with the meaning generation and revised some constraint-satisfaction models which introduce into the equation some dynamic mechanisms. The idea of these models is to take the amalgamated word vector from LSA and embed it into its discourse and semantic context, and by means of a dynamic mechanism, the appropriate features of it is are selected. To illustrate our arguments, we present some networks, providing evidence that polysemous words have separated representations for each sense only in presence of the linguistic context that involved it. We also present an example of how these mechanisms also contribute to support the visual heuristic searches in the visual information retrieval interfaces (VIRIs).

Análisis del tamaño y especificidad de los corpus en la evaluación de resúmenes mediante el LSA: Un análisis comparativo entre LSA y jueces expertos

Resumen: El Analisis Semantico Latente (LSA) es una sofisticada herramienta com-putacional de analisis semantico capaz de obtener una representacion matematica del significado de las palabras o textos. LSA, entre otras aplicaciones, ha demostrado ser eficiente en la evaluacion de textos. Esta herramienta adquiere la representacion matematica de los textos analizando previamente un corpus linguistico compuesto por documentos digitalizados. El principal objetivo de este estudio fue analizar que propie-dades han de tener distintos corpus linguisticos (general, condensado, diversificado, y corpus de base) para que las evaluaciones de los resumenes efectuadas por el LSA se parezcan lo maximo posible a las realizadas por 4 jueces humanos. Dichos resumenes fueron elaborados por 390 estudiantes de Primaria, ESO y universitarios espanoles. Los resultados indicaron que el tamano de los corpus no tiene por que ser tan generales ni tan grandes como los que se utilizan en Boulder (compuesto por millones de textos y mas de un millon de palabras), ni tampoco demasiado especificos (menos de 300 textos y 5000 palabras) para que la evaluacion que se desee hacer de ellos resulte satisfacto-riamente eficiente. Palabras Clave: Analisis Semantico Latente (LSA), resumenes, evaluacion del discurso, corpus linguistico, estudiantes universitarios.Recibido:18-VI-2007Aceptado:9-V-2008