Andrea Horbach

Computer-Assisted Scoring of Short Responses: The Efficiency of a Clustering-Based Approach in a Real-Life Task

We present an extrinsic evaluation of a clustering-based approach to computer-assisted scoring of short constructed response items, as encountered in educational assessment. Due to their open-ended nature, constructed response items need to be graded by human readers, which makes the overall testing process costly and time-consuming. In this paper we investigate the prospects for streamlining the grading task by grouping similar responses for scoring. The efficiency of scoring clustered responses is compared both with the traditional mode of grading individual test-takers’ sheets and with by-item scoring of non-clustered responses. Evaluation of the three grading modes is carried out during real-life language proficiency tests of German as a Foreign Language. We show that a system based on basic clustering techniques and shallow features yields a promising trend of reducing grading time and performs as well as a system displaying test-taker sheets for scoring.

Investigating Active Learning for Short-Answer Scoring

Active learning has been shown to be effective for reducing human labeling effort in supervised learning tasks, and in this work we explore its suitability for automatic short answer assessment on the ASAP corpus. We systematically investigate a wide range of AL settings, varying not only the item selection method but also size and selection of seed set items and batch size. Comparing to a random baseline and a recently-proposed diversitybased baseline which uses cluster centroids as training data, we find that uncertainty-based sampling methods can be beneficial, especially for data sets with particular properties. The performance of AL, however, varies considerably across individual prompts.

UdS-(retrain|distributional|surface): Improving POS Tagging for OOV Words in German CMC and Web Data

We present in this paper our three system submissions for the POS tagging subtask of the Empirist Shared Task: Our baseline systemUdS-retrain extends a standard training dataset with in-domain training data; UdSdistributional and UdS-surface add two different ways of handling OOV words on top of the baseline system by using either distributional information or a combination of surface similarity and language model information. We reach the best performance using the distributional model.