Debra Haley

e-assessment using latent semantic analysis in the computer science domain: a pilot study

Latent Semantic Analysis (LSA) is a statistical Natural Language Processing (NLP) technique for inferring meaning from a text. Existing LSA-based applications focus on formative assessment in general domains. The suitability of LSA for summative assessment in the domain of computer science is not well known. The results from the pilot study reported in this paper encourage us to pursue further research in the use of LSA in the narrow, technical domain of computer science. This paper explains the theory behind LSA, describes some existing LSA applications, and presents some results using LSA for automatic marking of short essays for a graduate class in architectures of computing systems.

Seeing the whole picture: evaluating automated assessment systems

Abstract This paper argues that automated assessment systems can be useful for both students and educators provided that its results correspond well with human markers. Thus, evaluating such a system is crucial. We present an evaluation framework and show why it can be useful for both producers and consumers of automated assessment. The framework builds on previous work to analyse Latent Semantic Analysis- (LSA) based systems, a particular type of automated assessment, that produced a research taxonomy that could help developers publish their results in a format that is comprehensive, relatively compact, and useful to other researchers. The paper contends that, in order to see a complete picture of an automated assessment system, certain pieces must be emphasised. It presents the framework as a jigsaw puzzle whose pieces join together to form the whole picture and provides an example of the utility of the framework by presenting some empirical results from our assessment system that marks questions about html. Finally, the paper suggests that the framework is not limited to LSA-based systems. With slight modifications, it can be applied to any automated assessment system.

CONSPECT: monitoring conceptual development

This paper describes and evaluates CONSPECT, a service that analyses states in a learner’s conceptual development. CONSPECT combines two technologies – Latent Semantic Analysis (LSA) and Network Analysis (NA) into a technique called Meaningful Interaction Analysis (MIA). It uses LSA for the language analysis and NA to provide visualisations of the semantic relatedness information calculated by LSA. CONSPECT was designed to help both learners and tutors monitor conceptual development. This paper reports on the verification activities undertaken to show how well LSA matches humans in clustering similar concepts. The verification used the edit distance between card sorts to quantitatively evaluate the service.