Masayuki Kameda

Automated Japanese Essay Scoring System based on Articles Written by Experts

We have developed an automated Japanese essay scoring system called Jess. The system needs expert writings rather than expert raters to build the evaluation model. By detecting statistical outliers of predetermined aimed essay features compared with many professional writings for each prompt, our system can evaluate essays. The following three features are examined: (1) rhetoric --- syntactic variety, or the use of various structures in the arrangement of phases, clauses, and sentences, (2) organization --- characteristics associated with the orderly presentation of ideas, such as rhetorical features and linguistic cues, and (3) content --- vocabulary related to the topic, such as relevant information and precise or specialized vocabulary. The final evaluation score is calculated by deducting from a perfect score assigned by a learning process using editorials and columns from the Mainichi Daily News newspaper. A diagnosis for the essay is also given.

Automated Japanese essay scoring system:jess

We have developed an automated Japanese essay scoring system named jess. The system evaluates an essay from three features: (1) rhetoric - ease of reading, diversity of vocabulary, percentage of big words (long, difficult words), and percentage of passive sentences; (2) organization - characteristics associated with the orderly presentation of ideas, such as rhetorical features and linguistic cues; (3) contents - vocabulary related to the topic, such as relevant information and precise or specialized vocabulary. The final evaluated score is calculated by deducting from a perfect score assigned by a learning process using editorials and columns from the Mainichi Daily News newspaper. A diagnosis for the essay is also given. Our system does not need any essays graded by human experts.

A portable & quick Japanese parser: QJP

QJP is a portable and quick software module for Japanese processing. QJP analyzes a Japanese sentence into segmented morphemes/words with tags and a syntactic bunsetsu kakari-uke structure based on the two strategies, a) Morphological analysis based on character-types and functional-words and b) Syntactic analysis by simple treatment of structural ambiguities and ignoring semantic information. QJP is small, fast and robust, because 1) dictionary size (less than 100KB) and required memory size (260KB) are very small, 2) analysis speed is fast (more than 100 words/sec on 80486-PC), and 3) even a 100-word long sentence containing unknown words is easily processed.Using QJP and its analysis results as a base and adding other functions for processing Japanese documents, a variety of applications can be developed on UNIX workstations or even on PCs.

Overwritable automated japanese short-answer scoring and support system

We have developed an automated Japanese short-answer scoring and support machine for new National Center written test exams. Our approach is based on the fact that accurate recognizing textual entailment and/or synonymy has been almost impossible for several years. The system generates automated scores on the basis of evaluation criteria or rubrics, and human raters revise them. The system determines semantic similarity between the model answers and the actual written answers as well as a certain degree of semantic identity and implication. Owing to the need for the scoring results to be classified at multiple levels, we use random forests to utilize many predictors effectively rather than use support vector machines. An experimental prototype operates as a web system on a Linux computer. We compared human scores with the automated scores for a case in which 3--6 allotment points were placed in 8 categories of a social studies test as a trial examination. The differences between the scores were within one point for 70--90 percent of the data when high semantic judgment was not needed.