Tsunenori Ishioka

Automated Japanese Essay Scoring System based on Articles Written by Experts

We have developed an automated Japanese essay scoring system called Jess. The system needs expert writings rather than expert raters to build the evaluation model. By detecting statistical outliers of predetermined aimed essay features compared with many professional writings for each prompt, our system can evaluate essays. The following three features are examined: (1) rhetoric --- syntactic variety, or the use of various structures in the arrangement of phases, clauses, and sentences, (2) organization --- characteristics associated with the orderly presentation of ideas, such as rhetorical features and linguistic cues, and (3) content --- vocabulary related to the topic, such as relevant information and precise or specialized vocabulary. The final evaluation score is calculated by deducting from a perfect score assigned by a learning process using editorials and columns from the Mainichi Daily News newspaper. A diagnosis for the essay is also given.

Automated Japanese essay scoring system:jess

We have developed an automated Japanese essay scoring system named jess. The system evaluates an essay from three features: (1) rhetoric - ease of reading, diversity of vocabulary, percentage of big words (long, difficult words), and percentage of passive sentences; (2) organization - characteristics associated with the orderly presentation of ideas, such as rhetorical features and linguistic cues; (3) contents - vocabulary related to the topic, such as relevant information and precise or specialized vocabulary. The final evaluated score is calculated by deducting from a perfect score assigned by a learning process using editorials and columns from the Mainichi Daily News newspaper. A diagnosis for the essay is also given. Our system does not need any essays graded by human experts.

Evaluation of criteria for information retrieval

We investigate van Rijsbergens F-measure, break-even point, and 11-point averaged precision, all of which can be translated into 1-dimensional scalar quantity from the precision and the recall. These investigations can be done by comparing to tetrachoric (four-fold) correlation coefficient and phi (four-fold point) coefficient, which are often used as the index of statistical association in a 2/spl times/2 contingency table. The results show that when a fallout rate is less than 0.1, (1) the F/sub 1/ measure has similar properties of the phi coefficient, (2) the break-even point is almost equivalent to a phi coefficient, and (3) the 11-point averaged precision should be a measure, which is larger than a phi coefficient and has a value smaller than a tetrachoric correlation coefficient.

Overwritable automated japanese short-answer scoring and support system

We have developed an automated Japanese short-answer scoring and support machine for new National Center written test exams. Our approach is based on the fact that accurate recognizing textual entailment and/or synonymy has been almost impossible for several years. The system generates automated scores on the basis of evaluation criteria or rubrics, and human raters revise them. The system determines semantic similarity between the model answers and the actual written answers as well as a certain degree of semantic identity and implication. Owing to the need for the scoring results to be classified at multiple levels, we use random forests to utilize many predictors effectively rather than use support vector machines. An experimental prototype operates as a web system on a Linux computer. We compared human scores with the automated scores for a case in which 3--6 allotment points were placed in 8 categories of a social studies test as a trial examination. The differences between the scores were within one point for 70--90 percent of the data when high semantic judgment was not needed.

Investigations into Missing Values Imputation Using Random Forests for Semi-supervised Data

This paper presents a revised procedure that imputes missing values by using random forests on semi-supervised data. The method has a feature that not only allows missing data to be found in a response variable but in a predictive variable, and furthermore, it can now deal with any types of data, i.e., numerical values, categories and categories with an order. By evaluating this method using Titanic data and eleven UC Irvine repository datasets, we found that our method performed fairly well, and a method of naive median imputation was also suitable in these cases.

Text Segmentation by Latent Semantic Indexing

We point out that the determination of the document boundary is possible by using Singular Value Decomposition (SVD) based on the idea of LSI (Latent Semantic Indexing), and show that the conditional entropy method is replaceable by the SVD method using illustrations from several well-known plays. If we use this entropy model, the homoscedasticity test can be available to detect the document boundary, because the entropy depends on the variance of k-variables of LSI.