Comparison of Rating Tendencies between Native and Non-native Speakers in Korean Speaking Test

Abstract

The purpose of this study was to compare the rating tendencies of Korean and Chinese raters in a Korean Speaking Test. For this purpose, graduate students majoring in Korean education were trained, and an individual rating process was performed. The results of the rating were then analyzed using the multifaceted Rasch model, focusing on rating consistency, severity, and bias. The results of the analysis indicate that rating severity differed among raters even in the same group, and rating consistency was either an overfit or a misfit in the Chinese rating (CR) group. The results also showed that the CR group tended to be more tolerant of assessment evaluation items with less difficulty than the Korean rating (KR) group, and to score assessment evaluation items with higher difficulty more strictly. In the analysis of criteria, the KR group scored more strictly than the CR group on the three criteria except vocabulary and grammar, and organization. In contrast, there was a statistically significant tendency toward the opposite for “organization”. The two groups differed in judging the proficiency of test takers, with differences between the test takers in cases where proficiency is relatively high and when specific languages are reflected in the test taker s pronunciation or intonation, some of which were analyzed as significant bias. Finally, the use of the evaluation scale showed that the CR group had a wider distribution, but the reliability of using the zero scale was low, and the difference between the mean scores of each scale was not uniform. This group also tended toward the middle point rather than toward the peak.