Sougo Tsuboi

Automatically generated captions: will they help non-native speakers communicate in english?

Many people find it difficult to communicate in a foreign language. In order to help these people, one approach being studied is the use of captions generated by automatic speech recognition (ASR). Captions are known to facilitate comprehension of foreign languages, but ASR-generated captions may be subject to problems attributable to recognition errors and recognition time. We conducted two experiments using subjects who are native Japanese speakers to determine how these differences caused by ASR affect understanding when listening to English. We found that captions with 80% accuracy will increase the understanding of the subjects with intermediate English skills, which would apply to about half of native Japanese users. Additionally, changing the display timing of the caption from after speech to before speech would contribute to improving the understanding more than increasing accuracy from 80% to 100%. These findings suggest that captions generated with todays ASR can help non-native speakers communicate in English when used carefully