Derrick Higgins

Automatic scoring of non-native spontaneous speech in tests of spoken English

This paper presents the first version of the SpeechRater^S^M system for automatically scoring non-native spontaneous high-entropy speech in the context of an online practice test for prospective takers of the Test of English as a Foreign Language^(R) internet-based test (TOEFL^(R) iBT). The system consists of a speech recognizer trained on non-native English speech data, a feature computation module, using speech recognizer output to compute a set of mostly fluency based features, and a multiple regression scoring model which predicts a speaking proficiency score for every test item response, using a subset of the features generated by the previous component. Experiments with classification and regression trees (CART) complement those performed with multiple regression. We evaluate the system both on TOEFL Practice data [TOEFL Practice Online (TPO)] as well as on Field Study data collected before the introduction of the TOEFL iBT. Features are selected by test development experts based on both their empirical correlations with human scores as well as on their coverage of the concept of communicative competence. We conclude that while the correlation between machine scores and human scores on TPO (of 0.57) still differs by 0.17 from the inter-human correlation (of 0.74) on complete sets of six items (Pearson r correlation coefficients), the correlation of 0.57 is still high enough to warrant the deployment of the system in a low-stakes practice environment, given its coverage of several important aspects of communicative competence such as fluency, vocabulary diversity, grammar, and pronunciation. Another reason why the deployment of the system in a low-stakes practice environment is warranted is that this system is an initial version of a long-term research and development program where features related to vocabulary, grammar, and content will be added in a later stage when automatic speech recognition performance improves, which can then be easily achieved without a re-design of the system. Exact agreement on single TPO items between our system and human scores was 57.8%, essentially at par with inter-human agreement of 57.2%. Our system has been in operational use to score TOEFL Practice Online Speaking tests since the Fall of 2006 and has since scored tens of thousands of tests.

A Comparison of Two Scoring Methods for an Automated Speech Scoring System.

This paper compares two alternative scoring methods – multiple regression and classification trees – for an automated speech scoring system used in a practice environment. The two methods were evaluated on two criteria: construct representation and empirical performance in predicting human scores. The empirical performance of the two scoring models is reported in Zechner, Higgins, Xi, & Williamson (2009), which discusses the development of the entire automated speech scoring system; the current paper shifts the focus to the comparison of the two scoring methods, elaborating both technical and substantive considerations and providing a reasoned argument for the trade-off between them. We concluded that a multiple regression model with expert weights was superior to the classification tree model. In addition to comparing the relative performance of the two models, we also evaluated the adequacy of the regression model for the intended use. In particular, the construct representation of the model was sufficiently broad to justify its use in a low-stakes application. The correlation of the model-predicted total test scores with human scores (r = 0.7) was also deemed acceptable for practice purposes.

Method and system for assessing pronunciation difficulties of non-native speakers by entropy calculation

The present disclosure presents a useful metric for assessing the relative difficulty which non-native speakers face in pronouncing a given utterance and a method and systems for using such a metric in the evaluation and assessment of the utterances of non-native speakers. In an embodiment, the metric may be based on both known sources of difficulty for language learners and a corpus-based measure of cross-language sound differences. The method may be applied to speakers who primarily speak a first language speaking utterances in any non-native second language.

Using Singular-value Decomposition on Local Word Contexts to Derive a Measure of Constructional Similarity

This paper presents a novel method of generating word similarity scores, using a term by n-gram context matrix which is compressed using Singular Value Decomposition, a statistical data analysis method that extracts the most significant components of variation from a large data matrix, and which has previously been used in methods like Latent Semantic Analysis to identify latent semantic variables in text. We present the results of applying these scores to standard synonym benchmark tests, and argue on the basis of these results that our similarity metric represents an aspect of word usage which is largely orthogonal to that addressed by other methods, such as Latent Semantic Analysis. In particular, it appears that this method captures similarity with respect to the participation of words in grammatical constructions, at a level of generalization corresponding to broad syntacticosemantic classes such as body part terms, kin terms and the like. Aside from assessing word similarity, this method has promising applications in language modeling and automatic lexical acquisition.

Reliability of human annotation of semantic roles in noisy text

This paper addresses the question of how to obtain consistent semantic annotation on the basis of a set of noisy texts. Many potential real-world applications of semantic computing are faced with the need to handle texts which are not well-edited, and for which a resource-intensive treebanking effort is not feasible. Student-produced short answers contain many grammatical and lexical errors, making consistent annotation a challenge. Nevertheless, this paper demonstrates that semantic role annotation can be done in a consistent and useful manner even under these constraints.Semantic concepts cement the ability to correlate visual information to higher-level semantic concepts. Traditional image search leverages text associated with images, a low-level content-based matching, or a combination of the two. We propose a new system that uses 374 semantic concepts (derived from the LSCOM lexicon (L. Kennedy, 2006)) to semantically facilitate fast exploration of a large set of video data. This new system, when coupled with traditional image search techniques produces a very intuitive and fruitful design for targeted user interaction.

An Unsupervised System for Visual Exploration of Twitter Conversations.

Social media provides a wealth of information regarding users’ perspectives on issues, public figures and brands, but it can be a timeconsuming and labor-intensive process to develop data pipelines in which those perspectives are encoded, and to build visualizations that illuminate important developments. This paper describes a system for quickly developing a model of the conversation around an issue on Twitter, and a flexible visualization system that allows analysts to interactively explore key facets of the analysis.