Astrid Schmidt-Nielsen

Speaker Verification by Human Listeners

Schmidt-Nielsen, Astrid, and Crystal, Thomas H., Speaker Verification by Human Listeners: Experiments Comparing Human and Machine Performance Using the NIST 1998 Speaker Evaluation Data, Digital Signal Processing10(2000), 249?266.The speaker verification performance of human listeners was compared to that of computer algorithms/systems. Listening protocols were developed to emulate as closely as possible the 1998 algorithm evaluation run by the U.S. National Institute of Standards and Technology (NIST), while taking into account human memory limitations. A subset of the target speakers and test samples from the same telephone conversation data was used. Ways of combining listener data to arrive at a group decision were explored, and the group mean worked well. The human results were very competitive with the best computer algorithms in the same handset condition. For same numbertesting, with 3-s samples, listener panels and the best algorithm had the same equal-error rate (EER) of 8%. Listeners were better than typical algorithms. For different numbertesting, EERs increased but humans had a 40% lower equal-error rate. Human performance in general seemed relatively robust to degradation.

A conversational test for comparing voice systems using working two-way communication links

A conversational test using live two-way communications provides a measure of the actual usability of voice systems, especially when voice quality is degraded. A conversational test developed at NRL was compared with two other communicability tests in a series of experiments using a variety of digital voice processors with data rates from 800 to 32 000 bits/s. All three tests ranked the voice processors very similarly, but they did not discriminate equally well among different processors. Other advantages and disadvantages of conversational test methods are discussed.

The effect of narrow-band digital processing and bit error rate on the intelligibility of ICAO spelling alphabet words

The diagnostic rhyme test (DRT) is widely used to evaluate digital voice systems. Would-be users often have no reference frame for interpreting DRT scores in terms of performance measures that they can understand, e.g., how many operational words are correctly understood. This research was aimed at providing a better understanding of the effects of very poor quality speech on human communication performance. It is especially important to determine how successful communications are likely to be when the speech quality is severely degraded. This paper compares the recognition of ICAO spelling alphabet words (ALFA, BRAVO, CHARLIE, etc.) to DRT scores for the same conditions. Confusions among the spelling alphabet words are also given. The voice conditions included unprocessed speech, speech processed through the DoD standard linear predictive coding algorithm operating at 2400 bits/s with random bit error rates of 0, 2, 5, 8, and 12 percent, and an 800 bit/s pattern matching algorithm. The results suggest that with distinctive vocabularies like the ICAO spelling alphabet, word intelligibility can be expected to remain very high even when DRT scores fall into the poor range; but once the DRT scores fall below about 75, the intelligibility can be expected to fall off rapidly; and at scores below 50, less than half the words will also be understood.

Comparing three numbers: The effect of number of digits, range, and leading zeros

The literature is abundant with results on the cognitive processes involved in determining the larger of two numbers. In the present experiment, range, number of digits, and leading zeros were varied to determine whether some of the major results for comparing two numbers generalize to judgments of the largest of three numbers. There were consistencies as well as inconsistencies between our results and previous two-number comparison data. For example, the distance effect(Moyer & Landauer, 1967) held for three-digit numbers but was not replicated for single-digit numbers. A two-stage process is suggested, with an encoding stage and a comparison stage. At the comparison stage, strategies may vary, depending on the nature of the comparison that is to be made.

Characterizing human ability to discriminate talkers over low data rate voice coders

Talker discrimination over low data rate voice coders (2400 bits/s) was evaluated in three separate experiments using a paired comparisons task. Test materials were Harvard sentences spoken by ten male and ten female talkers from the Boston area. Listeners were asked to decide whether two different sentences were spoken by the same person or by two different people. They then judged how dissimilar the two voices were using a five‐point scale. The ability to discriminate among the voices (measured by d′) was compared to the subjective judgments of the perceptual distance between the voices. The effect of different types of voice coders on talker discriminability and on the perceived talker space was compared with uncoded speech. Comparisons with traditional measures of intelligibility and voice quality suggest that higher intelligibility or quality scores are not necessarily related to better talker discrimination, but dissimilarity ratings of different talkers were more closely related to voice quality sc...

Dual-task performance using degraded speech in a sentence-verification task

The results of a preliminary study on the effects of reduced speech intelligibility on dual-task performance are reported. The speech task was a sentence-verification task, and speech degradation was accomplished using a narrowband digital voice-transmission system operating with and without random bit errors. The second task was a visual picture-sorting task. There was a dualtask decrement on the sorting task and, in addition, there was a further decrease in sorts per minute as the speech was increasingly degraded. Reaction time for the speech task increased with the concurrent sorting task, but the dual-task condition did not affect speech-task error rates.

Office of Naval Research funding strategies

The best strategies for obtaining military research funding can be quite different from NIH and NSF. Funding opportunities at the Office of Naval Research will be discussed. Different types of Navy/military funding will be described, and strategies for obtaining various types of funding, including the Young Investigator Program, will be discussed.

Tactical communications bring new challenges to automatic speech recognition

Automatic speaker‐independent recognition of conversational speech has been making significant progress in recent years. Military communications challenge the robustness of current recognition systems. Compared to telephone conversations or news broadcasts, tactical communication has a reduced vocabulary. It consists of short utterances, limited to the task at hand, with occasional chat words. What makes recognition difficult is high levels of background noise in tactical environments (helicopters, tanks) and the degradation of the signal by military microphones, including noise canceling microphones. Of perhaps more importance, people alter their speech to overcome these degradations and the loss of intelligibility from communicating over vocoders. The DARPA‐sponsored spine program is exploring these issues using speech from pairs of participants performing a collaborative task while communicating between separate sound booths. Each person sits in an accurately reproduced military background noise enviro...

Modulated noise reference unit (MNRU) tests using DRT, DAM, and spelling alphabet test materials

The modulated noise reference unit (MNRU) has been widely used for reference conditions in evaluating voice coders for telephone applications. Opinion scores using the mean opinion score (MOS) are referenced to equivalent modulated signal‐to‐noise level. The DoD Digital Voice Processor Consortium has conducted extensive tests of voice processors using the Diagnostic Rhyme Test (DRT) to evaluate intelligibility and the Diagnostic Acceptability Measure (DAM) to evaluate acceptability. Spoken materials for the DRT, the DAM, and the ICAO spelling alphabet words and digits were degraded by speech‐modulated noise at levels from + 30 to − 15 dB. The resulting tapes were evaluated by trained listeners. Even small amounts of noise degraded acceptability, DRT intelligibility degraded more slowly, and the spelling alphabet was moderately intelligible even at the worst speech to noise levels. Relations among the different types of tests will be discussed.

Listener recognition of unfamiliar speakers using conversational speech

Listeners attempted to recognize speakers from short excerpts of conversational speech. The listeners were given two repetitions of a 30‐s familiarization paragraph read by each of the five speakers in the recognition set and were told to associate each voice with the name given by the speaker at the beginning of the paragraph. The test consisted of 25 excerpts taken from tape recordings of each of the speakers playing a game of battleship, and the listeners were asked to identify each sample by writing down the name of the speaker. The average duration of the excerpts was slightly more than 2 s. Three sets of five speakers each were tested: one group of female speakers, one group of male speakers who had been rated high in voice distinctiveness (by a separate group of subjects), and one group of male speakers who had been rated low in voice distinctiveness. There were two sets of excerpts for each speaker group, one taken from recordings made while the speakers were conversing over a 2400‐bit/s LPC digit...