Cristina Delogu

A comparison between DTMF and ASR IVR services through objective and subjective evaluation

We evaluated some features of current automatic speech recognition (ASR) technology available in the marketplace for computer telephony applications, i.e., speaker independent vocal access through numbers and words, as well as some design techniques such as prompt and recovery. Three different ASR prototypes of the same application have been developed. Furthermore, a DTMF prototype that allows users to enter commands only through the telephone keyboard have been developed, with the purpose of making a comparison between touch tone and speech recognition technologies for telephony services.

New directions in the evaluation of voice input/output systems

A review of new directions in the evaluation of voice input/output systems is presented. Furthermore, recent studies on human factor aspects involved in speech communication are examined, with the idea that human factors research can improve current voice input/output devices and help to experimentally determine implementation guidelines for developing voice interface design. >

Spectral analysis of synthetic speech and natural speech with noise over the telephone line

In order to explain the different performance obtained with natural and synthetic speech at different linguistic levels over the telephone line, we analyzed the data collected in an experiment where 108 randomized stimuli were presented to 96 subjects. Subjects were required to identify the consonant in 51 CV and 57 VCV meaningful or meaningless words. There were 20 different listening conditions: 6 TTS systems (3 formant-based (SF) and 3 diphone-based (SD)), a pure natural voice (NV) and 3 signal-to-noise (S/N) ratios (6, 0, and -6 dB) for a total of 10 systems, presented both in good and in telephone conditions. The comparison between consonant confusions for natural and synthetic speech with comparable overall levels of intelligibility performance showed that the distributions of the consonant confusions for natural and synthetic speech were often quite different in each condition. Some analyses of different spectrograms suggests that such confusions are due to some problems in the phonetic rules and to the telephone line.

Comparative evaluation of spoken corpora acquired by presentation of visual scenarios and textual descriptions

The paper describes a system, in JAVA for written and visual scenario generation used to collect speech corpora in the framework of a tourism information system. Experimental evidence shows that the corpus generated with visual scenarios has a higher perplexity and a richer vocabulary than the corpus generated using the same conceptual derivations to produce textual scenarios. Furthermore, there is evidence that textual scenarios influence speakers in the choice of the lexicon used to express the concepts more than visual scenarios.

A field evaluation of the Italian “Automated reverse directory assistance” service

The paper describes a field evaluation of the automated ‘reverse directory assistance’ service presently in use in Italy in which information about names and addresses is provided by a TTS system. A simulation of the service using a natural voice was also run to get comparative data. Both services were accessed from an office room and a call-box on the street. Different evaluation metrics, such as intelligibility, task completion, task correctness, transaction success, and users reactions were used. The aim of the work was to evaluate TTS synthesis in real world use and to make a comparison between laboratory data and data on system performance in a real application. Such a comparison suggested that in laboratory tests more attention should be dedicated to simulate more closely the conditions that can be predicted in real world use, by including important aspects that are generally not taken into consideration in laboratory tests and that are likely to have a large influence on TTS system performance such as environmental noise, prosody, and task complexity. The results also underline the importance of field evaluations to get an overall view of the usability of a service in real applications and with users who are as similar as possible to actual users.

Towards a More Realistic Evaluation of Synthetic Speech: A Cognitive Perspective

In the past years we developed robust measures of different speech attributes [1], both at the segmental and suprasegmental level [2]. The ability of assessing synthetic speech with some degrees of confidence gives us the basis for more sophisticated studies of the way human beings interact with an artificially generated voice. For instance the length of the message, the degree of comprehension, the level of training of the listener, and the complexity of the task are examples of parameters affecting man-machine interaction [3].

Ecological tests: the impact of cognitive effort in quality assessment

The paper presents the results of a preliminary experiment in which we investigated some of the cognitive factors involved in viewing a video clip such as comprehension and cognitive load. The aim of our work is to have a measure which can help establish what visual quality is required by users to successfully assimilate and understand the informational content of a video clip with a minimum of user cost.