David O. Johnson

Evaluation of a Small Socially-Assistive Humanoid Robot in Intelligent Homes for the Care of the Elderly

The ageing population phenomenon is pushing the design of innovative solutions to provide assistance to the elderly. In this context a socially–assistive robot can act as a proactive interface in a smart-home environment, providing multimodal communication channels and generating positive feelings in users. The present paper reports results of a short term and a long term evaluation of a small socially assistive humanoid robot in a smart home environment. Eight elderly people tested an integrated smart–home robot system in five real–world scenarios. Six of the participants experienced the system in two sessions over a two week period; the other two participants had a prolonged experience of eight sessions over a three month period. Results showed that the small humanoid robot was trusted by the participants. A cross–cultural comparison showed that results were not due to the cultural background of the participants. The long term evaluation showed that the participants might engage in an emotional relationship with the robot, but that perceived enjoyment might decrease over time.

Human robot interaction through semantic integration of multiple modalities, dialog management, and contexts

Our research goal is to prove the hypothesis that using multiple modalities, dialog management, context, and semantics improves HRI over using a single modality by testing the hypothesis using a single implementation and application. We tested the hypothesis by simulating a domestic robot that can be taught to clean a house using a multi-modal interface. The multi-modal interface included speech, pointing, field of vision, and head nodding. The contexts included real world context and dialog history. We tested three different dialog-grounding strategies: optimistic, cautious, and pessimistic. We measured the learning process in two ways: dialog succinctness and learning accuracy. Experimental evidence shows that: multiple modalities and contexts improve HRI over speech by itself for various ratios of accuracy to succinctness; the dialog grounding strategy that improves HRI the best depends on the desired ratio of accuracy to succinctness, with optimistic grounding being the best for ratios ≤0.8, and pessimistic grounding being the best otherwise; and the Pareto principle applies to which grounding strategy is the best.

Exploring the Entertainment Value of Playing Games with a Humanoid Robot

Besides providing functional support, socially assistive robots can provide social support in the form of entertainment. Previous studies have shown that this type of social support improves elderly people’s well-being significantly. But it is still far from clear what underlying causes drive people’s judgment of entertainment value. Showing emotionally expressive behaviors seems to raise entertainment value, but what if these behaviors are not truly embodied, i.e., tied to processes in the environment? It would seem that it is important that expressive behaviors are tied to the proper events. In this study, we investigated whether multimodal behavioral patterns (i.e., combinations of gestures, eye LED patterns, and verbal expressions) based on the developments within a game aids the entertainment value. We chose a gaming situation where the robot plays Mastermind with a human, in which the robot could show no behavioral patterns, behavioral patterns tied to game progress, or randomly selected behavioral patterns from the same set. The behavioral patterns were designed to imitate four basic emotions (neutral, happy, angry, sad) in combination with five levels of surprise and five levels of confidence. In a pilot study we validated the correct interpretation of the behavioral patterns. The experimental setup was designed to collect information about which behaviors to choose. The results of our study confirmed that a robot playing games with people has entertainment value. The main technical contribution is the information we collected on the behaviors.

Improved automatic English proficiency rating of unconstrained speech with multiple corpora

The performance of machine learning classifiers in automatically scoring the English proficiency of unconstrained speech has been explored. Suprasegmental measures were computed by software, which identifies the basic elements of Brazil’s model in human discourse. This paper explores machine learning training with multiple corpora to improve two of those algorithms: prominent syllable detection and tone choice classification. The results show that machine learning training with the Boston University Radio News Corpus can improve automatic English proficiency scoring of unconstrained speech from a Pearson’s correlation of 0.677–0.718. This correlation is higher than any other existing computer programs for automatically scoring the proficiency of unconstrained speech and is approaching that of human raters in terms of inter-rater reliability.

Comparison of algorithms to divide noisy phone sequences into syllables for automatic unconstrained English speaking proficiency scoring

Four algorithms for syllabifying phones are compared in automatically scoring English oral proficiency. The first algorithm clusters consonants into groups with the vowel nearer to them temporally, taking into account the maximal onset principle. A Hidden Markov Model (HMM) predicts the syllable boundaries based on their sonority value in the second algorithm. The third one employs three HMMs which are tuned to specific categories of utterances. The final algorithm uses a genetic algorithm to identify a set of rules for syllabifying the phones. They were evaluated by: (1) how well they syllabified utterances from the Boston University Radio News Corpus (BURNC) and (2) how well they worked as part of a process to automatically score English speaking proficiency. A measure of the temporal alignment of the syllables was utilized to judge how satisfactorily they syllabified utterances. Their suitability in the proficiency process was assessed with the Pearson correlation between the computer’s predicted proficiency scores and the scores determined by human examiners. We found that syllabification-by-genetic-algorithm performed the best in syllabifying the BURNC, but that syllabification-by-grouping (i.e., syllables are made by grouping non-syllabic consonant phones with the vowel or syllabic consonant phone nearest to them with respect to time) performed the best in the English oral proficiency rating application.

Learning Macro Actions from Instructional Videos Through Integration of Multiple Modalities

We propose an architecture for a system that will “watch and listen to” an instructional video of a human performing a task and translate the audio and video information into a task for a robot to perform. This enables the use of readily available instructional videos from the Internet to train robots to perform tasks instead of programming them. We implemented an operational prototype based on the architecture and showed it could “watch and listen to” two instructional videos on how to clean golf clubs and translate the audio and video information from the instructional video into tasks for a robot to perform. The key contributions of this architecture are: integration of multiple modalities using trees and pruning with filters; task decomposition into macro-tasks composed of parameterized task-primitives and other macro-tasks, where the task-primitive parameters are an action (e.g., dip, clean, dry) taken on an object (e.g., golf club) using a tool (e.g., pail of water, brush, towel); and context, for determining missing and implied task-primitive parameter values, as a set of canonical task-primitive parameter values with a confidence score based on the number of times the parameter value was detected in the video and audio information and how long ago it was detected.

Investigating the Effect of a Humanoid Robot’s Head Position on Imitating Human Emotions

Humans show their emotions with facial expressions. In this paper, we investigate the effect of a humanoid robot’s head position on imitating human emotions. In an Internet survey through animation, we asked participants to adjust the head position of a robot to express six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. We found that humans expect a robot to look straight down when it is angry or sad, to look straight up when it is surprised or happy, and to look down and to its right when it is afraid. We also found that when a robot is disgusted some humans expect it to look straight to its right and some expect it to look down and to its left. We found that humans expect the robot to use an averted head position for all six emotions. In contrast, other studies have shown approach-oriented (anger and joy) emotions being attributed to direct gaze and avoidance-oriented emotions (fear and sadness) being attributed to averted gaze.