Jin Hou

A real-time interactive nonverbal communication system through semantic feature extraction as an interlingua

There has been a growing interest in the use of networked virtual environment (NVE) technology to implement telepresence that allows participants to interact with each other in shared cyberspace. In addition, nonverbal language has attracted increased attention because of its association with more natural human communication, and especially sign languages play an important role for the hearing impaired. This paper proposes a novel real-time nonverbal communication system by introducing an artificial intelligence method into the NVE. We extract semantic information as an interlingua from the input text through natural language processing, and then transmit this semantic feature extraction (SFE) to the three-dimensional (3-D) articulated humanoid models prepared for each client in remote locations. Once the SFE is received, the virtual human is animated by the synthesized SFE. Experiments with Japanese and Chinese sign languages show this system makes the real-time animation of avatars available for the participants when chatting with each other. The communication is more natural since it is not just based on text or predefined gesture icons. This proposed system is suitable for sign language distance training as well.

A real-time interactive non-verbal communication system through semantic feature extraction

This paper proposes a novel real-time non-verbal communication system from natural language instruction by introducing an artificial intelligence method into the networked virtual environment (NVE). We extract semantic information as an interlingua from the input text by natural language processing, and then transmit this semantic feature extraction (SFE), which actually is a parameterized action representation, to the 3-D articulated humanoid models prepared in each client in remote locations. Once the SFE is received, the virtual human will be animated by the synthesized SFE. Experiments between Japanese sign language and Chinese sign language show this system makes the real-time animations of avatars available for the participants when chatting with each other, not just based on text or predefined gesture icons, so the communication is more natural. This proposed system is suitable for sign language distance training as well.