Mohammad Rafayet Ali

LISSA — Live Interactive Social Skill Assistance

We present LISSA - Live Interactive Social Skill Assistance - a web-based system that helps people practice their conversational skills by having short conversations with a human like virtual agent and receiving real-time feedback on their nonverbal behavior. In this paper, we describe the development of an interface for these features and examine the viability of real time feedback using a Wizard of Oz prototype. We then evaluated our system using a speed-dating study design. We invited 47 undergraduate male students to interact with staff and randomly assigned them to intervention with LISSA or a self-help control group. Results suggested participants who practiced with the LISSA system were rated as significantly better in nodding when compared to a self-help control group and marginally better in eye contact and gesturing. The system usability and surveys showed that participants found the feedback provided by the system useful, unobtrusive, and easy to understand.

ROC comment: automated descriptive and subjective captioning of behavioral videos

We present an automated interface, ROC Comment, for generating natural language comments on behavioral videos. We focus on the domain of public speaking, which many people consider their greatest fear. We collect a dataset of 196 public speaking videos from 49 individuals and gather 12,173 comments, generated by more than 500 independent human judges. We then train a k-Nearest-Neighbor (k-NN) based model by extracting prosodic (e.g., volume) and facial (e.g., smiles) features. Given a new video, we extract features and select the closest comments using k-NN model. We further filter the comments by clustering them using DBScan, and eliminating the outliers. Evaluation of our system with 30 participants conclude that while the generated comments are helpful, there is room for improvement in further personalizing them. Our model has been deployed online, allowing individuals to upload their videos and receive open-ended and interpretative comments. Our system is available at http://tinyurl.com/roccomment.

The LISSA Virtual Human and ASD Teens: An Overview of Initial Experiments

We summarize an exploratory investigation into using an autonomous conversational agent for improving the communication skills of teenagers with autism. The system conducts a natural conversation with the user and gives real-time and post-session feedback on the user’s nonverbal behavior. We obtained promising results and ideas for improvements in preliminary experiments with five autism spectrum disorder teens.

Social skills training with virtual assistant and real-time feedback

Nonverbal cues are considered the most important part in social communication. Many people desire people; but due to the stigma and unavailability of resources, they are unable to practice their social skills. In this work, we envision a virtual assistant that can give individuals real-time feedback on their smiles, eye-contact, body language and volume modulation that is available anytime, anywhere using a computer browser. To instantiate our idea, we have set up a Wizard-of-Oz study in the context of speed-dating with 47 individuals. We collected videos of the participants having a conversation with a virtual agent before and after of a speed-dating session. This study revealed that the participants who used our system improved their gesture in a face-to-face conversation. Our next goal is to explore different machine learning techniques on the facial and prosodic features to automatically generate feedback on the nonverbal cues. In addition, we want to explore different strategies of conveying real-time feedback that is non-threatening, repeatable, objective and more likely to transfer to a real-world conversation.

Automated conversation skills assistant

Conversational skills training are getting popular now a days but often very hard to get due to expense and lack of accessibility. In this paper, we present the idea of an automated conversational skills training assistant, which provides both realtime and post summary feedback while having a conversation with a virtual agent. Our exploratory effort shows the applicability of this system and significant improvement of our participants conversational skills over a control group. We present our intended goals and the methodologies that we will use on our collected training data in order to automate this system in future.