Joon Hao Chuah

Exploring agent physicality and social presence for medical team training

Mixed reality and 3D user interface technologies have increased the immersion, presence, and physicality of user interactions. These technologies can also increase the physicality of embodied conversational agents (ECAs) by making the ECAs occupy and interact with the physical space. We propose that increasing the physicality of an ECA can increase the ECAs social presence, that is, the feeling that the ECA is a real person. In this paper, we examine existing research and formalize the idea of ECA physicality. We also explored the relationship between physicality and social presence by conducting two user studies (n 18 and n 29). Both user studies took place in a medical team training context and involved virtual human ECAs as fellow team members. The first studys results suggested that increasing physicality increased social presence and elicited more realistic behavior. The second studys results suggested that individual dimensions of physicality affect social presence to different extents.

Optimal learning in a virtual patient simulation of cranial nerve palsies: The interaction between social learning context and student aptitude

Background: Simulation in medical education provides students with opportunities to practice interviews, examinations, and diagnosis formulation related to complex conditions without risks to patients. Aim: To examine differences between individual and team participation on learning outcomes and student perspectives through use of virtual patients (VPs) for teaching cranial nerve (CN) evaluation. Methods: Fifty-seven medical students were randomly assigned to complete simulation exercises either as individuals or as members of three-person teams. Students interviewed, examined, and diagnosed VPs with possible CN damage in the Neurological Exam Rehearsal Virtual Environment (NERVE). Knowledge of CN abnormalities was assessed pre- and post-simulation. Student perspectives of system usability were evaluated post-simulation. Results: An aptitude-treatment interaction (ATI) effect was detected; at pre-test scores ≤ 50%, students in teams scored higher (83%) at post-test than did students as individuals (62%, p = 0.02). Post-simulation, students in teams reported greater confidence in their ability to diagnose CN abnormalities than did students as individuals (p = 0.02; mean rating = 4.0/5.0 and 3.4/5.0, respectively). Conclusion: The ATI effect allows us to begin defining best practices for the integration of VP simulators into the medical curriculum. We are persuaded to implement future NERVE exercises with small teams of medical students.

Increasing agent physicality to raise social presence and elicit realistic behavior

The concepts of immersion and presence focus on the environment in a virtual environment. We instead focus on embodied conversational agents (ECAs). ECAs occupy the virtual environment as interactive partners. We propose that the ECA analogues of immersion and presence are physicality and social presence. We performed a study to determine the effect of an ECAs physicality on social presence and eliciting realistic behavior from the user. The results showed that increasing physicality can elicit realistic behavior and increase social presence but there was also an interaction effect with plausibility.

Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit

Introduction: During critical incidents, teamwork failures can compromise patient safety. This study provides evidence that virtual humans can be used in simulated critical incidents to assess the learning needs of health professionals, and provide important information that can inform the development of continuing education programs in patient safety. We explored the effectiveness of information transfer during a devolving medical situation between postanesthesia care unit (PACU) nurses and a virtual attending physician. Methods: We designed a three‐stage scenario: tutorial, patient transfer, and critical incident. We developed 2 checklists to assess information transfer: Critical Patient Information and Interprofessional Communication Skills. All participants were videotaped; 2 raters reviewed all videos and assessed performance using the checklists. Results: Participants (n = 43) who completed all 3 stages scored 62.3% correct on critical patient information transfer and 61.6% correct on interprofessional communication skills. Almost 87% missed a fatal drug error. The checklists measured each item on a 1/0 (done/not) calculation. Additionally, no relationship was found between years of nursing experience and performance on either checklist. Discussion: The PACU nurses in this study did not consistently share critical information with an attending (virtual) physician during a critical incident, and most missed a fatal dosage error. These findings strongly suggest a crucial need for additional structured team training among practicing health care teams, and they demonstrate the utility of using virtual humans to simulate team members.

Hybrid virtual-physical entities

Mixed reality (MR) combines virtual elements with the real, physical world. Virtual elements can be more dynamic - easily moving around or changing appearance. Physical elements are more static but improve immersion and presence. The combination of the two creates dynamic, immersive MR environments. However, individual entities in MR environments, e.g. an object or a character, are typically either purely physical or purely virtual. We propose using both virtual and physical elements in a hybrid virtual-physical entity. We implemented this idea with a hybrid virtual human and ran several user studies. These studies helped us develop guidelines for implementing hybrid entities.

Pain Assessment and Treatment Decisions for Virtual Human Patients

Laypeople and healthcare professionals use demographic cues when making pain management decisions. These decisions can negatively affect patient outcomes. This study examined whether laypeople base their pain management decisions in part on pain-related postures and demographic cues. Virtual human (VH) technology was used to research whether sex and race, as well as body posture, influenced pain management decisions. Ninety-seven laypersons examined VH patients exhibiting low back pain related body postures whose demographic cues varied by VH sex and VH race. T tests validated that participants were able to distinguish between high pain related body postures and low pain related body postures. The participants assessed male VH patients to be experiencing more pain than female VH patients. This study suggests that participants use sex as a cue when assessing pain. Participants may perceive VH male patients as experiencing high pain intensity if the participants are willing to counter male stereotypes and acknowledge that the male VH patients display pain behaviors.

Applying Mixed Reality to Simulate Vulnerable Populations for Practicing Clinical Communication Skills

Health sciences students often practice and are evaluated on interview and exam skills by working with standardized patients (people that role play having a disease or condition). However, standardized patients do not exist for certain vulnerable populations such as children and the intellectually disabled. As a result, students receive little to no exposure to vulnerable populations before becoming working professionals. To address this problem and thereby increase exposure to vulnerable populations, we propose using virtual humans to simulate members of vulnerable populations. We created a mixed reality pediatric patient that allowed students to practice pediatric developmental exams. Practicing several exams is necessary for students to understand how to properly interact with and correctly assess a variety of children. Practice also increases a students confidence in performing the exam. Effective practice requires students to treat the virtual child realistically. Treating the child realistically might be affected by how the student and virtual child physically interact, so we created two object interaction interfaces - a natural interface and a mouse-based interface. We tested the complete mixed reality exam and also compared the two object interaction interfaces in a within-subjects user study with 22 participants. Our results showed that the participants accepted the virtual child as a child and treated it realistically. Participants also preferred the natural interface, but the interface did not affect how realistically participants treated the virtual child.

Automated Generation of Emotive Virtual Humans

Emotive virtual humans (VHs) are important for affective interactions with embodied conversation agents [1]. However, emotive VHs require significant resources and time. As an example, the VHs in movies and video games require teams of animators and months of work. VHs can also be imbued with emotion using appraisal theory methods that use psychology based models to generate emotions by using the VHs goals and beliefs to evaluate external events. These external events require manual tagging or natural language understanding [2]. As an alternative approach, we propose tagging VH responses with emotions using textual affect sensing methods. The method developed by Neviarouskaya et al. [3] uses syntactic parses and a database of words and associated emotion intensities.We use this database, and because these emotions are associated with specific words, we can combine the emotions with audio timing information to generate lip-synched facial expressions. Our approach, AutoEmotion, allows us to automatically add basic emotions to VHs without the need for manual animation or tagging or natural language understanding.