Heather Pon-Barry

Finding Eyewitness Tweets During Crises

Disaster response agencies incorporate social media as a source of fast-breaking information to understand the needs of people affected by the many crises that occur around the world. These agencies look for tweets from within the region affected by the crisis to get the latest updates on the status of the affected region. However only 1% of all tweets are “geotagged” with explicit location information. In this work we seek to identify non-geotagged tweets that originate from within the crisis region. Towards this, we address three questions: (1) is there a difference between the language of tweets originating within a crisis region, (2) what linguistic patterns differentiate within-region and outside-region tweets, and (3) can we automatically identify those originating within the crisis region in real-time?

Acoustic-Prosodic Entrainment and Rapport in Collaborative Learning Dialogues

In spoken dialogue analysis, the speech signal is a rich source of information. We explore in this paper how low level features of the speech signal, such as pitch, loudness, and speaking rate, can inform a model of student interaction in collaborative learning dialogues. For instance, can we observe the way that two peoples manners of speaking change over time to model something like rapport? By detecting interaction qualities such as rapport, we can better support collaborative interactions, which have been shown to be highly conducive to learning. For this, we focus on one particular phenomenon of spoken conversation, known as acoustic-prosodic entrainment, where dialogue partners become more similar to each other in their pitch, loudness, or speaking rate during the course of a conversation. We examine whether acoustic-prosodic entrainment is present in a novel corpus of collaborative learning dialogues, how people appear to entrain, to what degree, and report on the acoustic-prosodic features which people entrain on the most. We then investigate whether entrainment can facilitate detection of rapport, a social quality of the interaction. We find that entrainment does correlate to rapport; speakers appear to entrain primarily by matching their prosody on a turn-by-turn basis, and pitch is the most significant acoustic-prosodic feature people entrain on when rapport is present.

Disordered speech disrupts conversational entrainment: a study of acoustic-prosodic entrainment and communicative success in populations with communication challenges

Conversational entrainment, a pervasive communication phenomenon in which dialogue partners adapt their behaviors to align more closely with one another, is considered essential for successful spoken interaction. While well-established in other disciplines, this phenomenon has received limited attention in the field of speech pathology and the study of communication breakdowns in clinical populations. The current study examined acoustic-prosodic entrainment, as well as a measure of communicative success, in three distinctly different dialogue groups: (i) healthy native vs. healthy native speakers (Control), (ii) healthy native vs. foreign-accented speakers (Accented), and (iii) healthy native vs. dysarthric speakers (Disordered). Dialogue group comparisons revealed significant differences in how the groups entrain on particular acoustic–prosodic features, including pitch, intensity, and jitter. Most notably, the Disordered dialogues were characterized by significantly less acoustic-prosodic entrainment than the Control dialogues. Further, a positive relationship between entrainment indices and communicative success was identified. These results suggest that the study of conversational entrainment in speech pathology will have essential implications for both scientific theory and clinical application in this domain.

Effects of Voice-Adaptation and Social Dialogue on Perceptions of a Robotic Learning Companion

With a growing number of applications involving social human-robot interactions, there is an increasingly important role for socially responsive speech interfaces that can effectively engage the user. For example, learning companions provide both task-related feedback and motivational support for students with the goal of improving learning. As a learning companions ability to be socially responsive increases, so do learning outcomes. This paper presents a socially responsive speech interface for an embodied, robotic learning companion. We explore two methods of social responsiveness. The first method introduces social responses into the dialogue, while the second method augments these responses with voice-adaptation based on acoustic-prosodic entrainment. We evaluate the effect of a social, voice-adaptive robotic learning companion on social variables such as social presence and rapport, and we compare this to a companion with only social dialogue and one with neither social dialogue nor voice-adaptions. We contrast the effects against those of individual factors, such as gender. We find (1) that social presence is significantly higher with a social voice-adaptive speech interface than with purely social dialogue, and (2) that females feel significantly more rapport and are significantly more persistent in interactions with a robotic learning companion than males.

Naturalness and rapport in a pitch adaptive learning companion

Observed frequently in human-human interactions, entrainment is a social phenomenon in which speakers become more like each other over the course of a conversation. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic speech features, such as pitch and intensity. Correlated with communicative success, naturalness, and conversational flow as well as social variables such as rapport, a dialogue system which automatically entrains has the potential to improve verbal interactions by increasing rapport, naturalness, and conversational flow. In an application like the learning companion, such a socially responsive dialogue system may improve learning and motivation. However, it is not clear how to produce entrainment in an automatic dialogue system in ways that produce the effects seen in human-human dialogue. In this paper, we take the first steps towards implementing a spoken dialogue system which can entrain. We propose three methods of pitch adaptation based on analysis of human entrainment, and design and implement a system which can manipulate the pitch of text-to-speech output adaptively. We find a clear relationship between perceptions of rapport and different forms of pitch adaptations. Certain adaptations are perceived as significantly more natural and rapport-like. Ultimately, adapting by shifting the pitch contour of the text-to-speech output by the mean pitch of the user results in the highest reported measures of rapport and naturalness.

Scaling Introductory Courses Using Undergraduate Teaching Assistants

Undergraduates are widely used in support of Computer Science (CS) departments teaching missions as teaching assistants, peer mentors, section leaders, course assistants, and tutors. Those undergraduates engaged in teaching have the opportunity to deeply engage with CS concepts and develop key communication and social competencies. As enrollments surge, undergraduate teaching assistants (UTAs) play a larger role in student experience and outcomes. While faculty and graduate student instructional support does not necessarily increase with the number of students in our courses, the number of qualified undergraduate teaching assistants for introductory CS courses should scale with the number of students in our courses. With large courses, the significance of the UTAs role in students learning likely also increases. Students have relatively little interaction with the instructor, and faculty may have more challenges monitoring and supporting individual UTAs. UTAs have a major role in affecting climate in computer science courses. The climate in large courses has substantial implications for students from groups traditionally underrepresented in computing. This panel will discuss how undergraduate teaching assistants can serve as a scalable effective teaching resource that benefits both the students in the course and the UTAs themselves.

Expanding capacity and promoting inclusion in introductory computer science: a focus on near-peer mentor preparation and code review

Abstract A dilemma within computer science departments is developing sustainable ways to expand capacity within introductory computer science courses while remaining committed to inclusive practices. Training near-peer mentors for peer code review is one solution. This paper describes the preparation of near-peer mentors for their role, with a focus on regular, consistent feedback via peer code review and inclusive pedagogy. Introductory computer science students provided consistently high ratings of the peer mentors’ knowledge, approachability, and flexibility, and credited peer mentor meetings for their strengthened self-efficacy and understanding. Peer mentors noted the value of videotaped simulations with reflection, discussions of inclusion, and the cohort’s weekly practicum for improving practice. Adaptations of peer mentoring for different types of institutions are discussed. Computer science educators, with hopes of improving the recruitment and retention of underrepresented groups, can benefit from expanding their peer support infrastructure and improving the quality of peer mentor preparation.

A comparison of acoustic-prosodic entrainment in face-to-face and remote collaborative learning dialogues

Today, people are just as likely to have a business meeting remotely as they are face-to-face. Individuals obtain college degrees remotely and sick patients can visit the doctor from home. Especially important in light of this popularity, remote settings are posing communication challenges that are not present in face-to-face settings. Visual cues such as facial expressions and body language are either degraded or nonexistent. In this paper, we are interested in how remote settings affect spoken dialogue when compared to face-to-face settings. We focus on entrainment, a phenomenon of conversation where individuals adapt to each other during the interaction. Specifically, we investigate acoustic-prosodic entrainment, where individuals become more similar in their pitch, loudness, or speaking rate. We explore three different measures of acoustic-prosodic entrainment, comparing remote settings to face-to-face settings on a turn-by-turn basis. Our results indicate that the two settings do differ for different forms of entrainment, suggesting that the presence or absence of visual cues such as facial expressions and body language has an impact on the degree of entrainment.

Discourse Analysis of User Forums in an Online Weight Loss Application

Online social communities are becoming increasingly popular platforms for people to share information, seek emotional support, and maintain accountability for losing weight. Studying the language and discourse in these communities can offer insights on how users benefit from using these applications. This paper presents a preliminary analysis of language and discourse patterns in forum posts by users who lose weight and keep it off versus users with fluctuating weight dynamics. Our results reveal differences about how the types of posts, polarity of sentiments, and semantic cohesion of posts made by users vary along with their weight loss pattern. To our knowledge, this is the first discourse-level analysis of language and weight loss dynamics.

Dyadic Stance in Natural Language Communication with a Teachable Robot

Learning companion robots can provide personalized learning interactions to engage students in many domains including STEM. For successful interactions, students must feel comfortable and engaged. We describe an experiment with a learning companion robot acting as a teachable robot; based on human-to-human peer tutoring, students teach the robot how to solve math problems. We compare student attitudes of comfort, attention, engagement, motivation, and physical proximity for two dyadic stance formations: a face-to-face stance and a side-by-side stance. In human-robot interaction experiments, it is common for dyads to assume a face-to-face stance, while in human-to-human peer tutoring, it is common for dyads to sit in side-by-side as well as face-to-face formations. We find that students in the face-to-face stance report stronger feelings of comfort and attention, compared to students in the side-by-side stance. We find no difference between stances for feelings of engagement, motivation, and physical proximity.