Alfie: An Interactive Robot with a Moral Compass
Cigdem Turan, Patrick Schramowski, Constantin Rothkopf, Kristian Kersting
AAlfie: An Interactive Robot with a Moral Compass
Cigdem Turan ∗ [email protected] Darmstadt, Dept. of Computer ScienceDarmstadt, Germany Patrick Schramowski ∗ [email protected] Darmstadt, Dept. of Computer ScienceDarmstadt, Germany Constantin Rothkopf [email protected] Darmstadt, Institute of Psychologyand Centre for Cognitive ScienceDarmstadt, Germany
Kristian Kersting [email protected] Darmstadt, Dept. of Computer Scienceand Centre for Cognitive ScienceDarmstadt, Germany
ABSTRACT
This work introduces Alfie, an interactive robot that is capable ofanswering moral (deontological) questions of a user. The interac-tion of Alfie is designed in a way in which the user can offer analternative answer when the user disagrees with the given answerso that Alfie can learn from its interactions. Alfie’s answers arebased on a sentence embedding model that uses state-of-the-artlanguage models, e.g. Universal Sentence Encoder and BERT. Alfieis implemented on a Furhat Robot, which provides a customizableuser interface to design a social robot.
CCS CONCEPTS • Human-centered computing → Interactive systems and tools . KEYWORDS interactive robot; bias in machine learning; text-embedding models;human-centered artificial intelligence
ACM Reference Format:
Cigdem Turan, Patrick Schramowski, Constantin Rothkopf, and KristianKersting. 2020. Alfie: An Interactive Robot with a Moral Compass. In
Pro-ceedings of the 2020 International Conference on Multimodal Interaction (ICMI’20), October 25–29, 2020, Virtual event, Netherlands.
ACM, New York, NY,USA, 2 pages. https://doi.org/10.1145/3382507.3421163
There is a broad consensus that artificial intelligence (AI) researchis progressing steadily and has pronounce impact on our daily life.Keeping the impact beneficial for society is of most importance. Weall remember the unfortunate event that happened when MicrosoftResearch (MSR) decided to release a chatbot for Twitter . Aftermany interactions with Twitter users, the bot started creating racistand sexually inappropriate posts. This resulted in the suspension ∗ Both authors contributed equally to this research. https://twitter.com/tayandyouPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s). ICMI ’20, October 25–29, 2020, Virtual event, Netherlands © 2020 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-7581-8/20/10.https://doi.org/10.1145/3382507.3421163
Figure 1: The interactive robot Alfie has a moral compass. of the bot for the users. This clearly shows the potential dangers ofunattended AI models.Recent studies have shown that language representations encodenot only human knowledge but also biases such as gender bias [1, 2],and according to more recent studies [5, 8, 9] also the moral anddeontological values of our culture. Schramowski et al. [9] haveshown that language models such as BERT [4] and the UniversalSentence Encoder [3] cannot only reflect the accurate imprints ofmoral and ethical choices of actions such as “kill” and “murder”,but also understand the context of the action, e.g., “killing time” ispositive whereas “killing humans” is negative. This, in turn, canbe used to compute a moral score of any (deontological) questionat hand, measuring the rightness of taking an action. This “MoralChoice Machine” (MCM) [9] can be used to determining the moralscore of any given sentence and in turn paves the way to avoidincidents like the MSR chatbot.Unfortunately, the MCM approach is purely unsupervised, justmaking use of the knowledge encoded in the language modelstrained without any supervision. This makes it difficult—if notimpossible—to correct the score and, in turn, help avoiding “MSRchatbot” moments. An attractive alternative would be to revise themoral choice via interacting with the MCM algorithm in a user-centric and easy way. In this demonstration, we investigate the useof the MCM algorithm in the context of an interactive robot, called a r X i v : . [ c s . H C ] S e p lfie and shown in Fig. 1. Alfie is giving us a great opportunity toinvestigate individuals’ reactions to the moral and deontologicalvalues of our culture encoded in human text. Alfie can also learnfrom the users and adjust its moral score based on human feedback.The rest of this paper is as follows: Section 2 presents the ar-chitecture of the system including the Moral Choice Machine, theemployed Furhat Robot and the dialog model. Section 3 concludesthe paper with a discussion and future work. Alfie is a Furhat Robot , which provides a customizable user inter-face. We can customize the speech production and facial expressionsas well as the human face presented through Furhat’s Software De-velopment Kit. There are a side microphone and a camera in frontof the Furhat Robot that allows the robot to follow the user andprovides the opportunity to access the camera feed so that one canperform more sophisticated computer vision algorithms.The interacting users are able to ask questions (user queries)to Alfie to get a moral score of the corresponding question. Inthe current version, the questions have to be in a certain form,e.g. Should I [action] [context] or Is it okay to [action] [context] .The Furhat Software preprocesses the speech input. The resultingtext output is then passed to the Moral Choice Machine (MCM)algorithm presented in [8, 9] as an input to calculate a moral score.The moral score computed is a real number normalized to [− , ] .In our current design, the range of moral scores is divided into threeintervals: [− , − . ] is no , [− . , . ] is neutral , and [ . , ] is yes .Both MCM variants [8, 9] employ current state-of-the-art sentenceembeddings computed using transformer architectures [3, 4, 6]and determine the moral score based on sentence similarities in theembedding space. This is an unsupervised method and consequentlythe quality of the moral score heavily depends on the performanceof the language models. In the current version of Alfie, we use thealgorithm described in [8].Additionally, we compute an emotional state corresponding tothe user query based on sentence similarities in the embeddingspace, i.e. finding the emotion with the highest similarity scoreto the question asked. In the current version, possible emotionsare Anger, Confusion, Disgust, Fear, Joy, Sadness, Satisfaction, Sur-prise. We change the facial expressions of Alfie based on theseemotions and adapt the pitch and the speech’s speed to fit thecorresponding emotion the best. According to the answer—"yes","no", or "indecisive"—we also add the respective head movementto make the conversation engaging. Due to the computational re-source limitations of the Furhat Robot, the MCM algorithms andother operations on the embedding space are computed on a sep-arate server. The resulting moral score is passed to Alfie again sothat the Furhat Software produces the speech as an output in formof a corresponding answer. We save all the questions asked to Alfieto a database in our servers for statistical purposes.Once in a while (as determined with a percentage value in thescript), Alfie asks for feedback about whether the user agrees withits answer. This response is also saved to the database. Of particularinterest are the responses when the user disagrees with Alfie. Thisgives us the opportunity and the data to retrain Alfie to adjust its https://furhatrobotics.com/ moral score with data collected during interactions or even onlineduring the interaction. We also created a training mode whereAlfie asks users many moral questions listed in our database. It ismeant for collecting feedback from the user for moral questions weare interested in human feedback. This data can later be used foradapting Alfie’s moral scores. As mentioned earlier, Alfie’s capabilities on the moral score dependon the performance of the language model, as well as the algorithmwe use to calculate the moral score. Also, since there is no absoluteagreement of right and wrong in general, it is difficult to qualita-tively evaluate the computed moral score. These are the reasonswhy we designed an interactive robot that is able to interact withhumans and collect their responses to learn from them. We aimto extend the interactions of simple feedback to explanatory in-teractive learning [7], i.e. adding the capability to explain Alfie’sdecisions and revising them based on user feedback. Although wecurrently focus on explicit feedback from users, i.e. their directfeedback on whether they agree or not, we aim to obtain implicitfeedback using the channels like gaze and body movement andfacial expressions similar to the study [10].
ACKNOWLEDGMENTS
We would like to thank Dustin Heller, Philipp Lehwalder, JonasMüller, Steven Pohl for their work on programming the initialversion of Alfie by transferring the Moral Choice Machine.
REFERENCES [1] Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, andAdam Tauman Kalai. 2016. Man is to Computer Programmer as Woman is toHomemaker? Debiasing Word Embeddings. In
Proceedings of Neural informationProcessing (NIPS) . Curran Associates Inc., USA, 4349–4357.[2] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derivedautomatically from language corpora contain human-like biases.
Science arXiv preprint arXiv:1803.11175 (2018).[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding. In
Proceedings of the 2019 Conference of the North American Chapter of the Associa-tion for Computational Linguistics: Human Language Technologies, NAACL-HLT .Association for Computational Linguistics, 4171–4186.[5] Sophie Jentzsch, Patrick Schramowski, Constantin Rothkopf, and Kristian Kerst-ing. 2019. Semantics Derived Automatically from Language Corpora ContainHuman-like Moral Choices. In
Proceedings of the AAAI/ACM Conference on AI,Ethics, and Society (AIES) . ACM, 37–44.[6] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddingsusing Siamese BERT-Networks. In
Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing, EMNLP . Association for ComputationalLinguistics, 3980–3990.[7] Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, FranziskaHerbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and KristianKersting. 2020. Making deep neural networks right for the right scientific reasonsby interacting with their explanations.
Nature Machine Intelligence
2, 8 (2020),476–486.[8] Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin Rothkopf, andKristian Kersting. 2019. BERT has a Moral Compass: Improvements of ethicaland moral values of machines. arXiv preprint arXiv:1912.05238 (2019).[9] Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin Rothkopf,and Kristian Kersting. 2020. The Moral Choice Machine.
Frontiers in ArtificialIntelligence