Challenges and Opportunities for Replication Science in HRI: A Case Study in Human-Robot Trust

Abstract

As human-robot interaction (HRI) researchers, like all scientists, we must demonstrate the reproducibility of findings-especially across robots. We present a three-study replication effort that illustrates the challenges and opportunities for replication science in HRI. A recent human-robot trust study (Ullman & Malle, 2017) suggested that people in the loop with a robot via a simple button press trusted the robot (a Thymio) more than those who observed the robot complete the task autonomously. This intriguing finding was based on a small sample (n = 40) and was therefore greatly underpowered (observed power 1 - β = .35), prompting replication. To test whether the in-the-loop effect generalizes to similar robots, we conducted a conceptual replication (Study 1) using the Create robot. The effect did not replicate, despite a large sample (n = 140) and expected power of .86. This result called for a direct replication (Study 2) using the original Thymio robot. The effect again did not replicate, despite a large sample (n = 200) and expected power of .96. We then conducted an online study (Study 3) with videos of both robots to examine whether different expectations for each robot drove the divergent results, but the hypothesis was disconfirmed (n = 400). However, one finding held across all studies: Participants consistently trusted imagined future robots far less in social than nonsocial use contexts (effect sizes of ds = -0.71, -0.78, and -0.79 in the lab studies; d = -0.38 in the online study).