Remote VR Studies -- A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs
Radiah Rivu, Ville Mäkelä, Sarah Prange, Sarah Delgado Rodriguez, Robin Piening, Yumeng Zhou, Kay Köhle, Ken Pfeuffer, Yomna Abdelrahman, Matthias Hoppe, Albrecht Schmidt, Florian Alt
RRemote VR Studies
A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs
RIVU RADIAH,
Bundeswehr University Munich, Germany
VILLE MÄKELÄ,
LMU Munich, Germany
SARAH PRANGE,
Bundeswehr University Munich, Germany
SARAH DELGADO RODRIGUEZ,
Bundeswehr University Munich, Germany
ROBIN PIENING,
LMU Munich, Germany
YUMENG ZHOU,
LMU Munich, Germany
KAY KÖHLE,
LMU Munich, Germany
KEN PFEUFFER,
Bundeswehr University Munich, Germany
YOMNA ABDELRAHMAN,
Bundeswehr University Munich, Germany
MATTHIAS HOPPE,
LMU Munich, Germany
ALBRECHT SCHMIDT,
LMU Munich, Germany
FLORIAN ALT,
Bundeswehr University Munich, GermanyWe investigate opportunities and challenges of running virtual reality (VR) studies remotely. Today, manyconsumers own head-mounted displays (HMDs), allowing them to participate in scientific studies from theirhomes using their own equipment. Researchers can benefit from this approach by being able to reach a morediverse study population and to conduct research at times when it is difficult to get people into the lab (cf. theCOVID pandemic). We first conducted an online survey (N=227), assessing HMD owners’ demographics, theirVR setups and their attitudes towards remote participation. We then identified different approaches to runningremote studies and conducted two case studies for an in-depth understanding. We synthesize our findingsinto a framework for remote VR studies, discuss the strengths and weaknesses of the different approaches,and derive best practices. Our work is valuable for HCI researchers conducting VR studies outside labs.CCS Concepts: •
Human-centered computing → User studies ; Virtual reality .Additional Key Words and Phrases: Virtual Reality, User Studies, Remote Studies, Data Collection Methods
ACM Reference Format:
Rivu Radiah, Ville Mäkelä, Sarah Prange, Sarah Delgado Rodriguez, Robin Piening, Yumeng Zhou, Kay Köhle,Ken Pfeuffer, Yomna Abdelrahman, Matthias Hoppe, Albrecht Schmidt, and Florian Alt. 2021. Remote VRStudies: A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs. 1, 1(February 2021), 29 pages. https://doi.org/10.1145/1122445.1122456
Authors’ addresses: Rivu Radiah, [email protected], Bundeswehr University Munich, Germany; Ville Mäkelä,[email protected], LMU Munich, Germany; Sarah Prange, [email protected], Bundeswehr UniversityMunich, Germany; Sarah Delgado Rodriguez, [email protected], Bundeswehr University Munich, Munich, Germany;Robin Piening, [email protected], LMU Munich, Germany; Yumeng Zhou, LMU Munich, Germany; Kay Köhle,LMU Munich, Germany; Ken Pfeuffer, [email protected], Bundeswehr University Munich, Munich, Germany;Yomna Abdelrahman, [email protected], Bundeswehr University Munich, Germany; Matthias Hoppe,[email protected], LMU Munich, Germany; Albrecht Schmidt, [email protected], LMU Munich,Germany; Florian Alt, [email protected], Bundeswehr University Munich, Germany.© 2021 Association for Computing Machinery.XXXX-XXXX/2021/2-ART $15.00https://doi.org/10.1145/1122445.1122456 , Vol. 1, No. 1, Article . Publication date: February 2021. a r X i v : . [ c s . H C ] F e b Rivu et al.
Virtual Reality (VR) has become a widely adopted technology in the Human-Computer Interaction(HCI) research community and beyond [43]. Many subtopics emerged around this technology,including but not limited to, novel interaction techniques, presence and immersion, avatar modeling,navigation, and locomotion. More recently, researchers started looking into how VR can substituteor complement other research approaches [29, 40], for example, in-the-wild studies that require alot of effort to maintain the technology used, evaluations in potentially dangerous environments(e.g., automotive or pedestrian user interfaces), or the investigation of situations that occur rarely.In the past decades, VR headsets were not widely available and their setup required a substantialdegree of technical knowledge. Hence, the vast majority of research was conducted in lab settings,where researchers took care of the setup and guided participants through (controlled) studies. As aresult, participants of such studies were often part of a rather homogeneous university population– a well-known challenge in HCI research [7].The past years witnessed a proliferation of VR head-mounted displays (HMDs) in the consumermarket and while the number of worldwide shipped VR units with 6 million units in 2019 isstill marginal from a commercial perspective, a considerable number of end users now own a VRheadset. This creates an unprecedented chance for researchers to move VR research out of the laband reach a more diverse community – similar to what happened with other technologies thatreached the consumer market, such as smartphones [20–22] or public displays [14].In this work, we contribute an in-depth investigation of an emerging research paradigm, thatis conducting out-of-the-lab research with owners of VR headsets. In particular, we explore thechallenges and pitfalls researchers are facing as they are shifting or complementing their researchusing this new paradigm and synthesize them into a framework for use by other researchers. Tothis end, our work is guided by the following research questions: • RQ1:
How suitable are HMD owners and their VR setups as subjects for remote studies, andwhat limitations do home setups have? – 1.1:
What are the demographics of HMD owners? – 1.2:
Which VR equipment do they own? – 1.3:
What are their home VR setups like? – 1.4:
How do they feel about participating in remote studies from home? • RQ2:
What suitable approaches exist for conducting remote VR studies, and what are theiradvantages and disadvantages? – 2.1:
What options exist for developing and setting up VR research applications? – 2.2:
How can VR research prototypes be distributed to study participants? – 2.3:
Through which channels can owners of VR headsets be reached and recruited? – 2.4:
What special considerations are there when designing and running a remote study?Our research approach is as follows: we set out with conducting an online survey (N=227)among owners of VR headsets to understand their demographics, the equipment they own, thesetting in which they use it, and whether they would be willing to participate in remote VR studies.Subsequently, based on a review of existing VR distribution channels, we describe different ways inwhich VR applications for remote studies can be implemented and distributed as well as how datacollection can be realized. We then report on our experiences from two remote case studies, wherewe experimented with different approaches. We then synthesize all our findings into a frameworkthat guides researchers through different ways of designing and conducting remote VR studies. emote VR Studies 3 Our framework consists of four primary approaches to remote studies:(1) Researchers build a standalone VR application that is distributed directly to participants.(2) Researchers build a VR application that is distributed through existing vendor platforms("app stores") such as Steam and the Play Store.(3) Researchers build a VR application using an API of an existing social VR platform (e.g., RecRoom, VRChat) and upload it to the corresponding platform.(4) Researchers set up their study environment directly within an existing social VR platformusing the tools provided by these platforms.All of these approaches come with unique advantages, drawbacks, and considerations. Some ofthe clearest differences are in the level of autonomy and the amount of required effort. Using existingvendor platforms and social VR platforms requires that researchers conform to the regulations andlimitations of these platforms. This often means that options for data collection are limited, and theplatforms may also be more limited in their functionalities. However, using such platforms can oftenmake setting up the study far easier and faster, and existing platforms ensure an existing user base,and often alleviate compatibility issues. Building custom software and distributing it independentlymay get around many of the limitations with existing platforms, but the workload is often higherand may result in other challenges (e.g., ensuring the safety and privacy of participants).Another dimension is whether or not the VR study is conducted with a remote study experimenter,or whether participants run the study independently, i.e., without remote guidance. We discusshow well the four main approaches lend themselves to guided studies and independent studies.Our research is valuable for researchers considering to conduct virtual reality studies remotely.Beyond an opportunity to reach a larger and more diverse audience, our research is also of valuefor other reasons. Most prominently, the recent COVID-19 pandemic affected the HCI communityaround the globe, making it difficult for many researchers to recruit participants for their workin the lab. This unprecedented event raised the question: "How can VR researchers continue toconduct studies?" Here, we believe that our work can provide useful guidance. Our findings are notonly valuable in a pandemic but also opens up opportunities beyond these times.
Contribution Statement.
We contribute a holistic view on remote VR studies. Specifically, (1) wepresent the results of an online survey assessing users’ view towards such studies and their VRsettings; (2) we provide a review of research approaches with a focus on how they influence thedevelopment / study setup, software distribution, and recruitment challenges; (3) we report on twostudies and derive lessons learned; and (4) we synthesize our findings into a framework.
Several strands of prior work are relevant to our research. After briefly introducing the term remoteVR studies , we review (a) work involving ubiquitous technologies for conducting out-of-the-labresearch as well as (b) prior work on conducting VR studies outside the lab.
Over many decades, the HCI research community adapted or came up with new approaches toresearch, which can be broadly classified into research conducted in controlled settings (i.e. in thelab) vs. in less controlled settings (i.e. in the field). As Virtual Reality is being appropriated as a toolfor conducting research, another dimension – in addition to where the study physically takes place– emerges, that is the virtual setting. Here, researchers are provided rich opportunities to situatetheir research in almost any setting they like – be it the reconstruction of a lab, a public space, areplica of a person’s home, or an entire world (cf. Second Life ). Second Life: https://secondlife.com, last accessed January 21, 2021, Vol. 1, No. 1, Article . Publication date: February 2021.
Rivu et al.
We focus on studies taking place physically in users’ homes but not being restricted with regardto the virtual setting. In the following, we refer to these type of studies as remote VR studies . The idea of leveraging consumers’ devices for ’bringing the lab to participants’ is already establishedpractice in several areas of human-computer interaction research. Among the most prominentexamples is work by Henze et al. who used smart phones as a tool to conduct out-of-the-lab studies,initially on touch targeting and typing behavior [20–22]. Over the following years, the communitysaw more studies focusing on various aspects, including but not limited to an exploration of userinteraction with notifications [34], authentication behavior [4, 10, 35], and keystroke dynamics [9].Prior work has identified both strength and challenges of such types of studies. Of particularinterest is the work of Gustarini et al. [19] who categorize challenges into the design, development,execution, and data analysis phase. While their work is specific to studies focusing on smartphones, several of the challenges apply to other technologies as well and, hence, informed our work.These challenges include the question of which data can be collected, where data can be stored,how researchers can deal with device heterogeneity, recruiting bias, remunerating participants,participant cheating, handling participants’ questions, considering participants’ motivation andprivacy needs, conducting interviews and how to synchronize data.Studies have also been conducted to understand the extent of ecological validity in remoteexperiments. For example, Andreasen et al. [5] conducted an empirical comparison of remoteusability testing and conventional lab testing, and Germine et al. [18] investigated data quality acrosslab-based and web-based experiments. Research shows that it is possible to achieve ecologicallyvalid results using out-of-lab studies [5, 18].
In the following section we summarize key insights obtained from prior research. We take achronological view on how out-of-the-lab studies have progressed over the years.In 2012, Hodgson el at. presented ways for ‘portable’ VR studies, using systems that can be easilycarried and used in-the-wild [25]. The authors introduced a self-contained backpack, capable ofrunning VR simulations and demonstrated how such portable systems can be used by researchers.Four years later, Steed et al. [39] experimented on presence and embodiment in VR throughremote studies by recruiting Samsung Gear VR and Google Cardboard owners. They argue thatfor remote VR studies it is generally easier to recruit more participants than for typical lab-basedstudies. However, their work suggests that the diversity of participants was still defined by thenature of the study. The authors also highlighted that remote studies might require more effortthan lab studies in terms of, for example, preparation and app development.In 2017, Mottelson et al. [31] conducted experiments both in and out of the lab to understand theadvantages and pitfalls between the two approaches. The experiments generally yielded comparableresults as both in-lab and remote studies were able to collect reliable data and similar output. Atthe same time, out-of-the-lab experiments were characterized by a greater heterogeneity amongparticipants such as sample population. They also outline several limitations of remote studiesincluding demographics validity, ethical concerns, and the lack of experiment control.Ma et al. [28] conducted three behavioral experiments in VR using crowd sourcing in 2018.Their evaluation suggests the feasibility of conducting web-based VR experiments albeit severalchallenges, including few participants with access to VR-capable devices. The authors also highlightmany advantages, including obtaining a more diverse sample compared to lab VR studies. In astudy about social interactions in weekly VR get-togethers, Moustafa and Steed [32] in the sameyear also highlighted the potential of reaching broad and diverse audiences through remote studies. , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 5
Recently, Saffo et al. investigated the feasibility of running VR studies via a popular socialplatform, VRChat [33]. They identified the ability to enable researchers to create their own content,the large user base, and the ease of recruiting participants from within VRChat as advantages ofthe platform as it is easier to recruit participants within the platform. At the same time, Saffo etal. reported on data collection to be limited when using online platforms as each online platformreviews the content before researchers are able to upload content.Huber and Gajos [26] worked on conducting online VR studies with uncompensated and un-supervised participants. While they identified advantages, such as recruiting participants fromall over the world with various backgrounds, there are challenges: for example, in this study thesample count was much smaller compared to other online studies.
Our review shows that while researchers have recognized and acknowledged the potential of remoteVR studies, for many years they have struggled with challenges, such as the available technology,the number of people who had access to the technology, means to reach out to them, and ways ofdelivering software to the participants. Not only with the current pandemic, we see that interestin this type of studies is increasing at a rapidly accelerating pace. We believe to be currently at aturning point where with advances in technology and due to the increasing proliferation of thetechnology for the first time it is finally possible for VR researchers to strongly benefit from thisapproach. Yet there is still a lack of a comprehensive understanding of how such studies can beimplemented. This is at the focus of our work.While much can be learned from other sub fields of HCI where technology that became widelyavailable to end users was leveraged for conducting remote studies, there are many aspects uniqueto virtual reality and head-mounted displays that remain under-explored, including but not limitedto the available technology, the environment, data collection methods, and distribution channels.Some of the aforementioned aspects have been touched upon in prior work. Yet, our analysisshows that pre-requisites have changed over the years and new opportunities and challengeshave emerged. Our work provides an understanding of the current state-of-the-art in runningremote VR studies as well as a comprehensive assessment of this study paradigm that we synthesizeinto a framework. The framework explains different approaches to remote VR studies and alsosummarizes best practices and lessons learned.
We conducted an online survey among VR users with the goal of better understanding (a) theirdemographics, (b) how they can be reached and recruited, (c) what their VR setups look like, (d)which technology they own, and (e) whether they are willing to participate in remote VR studies.
In particular, our survey focuses on the following aspects:
Demographics
We assessed the participants’ age, gender, background, and their reasons forowning a VR device.
VR Equipment & Platforms
We asked about the devices they own and the platforms theyare using.
VR Setting and Use
We asked them to describe their VR setup, the space (with an option toupload a picture of the space), how often they use it, and how likely they are to be interruptedwhile being in VR. , Vol. 1, No. 1, Article . Publication date: February 2021.
Rivu et al.
Willingness to participate
We asked the participants how willing they are to participate inremote VR studies, both in general as well as in situations such as the COVID-19 lockdown,and what kind of payment they might expect.
We distributed the survey via different channels, in particular such that would also be used byresearchers to reach out to potential participants of a VR study. To this end, we included research-oriented crowd sourcing platforms, VR online forums, social media and university mailing lists. Inthe following we report some lessons learned.
We recruited through two platforms: Prolific and XRDRN . Prolific is a platform primarily meant for recruiting study participants, for example,for surveys. It allows participants to be pre-selected by demographics (for example, owners of VRdevices). XRDRN is a dedicated platform to connect researchers and participants with a specificfocus on mixed reality. The platform was built specifically in response to the COVID-19 pandemic.Response rates from Prolific were considerably higher and faster. This was likely because (a) Prolificintegrates means for payment directly with the platform whereas for XRDRN this would be handledout of the platform between researchers and participants, and (b) XRDRN is still in the early stagesof building a community. Another aspect is that since all participants on Prolific are paid, many ofthem can be expected to be ‘professional’ study takers. We reached out to VR users through Facebook, Reddit and platform-specific forums (e.g., Steam, Rec Room). Within these platforms, we identified suitable subgroups(e.g., VR-related subreddits in Reddit).Within platforms, subgroups could be easily identified. However, it took some time to ensure thatwe complied with the rules of each subgroup and that posting an invitation to an online survey wasappropriate. Where groups did not specifically allow or disallow for these kinds of posts, we reachedout to moderators to seek permission. While for some groups we obtained permission in this way,many moderators were unsure and suggested we rather post in other channels, for example, suchthat were concerned with the organization of VR events or for discussing miscellaneous topics. Wenoticed that such groups often had fewer active members, which limited the visibility of the survey.Some subgroups employed specific posting procedures. Some required specific tags or featuresto be assigned to messages (for example, on Reddit or Rec Room) – yet the pre-defined tags wereoften a poor match for our recruitment messages. Many forums also make it mandatory for users toachieve a certain number of points (points are acquired based on several metrics such as number ofcomments and posts on other discussion threads) before they are eligible to create their own threads,while others required a user account to be at least two months old before being eligible to post.When posting links to different subforums simultaneously, there is a chance that auto-moderatorsflag the posts as spam. Established VR products (such as Unity and Rec Room) run forums andgroups on different social platforms (such as Reddit, Facebook, Twitter, Discord). Those could beused in addition to more general groups on VR.
This method helped to reach out to a large number of recipients.In particular, we used mailing lists allowing a general university population to be reached. Thispopulation included many non-technical subjects. Our experience shows that only few of themwere eligible for the survey (as they were required to own an HMD to participate). Mailing liststargeting students from computer science subjects might reach more participants. emote VR Studies 7 Table 1. Online survey: participants’ demographics including their relationship to VR, which VR platformsthey use (left), their purposes of using VR, and possible new uses discovered during the pandemic (multiplechoice questions each) (right). G e n d e r Male 170Female 52Prefer not to say 1Other 4 R o l e s I am a consumer of VR products and ap-plications. 214I am a developer of VR products and ap-plications 21I create content for VR. 20I conduct research in the area of VR. 16Other 3 U s e d V R P l a t f o r m s VRChat 91None 77Rec Room 57AltspaceVR 26Mozilla Hubs 23Other 12Bigscreen VR 8Hologate VR 7Cluster 5Playstation VR 5The Wild 4A Township Tale 2 R e a s o n s f o r U s i n g V R Gaming 206Social platforms 63Socializing with friends and family 43Development or Research 33Work (e.g meetings, presentation) 17Other 14Watch Videos (e.g. Netflix or films) 10 U s e s D i s c o v e r e dd u r i n g P a n d e m i c I discovered new VR games. 130I discovered new virtual social plat-forms. 35I started VR meetings with friends andrelatives. 24I started watching VR movies togetherwith friends. 23My usage did not change. 18I started VR collaboration with col-leagues. 12Other 11
We received 227 complete surveys (out of 276). Respondents were recruited through Prolific (97),Reddit (95), mailing lists (14), Facebook (12), Discord (4), VR platforms (3) and other (2).
A total of 227 participants responded to our online survey, 52 female (cf.Table 1 for an overview) Participants were between 18 and 56 years of age (
𝑀𝑒𝑎𝑛 = . , 𝑆𝐷 = . 𝑁 = 𝑁 = 𝑁 = consumers of VR products ( 𝑁 = 𝑁 =
21, 9%), VR content creators ( 𝑁 =
20, 9%), and VR researchers ( 𝑁 =
16, 7%).They mainly used their VR setup for gaming ( 𝑁 = 𝑁 = 𝑁 =
35, 15%). Some users had startedusing their setups for entirely new purposes, like meeting with friends and relatives ( 𝑁 =
24, 11%),watching movies socially ( 𝑁 =
23, 10%), and collaborating with colleagues ( 𝑁 =
12, 5%).
We first asked participants about the devices they owned .Most participants used Oculus devices ( 𝑁 = 𝑁 = 𝑁 = 𝑁 = 𝑁 = 𝑀𝑒𝑑𝑖𝑎𝑛 =
2, 1=not at all, 5=almost all the time). Consequently, mostparticipants could use their VR setups without any notable constraints ( 𝑁 = 𝑀𝑒𝑑𝑖𝑎𝑛 =
2, 1=veryunlikely, 5=very likely). Furthermore, participants reported that getting their VR setup ready foruse (Figure 2c) did not require much effort (
𝑀𝑒𝑑𝑖𝑎𝑛 =
2, 1=no effort at all, 5=a lot of effort). , Vol. 1, No. 1, Article . Publication date: February 2021.
Rivu et al. (a) Total time (hours) spent in VR on weekdays andweekends. Most users spend at most two hours withthe VR setup on weekdays and on weekends undernormal circumstances. During the pandemic, partici-pants reported somewhat higher usage. (b) Space available for users’ VR setups, showingmost users have a space of up to 25 square meters.26 participants stated a larger than 40 square metersspace (excluded in this plot).Fig. 1. Results on how much time participants spend in VR and how large their VR setups are.
Regarding usage times (Figure 1a), most participants reported to normally use their setups for upto two hours on weekdays (in total during an average week) ( 𝑁 = 𝑁 = size , providing thewidth and length in meters (Figure 1b). 65 participants (29%) reported having up to 5 𝑚 , 71 (31%)from 5 to 10 𝑚 , 26 (11%) from 10 to 15 𝑚 , 21 (9%) from 15 to 20 𝑚 , and 15 (7%) from 20 to 25 𝑚 . We inquired about participants’ willingness to participate inVR studies (‘I am willing to participate in VR studies.’; 5-Point Likert scale; 1=strongly disagree,5=strongly agree). To better understand whether this was a result of the current pandemic situation ,we asked the question for both during and after the lockdown (Figure 2a). 152 participants (67%)agreed or strongly agreed to be willing to participate in VR studies after the pandemic (
𝑀𝑒𝑑𝑖𝑎𝑛 = 𝑀𝑒𝑑𝑖𝑎𝑛 = comfortable participants felt about participating in VR studies in differentstudy locations (Figure 2b). Participants felt more comfortable about the idea of participating fromhome than attending a lab study (home: 𝑀𝑒𝑑𝑖𝑎𝑛 =
5, lab:
𝑀𝑒𝑑𝑖𝑎𝑛 = compensation (multiple choices), participants were mostly willing to accept cash( 𝑁 = 𝑁 = Our survey provided insights on the current mainstream users of VR. First, the survey confirmsthe ‘cliche’ of VR users being primarily gamers (91%). Furthermore, users were predominantlymale (77%) and mostly consumers (94%). Besides gaming, a rather prominent use case for VR wassocializing, with 28% of participants stating they use social platforms and 19% that they use VR tomeet with friends and family. We also found that participants used many different VR devices andplatforms. The most popular
VR platforms were VRChat (40%) and Rec Room (25%) followed byAltspaceVR (11%) and Mozilla Hubs (10%), indicating that there is a wide range of platforms withpotential for researchers to conduct studies. Many participants used more than one platform. , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 9 (a) Willingness to participate in studies (b) Comfort in participating in studies(c) Required effort for VR setupFig. 2. Results on how willing participants are to participate in remote studies, how comfortable they feelabout it, and how much effort it is for them to use their VR setups.
Most HMD owners used their VR setups exclusively without notable constraints. Participantsreported that the effort required to get their setup ready for use was low, suggesting that VRsetups are typically permanent setups with a dedicated space. Respondents also estimated that thelikelihood of being interrupted during VR use was low, again suggesting that the setups are mainlyused in personal spaces (e.g., bedrooms) rather than in shared spaces (e.g., living rooms). Thesefindings indicate that home VR setups are generally well suited for remote studies.Furthermore, we assessed the size of the users’ VR setups. The majority of participants (60%)seemed to operate withing a space of up to 10 𝑚 . Around half of this group (29% of all respondents)reported having a small space of up to 5 𝑚 , likely indicating a seated setup. These findings suggestthat remote studies requiring active movement in a larger space (e.g., more than 15 𝑚 ) might bechallenging as many HMD owners do not have the necessary space, potentially risking skewedresearch results as well as participants getting injured. To work around the problem of small spaces,researchers might consider various locomotion techniques, such as teleportation techniques [8, 17],to reduce the need for physical movement. However, such techniques might not always be desirable,particularly if "real" physical movement is critical to the phenomena being studied.The clear majority of participants expressed willingness to participate in remote VR studies (67%under normal conditions and 73% during the pandemic). Participants also felt very comfortable aboutthe idea of participating in studies from their own homes, noticeably more so than participating inlab studies. The survey attempts to assess the willingness of a potential participant to participate. Inthe end, participation is an amalgam of many factors and is dependent on each specific study, suchas the participants actually signing up for the study and participating on it, the setup requirementsand data collection methods based on the specific research question. Thus, our obtained results , Vol. 1, No. 1, Article . Publication date: February 2021. indicate an initial interest to participate from home. Our survey also indicates that participants arecomfortable about remote participation and they have VR setups which are likely to be permanent,private and uninterrupted during participation. We believe these initial findings are beneficial whenresearchers are planning remote studies.Our results also suggest that the COVID-19 pandemic affected people in that they reported moreVR usage, and some reported having started to use their VR setups in new ways, like meetings withfriends and family. This suggests that users are open to new uses of VR, further strengthening ourbelief that HMD owners are well willing to participate in remote studies.We distributed the survey over numerous channels to limit a potential recruiting bias. Neverthe-less, we acknowledge that it is hard to fully avoid a recruitment bias. This may have impacted thesurvey results depending on the popularity of each channel chosen, as some VR platforms mayhave been more/less popular.Our results also indicate that HMD owners are highly suitable to be recruited for remote studies.Most respondents were willing to participate in remote studies and felt comfortable about it. TheirVR setups and use also support this, as most setups seem to be permanent and used exclusively,with a low chance of interruptions. The most notable challenge identified through our results isthat most participants have relatively small setups, which might be challenging for some studies. In this section, we discuss potential ways to conduct remote VR studies and hypothesize abouttheir strengths and weaknesses. We also provide a brief discussion on app stores and social VRplatforms that could be utilized for remote VR studies.
For most VR research projects, researchers develop a custom appli-cation that runs independently. This often involves designing and modeling a digital environment,programming the desired functionality, and designing and programming the methods for datacollection [29]. The advantage of custom-built applications it that they provide maximum flexibilityand freedom when it comes to designing the visuals, means for interaction, data collection, anddata storage, among other aspects.Many modern tools are of great help with building VR applications. Particularly popular in thisregard is the Unity engine , complemented with the SteamVR plugin . The Unity engine is amassively popular tool for creating games, but it is also widely used for industrial purposes, films,and architecture, among others. As a result of its popularity, the internet is full of tutorials andtips for Unity developers, and communities for supporting developers are plentiful and active. TheSteamVR plugin takes care of the basic communication with the head-mounted display and theVR application – many HMDs, like the HTC Vive, support SteamVR. This greatly alleviates thehurdles related to developing for HMDs. For the most part, developing VR applications is not anydifferent from general 3D development.Despite these great tools and related resources at our disposal, it is clear that building VRapplications takes time and requires considerable software engineering expertise, and in manycases, also expertise in other areas like visual design and 3D modeling. We also anticipate furtherchallenges with remote studies, as researchers have less control over the study and are more in thedark with how participants behave. Researchers may therefore need to develop additional tools tomonitor participants or consider other ways to ensure smooth procedures. https://unity.com/, last accessed January 21, 2021 https://assetstore.unity.com/packages/tools/integration/steamvr-plugin-32647, last accessed January 21, 2021, Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 11 Platform Interactions Data Collection CustomizableAvatars External Sources Viability
Altspace Programmable Full (web-hosted) Yes (in-game) Yes (browser) HighVRChat Programmable Limited Yes (model upload) Yes (Youtube player& browser) HighRec Room In-game rules Limited Yes (model upload) No HighMozilla Hubs No Influence No Yes (in-gamecustomisation ormodel upload) Yes (share mediafrom PC or web) MediumThe Wild No influence
Insufficient data
No Revit metadata LowEonReality Lesson rules
Insufficient data
No avatar Models/Media LowMinecraft In-game scripts(Command Blocks) No Yes (reskin) No Low
Table 2. Comparison of social VR platforms.
Another, emerging possibility for remote studies is to use social VRplatforms. Many of these platforms, like Rec Room and VRChat, offer diverse customization optionsthat might be enough for many studies. There are considerable differences between such VRplatforms in terms of their user base and what features they provide. Based on our analysis of theseplatforms, we see two distinct approaches to implementing virtual environments for user studies.First, some VR platforms allow custom applications to be uploaded . For example, VR applica-tions built with the Unity engine can be uploaded to VRChat, as long as the applications usethe VRChat Software Development Kit (SDK). Publishing applications in VR platforms requiresthat the application fulfills requirements and follows rules put forth by the platform. This mayresult in more restricted possibilities as opposed to fully independent VR applications. At the sametime, these platforms are likely to bring various advantages. For example, installing and runningthe application is likely less error-prone and more novice-friendly, and compatibility of differenthardware is handled by the platform rather than the application. Furthermore, it might be possibleto attract study participants from the existing user base of the platform.Second, many VR platforms allow users to build their own environments inside the platform usingsimple built-in tools, and offer customization options. This includes choosing between variousindoor and outdoor settings and adding 3D objects like furniture, interactive objects like drawingboards, and even questionnaires. Although it is clear that these tools offer less flexibility and freedomfor researchers than fully custom-built applications, these tools make building VR environmentsconsiderably faster and easier. Another significant advantage is that these tools are more accessibleto researchers from other fields who might have less technical expertise.According to our survey results, the most popular VR platforms among HMD owners wereVRChat, Rec Room, and AltspaceVR. Furthermore, based on our analysis of the features of variousVR platforms (Table 2), these same platforms stand out from the others as they offer the highestflexibility for researchers. Hence, we provide a brief overview of these three platforms. Other VRplatforms that are suitable for research purposes may emerge in the future.
Rec Room is a popular platform with more than 1 million VR users [41]. It allows users toplay and create VR games and it is compatible with several operating systems and devices.Users can program simple logic in their custom rooms [1], making Rec Room a powerfulplatform for studies that only need simple interactive features.
VRChat is a highly customizable platform. Custom worlds can be uploaded into the platform(e.g., worlds built using Unity and the VRChatSDK), offering many features (e.g., spawnpoints, interactive objects). Prior research used VRChat for studies [33], finding that this , Vol. 1, No. 1, Article . Publication date: February 2021. was generally a useful approach that, however, came with several limitations. For example,developers with new accounts cannot upload content without earning a certain ’trust’ level.Furthermore, there are compatibility issues with some devices and custom content cannotcommunicate with outside services.
AltspaceVR is a social VR platform supporting several devices (e.g, HTC Vive, Oculus). Itallows new world templates and world objects to be created via Unity Uploader, which canbe uploaded and shared on the official hub. Due to security reasons, no scripts are allowed .Developers can, however, place so-called "extensions" into their world through external webpages that get translated into 3D content. In addition to developing and setting up their VR applications, researchers need to think abouthow the applications will be distributed to participants as well as how potential participants couldbe reached.
Perhaps the most straightforward option is to distribute the applicationdirectly to participants, for example, by sending them a download link. Participants then install theapplication on their own.It is likely that this approach is relatively effortless and offers the most flexibility, as the applicationdoes not need to conform to the rules and regulations of any external services. However, somechallenges are likely present. For example, direct installation requires users to install software fromuntrusted sources (which might limit their willingness to participate) and it might generally bemore error-prone than other ways of distribution (e.g., due to lack of support during installation).
As already discussed, social VR platforms might be utilized for remotestudies, as study environments can be set up through them. Since the VR application resides withinan existing platform, there is no need to explicitly distribute the application. Here, we might utilizethe existing user base of the social VR platform, or attempt to attract new users to it.
Application stores offer another, potentially effective way to distribute VRapplications. Here, applications are published in app stores, such as Google Play , Steam , or theOculus Go Store . This approach poses certain requirements to researchers. Firstly, app storesmight require the application to be developed using certain tools and programming languages.Secondly, applications are usually subject to a review and, hence, researchers must fulfill certaincriteria regarding data protection, user interface design, etc. Thirdly, publication on app stores areoften not free of charge.At the same time, publishing in app stores is attractive because a large audience can be reached.Indeed, prior research has successfully distributed research prototypes through app stores in moreconventional contexts like mobile interaction [20–22]. Furthermore, since the application storestake care of the installation process, it tends to be safe, free of errors, and easy for consumers.We provide a brief overview of some popular application stores and their viability for publishingapplications for research purposes. Our list is not intended to be exhaustive, but rather, to provide ageneral idea of the possibilities and challenges with app stores. Other app stores might be availableor emerge in the future for research purposes. https://help.altvr.com/hc/en-us/articles/360015560614-Unity-Uploader-FAQacc:15.09.2020 https://support.google.com/googleplay/android-developer/answer/6334282?hl=en, last accessed January 21, 2021 https://partner.steamgames.com/doc/store/creator_homepage, last accessed January 21, 2021 https://developer.oculus.com/distribute/latest/tasks/publish-submit-app-review/, last accessed January 21, 2021, Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 13 Steam is an app store particularly popular with games and VR games that support SteamVR.One caveat of Steam is its strong focus on (commercial) games. There are only a few exceptions,such as software for content creation. Nevertheless, it might be interesting if the researchprototype can be embedded into a game, which is particularly popular in the HCI community.A strong advantage of Steam is that it has millions of users and that many HMDs supportingSteamVR exist, hence allowing a large audience to be reached. Researchers need to be awarethat publishing takes at least 30 days and has a charge of about 100$ per application.
The Oculus Store is targeted at Oculus devices and, hence, cannot be used by owners of someother popular HMDs. At the same time, the consumer base is fairly large. The store providesclear guidelines for content publishing and the review process takes around two weeks.After review, the content may be published live on the Oculus store or it can be shared via adownload key with participants.
The Google Play Store is immensely popular, connecting developers to millions of Androidusers. The Play Store allows for publishing Android-based VR apps, which can be run on afew HMDs and, more importantly, on any Android smart phone serving as a low-cost HMDin a cardboard. A Google Play developer account is needed, which is available for a onetime fee of 25$. One can choose to publish based on several available modes (internal test,closed test, open beta, full publish), called tracks. There are also other options to control theavailability of the app, like country-specific availability. These options are very useful forresearchers who might want to retain some control over how many people install their appor who they recruit as participants. Overall, the Google Play Store is a highly viable optionfor researchers, with the downside of not being an option for most high-end VR HMDs.
We identified several different ways to conduct remote VR studies. On one end of the spectrum,researchers can produce isolated VR prototypes and distribute them directly to participants. Onthe other end, researchers can set up their studies in existing VR platforms and make full use oftheir features. At the same time, there are options in-between, like distributing a custom-builtapplication through app stores, or uploading it to a VR platform that supports custom applications.These approaches offer a good starting point for our investigation. It is worth noting that thereare other considerations involved as well, such as how participants can best be recruited for remotestudies. We will get back to these considerations in the later chapters, when we synthesize ourfindings from the online survey and the two case studies.With these possibilities in mind, we then conducted the two case studies, which we present next.For these studies, we identified the best ways to run them remotely, which were very differentfrom each other. This also allowed us to report on a wide range of experiences.
To complement the online survey as well as our investigation of different approaches, we presenttwo case studies to gather first-hand experience. The studies we present are part of independentresearch projects, in which some of the co-authors of this submission are involved. In this paper,we focus on aspects related to the study methodology in the description of the case studies, ratherthan on the main research questions of the projects and the obtained results.
In Case Study 1, we developed a VR shooting game (Figure 3) where study participants attempted toshoot at drones using their VR controllers, and avoid getting hit by the drones’ lasers. Our researchfocus was on investigating different ways to convey the player’s health . The player’s health was , Vol. 1, No. 1, Article . Publication date: February 2021.
Fig. 3. The VR shooting game and instructions used in Case Study 1.
LEFT: players shoot drones using theirVR contollers as pistols. One of the tested health interfaces, the watch, is visible on the user’s wrist.
MIDDLE:
The starting instructions and a menu where the players choose which health interface to use. After theirselection, they can begin the game.
RIGHT:
Instructions and menu displayed after playing a session with oneof the health interfaces. Study participants repeated this procedure three times. reduced each time they got hit by lasers, and would slowly restore if they did not get hit for enoughtime. The game ended when health was reduced to zero.We designed and implemented three different interfaces to communicate the player’s healthstatus. Study participants played three rounds of the game, each time with a different healthinterface, and provided feedback.In this study, the most critical aspects of the remote VR methodology were: • Standalone VR application.
We built a custom, standalone VR application. We did not useexisting platforms like Rec Room or VRChat. • Direct download.
We hosted the application on a web server and provided a link to it toparticipants. Participants then downloaded the application and ran it on their own. • Independent study method.
The study was asynchronous, i.e., participants ran the studyon their own any time they wanted. No experimenter was present during the study.
We investigated whether diegetic representations of player’s health statusin VR shooting games would improve the sense of presence and add to a sense of danger duringintense gaming sessions. As a non-diegetic baseline condition, the player’s health was displayed asa health bar, which was visible on the screen at all times. As diegetic conditions, we implemented awristwatch displaying the player’s health (players would need to lift their wrist and really look atthe watch) (Figure 3, left) and a movement-based method, where hurt players moved slower andtheir firearm was shaking, to simulate the condition of being physically hurt. We hypothesized thatthese diegetic conditions would improve certain aspects of the gaming experience.
Study Design.
We built a custom VR shooter game using Unity and SteamVR. In the game’smain menu, instructions were displayed for the player. They could choose any of the three healthinterfaces and start the game (Figure 3, middle). In the game, drones appeared around the playerwhich attempted to shoot at the player. The player’s task was to avoid getting hit and try to shootas many drones as possible. The game session was set to last for four minutes. To support playersof different skill levels, we implemented a dynamic difficulty system that adjusted various factorsbased on how well the player performed: the rate at which the drones appeared, how spread outthey would be, and how fast and frequently they would shoot at the player. Also, we added a curveto player health – a typical trick in videogames – so that players would not die as easily as thesituation might suggest. These were important factors in ensuring that players of all skill levelswould experience a challenge and the thrill of being low on health, which in turn was importantfor our research goals. At the same time, the dynamic difficulty and the curved health would notlet players die very easily and frustrate them. Players regained their health during the session ifthey remained unhurt for a brief period. , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 15
Participants played a short session using all three health interfaces. After each session, a feedbackquestionnaire was automatically opened on their PC on their default browser, and the gameinstructed participants to take off their HMD, fill in the form, and then return to the game (Figure3, right). We also used the form as a means to log data from the game: we pre-filled some of theform questions with game data and prevented users from modifying them.Even though current research suggests filling questionnaires inside VR [3, 36], we opted forexternal questionnaires. This was partly because we wanted to give players a proper break aftera potentially intense game session, but more importantly because the questionnaires were solong that they would have been very tedious to fill in VR. The questionnaires after each sessionincluded the Game Experience Questionnaire [27], the Slater-Usoh-Steed presence questionnaire(e.g., [37, 42]), custom statements, and also some optional open-ended questions.When participants filled in the form for the third and final time, they ticked a checkbox statingthat they had finished playing with all three health interfaces. At this point participants werepresented with a brief set of additional questions about their background and their experienceabout participating in a remote VR study.We took special care to provide clear instructions to participants and support them throughout thestudy process. In our advertisement, we provided a brief overview of what the study would contain.In our recruitment email, we had instructions for downloading and running the VR applicationas well as the high-level steps they should take in the study. Inside the VR game, we had a textbox containing the same instructions, which the users would see immediately after starting theapplication (Figure 3, middle). Further, there was a brief explanation about each health interfacebefore they started the game. After the game, there were instructions to take off the HMD and fillin the questionnaire on their computer, and then return to the game (Figure 3, right). The samereminder was added at the end of the questionnaire, that told participants to return to the game.In addition, we set up a separate issue report form, that prospective participants were encouragedto submit in case they experienced any problems with the game or the study. This issue reportform was added in the advertisements as well as in the emails that contained the download link.
Recruitment.
We did two rounds of recruitment. In the first round, participation was voluntary,i.e., no rewards were offered. We advertised the study through channels we had identified assuitable for this research, such as Reddit. We also advertised the study to a practical course aboutVR technologies and programming that took place at the same time at a local university. Some ofthe students owned a private HMD, and some had been loaned an HMD by the university.In the recruitment call, we included instructions for the study procedure (even though all theinstructions were in the game as well), because the study was designed to be run fully independentlyand no experimenters were present at any point. It was therefore important for participants toknow exactly what to expect in the study. The call also included a direct download link to the gameas well as a link to the issue report form.In the second round, we advertised the study through the same channels with a 10€/12$ reward,paid via PayPal. We removed the download link from recruitment calls and instead set up a separateregistration form where participants provided basic user information as well as an email address.We then sent the study instructions along with the download link to the registered participants.We did this to retain control, so that no one would try to cheat (e.g, by completing the studyseveral times), and that we would not be flooded with too many participants who all needed to becompensated.This way, we recruited 24 participants (18 male, 5 female, 1 undisclosed). Their average age was24 (SD = 7.9). The participants used a variety of different VR devices: HTC Vive Pro (7), Valve Index(7), HTC Vive (6), Oculus Quest (2), Oculus Rift S (1), and Asus Windows Mixed Reality Headset (1). , Vol. 1, No. 1, Article . Publication date: February 2021.
The participants also reported their estimates on how much they play VR games on an averageweek, and how much they play other digital games. With regards the VR gaming, 12 participants(50%) reported that they play 0–5 hours on an average week. Seven (29%) estimated 5–10 hours,four (17%) estimated 10–15 hours, and one participant (4%) estimated that they play 20+ hoursper week. With regards to other digital games, seven participants (29%) estimated that they play0–5 hours a week, eight (33%) estimated 5–10 hours, three (13%) estimated 10-15 hours, two (8%)estimated 15-20 hours, and four (17%) estimated that they play 20+ hours per week.
In addition to collecting data to answer our study-specific researchquestions, we gathered impressions on how the participants felt about the remote study. We askedthem to respond to three statements on a 7-point Likert scale, where 1 = strongly disagree, 4 =neither agree nor disagree, and 7 = strongly agree.Participants were very positive about their experience, strongly agreeing with all three statements(MD = 7). They felt that (1) they were comfortable participating from home, (2) the instructions forhow to run the study independently were clear, and (3) they did not need anyone in the same roomto consult. One participant wrote a comment, commending us for a "smooth remote study".The positive feedback from participants is encouraging. It supports our findings from the onlinesurvey, where participants felt comfortable about the thought of remote participation. We weresomewhat surprised by this. Rather, we expected that the fully independent study procedure mightmake some participants stressed or uncomfortable, but this did not seem to be the case. Thisstrengthens our belief that our procedure was well thought out and our instructions prior to andduring the study were clear (as also reported by the participants). Still, it is possible that the fullyindependent procedure drove away some potential participants in the recruitment phase.In addition to the clarity of instructions, we believe that convenience and familiarity played keyroles in the positive experience. In remote studies, users do not have to spend money and timetraveling to a lab and they do not have to get familiar with new equipment or meet with newpeople. In contrast, users can participate from home. This familiar setting likely provides supportand safety. They also use their own equipment that they are familiar with before the study.
In the first recruitment round, wefaced a major challenge with getting a sufficient number of participants. Logically, this was becausewe did not compensate participants, and we required a SteamVR-compatible HMD, so low-enddevices were not suitable. A total of 10 people participated in the study. Five of them came fromthe practical university course, three came from Reddit, and two from other channels.In the second round, we immediately received 33 registrations overnight, which resulted in ushaving to close the recruitment form. All who registered were sent the instructions, and 15 ofthem participated. Therefore, offering a reward solved the recruitment issues. Had we kept theregistration form open for longer, we would have likely been able to recruit many more participants,despite the pre-requisite of a SteamVR-compatible HMD. A separate registration form also seemedto be an good choice for controlling the number of participants. Direct access to the study couldresult in issues for researchers, who often have a limited budget for rewarding study participants.One of our initial concerns was that we would suffer from a high drop-out rate (participantsquitting mid-study). However, out of the total 25 people who ran our VR application, 20 participants(80%) completed the study in full. Four participants (16%) completed two of the three conditions,and one participant completed only one condition. We see this as a very high completion ratefor an entirely remote and independent study procedure that took approximately 30–40 minutes.We believe that this success was due to our efforts to provide as clear instructions as possible forthe participants to perform independently, and to make the transitions between study phases aseffortless as possible (e.g., moving from the game to a pre-opened, pre-filled questionnaire). , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 17
However, the number of second-round participants (15, of which 12 completed the study infull) was low compared to the number of registered prospective participants (33). It is difficult tosay why so many registrants did not follow up in the end, but our experience would indicate thatresearchers should prepare for this happening.Three people reported issues through our issue report form. Those were of technical nature.Two participants, who both had Valve Index controllers, reported that they were not able to shootusing the triggers, but they were able to shoot by pressing other buttons. One participant reportedthat their pistols were slightly misaligned with their hands. The reported issues were minor, asparticipants were still able to complete the study. Still, such instances are testament to the challengeswith remote studies, as things that researchers typically have control over (e.g., calibration, testing,solving issues on the spot) are outside of their grasp.
Conclusion.
Besides initial recruitment issues (which we overcame later), the study procedureworked very well. We did not identify major pitfalls in our procedure, which we had designedcarefully, keeping in mind that the participants do not have anyone to consult during the experiment,and that they must be motivated to finish on their own. Furthermore, the participants gave verypositive feedback about the remote study.As we could expect based on our survey data, our participants were dominantly male (75%).Although there is a prevalent gender imbalance in HCI studies in general [11], this bias is stillnoticeably strong. The average age of our participants was 24 (SD = 7.9). This is indeed not high,but it is likely no lower than what HCI studies typically have. While we are not aware of anaverage participant age across HCI studies, we do know that study participants are typicallystudents [11], and they likely fall in this range. It is also worth noting that our standard deviationwas relatively high (7.9), with some participants being over 40 years old. With respect to theparticipants’ gaming habits, most were rather moderate with their VR use (0–5 hours or 5–10 hoursper week). Instead, they were more actively gaming on other devices (e.g., PC, consoles). This is notsurprising, considering the potential fatigue in VR gaming as well as the fact that VR gaming is notyet as mature as other forms of digital gaming. In any case, our participants seemed to represent arelatively balanced group of moderate games and hardcore gamers (judging from their play time).
In Case Study 2, we set up a VR environment (Figure 4) where two participants completed collabo-rative tasks. Our research focus was on investigating how the gender of virtual avatars affects theinterpersonal distance (IPD) between people. We measured the distance between the participantsat specific points during the study.We designed a virtual environment in Rec Room and implemented two different collaborativetasks, as well as implemented a sophisticated system to track the distance between participants.Participants switched their avatar genders halfway through the study.In this study, the most critical aspects of the remote VR methodology were: • VR application built within an existing VR platform.
We built a custom VR environ-ment in Rec Room, using the tools provided by the platform. • Distribution through the VR platform.
Because our application was built inside RecRoom, there was no need to download or install an application. Instead, our study wasaccessible through Rec Room. • Remotely guided study method.
A remote experimenter was present during the study torecruit users and to set up and manage the study environment in each session. , Vol. 1, No. 1, Article . Publication date: February 2021.
Fig. 4. Proxemics study (Case Study 2). a: Players playing inside the VR game. b: Range finder feature of RecRoom (used to calculate the distance between players). c: Conducting interviews at the end of the study.
We investigated proxemics in virtual reality using Rec Room. The prox-emics theory studies the interpersonal distance between humans during interaction [13]. Weexplored to what extent this theory applies to virtual reality. In particular, we investigated whetherthe relationship between people (strangers and friends) as well as gender affects proxemics.
Study Design.
We designed a collaborative two-player game in which participants had to solvepuzzles together to unlock rewards (Figure 4a). To understand the influence of the relationshipbetween people on proxemics, we recruited both pairs of friends as well as pairs of strangers. Toadditionally investigate the role of gender, the study consisted of two phases. In the first phase,participants collaborated on the puzzle-solving task with an avatar matching their gender. Inthe second phase, participants were represented by an avatar of the opposite gender. The studyconsisted of two sessions. After the first session participants were required to go back to their ownroom and change their avatar gender and then return to join for the second session of the study.The experimenter was present in the room throughout the study. The experimenter embodiedtheir own avatar in the study space, provided instructions to participants and oversaw the procedure,very much like an experimenter would do during a lab study.The distance between the players was logged using the Rec Room feature "Range Finder" (Figure4b), that can track the distance from a certain point to a player. However, because Rec Room doesnot allow storing such information, we had to implement a workaround. We set up the RangeFinder data on a virtual display in Rec Room and recorded the data using screen capture. We thenused Optical Character Recognition (OCR) to extract the data.After the players finished the game, they filled in a demographics questionnaire and the experi-menter interviewed them inside Rec Room (Figure 4c). Different rooms were designed for eachpurpose so participants played the game in one room and completed the interview in a separateroom. Participants were compensated with 5€ via PayPal.
Recruitment.
We distributed a call for participation via various social media platforms, universitymailing lists, in the Rec Room community on Reddit, and via Facebook. In addition, we reached outto people active in the Rec Room community. In particular, one moderator from the Reddit RecRoom group advertised the study via social media (Facebook). One Rec Room tutor, who hosts atutorial class on circuit and logic design inside the platform, suggested this study to his followers.Furthermore, we asked participants to advertise the study to friends who might be interested. Thisled to additional participants. Finally, we recruited active players in Rec Room on the fly. Thisturned out to be an effective method, since those players were already in Rec Room and were oftenavailable for participation. , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 19
For this study, we struggled to find female participants. This led us to have an imbalancedgender ratio and, therefore, we could not conduct studies between three gender groups (male-male,male-female, female-female) as planned initially. Eventually we had only male-male pairs (15) andmale-female pairs (5). Therefore, we had 40 participants (35 males) with an average age of 23.7years (SD = 7.75). In general, we found the recruitment process to be fairly easy for the Rec Roomstudy. One drawback was a difference in time zones between the experimenter and the participants,and thus the experimenter had to conduct some sessions at unusual hours (e.g., very late at night).
In addition to collecting data to answer our study-specific researchquestions, we gathered impressions on how the participants felt about the remote study. Hence, inthe interview at the end of the study, we asked them open-ended questions about their experience.All 40 participants were overall very positive about the experience. All of them stated that theyfelt comfortable about participation in a remote study. Furthermore, 39 participants stated that theremote study in Rec Room was a novel experience to them; one participant mentioned to havepreviously attended a focus group in Rec Room. We asked them if they faced any challenges inunderstanding instructions or in participating. For all participants, it was an easy experiment andthey did not face any difficulty during the experiment.Similar to case study 1, it is likely that participants felt positive about the experience due to theconvenience (participate from home, no need for travel or other inconveniences) and familiarity(familiar and safe home setting, familiar equipment and platform). The difference to the first studywas that we used Rec Room with a remote experimenter, which also seemed to function verywell from the participants’ perspective. It is notable that Rec Room is a social platform whereparticipants likely had many interactions with other people before. Therefore, interacting with astranger (the remote experimenter as well as the other participants in the stranger condition) wasnot new to them.
In total we recruited 40 participantsin pairs. As many friends play together in Rec Room it was easy to recruit friends in pairs. It wasalso easy to recruit strangers in pairs. For recruitment, the experimenter would generally joinplaying rooms and approach other players about the study. Since each game room consists of manyplayers, both friends and also strangers playing altogether, recruiting was easily possible.Our main challenge was the balanced recruitment of male and female participants. Ultimately, wefailed to do this, as recruiting female-female pairs proved to be difficult. Even though we otherwiseran a very successful remote study, the lack of female participants limited our findings regardinggender differences. While we also faced a bias towards males in the first case study, the issue wasmore clear in this second study. This is likely due to two factors. First, we needed participants toattend the study in pairs; finding two female participants for one session as opposed to just onefemale participant is naturally more difficult. Second, according to your survey the general pool ofHMD owners (and likely Rec Room users) is male-dominated.One important aspect to consider is that the Rec Room community consists of a large number ofusers below the age of 18. Hence, proper verification of the participants’ age should be put in placeupon recruitment. At least, participants should be told that they are required to be at least 18 yearsof age to be eligible to participate. But this also makes it easy for researchers to recruit underageparticipants if required for the study. In this case, researchers would need to find ways of obtainingconsent from participants’ parents.One challenge during recruitment and conducting the study sessions was that potential partic-ipants were spread across the globe. Therefore, some participants were not available at regularworking hours for the experimenter. As a result, the experimenter ran some sessions in the middleto accommodate for different time zones. , Vol. 1, No. 1, Article . Publication date: February 2021.
We planned for position tracking in VR to measure the interpersonal distance of participants.Rec Room allows the position of each player to be visualized; however, logging this data was notpossible. We worked around this by recording the visualization during the study and then usingoptical character recognition to extract the numbers. This worked well in the end, but requiredsome experimenting to get right, so that OCR would produce the correct output. Depending onthe study being conducted, implementation challenges are likely to occur when using existing VRplatforms, as their features can be limited.Another design challenge specific to Rec Room was the limited ink available for designers. Thepen system in Rec Room allows creators to design a room and this has an ink limit to facilitaterobust processing, i.e., the rooms have a limit to how many objects they can contain. Thus, whendesigning the room, we had to plan economically.The study was interrupted a few times when the participant’s game crashed and they left theroom. Additionally, since the experimenter was also virtually present wearing an HMD device, itwas difficult for the experimenter to check their personal notes during the study. After some creative workarounds, our overall study procedure worked verywell. We had no problems during the study sessions, the participants’ were positive about theirexperience, and our data collection methods worked as intended. Recruiting participants turnedout to be easy in several ways. Most importantly, the option to directly recruit participants frominside Rec Room was very helpful.As noted earlier, though, it was critical for our research to also recruit female pairs as partic-ipants, and this was a major challenge. This showcases the limitations in the demographic ofHMD users, although in most studies the issue should not be as pronounced as here. Two minorinconveniences were that some participants were in far-away time zones, and that Rec Room hadlimited data collection features, so we could not record the exact distance between participantsdirectly. Researchers should be prepared for creative workarounds in such cases.
In this section, we synthesize the findings from our online survey, the analysis of different ap-proaches (standalone, app store, VR platform), and two case studies into a framework for remoteVR studies (Figure 5). We discuss the strengths and weaknesses of each approach, and when eachapproach might be appropriate. The framework is meant to assist researchers in planning theirown remote VR studies.
We distinguish four primary approaches to conducting remote VR studies – we refer to these asPaths 1–4. In the following, we present these four approaches and discuss their differences. Inparticular, we discuss whether each path is more suitable for independent studies that participantsrun without an experimenter, or remotely guided studies, where an experimenter is present. Wealso discuss the strengths and weaknesses of these approaches.
In Path 1, which we experimented within case study 1, researchers build a standalone, independent VR application, which participantsdownload and install themselves. The study can then be advertised through various channels. Inour experience, VR-related forums (like VR subreddits in Reddit) worked best in attracting HMDowners. https://rec-room.fandom.com/wiki/Maker_Pen, Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 21 Fig. 5. We envision four high-level procedures to conducting remote VR studies. Paths 1 and 2 are moreideal for studies that participants complete independently, while Paths 3 and 4 are better suited for remotelyguided studies.
PATH 1:
Standalone VR application that participants download and install independently.
PATH 2:
Standalone VR application that is published in an app store.
PATH 3:
VR application that is uploadedto an existing VR platform, which can also be used to recruit participants.
PATH 4:
VR application that is builtusing the built-in tools of an existing VR platform, which can also be used to recruit participants.
The strengths of this approach are that independent applications can have the most extensivefunctionality and the most options for collecting data, because the applications do not need toconform to the limitations and regulations of app stores and VR platforms. In our study, we loggedextensive quantitative and qualitative data with relative ease.Still, this approach might not be ideal for every development platform. For example, installingexternal applications for Android devices requires extra steps, such as turning on developer mode,which not every consumer is willing or even capable of doing. In our case study, this worked well,as participants only needed to download our game package, after which they could immediatelyrun the game through an executable.Researchers should consider whether or not they should distribute the download link openly.In case participants are rewarded, researchers might consider mechanisms to control that not toomany participants take the study (e.g., so that their budget is not exceeded), and to reduce thechances of people "hacking" the system (e.g., by completing the study and claiming the rewardmultiple times). Researchers could, for example, set up a website for pre-registration, and onlyregistered participants would receive the download link. In our case study, this approach workedwell, although it is likely that not all pre-registrants will complete the study.We believe Path 1 to be particularly suitable for independent studies, where participants gothrough the study procedure alone, without consulting an experimenter. In our case study, thisapproach worked extremely well, although the procedure should be planned meticulously. Extracare should be paid to providing clear instructions. In best case scenarios using Path 1, the VRapplication is distributed and advertised, after which the entire study runs on its own.It is certainly possible to run experiments with a remote experimenter through Path 1, but thisrequires additional steps like agreeing on specific time slots with each participant. Depending on thestudy, a remote connection to the VR application would need to be developed for the experimenter,or a connection would need to be established via other means like video conferencing tools (whichwould, in turn, limit how much control the experimenter has and what insights they can gather). , Vol. 1, No. 1, Article . Publication date: February 2021.
In Path 2, researchers builda VR application and publish it in an app store. In some cases, it is possible that the app storeitself attracts participants, but researchers should be prepared to reach out to potential participantsthrough additional channels.The main strengths of this approach are that distributing and installing the application is easierand less error-prone (as opposed to independent installation), and might help in reaching a wideand diverse audience. App stores have been commonly used in other research areas, for example,in mobile interaction [20–22], where large participant numbers were reached. Opportunities fordata collection are still relatively extensive, although somewhat more limited than in Path 1, as theapplication must conform to the rules and regulations of the app store.Based on our analysis of existing app stores as well as own experience with publishing a VR appon Google’s Play Store, there are considerable differences between app stores in terms of how easyit is to publish apps (some require a review by the provider) as well as how much effort and timethey require (for some of them it takes up to 1 month to receive feedback). Researchers shouldinform themselves prior to choosing a particular app store.For the same reasons as Path 1, we believe that Path 2 is more suitable for independent studies;in the best case scenario, studies using Path 2 could also run on their own after being advertised.
In Path 3, researchers build a VRapplication and upload it to an existing VR platform, such as VRChat. The study can be advertisedthrough the platform itself, but also through conventional channels.The main strengths of this approach come from the social aspects of the VR platforms. Theyare built for multiple users to connect to, and interact with, each other. As such, many thingscritical to some users studies are already there or can be implemented with little effort. For example,experimenters and additional users can connect to and interact with the environment withoutadditional effort. In addition, the VR platforms serve as excellent recruitment channels, evidencedby our experience with Rec Room and a prior study in VRChat [33]. Experimenters can lookfor participants within the platform and recruit them on the spot. These strengths make Path 3well-suited for remotely guided studies where an experimenter is present, and perhaps particularlysuitable for studies where multiple participants are needed simultaneously.Publishing custom applications in VR platforms requires the application to follow certain regula-tions, and requires that the application is implemented correctly (e.g., that it uses specific APIs andlibraries). This somewhat limits the features that can be included in the application and what kindsof data can be collected. On the other hand, connecting an experimenter allows them to easilyconduct in-depth observations and interviews, which are more difficult with Paths 1 and 2. Using aVR platform also helps in overcoming compatibility issues.
In Path 4, researchers build the VR "application" directlywithin an existing VR platform, such as Rec Room, using the platform’s built-in tools (like wedid in case study 2). The study can be advertised through the platform itself, but also throughconventional channels.Similar to Path 3, the strengths of this approach are that experimenters can easily connect to thestudy sessions and the platforms serve as excellent recruitment channels. A unique strength ofPath 4 is that the tools offered by the platforms make setting up study environments fast and easy,as opposed to implementing custom VR apps. For example, Rec Room offers a simple user interfacefor building rooms inside the platform, adding and modifying objects, and setting up interactions.Still, the possibilities offered by VR platforms are limited, as they are rarely built for researchpurposes, but for simple social interactions between users. Therefore, when considering whichpath to take, researchers should look into existing VR platforms to see whether any of them offer , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 23 the required functionality. A related limitation is that VR platforms offer limited ways to collect(especially quantitative) data from users. Still, it is often possible to build questionnaires within theplatforms (like we did in Rec Room). Moreover, like in Path 3, connecting an experimenter allowsin-depth observations and interviews to be conducted.
The four approaches balance different needs for user studies. While most approaches can be adaptedto fit different studies one way or another, there are still general considerations that can be used toidentify the best approach. It is also worth noting that for certain VR studies, remote approachesmight not be feasible at all, or they might present significant challenges. Below, we discuss someperspectives for evaluating which remote approach is the most feasible:
In deciding which approach to take with a remoteVR study, researchers can consider the interactive features that their study requires, and investigatewhether existing VR platforms meet their criteria. We believe that Path 4 is generally the easiestpath to take, and, therefore, it might be useful to evaluate whether Path 4 is a feasible option in thefirst place. Path 1 offers the most control over what interactive features can be included, with Paths2 and 3 situating in the middle, being also dependent on the targeted app store or VR platform.
Similar to the required interactive features,researchers should consider the data that they need to collect. Remote studies in general are morelimited than lab studies, and collecting data through specialized hardware such as biosensors,electroencephalography (EEG) and electromyography (EMG) is very challenging. Despite currenttechnology making a greater range of data available (such as eye tracking data), additional hardwarerequirements constrain remote studies. Hence, remote studies are not always possible.Setting specialized hardware requirements aside, more typical data can certainly be collectedremotely. Much like with with interactive features, Path 1 offers the most control over what datacan be logged from the application. Path 2 also offers much control over logged data, althoughsome app stores might impose some limitations. Paths 1 and 2 are therefore well suited particularlyfor gathering quantitative data.Because VR platforms limit data logging – for privacy and safety reasons – Paths 3 and 4 are lessideal for collecting quantitative data. However, some limited options may still be available, andalso some workarounds exist. For example, as demonstrated in case study 2, we were unable to logparticipant positions from Rec Room. We obtained the required data using screen capture and OCRto extract the data. This was a successful workaround, yet required additional effort from us interms and planning and experimentation, and we also had to purchase additional OCR software.In contrast, with qualitative data the advantages may be flipped. Because Paths 3 and 4 arewell suited for studies where an experimenter is present, they are — by extension — suitablefor collecting qualitative data through observations and interviews. Collecting qualitative datathrough Paths 1 and 2 is certainly possible, but may require additional effort. For example, forstandalone applications, the possibility for a remote experimenter to connect to the same sessionas the participant must be implemented, or they must use additional software to connect for thesessions. Video conferencing tools may be helpful here. However, despite their ease of use they stilladd additional steps to the procedure and might not be enough to capture all details in the session.Therefore, studies relying heavily on quantitative data (or the collection of large amounts of data)might benefit more from Paths 1 and 2. For studies with a qualitative focus, Paths 3 and 4 could beconsidered. We again emphasize that these are not set in stone. For example, with Paths 1 and 2,video conferencing tools, screen sharing tools, etc. could be used to connect with participants, butthey are still limited and require extra steps in the procedure. , Vol. 1, No. 1, Article . Publication date: February 2021.
Asalready discussed, independent studies likely support quantitative data, while remotely guidedstudies might support qualitative data. There are also other considerations to these two approaches.For independent (asynchronous) studies, the benefits are that since the actual procedure requiresno resources, the study can run day and night and potentially reach a large number of participants.For certain studies, it is also considerably easier to develop a VR application if one does nothave to worry about functionality surrounding the experimenter. However, due to the lack ofan experimenter, independent studies might be more prone to malfunctions (e.g., bad calibration,hardware issues) and misunderstandings (e.g., not following the procedure correctly), or participantsmight simply be overwhelmed or discouraged by the procedure.In contrast, through a remote connection, experimenters can instruct participants and overseethat the procedure is followed accordingly. Especially, they can tackle any unforeseen events andprovide support if problems occur, and conduct observations and interviews. Remote guidancemight be particularly desirable if the participants need to learn or train new things, if the procedurehas several phases with instructions, or if the study has multiple things running in parallel, likeseveral participants. A case in point is our case study 2, where participants attended the sessions inpairs. This study would have been very difficult to run without an experimenter who was requiredto monitor each session and instruct participants accordingly, especially when considering thelimitations of the used VR platform. The obvious drawback is that an experimenter is then tiedto running each study session, which takes time. Experimenters might also face challenges withdifferent time zones; flexibility with schedules might be needed to reach participants located inother countries.
Here, we briefly reflect on how we believethis framework and the different approaches to remote VR studies might evolve in the long term.It is inevitable that the VR landscape, the tools, and the user base will evolve. However, we haveformulated most of our findings and considerations in such a way that they will still hold despitethe changes that we expect to see in the future.We have only recently arrived at a situation where a notable enough number of people ownhead-mounted displays. But VR technologies and the related tools are still relatively immature, andthe number of people who own HMDs is still marginal to that of mainstream devices. We believethat in the future, remote VR studies in general will become easier, as VR technologies develop anda larger audience becomes equipped and acquainted with head-mounted displays—and possiblyother VR devices in the far future.Through the future advancement of VR technologies, we may eventually have more data col-lection opportunities in remote studies. However, we believe that studies using app stores andespecially studies using social VR platforms will continue to be limited in their data collection.Such limitations are in place for good reasons—to protect the privacy and safety of the users—sothis is unlikely to change. Hence, Path 1 (standalone application + direct download) might benefitthe most from advanced data collection, which we believe is already the most flexible in this regard.Data collection limitations aside, it is reasonable to assume that social VR platforms will continueto develop alongside HMDs, and offer more advanced and comprehensive interactive options aswell as tools for content creators. Therefore, we believe that Paths 3 and 4 (that utilize social VRplatforms) will become a feasible option for a wider range of studies.
In this section, we draw upon our results and experiences, and provide some best practices andlessons learned that we hope help researchers while planning their remote VR studies. , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 25
Even though it is possible to reach a large number of participants remotely compared to a lab study,remote studies may suffer from certain demographic limitations. As we learned from our survey,HMD owners are biased towards young males. This is also evidenced by our two case studies,where the majority of participants were male (75% and 87.5%, respectively).The bias towards male participants was particularly a challenge in our second case study, wherewe initially attempted to investigate gender differences. However, due to the difficulties of findingfemale pairs, we had to exclude them and opt for an investigation between male-male pairs andmixed gender pairs. Hence, we received only limited insight into gender differences.It is worth noting that the bias towards males is not necessarily solely due to the imbalanceamong HMD owners, but also due to the gender imbalance in suitable recruitment platforms (e.g.,Reddit, Rec Room). In any case, researchers looking to utilize HMD owners as study participantsshould be aware of this limitation, particularly if gender balance is important with respect to theinsights of the planned study.As an advantage, however, online recruitment platforms allow pre-selecting participants based ondemographics (e.g selecting participants who own a VR HMD). Recruitment in online VR platformsis easy as experimenters are likely to find available players online.
According to our survey, many home VR setups are smaller than what researchers might be usedto in lab conditions. 29% of respondents reported a small space of up to only 5 𝑚 , while another31% reported a space between 5 and 10 𝑚 . Therefore, it might be challenging to conduct studiesthat require more space. This may be especially true for user studies researching locomotion inVR or collaboration in social VR. At the very least, researchers should be clear about their spacerequirements and implement mechanisms to ensure the safety of remote participants. In case study 2, where we set up our study in Rec Room, an effective way to recruit participants wasto look for them from within Rec Room. Since Rec Room users are already using their HMD andare already within the platform, many of them were available for participation on the spot. Thisindicates that studies utilizing VR platforms (Paths 3 and 4), researchers might want to considerrecruiting participants from within the platform.
VR-related groups on various forums and social media platforms, especially Reddit, were also ideallocations for recruiting participants. However, many such platforms not only have their generalcommunity rules, but many subgroups have their own rules. Researchers should identify groupswhere posting study advertisements is allowed, and contact moderators in cases where the rules donot provide the necessary information.
In case study 1, we set up a separate issue report form that we linked to in the original advertisement.Although we luckily did not run into any major issues, some participants still used it to reportminor things that they had trouble with. Because of the lack of control and knowledge overhow participants run the study, we recommend that researchers set up a similar channel forindependent studies. This is particularly valuable with issues that altogether prevent a participantfrom completing the study, potentially making researchers aware of major issues early on. , Vol. 1, No. 1, Article . Publication date: February 2021.
Potential study participants can be located anywhere in the world and hence they might be in atotally different time zone. In our case study 2, where we used a remote experimenter, this meantthat some studies had to be run during the night to suit the participants’ schedules. Of course,recruiting participants from several continents is not always necessary, but being able to recruit amore diverse set of participants is one of the strengths of remote studies.
In our case study 2, an unforeseen challenge was that it was occasionally difficult for the exper-imenter to check their notes while wearing an HMD. There are several solutions to this. Onepractical consideration is for experimenters to have the notes on their person instead of, e.g., on atable. In the latter case, experimenters may be disoriented when taking off the HMD, or they maybe far away from the table, and may, therefore, need some time to locate their notes. A secondconsideration is that many VR platforms, such as Rec Room, can be used on desktop machines aswell. This might be a more ideal way in some studies for the experimenters to connect from, insteadof using an HMD. A third consideration is that it might be possible to put the experimenter’s noteswithin the virtual world, so that the notes are only visible to them.
VR technologies are associated with health and privacy risks [6, 12, 23, 38], for example, throughcybersickness [12, 15], physical injuries [30]), and effects on mental health [2, 24]. The privacyrisks of VR technology should also not be neglected, as large amounts of sensitive informationmay be involved, while access to and transmission of this data is often not transparently regulated[6, 16]. These risks imply a need for reflections on the ethical consequences of VR studies, sincethey can affect participants. Remote VR studies could even aggravate the moral concerns as theyare less controlled.Ethical concerns must be addressed to ensure safety and security of the participants. For example,the absence of an on-site researcher means no assistance to the participants in case of healthissues. For remote studies, having a remote experimenter present during the study can improve thesafety of the participants as the remote experimenter can monitor the participants and stop thestudy if required. Using internet to download study links or participating online may pose securitythreats to participants. We strongly suggest that researchers make their best effort to minimizesuch threats by using verified and trusted sources, such as official institution websites, to host studyadvertisements, information, and materials (like download links), instead of public sources such aslinks on Google Drive. Researchers can also focus on distributing studies in forums where safetymeasures exist (e.g., new accounts cannot make new posts, a certain trust level must be earned, orthe community can help by marking safe and valid posts with badges or tags).Therefore, we recommend a thoughtful consideration of all health and privacy risks and theirmitigation. Moreover, transparent communication of risks to the participants is necessary to allowfor an informed consent.
We investigated different ways to conduct VR studies remotely, using the participants’ own VRequipment. We first conducted an online survey (N=227) to understand HMD owners demographics,their VR setups and usage, and their willingness to participate in remote VR studies. Second, weanalyzed existing app stores and social VR platforms to assess how suitable they would be tobe utilized in remote VR studies, and what advantages and limitations they might bring. Third, , Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 27 we conducted two case studies to gather experiences from different approaches. Based on thesecontributions, we derived a framework, guiding researchers planning to conduct remote VR studies.Our framework features four plausible paths to conduct remote VR studies. We learned thatremote studies come with unique opportunities, challenges and considerations. The most definingfactors in designing a suitable VR study for each purpose are 1) whether to utilize existing VRplatforms or app stores, or to implement a standalone VR application, and 2) whether the study isrun independently by participants, or guided by a remote experimenter.We also identified best practices and lessons learned. Together with them and the framework,we believe that our work helps researchers to identify the best possible approach for their remoteVR studies, and to anticipate potential pitfalls. As VR technologies keep advancing and more VRdevices find their way into people’s homes, we believe our work to be valuable to a growing numberof researchers and practitioners.
REFERENCES [1] [n.d.]. . https://recroom.com/2018inreview/[2] Frederick Aardema, Kieron O’Connor, Sophie Côté, and Annie Taillon. 2010. Virtual reality induces dissociation andlowers sense of presence in objective reality.
Cyberpsychology, Behavior, and Social Networking
13, 4 (2010), 429–435.https://doi.org/10.1089/cyber.2009.0164[3] Dmitry Alexandrovsky, Susanne Putze, Michael Bonfert, Sebastian Höffner, Pitt Michelmann, Dirk Wenig, RainerMalaka, and Jan David Smeddinck. 2020. Examining Design Choices of Questionnaires in VR User Studies. In
Proceedingsof the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20) . Association forComputing Machinery, New York, NY, USA, 1–21. https://doi.org/10.1145/3313831.3376260[4] Florian Alt, Stefan Schneegass, Alireza Sahami Shirazi, Mariam Hassib, and Andreas Bulling. 2015. Graphical Passwordsin the Wild: Understanding How Users Choose Pictures and Passwords in Image-Based Authentication Schemes.In
Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (Copenhagen, Denmark) (MobileHCI ’15) . Association for Computing Machinery, New York, NY, USA, 316–322.https://doi.org/10.1145/2785830.2785882[5] Morten Sieker Andreasen, Henrik Villemann Nielsen, Simon Ormholt Schrøder, and Jan Stage. 2007. What Happenedto Remote Usability Testing? An Empirical Study of Three Methods. In
Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems (San Jose, California, USA) (CHI ’07) . Association for Computing Machinery, New York,NY, USA, 1405–1414. https://doi.org/10.1145/1240624.1240838[6] Roya Bagheri. 2016. Virtual Reality: The Real Life Consequences.
UC Davis Bus. LJ
17 (2016), 101. https://heinonline.org/HOL/LandingPage?handle=hein.journals/ucdbulj17&div=7[7] Louise Barkhuus and Jennifer A. Rode. 2007. From Mice to Men - 24 Years of Evaluation in CHI. In
Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07) . Association forComputing Machinery, New York, NY, USA, 1. https://doi.org/10.1145/1240624.2180963[8] Evren Bozgeyikli, Andrew Raij, Srinivas Katkoori, and Rajiv Dubey. 2016. Teleport Locomotion Technique for VirtualReality. In
Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (Austin, Texas, USA) (CHI PLAY ’16) . Association for Computing Machinery, New York, NY, USA, 205–216. https://doi.org/10.1145/2967934.2968105[9] Daniel Buschek, Benjamin Bisinger, and Florian Alt. 2018. ResearchIME: A mobile keyboard application for studyingfree typing behaviour in the wild. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems .1–14.[10] Daniel Buschek, Fabian Hartmann, Emanuel Von Zezschwitz, Alexander De Luca, and Florian Alt. 2016. Snapapp:Reducing authentication overhead with a time-constrained fast unlock option. In
Proceedings of the 2016 CHI Conferenceon Human Factors in Computing Systems . 3736–3747.[11] Kelly Caine. 2016.
Local Standards for Sample Size at CHI . Association for Computing Machinery, New York, NY, USA,981–992. https://doi.org/10.1145/2858036.2858498[12] Sue V. G. Cobb, Sarah Nichols, Amanda Ramsey, and John R. Wilson. 1999. Virtual Reality-Induced Symptoms andEffects (VRISE).
Presence: Teleoper. Virtual Environ.
8, 2 (April 1999), 169–186. https://doi.org/10.1162/105474699566152[13] M. Danesi. 2006. Proxemics. In
Encyclopedia of Language & Linguistics (Second Edition) (second edition ed.), KeithBrown (Ed.). Elsevier, Oxford, 241 – 243. https://doi.org/10.1016/B0-08-044854-2/01441-3[14] Nigel Davies, Sarah Clinch, and Florian Alt. 2014. Pervasive displays: understanding the future of digital signage.
Synthesis Lectures on Mobile and Pervasive Computing
8, 1 (2014), 1–128., Vol. 1, No. 1, Article . Publication date: February 2021. [15] Simon Davis, Keith Nesbitt, and Eugene Nalivaiko. 2014. A Systematic Review of Cybersickness. In
Proceedings ofthe 2014 Conference on Interactive Entertainment (Newcastle, NSW, Australia) (IE2014) . Association for ComputingMachinery, New York, NY, USA, 1–9. https://doi.org/10.1145/2677758.2677780[16] Jaybie A. De Guzman, Kanchana Thilakarathna, and Aruna Seneviratne. 2019. Security and Privacy Approaches in MixedReality: A Literature Survey.
ACM Comput. Surv.
52, 6, Article 110 (Oct. 2019), 37 pages. https://doi.org/10.1145/3359626[17] Markus Funk, Florian Müller, Marco Fendrich, Megan Shene, Moritz Kolvenbach, Niclas Dobbertin, Sebastian Günther,and Max Mühlhäuser. 2019. Assessing the Accuracy of Point; Teleport Locomotion with Orientation Indication forVirtual Reality Using Curved Trajectories. In
Proceedings of the 2019 CHI Conference on Human Factors in ComputingSystems (Glasgow, Scotland Uk) (CHI ’19) . Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300377[18] Laura Germine, Ken Nakayama, Bradley C Duchaine, Christopher F Chabris, Garga Chatterjee, and Jeremy B Wilmer.2012. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments.
Psychonomic bulletin & review
19, 5 (2012), 847–857.[19] Mattia Gustarini, Selim Ickin, and Katarzyna Wac. 2013. Evaluation of Challenges in Human Subject Studies “in-the-Wild” Using Subjects’ Personal Smartphones. In
Proceedings of the 2013 ACM Conference on Pervasive and UbiquitousComputing Adjunct Publication (Zurich, Switzerland) (UbiComp ’13 Adjunct) . Association for Computing Machinery,New York, NY, USA, 1447–1456. https://doi.org/10.1145/2494091.2496041[20] Niels Henze and Susanne Boll. 2010. Push the Study to the App Store: Evaluating off-Screen Visualizations for Maps inthe Android Market. In
Proceedings of the 12th International Conference on Human Computer Interaction with MobileDevices and Services (Lisbon, Portugal) (MobileHCI ’10) . Association for Computing Machinery, New York, NY, USA,373–374. https://doi.org/10.1145/1851600.1851671[21] Niels Henze, Martin Pielot, Benjamin Poppinga, Torben Schinke, and Susanne Boll. 2011. My app is an experiment:Experience from user studies in mobile app stores.
International Journal of Mobile Human Computer Interaction (IJMHCI)
3, 4 (2011), 71–91.[22] Niels Henze, Enrico Rukzio, and Susanne Boll. 2011. 100,000,000 Taps: Analysis and Improvement of Touch Performancein the Large. In
Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices andServices (Stockholm, Sweden) (MobileHCI ’11) . Association for Computing Machinery, New York, NY, USA, 133–142.https://doi.org/10.1145/2037373.2037395[23] Marc Herz and Philipp A. Rauschnabel. 2019. Understanding the diffusion of virtual reality glasses: The role of media,fashion and technology.
Technological Forecasting and Social Change
138 (2019), 228 – 242. https://doi.org/10.1016/j.techfore.2018.09.008[24] Juan Luis Higuera-Trujillo, Juan López-Tarruella Maldonado, and Carmen Llinares Millán. 2017. Psychologicaland physiological human responses to simulated and real environments: A comparison between Photographs, 360Panoramas, and Virtual Reality.
Applied Ergonomics
65 (2017), 398–409. https://doi.org/10.1016/j.apergo.2017.05.006[25] E. Hodgson, E. Bachmann, D. Waller, A. Bair, and A. Oberlin. 2012. Virtual reality in the wild: A self-contained andwearable simulation system. In . 157–158.[26] Bernd Huber and Krzysztof Z. Gajos. 2020. Conducting online virtual environment experiments with uncompensated,unsupervised samples.
PLOS ONE
15, 1 (01 2020), 1–17. https://doi.org/10.1371/journal.pone.0227629[27] Wijnand A IJsselsteijn, Yvonne AW de Kort, and Karolien Poels. 2013. The game experience questionnaire.
Eindhoven:Technische Universiteit Eindhoven (2013), 3–9.[28] Xiao Ma, Megan Cackett, Leslie Park, Eric Chien, and Mor Naaman. 2018. Web-Based VR Experiments Powered by theCrowd. In
Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW ’18) . International World WideWeb Conferences Steering Committee, Republic and Canton of Geneva, CHE, 33–43. https://doi.org/10.1145/3178876.3186034[29] Ville Mäkelä, Rivu Radiah, Saleh Alsherif, Mohamed Khamis, Chong Xiao, Lisa Borchert, Albrecht Schmidt, andFlorian Alt. 2020. Virtual Field Studies: Conducting Studies on Public Displays in Virtual Reality. In
Proceedingsof the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20) . ACM, New York, NY, USA. https://doi.org/10.1145/3313831.3376796[30] RJ Menzies, SJ Rogers, AM Phillips, Elodie Chiarovano, C De Waele, Frans AJ Verstraten, and H MacDougall. 2016. Anobjective measure for the visual fidelity of virtual reality and the risks of falls in a virtual environment.
Virtual Reality
20, 3 (2016), 173–181. https://doi.org/10.1007/s10055-016-0288-6[31] Aske Mottelson and Kasper Hornbæk. 2017. Virtual Reality Studies Outside the Laboratory. In
Proceedings of the23rd ACM Symposium on Virtual Reality Software and Technology (Gothenburg, Sweden) (VRST ’17) . Association forComputing Machinery, New York, NY, USA, Article 9, 10 pages. https://doi.org/10.1145/3139131.3139141[32] Fares Moustafa and Anthony Steed. 2018. A Longitudinal Study of Small Group Interaction in Social Virtual Reality. In
Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (Tokyo, Japan) (VRST ’18) . Associationfor Computing Machinery, New York, NY, USA, Article 22, 10 pages. https://doi.org/10.1145/3281505.3281527, Vol. 1, No. 1, Article . Publication date: February 2021. emote VR Studies 29 [33] David Saffo, Caglar Yildirim, Sara Di Bartolomeo, and Cody Dunne. 2020. Crowdsourcing Virtual Reality ExperimentsUsing VRChat. In
Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu,HI, USA) (CHI EA ’20) . Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3382829[34] Alireza Sahami Shirazi, Niels Henze, Tilman Dingler, Martin Pielot, Dominik Weber, and Albrecht Schmidt. 2014.Large-Scale Assessment of Mobile Notifications. In
Proceedings of the SIGCHI Conference on Human Factors in ComputingSystems (Toronto, Ontario, Canada) (CHI ’14) . Association for Computing Machinery, New York, NY, USA, 3055–3064.https://doi.org/10.1145/2556288.2557189[35] Stefan Schneegass, Frank Steimle, Andreas Bulling, Florian Alt, and Albrecht Schmidt. 2014. SmudgeSafe: GeometricImage Transformations for Smudge-Resistant User Authentication. In
Proceedings of the 2014 ACM International JointConference on Pervasive and Ubiquitous Computing (Seattle, Washington) (UbiComp ’14) . Association for ComputingMachinery, New York, NY, USA, 775–786. https://doi.org/10.1145/2632048.2636090[36] Valentin Schwind, Pascal Knierim, Nico Haas, and Niels Henze. 2019. Using presence questionnaires in virtual reality.In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . 1–12.[37] Mel Slater, John McCarthy, and Francesco Maringelli. 1998. The Influence of Body Movement on Subjective Presencein Virtual Environments.
Human Factors
40, 3 (1998), 469–477. https://doi.org/10.1518/001872098779591368arXiv:https://doi.org/10.1518/001872098779591368 PMID: 9849105.[38] James S Spiegel. 2018. The ethics of virtual reality technology: social hazards and public policy recommendations.
Science and engineering ethics
24, 5 (2018), 1537–1550. https://doi.org/10.1007/s11948-017-9979-y[39] A. Steed, S. Frlston, M. M. Lopez, J. Drummond, Y. Pan, and D. Swapp. 2016. An ‘In the Wild’ Experiment on Presenceand Embodiment using Consumer Virtual Reality Equipment.
IEEE Transactions on Visualization and Computer Graphics
22, 4 (2016), 1406–1414.[40] Anthony Steed, Francisco R Ortega, Adam S Williams, Ernst Kruijff, Wolfgang Stuerzlinger, Anil Ufuk Batmaz,Andrea Stevenson Won, Evan Suma Rosenberg, Adalberto L Simeone, and Aleshia Hayes. 2020.
Evaluating immersiveexperiences during Covid-19 and beyond . https://interactions.acm.org/blog/view/evaluating-immersive-experiences-during-covid-19-and-beyond[41] The Against Gravity Team. 2019.
Rec Room in 2018 . https://recroom.com/2018inreview/[42] Martin Usoh, Kevin Arthur, Mary C Whitton, Rui Bastos, Anthony Steed, Mel Slater, and Frederick P Brooks. 1999.Walking > Walking-in-Place > Flying, in Virtual Environments. In
Proceedings of the 26th Annual Conference onComputer Graphics and Interactive Techniques (SIGGRAPH ’99) . ACM Press/Addison-Wesley Publishing Co., USA,359–364. https://doi.org/10.1145/311535.311589[43] FengJun Zhang, GuoZhong Dai, and Xiaolan PENG. 2016. A survey on human-computer interaction in virtual reality.