Evaluating User Experiences in Mixed Reality
Dmitry Alexandrovsky, Susanne Putze, Valentin Schwind, Elisa D. Mekler, Jan David Smeddinck, Denise Kahl, Antonio Krüger, Rainer Malaka
EEvaluating User Experiences in Mixed Reality
Dmitry Alexandrovsky ∗ [email protected], University of BremenBremen, Germany Susanne Putze ∗ [email protected], University of BremenBremen, Germany Valentin Schwind
Frankfurt University of AppliedSciencesFrankfurt, Germany
Elisa D. Mekler elisa.mekler@aalto. Aalto UniversityHelsinki, Finnland
Jan David Smeddinck [email protected] UniversityUK
Denise Kahl [email protected], Saarland Informatics CampusSaarbrücken, Germany
Antonio Krüger [email protected], Saarland Informatics CampusSaarbrücken, Germany
Rainer Malaka [email protected], University of BremenGermany
ACM Reference Format:
Dmitry Alexandrovsky, Susanne Putze, Valentin Schwind, Elisa D. Mekler,Jan David Smeddinck, Denise Kahl, Antonio Krüger, and Rainer Malaka.2021. Evaluating User Experiences in Mixed Reality. In
CHI Conference onHuman Factors in Computing Systems Extended Abstracts (CHI ’21 ExtendedAbstracts), May 8–13, 2021, Yokohama, Japan.
ACM, New York, NY, USA,5 pages. https://doi.org/10.1145/3411763.3441337
Recent advances of Mixed Reality (MR) technology have enablednew research methods and interventions across various elds andallow for the design of highly immersive user experiences. By this,Virtual Reality (VR) and Augmented Reality (AR) research havebecome central topics in HCI. To measure these experiences, re-searchers apply a wide range of research methods using objective orsubjective metrics [2]. Objective measures include behavioural met-rics (e.g., gaze direction, movement amplitude), physiological mea-sures, (e.g., EEG, EDA, ECG), and performance measures (e.g., timelogging, success rates). Subjective self-reports through standardizedor custom questionnaires remain a widely applied method for ad-ministering mid- and post-experience measures, such as the sense ofpresence [30] or being embodied using virtual avatars [29]. Alterna-tively, VR o ers a wide range of opportunities for non-obstructiveassessment methods of user experience, like objective measure-ments using biosignals [26, 27], or behavioural measures [32, 36].Many of these measurement methods were adapted from use-casesoutside of MR, in which interactions are often less immersive, andtheir validity of usage in MR experiments has not yet been vali-dated. However, researchers are faced with various challenges and ∗ Both authors contributed equally to this research.Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro t or commercial advantage and that copies bear this notice and the full citationon the rst page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s). CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan © design alternatives when measuring immersive experiences. Thesechallenges become even more diverse when running out-of the labstudies [20, 39]. Measurement methods for VR experience receivedrecently much attention and research has already started to embedquestionnaires in the Virtual Environment (VE) for various appli-cations (e.g.,[14, 23]) as this allows to stay closer to the ongoingexperience while lling out the survey [2, 7, 12, 27, 30]. However,there is a diversity in the interaction methods and practices on howthe assessment procedure is conducted. This diversity in methodsshows that there is no shared agreement on standardized methodsof assessing the experience of being in the VR. Moreover, researchpointed towards a multitude of open questions around method-ological [2, 30], technical [26], social [41], and other challengesthat require a focused investigation [20]. It appears crucial to worktowards a shared agreement on assessment methods of VR userstudies as researchers in the HCI community have to be aware ofbiases that may exist for their research methods of choice. AR re-search strongly orients on the research methods from VR, e.g., usingthe same type of subjective questionnaires. However, there are somecrucial technical di erences that require deliberate considerationsduring the evaluation. In this workshop, we exchange experienceswith research methods in MR (i.e., AR/VR) user studies and examinethe particular challenges of the di erent research methods. By this,our workshop launches a discussion of research methods whichshould lead towards standardizing assessment methods in MR userstudies. The outcomes of the workshop will be aggregated into acollective special issue journal article. Due to its immersive nature and a wide variety in technical setups,MR requires careful deliberation of the assessment methods whenaiming to conduct immersive studies with human subjects. WhileMR allows for the implementation of diverse research settings, thetechnology itself a ects the research results [40]. Research tries tocounteract the disengaging and tedious qualities of (VR) user stud-ies by making the tasks more appealing [42, 43]. The assessmentof User Experience (UX) falls into two categories of subjective and © the authors, 2021. This is the author’sversion of the work. It is posted here foryour personal use. Not for redistribution.The definitive version to be published asnoted above (ACM Reference Format). HI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan Trovato and Tobin, et al. objective metrics [24]. Most research attributes a sense of presence[32] and immersion as the central characteristic of UX in VR. Thereis a variety of standardized questionnaires to assess the presence,c.f., [30]. The major advantage of questionnaires is that they areeasy to administer and generally don’t require modi cations of theVE [32]. However, post-experience questionnaires are not sensitiveto state changes during the ongoing experience [16, 34]. Moreover,the existing scales (on presence) are often long and the items arenot always t well to the experiences. Further, it remains openfor discussion if presence is actually a good candidate to describethe quality of a VR experience since a) it is di cult to measureand b) its relationship with user performance [15, 19, 44] or the -delity [4, 35, 44] of the environment is ambiguous. Particularly whilelooking at applications in the mixed reality using a construct suchas presence requires critical discussion. Yet, post-experience pres-ence questionnaires remain the predominant method applied in theliterature [32, 36]. Surveying UX within the VR experience receivedrecent attention in the literature. Schwind et al. [30] contrasted thescreen-based questionnaires against VR-embedded questionnairesand found that with embedded assessment the subjective responsesin VR are more consistent. In contrast, others have shown thatin-VR questionnaires may lead to inconsistencies [11]. To coun-teract for such inconsistencies, Alexandrovsky et al. [2] presentedimportant usability criteria for in-VR questionnaires. Other toolsthat allow administering questionnaires in VR are the VR Ques-tionnaire Toolkit [7], VRate [28]. Similarly, MRAT [21] is a toolkitfor AR studies. These tools aim for a less-disruptive study owand target problems of context-dependent forgetting [1, 10] due toenvironment change [25] which may bias responses.Several approaches have been proposed for behavioral measuresof UX, including gaze direction [22] responses to social [38], orthreatening events [31], perception of discrepancy between VR andthe physical space [37], or magnitude of postural responses [8].Skarbez et al. point out that behavioral measures are objective, con-temporaneous and non-intrusive and thus, they overcome some ofthe shortcomings of the subjective measures. However, in order totrigger speci c behavioral responses the VE or evaluation procedureof the ongoing study requires speci c manipulations, which are notalways applicable [32]. Highly immersive experiences are expectedto facilitate speci c reaction patterns from the autonomous ner-vous system [6]. Physiological responses provide information aboutspeci c episodes of the experience [3, 16, 18, 26] and allow a betterinterpretation of subjective ratings and task performance [5]. How-ever, these physiological signals are challenging to administer inMR scenarios. For example, assessing brain activity using Electroen-cephalography (EEG) sensor with Head Mounted Displays (HMDs)is cumbersome for both participants and researcher, as they may beuncomfortable to wear together and the electrical signals from theHMD can interfere with the EEG sensors [26]. Although researchhas shown that physiological measures are well applicable, Slaterand Steed argue that physiological measures of presence can onlybe applied in anxious scenarios (e.g., a response to a threat) but thatthey are ine ective in mundane situations [33]. While measuringVR experience in the lab is diverse, measuring becomes even moretechnically and methodology challenging when running out-of thelab studies. Out-of the lab VR studies allow for larger variationsin the settings [20] and require researchers for complex technical solutions [38, 39]. Ma et al. investigated how to enable telemetricweb VR studies and to address the technical obstacles [17].While AR research strongly orients on the research methodsfrom VR (i.e., presence as a quality outcome of an experience),there are some crucial di erences that require deliberate considera-tions. Especially optical see-through AR includes a high degree ofinteraction with the physical reality. Therefore, a strong focus onAR content might be disturbing [13] and a balanced fusion of real-ity with the virtual information is desired which should be ideallyindistinguishable for the users. Therefore, measurement methods ofimmersive technology should account for both AR and VR. Whilea signi cant body of work developed standardized scales for mea-suring presence in VR (c.f., [32]), little research has been done onthe development and adoption of the questionnaires for AR expe-riences. Georgiou and Kyza [9] developed the Augmented RealityImmersion (ARI) questionnaire, which conceptualizes immersion inAR applications on the three levels of engagement , engrossment and total immersion including subscales of interest, usability, emotionalattachment, attention, presence and ow.The presented literature outlines a series of challenges and pos-sible pitfalls HCI faces in the context around measuring UX inimmersive environments. Various toolkits and frameworks existswhich address some of those challenges. However, there is still noagreement on assessing methods for UX in MR applications. Thisworkshop targets at general and speci c problems of UX researchmethods in MR and opens a critical discussion of existing researchmethods aiming to retain valid results when evaluating immersivetechnologies. The objectives of the workshops are nding a com-mon ground of research practices and layout a research agendatowards standardized research methods of MR experiences. The organizers are all experienced researchers in the area of MR,evaluation of immersive experience, and the development of re-search methods. The co-organizers bring multiple perspectivesfrom computer science, interaction design, psychology, and userengagement.
Dmitry Alexandrovky is a nal-year doctoral student at the Dig-ital Media Lab, University of Bremen, Germany. His research inter-ests are immersive interaction, user engagement, and game designresearch. He works on interface designs for questionnaires in VRand developed an in-VR questionnaire toolkit. His research wasawarded with ’Honorable Mentions’ at CHI PLAY conferences. Susanne Putze is a nal-year doctoral student at the Digital MediaLab at the University of Bremen. Her research interests are in HCI,improvement of research work ows, and research communicationmethods. She works on measuring VR experiences using subjectivequestionnaires and physiological signals. Valentin Schwind is professor for human-computer interactionat the Frankfurt University of Applied Sciences. His work exploresimmersive and multimodal user experiences in virtual and aug-mented reality. He is expert in research of quantifying immersionand presence. Valentin has received multiple awards at CHI andother HCI conferences for his research of avatars and virtual char-acters. valuating User Experiences in Mixed Reality CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan
Elisa D. Mekler is an assistant professor at the Aalto UniversityDepartment of Computer Science. Her research interests includethe applications of psychological theories and methods in HCI, aswell as the development and validation of UX questionnaires. Elisa’swork has garnered multiple awards at CHI and CHI PLAY.
Jan David Smeddinck is an assistant professor at Open Lab andthe School of Computing at Newcastle University in the UK. Build-ing on his background in interaction design, serious games, webtechnologies, human computing, machine learning, and visual ef-fects, his research interests include virtual-, mixed- and augmented-reality with a focus on applications in digital health and education.
Denise Kahl is a doctoral student at the German Research Centerfor Arti cial Intelligence (DFKI). In her research she explores therelationship between virtual objects and their physical represen-tations for tangible interaction in optical see-through AugmentedReality. She evaluates AR visualizations by measuring presenceusing subjective questionnaires. Antonio Krüger is the CEO of the German Research Center forArti cial Intelligence (DFKI) and a professor of computer scienceat Saarland University heading the Ubiquitous Media TechnologyLab (UMTL). He is an internationally renowned expert on human-machine interaction and arti cial intelligence. His research focuseson Mobile and Ubiquitous Spatial Assistance Systems, combiningthe research areas Intelligent User Interfaces, User Modeling, Cog-nitive Sciences and Ubiquitous Computing. Rainer Malaka is professor for Digital Media at the University ofBremen. He is managing Director of the Center for Computing Tech-nologies (Technologiezentrums Informatik und Informationstech-nik, TZI) and Director of the PhD program Empowering DigitalMedia that is funded by the Klaus Tschira foundation. His researchfocus is on multimodal interaction in MR, language understanding,entertainment computing, and arti cial intelligence. Rainer is coun-cillor of IFIP (International Federation for Information Processing)and chair of IFIP’s technical committee on Entertainment Comput-ing. He has an extensive experience in VR research and evaluationof VR applications from various research projects, including H2020s" rst stage". To advertise the workshop, we will make our workshop website(http://evaluating-mr-ws.com/ ) available upon the workshop ac-ceptance, which features organizational aspects such as a Call forParticipation, information about organizers, paper submission in-structions and a workshop agenda, as well as later all contributions,presentations, and discussion outcomes (included the annotatedMiro boards) of the workshop. We plan to broadly advertise our Call for Participation via distribu-tion lists and on social media (e.g., Twitter, Facebook). Meanwhile,we will also send personal invitations to potential researchers andpractitioners from our network in the research community. The sub-mission of workshop papers will be handled through a conferencemanagement system. All submitted workshop papers will be re-viewed and selected by the workshop organizers (juried selection). URL will change after acceptance
We will share accepted workshop papers with the participants inadvance of the conference. Participants are encouraged to publisha pre-print of their work, e.g., on arXiv or OSF.
The workshop is planned as a single day workshop. The scheduleconsists of a mixture of prerecorded talks by the participants as wellas active discussions and a breakout session. We expect around 15-20 participants, where 20 is the maximum. This group size allowson the one side for a versatile perspective on research methodsand their challenges, and on the other side it enables intensivediscussions with active participation of all participants. Preliminaryschedule (CET):
Welcome (15:00 – 15:15): Opening presentation to outline theworkshop motivation and goals.
Paper session 1 (15:15 – 16:30): Current challenges and barriersof measuring UX in MR Co ee break (10 min) Paper session 2 (16:40 – 17:55): Future directions of measuringmethods for UX in MR We will have two paper sessions with max.10 participants in each session. Each paper session is split up intotwo blocks of about ve papers. In the blocks we will show thepre-recorded video presentations, including a short introductionof the presenter. The blocks end with an open discussion on thesession’s topic. To engage the participants into the discussion, wewill prepare ice breaking questions. Co ee break (15 min) Breakout session (18:10 – 18:50): In the breakout session, allparticipants will discuss in small groups of 3-4 people for 20 min.The groups will be assigned in advance according to the paper topics.During the breakout session, the participants should brainstormand aggregate their discussions in mind maps and charts. After that,the groups will present their outcomes to the workshop and openthe discussion.
Closing and wrap up (18:50 – 19:00): Workshop results, includ-ing best practices, and experiences from the eld trip will be docu-mented. Remaining open questions will be wrapped up, follow-upactivities will be discussed.We will end the workshop day with a virtual social event in a onAltspace, e.g., a basketball tournament. To facilitate the discussionin the breakout session and on the paper presentation videos as wellas to capture their outcomes we will deploy collaborative onlineMiro boards which allows for collaborative discussions remotely,and persist discussion results. To incorporate both participants remotely, we will base the meetingon an online platform (Zoom or in line with the CHI 2021 centralizedcommunication system) which allows for showing pre-recordedpresentation videos including presenter introductions, discussionsas well as breakout sessions for a subset of the participants. Toensure accessibility, we asked all participants to pre-record andsubtitle their presentation, and upload them before the workshop.This allows all participants to watch the presentations regardless ofbad connectivity. To support the discussion in the breakout sessionand on the papers, we will deploy collaborative Miro boards which
HI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan Trovato and Tobin, et al. allows for collaborative discussions remotely. For interactive butdistance socializing we will organize a virtual get-together at theend of the workshop day. Further, we will provide a chat platform(Slack, or other platforms in line with the CHI 2021 centralizedcommunication system) for communications during co ee breaks,as well as before and after the workshop. We expect our workshop to • Connect a community of researchers and practitioners ininvestigating the challenges and opportunities of MR mea-surement and evaluation methods of UX. • Identify current challenges and barriers of MR evaluationmethods. • Outline guidance for research methods and interaction de-sign for MR user studies.To ensure these outcomes of the workshop, we will: • Summarize the workshop outcomes and share all presentedmaterials on the workshop website. • The outcomes will be aggregated on the workshop website. • Disseminate the workshop results to the Human-ComputerInteraction (HCI) community in form of a collective journalarticle co-authored by the organizers and the participants.
To measure MR experiences, researchers apply a range of researchmethods using objective (e.g., biosignals, logging, behavioural), andsubjective (e.g., questionnaires) metrics. However, the assessmentmethods are heterogeneous and miss consistency among the userstudies which impedes transferability of the results.This one-day virtual CHI2021 workshop will focus on commonpractices of evaluation methods and their methodological, technical,and design challenges. We invite researchers and practitioners fromall sub elds of HCI to drive the research agenda of the researchpractices, technologies, and challenges of MR user studies. Thisworkshop invites submissions of position papers (2-4 pages exclud-ing references according to the (single column) ACM Templates),covering but not limited to the following topics: • Measurement methods (behavioural, objective, subjective)for single- or multi-user MR • Technical challenges/solutions/artifacts for assessment meth-ods in and outside the lab. E.g., interaction for in-VR mea-surements, use of biosignals, assessing behavioral measures • Experimenter-participant communication (e.g., telepresence,avatarization)Submissions will be selected by the workshop organizers based onthe relevance to the workshop topic and their potential to engenderinsightful discussions at the workshop. At the workshop, acceptedpapers will have a 3-4 minutes video presentation. At least oneauthor of the accepted paper must attend the virtual workshop. Allparticipants must register for both the workshop and for at leastone day of the conference.
Important Dates • Submission Deadline: February 21st, 2021 at 12pm PT • Noti cation: TBA • Workshop Day: May 7t/h8th/9th, 2021, virtualFor submission and further information, please visit: http://evaluating-mr-ws.com/
REFERENCES [1] Ethel Mary Abernethy. 1940. The E ect of Changed Environmental ConditionsUpon the Results of College Examinations. The Journal of Psychology
10, 2 (Oct.1940), 293–301. https://doi.org/10.1080/00223980.1940.9917005[2] Dmitry Alexandrovsky, Susanne Putze, Michael Bonfert, Sebastian Hö ner, PittMichelmann, Dirk Wenig, Rainer Malaka, and Jan D. Smeddinck. 2020. ExaminingDesign Choices of Questionnaires in VR User Studies. In Proceedings of the 2020CHI Conference on Human Factors in Computing Systems (CHI ’20) . Honolulu, HI,USA. https://doi.org/10.1145/3313831.3376260[3] Chris Bateman and Lennart E. Nacke. 2010. The Neurobiology of Play. In
Proceed-ings of the International Academic Conference on the Future of Game Design andTechnology - Futureplay ’10 . ACM Press, Vancouver, British Columbia, Canada, 1.https://doi.org/10.1145/1920778.1920780[4] Doug A. Bowman and Ryan P. McMahan. 2007. Virtual Reality: How MuchImmersion Is Enough?
Computer
40, 7 (July 2007), 36–43. https://doi.org/10.1109/MC.2007.257[5] Andrea Brogni, Vinoba Vinayagamoorthy, Anthony Steed, and Mel Slater. 2007.Responses of Participants during an Immersive Virtual Environment Experience.
IJVR ect, Physiology, and Presence. In Proceedings of the Fifth Annual InternationalWorkshop on Presence, Porto, Portugal .[7] Martin Feick, Niko Kleer, Anthony Tang, and Krueger Antonio. 2020. TheVirtual Reality Questionnaire Toolkit. In
The 33st Annual ACM Symposiumon User Interface Software and Technology Adjunct Proceedings (UIST ’20 Ad-junct) . Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3379350.3416188[8] Jonathan Freeman, S. E. Avons, Ray Meddis, Don E. Pearson, and Wijnand IJs-selsteijn. 2000. Using Behavioral Realism to Estimate Presence: A Study of theUtility of Postural Responses to Motion Stimuli.
Presence: Teleoperators andVirtual Environments
9, 2 (April 2000), 149–164. https://doi.org/10/drkrr8[9] Yiannis Georgiou and Eleni A Kyza. 2017. The Development and Validation of theARI Questionnaire: An Instrument for Measuring Immersion in Location-BasedAugmented Reality Settings.
International Journal of Human-Computer Studies
98 (2017), 24–37. https://doi.org/10.1016/j.ijhcs.2016.09.014[10] D. R. Godden and A. D. Baddeley. 1975. Context-Dependent MemoryIn TwoNatural Environments: On Land and Underwater.
British Journal of Psychology
66, 3 (Aug. 1975), 325–331. https://doi.org/10/cjgxhp[11] Sarah Graf and Valentin Schwind. 2020. Inconsistencies of Presence Question-naires in Virtual Reality. In . Virtual Event, Canada. https://doi.org/10.1145/3385956.3422105[12] Jason Jerald. 2016.
The VR Book: Human-Centered Design for Virtual Reality ( rst edition ed.). Number 8 in ACM Books. acm, Association for ComputingMachinery, New York.[13] S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S. Feiner, T. Hollerer, and S.Sestito. 2000. Information Filtering for Mobile Augmented Reality. In ProceedingsIEEE and ACM International Symposium on Augmented Reality (ISAR 2000) . IEEE,Munich, Germany, 3–11. https://doi.org/10.1109/isar.2000.880917[14] HyeongYeop Kang, Geonsun Lee, Seongsu Kwon, Ohung Kwon, Seongpil Kim, andJungHyun Han. 2018. Flotation Simulation in a Cable-Driven Virtual Environment– A Study with Parasailing. In
Proceedings of the 2018 CHI Conference on HumanFactors in Computing Systems (CHI ’18) . ACM, New York, NY, USA, 632:1–632:11.https://doi.org/10.1145/3173574.3174206[15] Tina Kjær, Christo er B. Lillelund, Mie Moth-Poulsen, Niels C. Nilsson, RolfNordahl, and Stefania Sera n. 2017. Can You Cut It?: An Exploration of theE ects of Editing in Cinematic Virtual Reality. ACM Press, 1–4. https://doi.org/10.1145/3139131.3139166[16] Benny Liebold, Michael Brill, Daniel Pietschmann, Frank Schwab, and PeterOhler. 2017. Continuous Measurement of Breaks in Presence: Psychophysiologyand Orienting Responses. Media Psychology
20, 3 (July 2017), 477–501. https://doi.org/10/gf7c8f[17] Xiao Ma, Megan Cackett, Leslie Park, Eric Chien, and Mor Naaman. 2018. Web-Based VR Experiments Powered by the Crowd. In
Proceedings of the 2018 WorldWide Web Conference on World Wide Web - WWW ’18 . ACM Press, Lyon, France,33–43. https://doi.org/10.1145/3178876.3186034 valuating User Experiences in Mixed Reality CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan [18] Regan Lee Mandryk. 2004. Objectively Evaluating Entertainment Technology.In
Extended Abstracts of the 2004 Conference on Human Factors and ComputingSystems - CHI ’04 . ACM Press, Vienna, Austria, 1057. https://doi.org/10.1145/985921.985977[19] David Modjeska and John Waterworth. 2000. E ects of Desktop 3D WorldDesign on User Navigation and Search Performance. In . 215–220. https://doi.org/10/fkwwn4[20] Aske Mottelson and Kasper Hornbæk. 2017. Virtual Reality Studies Outside theLaboratory. ACM Press, 1–10. https://doi.org/10.1145/3139131.3139141[21] Michael Nebeling, Maximilian Speicher, Xizi Wang, Shwetha Rajaram, Brian D.Hall, Zijian Xie, Alexander R. E. Raistrick, Michelle Aebersold, Edward G. Happ,Jiayin Wang, Yanan Sun, Lotus Zhang, Leah E. Ramsier, and Rhea Kulkarni.2020. MRAT: The Mixed Reality Analytics Toolkit. In Proceedings of the 2020 CHIConference on Human Factors in Computing Systems . ACM, Honolulu HI USA,1–12. https://doi.org/10/ghbdtp[22] Joshua Newn, Eduardo Velloso, Fraser Allison, Yomna Abdelrahman, and FrankVetere. 2017. Evaluating Real-Time Gaze Representations to Infer Intentions inCompetitive Turn-Based Strategy Games. ACM Press, 541–552. https://doi.org/10.1145/3116595.3116624[23] Sebastian Oberdörfer, David Heidrich, and Marc Erich Latoschik. 2019. Usabilityof Gami ed Knowledge Learning in VR and Desktop-3D. In Proceedings of the2019 CHI Conference on Human Factors in Computing Systems (CHI ’19) . ACM,New York, NY, USA, 175:1–175:13. https://doi.org/10/gf3k86[24] Heather L. O’Brien and Elaine G. Toms. 2008. What Is User Engagement? AConceptual Framework for De ning User Engagement with Technology. Journalof the American Society for Information Science and Technology
59, 6 (2008), 938–955. https://doi.org/10.1002/asi.20801[25] Rüdiger Pohl and Rüdiger F. Pohl. 2004.
Cognitive Illusions: A Handbook onFallacies and Biases in Thinking, Judgement and Memory . Psychology Press.[26] Felix Putze. 2019. Methods and Tools for Using BCI with Augmented and VirtualReality. In
Brain Art: Brain-Computer Interfaces for Artistic Expression , AntonNijholt (Ed.). Springer International Publishing, Cham, 433–446. https://doi.org/10.1007/978-3-030-14323-7_16[27] Susanne Putze, Dmitry Alexandrovsky, Felix Putze, Sebastian Hö ner, Jan D.Smeddinck, and Rainer Malaka. 2020. Breaking The Experience: E ects ofQuestionnaires in VR User Studies. In Proceedings of the 2020 CHI Confer-ence on Human Factors in Computing Systems (CHI ’20) . Honolulu, HI, USA.https://doi.org/10.1145/3313831.3376144[28] Georg Regal, Jan-Niklas Voigt-Antons, Steven Schmidt, Johann Schrammel, TanjaKoji ć , Manfred Tscheligi, and Sebastian Möller. 2019. Questionnaires Embeddedin Virtual Environments: Reliability and Positioning of Rating Scales in VirtualEnvironments. Quality and User Experience
4, 1 (Dec. 2019), 5. https://doi.org/10.1007/s41233-019-0029-1[29] Daniel Roth and Marc Erich Latoschik. 2020. Construction of the Virtual Embod-iment Questionnaire (VEQ).
IEEE Transactions on Visualization and ComputerGraphics (2020), 1–1. https://doi.org/10.1109/tvcg.2020.3023603[30] Valentin Schwind, Pascal Knierim, Nico Haas, and Niels Henze. 2019. Using Pres-ence Questionnaires in Virtual Reality. In
Proceedings of the 2019 CHI Conferenceon Human Factors in Computing Systems (CHI ’19) . ACM, New York, NY, USA,360:1–360:12. https://doi.org/10/gfz8sr[31] Thomas B. Sheridan. 1992. Musings on Telepresence and Virtual Presence.
Pres-ence: Teleoperators and Virtual Environments
1, 1 (Jan. 1992), 120–126. https://doi.org/10/gdcftg[32] Richard Skarbez, Frederick P. Brooks, Jr., and Mary C. Whitton. 2017. A Survey ofPresence and Related Concepts.
ACM Comput. Surv.
Annual International Workshop On Presence ,Vol. 6. 4.[35] Mel Slater, Pankaj Khanna, Jesper Mortensen, and Insu Yu. 2009. Visual Real-ism Enhances Realistic Response in an Immersive Virtual Environment.
IEEEComputer Graphics and Applications
29, 3 (May 2009), 76–84. https://doi.org/10/d6vn2k[36] Mel Slater and Anthony Steed. 2000. A Virtual Presence Counter.
Presence:Teleoperators and Virtual Environments
9, 5 (2000), 413–434. https://doi.org/10/cd5kxg arXiv:https://doi.org/10.1162/105474600566925[37] Mel Slater, Martin Usoh, and Yiorgos Chrysanthou. 1995. The In uence ofDynamic Shadows on Presence in Immersive Virtual Environments. In VirtualEnvironments ’95 (Eurographics) , Martin Göbel (Ed.). Springer Vienna, 8–21.[38] Anthony Steed, Sebastian Frlston, Maria Murcia Lopez, Jason Drummond, YePan, and David Swapp. 2016. An ‘In the Wild’ Experiment on Presence andEmbodiment Using Consumer Virtual Reality Equipment.
IEEE Transactionson Visualization and Computer Graphics
22, 4 (April 2016), 1406–1414. https://doi.org/10.1109/tvcg.2016.2518135 [39] Anthony Steed, Francisco R. Ortega, Adam S. Williams, Ernst Kruij , WolfgangStuerzlinger, Anil Ufuk Batmaz, Andrea Stevenson Won, Evan Suma Rosen-berg, Adalberto L. Simeone, and Aleshia Hayes. 2020. Evaluating ImmersiveExperiences during Covid-19 and Beyond. Interactions
27, 4 (July 2020), 62–67.https://doi.org/10/gh pq[40] Alexandra Voit, Sven Mayer, Valentin Schwind, and Niels Henze. 2019. On-line, VR, AR, Lab, and In-Situ: Comparison of Research Methods to Evalu-ate Smart Artifacts. In Proceedings of the 2019 CHI Conference on Human Fac-tors in Computing Systems (CHI ’19) . ACM, New York, NY, USA, 507:1–507:12.https://doi.org/10.1145/3290605.3300737[41] Nadine Wagener and Johannes Schöning. 2020. Symmetric Evaluation: An Evalua-tion Protocol for Social VR Experiences. In
Workshop on Social VR: A New Mediumfor Remote Communication Collaboration at CHI ’20, April 26, 2020, Honolulu, HI,USA. [42] Xinghui Yan, Katy Madier, Sun Young Park, and Mark Newman. 2019. TowardsLow-Burden In-Situ Self-Reporting: A Design Space Exploration. In