All Factors Should Matter! Reference Checklist for Describing Research Conditions in Pursuit of Comparable IVR Experiments
Kinga Skorupska, Daniel Cnotkowski, Julia Paluch, Rafa? Mas?yk, Anna Jaskulska, Monika Kornacka, Wies?aw Kope?
AAll Factors Should Matter! Reference Checklistfor Describing Research Conditions in Pursuit ofComparable IVR Experiments (cid:63)
Kinga H. Skorupska , − − − , Daniel Cnotkowski ,Julia Paluch − − − , Rafa(cid:32)l Mas(cid:32)lyk − − − ,Anna Jaskulska − − − , Monika Kornacka − − − ,and Wies(cid:32)law Kope´c , − − − Polish-Japanese Academy of Information Technology [email protected] SWPS University of Social Sciences and Humanities National Information Processing Institute KOBO Association
Abstract.
A significant problem with immersive virtual reality (IVR)experiments is the ability to compare research conditions. VR kits andIVR environments are complex and diverse but researchers from differ-ent fields, e.g. ICT, psychology, or marketing, often neglect to describethem with a level of detail sufficient to situate their research on the IVRlandscape. Careful reporting of these conditions may increase the ap-plicability of research results and their impact on the shared body ofknowledge on HCI and IVR. Based on literature review, our experience,practice and a synthesis of key IVR factors, in this article we present areference checklist for describing research conditions of IVR experiments.Including these in publications will contribute to the comparability ofIVR research and help other researchers decide to what extent reportedresults are relevant to their own research goals. The compiled checklist isa ready-to-use reference tool and takes into account key hardware, soft-ware and human factors as well as diverse factors connected to visual,audio, tactile, and other aspects of interaction.
Keywords: VR · Empirical Studies in HCI · Research practices.
VR has a long history of development on the reality-virtuality continuuum [17]which may include any system that makes use of artificial elements to create avirtual experience, from the Sensorama machine of 1956, through 3D graphicson a single flat screen or immersive CAVE environments (Figure 1) to modernHead Mounted Displays (HMDs).Consumer solutions were deemed mature enough to conduct scientific exper-iments as of 2016 [2] and have made it possible to perform immersive virtual (cid:63)
Supported by KOBO Association and XR Lab at PJAIT. a r X i v : . [ c s . H C ] J a n Skorupska et al.
Fig. 1.
An Immersive Virtual Reality CAVE, where the IVR experience can be sharedwith others and the ”hamster wheel” can simulate covering large distances in VR. reality (IVR) studies on a budget. This created an illusion that the experimentsconducted with IVR are easily comparable. For example, Buschet al. conclude“virtual environments can be an alternative to real environments for user experi-ence studies, when a high presence is achieved.” [7] and as such, they can be usedto simulate real experiences. The excitement of human subjects experiments inIVR environments in some cases overshadowed the human, environmental andtechnological factors which go hand in hand for creating specific research con-ditions. Researchers often omit or ignore some of these factors or at least ne-glect to mention them in their publications, when it comes to IVR experiments.Such omissions have a lasting effect on the quality of reporting, and the conclu-sions others can draw based on them. Andujar and Brunet [1] list issues withVR experiment validation related to ”lack of background on experimentation,psychology and psychophysics, time-consuming nature of human-subject exper-iments, and the difficulties to fulfill all requirements (double-blind experiments,informed consent, representative users, representative datasets/models/tasks, re-producibility)”. These issues are in part related to the knowledge of IVR experi-menters who have to exhibit interdisciplinary competence and take into accountboth human and technological factors. Andujar and Brunet also noticed that theperformance of participants of VR experiments may be related to hardware fac-tors. Continuous development in the field of dimensional tracking, high fidelitydisplays, wireless transmission of data and motion controls, has brought manyimprovements over each new generation of previous VR hardware. Some aspectsmay even be hard to compare across different HMD versions, not only manufac-turers, as the specificity of each of them is reflected in the quality of experiencethey offer. This fast development pace of IVR solutions may cause older researchto become obsolete - old Oculus or first Vive HMDs offered an inferior experience ll Factors Should Matter! 3 to their newest counterparts, but are leaps ahead of Google Cardboard. Thus,the validity of conclusions is connected to the specific setup, participants’ psy-chological features (eg. executive function and attention) and domain. This allcontributed to the significant problem with IVR experiments which is related tothe ability to compare research conditions and, in consequence, the applicabilityof research results and their contribution to the shared body of knowledge onhuman factors and IVR. This is a very important limitation which is difficult toovercome, but it can be mitigated.There are multiple guidelines on good research practices in the EuropeanCode of Conduct for Research Integrity , but such documents are written inbroad strokes and do not cover the specifics of each subfield, especially one ascomplex as the IVR landscape now. Within it, there is diverse research being con-ducted from case studies of creating interactive VR archaeological exhibits [4],to making low-cost VR videos to provide higher level of comfort to pre-surgerypatients [18] or coping with stress and regulating emotions [14]. Researchers ex-plore various aspects of VR, such as the feeling of presence, connected to minutedetails like realistic light, sounds or movements [15]; the sense of agency [3], oreven attitudes towards entering and exiting VR environments [12].This problem is also directly related to the ”reproducibility crisis”, one of thedriving forces of the open science movement. In psychology research, in responseto some of these problems, checklists, such as the one created by Liao et al.[16],started to appear. Software engineering also saw its share of checklists, as in thereview by Wieringa [21]. The same is needed in the area of IVR experiments,which has its own specificity.Therefore, we have decided to draft an IVR checklist that may guide re-searchers to report their research in a more comprehensive way. First of all, Virtual Reality is a very broad concept, and it is important to dis-tinguish Immersive Virtual Reality (IVR), usually entered with a head mounteddisplay (HMD), from Virtual Reality (VR) simulated on a traditional computerscreen, or in a massive VR installation like CAREN (Figure 2).Grouping insights from these areas together is a somewhat common mistakeof novice practitioners caused by the common shorthand of VR for IVR.
Hardware importance can be also seen in a study [8] which compared video-conferencing via Skype and VR-mediated communication in financial servicesbased on surveys filled out by participants after their two meetings, which wererandomly held first either in VR, or via Skype. VR communication achievedsignificantly higher presence and closeness (to the other person) scores, however,participants rated Skype higher, possibly due to the VR gear being uncomfort-able to wear. The code was developed by ALLEA - the European Federation of Academies ofSciences and Humanities and more on this can be found here: https://allea.org/code-of-conduct/ Skorupska et al.
Fig. 2.
The CAREN system - is a large scale immersive installation with a movementsensitive balance platform and a set of diagnostic tools to track body movement.
So, even within IVR proper, entered with a HMD, there exist multiple dif-ferences which rely on plethora of other factors: – Head mounted display: while most of the time HMD’s specification is pub-licly available, when it is not, these properties should be reported: per eyeor combined resolution, native refresh rate, field of view (horizontal and ver-tical), display’s pixels per inch, display’s technology, lenses, eye tracking,recommended user’s IPD or IPD range, HMD’s weight, tracking system. – Controllers: their model information is necessary to assess the ease of use. – Graphics processing unit: Even simple VR applications can give very poorexperience if GPU’s performance is not sufficient to display graphics on twoor even three (with on screen reproduction) high resolution screens withintarget frame rate (e.g. Varjo, Valve Index, Pimax, HTC Vive Pro) – Central processing unit: VR applications tend to be GPU intensive due tocombined resolution that needs to be rendered, CPU performance might bea limiting factor when we incorporate various media streaming methods orwireless HMD solutions (e.g. HTC Vive, HTC Vive Pro). – Media storage: due to asset loading, it can negatively impact research thatrelies on e.g. high resolution textures coupled with movement. – Audio accessories: features like active noise cancellation are absent even inhigh-end VR solutions, so use of external audio solutions should be reported. – Cable management systems and extensions: these should be reported ashardware or software when talking about play space area, especially in re-gards to room scale VR environments.If wireless solution was used, information about latency should be provided. In our research we observed that dynamic IVR experiences tend to be more immer-sive with higher refresh rates, which are not critical for non-dynamic HQ IVREs.ll Factors Should Matter! 5 – External accessories and modifications: Most HMD accessories and modifi-cations alter the factors making up the user experience.While other PC components are important when configuring systems, theirimportance from research perspective is little to none as long as complete sys-tems behave as expected. Given that: motherboard chipset, RAM speed, cableextension properties etc, do not need to be reported unless they were in someway the subject of experiment.
Software plays an important role in creating coherent experiences and exper-iments and key affecting solutions should be reported, even if setup reproductionmay not be possible: – Framework used: Software like Steam VR, Oculus VR app, proprietaryframeworks. These can change HMDs’ and controller capabilities. – GPU drivers: GPU’s are one of the most important pieces in hardware setupfor IVR. Different versions can cause a performance boost or instability. – Application engine: Application engine decision is largely based on compat-ibility, features and team experience. It is important to measure overall per-formance within our application, to allow for better reproducibility withoutneed for code publication. – Boundary system: Most VR frameworks offer some kind of boundary sys-tem, with various levels of customization. Those settings should be noted,especially when exploring immersion-related aspects. – User calibration: With ambiguity regarding if users experience content prop-erly, a set of actions should be taken into consideration when fitting equip-ment to participants. HMD fitting, IPD adjustments, eye tracking calibrationare important from experiment validity perspective. Proper vision and com-fort validation processes should not only be incorporated into experiments,but also reported if unusual or innovative steps were taken.
Environment design is another critical factor impacting user experience andimmersion. While environment reproduction might be impossible in derivativeworks, or when the environment was co-designed in participatory studies [13], itis important convey enough detail to encourage other researchers to refer to ourwork, or to become a source of inspiration for them. Sufficient reporting shouldinclude: – Environment type: Type of environment used - with its features and content- is the most important information. – Interactions: While subset of possible meaningful interactions is defined byenvironment type and hardware used, it is crucial to describe the subset usedin as much detail as possible. Haptic feedback presence and design shouldbe also included in interaction description as it can be highly impactful forcertain user groups [11]. Naturally environment interactivity can be close tonone in e.g. 360 videos.
Skorupska et al. – Locomotion system: If used one of most important and troublesome aspectof bigger than physical space environments. There is plethora of locomotiontechniques to choose from [5], not including in-house solutions. Not only weshould describe how locomotion was solved but also what complementarysystems supported it, such as screen dimming, field of view reductions etc. – Avatar: Usage of certain styles of avatars can increase virtual body owner-ship and greatly impact immersion. Avatar style, body parts’ tracking andsimulation, and customization options necessitate a detailed description. [20] – Control scheme: Control scheme in conjunction with controllers and interac-tions can make up very unique experiences within identical environments. – Graphic fidelity: Graphic fidelity can impact immersion and overall presence,even to the point of increasing emotional response to environments [10]. Itsaspects such as graphical style used, content resolution or model complexityshould be described, or visually best presented in research description. – Sound design: Use of proper sound technology and mixing can lead to moreimmersive environments and achievement of higher presence. [19] – Ethical factors: The ethical factors [6] conveyed by the environment feel(architectural style, presence of minorities, age groups etc) and interactiondesign choices (what you can do with items) may be subject to unconsciousbiases and impact the experience for different groups of users.Finally, there is a wide array of possible factors related to participants .For example, a comparative study on obedience in VR-mediated communica-tion [9] in which 51 participants were divided into two groups (VR and RR)to solve problems and encouraged to change their answers by the researcherwho was with them shows that in the VR setting the participants were moresusceptible to manipulation. This effect appears despite the VR avatar of theresearcher having limited emotive capacity. However, the study did not accountfor the participants’ familiarity and the perceived presence felt in VR. So theremay have been interference caused by lack of experience with VR. Therefore,recruitment strategies and screening for IVR experiments and human factorsare critical. Participants can vary considerably, not only in age, socio-economiccircumstances, executive and attentional functioning, emotional reactivity butalso in how receptive and experienced they are with IVR.Some studies show differences in behaviors in Virtual Reality and Real Real-ity, which can be explained by the extent to which the VR scenarios imitate theRR. Even more differences are found comparing results of one IVR study withanother with a different setup. Without a reference to a detailed checklist of fac-tors affecting these experiments conclusions from one study may not be relevantto other studies as the quality of the IVR experience may differ significantly.
Based on literature review, our experience, practice and a synthesis of key IVRfactors we present a reference checklist for describing research conditions of IVRexperiments. ll Factors Should Matter! 7
In creating this preliminary checklist we propose a structured way to presentIVR papers to include, at least, the following core factors affecting the VRexperience and the results:1.
HARDWARE (cid:3)
VR HMD and controllers used (cid:3)
Information about GPU, CPU, Media storage (cid:3)
Audio sources and inputs (external microphones, headphones with ANC) (cid:3)
Wire extensions, cable management systems, wireless solutions (ceilingmounted cables, wireless adapters) (cid:3)
Non-standard HMD and controller qualities (lens mods, prescription) (cid:3)
Accessories used and modifications (silicone grips, foam replacement)2.
SOFTWARE (cid:3)
Framework used: Steam VR, Oculus VR or other (cid:3)
GPU drivers’ versions (cid:3)
Application engine (cid:3)
Boundary system and customization (cid:3)
User calibration3.
ENVIRONMENT DESIGN (cid:3)
Environment type (cid:3)
Types of interactions (cid:3)
Locomotion solutions (cid:3)
Avatar look and features (cid:3)
Control scheme (cid:3)
Graphic fidelity and style illustrated with example assets (cid:3)
Sound design (cid:3)
Ethical factors and other unique features.4.
PARTICIPANTS (cid:3)
Recruitment procedures, ethics and informed consent (cid:3)
Prior IVR experience and any training received (cid:3)
Participant age group and socioeconomic status (cid:3)
Exclusions (e.g. participants leaving or finishing early) (cid:3)
Glasses/hearing aids (cid:3)
VR sickness inducing predispositions (e.g. travel sickness) (cid:3)
General attitude towards VR (cid:3)
Momentary mood (cid:3)
Use of physiological measures of emotional activation (skin conductance)5.
DISTRACTIONS (cid:3)
Obstacles and problems encountered (cid:3)
Waiting times and mean calibration time per participant (cid:3)
Other people present during the experiment (e.g. “spotter”) (cid:3)
Environmental factors such as wind, smells, temperature and sounds
Skorupska et al.
Relevance of conducted research to other research is of key importance for build-ing a solid foundation of knowledge. For this reason we are excited to presentthis preliminary checklist that takes into account the specificity of IVR experi-ments. We hope that this list will be field-tested and expanded according to thediverse experience of the IVR research community. It is our goal to initiate thediscussion which would result in a more standardized and comprehensive way ofdescribing IVR experiments, so that researchers with all levels of experience andfrom very different disciplines can more easily situate the research conditions ofeach reported IVR experiment on the IVR landscape.
We would like to thank Kobo Association and all transdisciplinary experts in-volved with the HASE research initiative (Human Aspects in Science and En-gineering) including XR Lab at PJAIT, VR Lab at IP PAS, EC Lab at SWPSUniversity and LIT of the NIPI.
References
1. Andujar, C., Brunet, P.: A Critical Analysis of Human-Subject Experiments in Vir-tual Reality and 3D User Interfaces, pp. 79–90. Springer International Publishing,Cham (2015)2. Anthes, C., Garc´ıa-Hern´andez, R.J., Wiedemann, M., Kranzlm¨uller, D.: State ofthe art of virtual reality technology. In: 2016 IEEE Aerospace Conference. pp. 1–19(March 2016). https://doi.org/10.1109/AERO.2016.75006743. Argelaguet, F., Hoyet, L., Trico, M., Lecuyer, A.: The role of interaction in virtualembodiment: Effects of the virtual hand representation. In: 2016 IEEE VirtualReality (VR). pp. 3–10 (March 2016). https://doi.org/10.1109/VR.2016.75046824. Barbieri, L., Bruno, F., Muzzupappa, M.: User-centered design of a vir-tual reality exhibit for archaeological museums. International Journal onInteractive Design and Manufacturing (IJIDeM) (2), 561–571 (6 2018).https://doi.org/10.1007/s12008-017-0414-z5. Boletsis, C.: The new era of virtual reality locomotion: A systematic literaturereview of techniques and a proposed typology. Multimodal Technologies and Inter-action (4) (2017). https://doi.org/10.3390/mti10400246. Brey, P.: The ethics of representation and action in virtual re-ality. Ethics and Information Technology (1), 5–14 (Mar 1999).https://doi.org/10.1023/A:10100699074617. Busch, M., Lorenz, M., Tscheligi, M., Hochleitner, C., Schulz, T.: Being therefor real – presence in real and virtual environments and its relation to usability.In: Proceedings of the NordiCHI 2014: The 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (10 2014)8. Campbell, A., Holz, T., Cosgrove, J., Harlick, M., O’Sullivan, T.: Uses of VirtualReality for Communication in Financial Services: A Case Study on Comparing Dif-ferent Telepresence Interfaces: Virtual Reality Compared to Video Conferencing,pp. 463–481 (01 2020)ll Factors Should Matter! 99. Dzardanova, E., Kasapakis, V., Gavalas, D., Sylaiou, S.: Exploring aspects ofobedience in vr-mediated communication. In: 2019 Eleventh International Con-ference on Quality of Multimedia Experience (QoMEX). pp. 1–3 (June 2019).https://doi.org/10.1109/QoMEX.2019.874319610. Hvass, J., Larsen, O., Vendelbo, K., Nilsson, N., Nordahl, R., Serafin, S.: Visualrealism and presence in a virtual reality game. In: 2017 3DTV Conference: TheTrue Vision (3DTV-CON). pp. 1–4 (2017)11. H˚akonsson, Jakob: Haptic-enhanced presence in vr: Exploring the importance ofhaptic feedback in virtual environments to achieve presence (Jan 2018), http://muep.mau.se/handle/2043/25952
12. Knibbe, J., Schjerlund, J., Petraeus, M., Hornbæk, K.: The dream is collapsing:The experience of exiting vr. In: Proceedings of the 2018 CHI Conference on HumanFactors in Computing Systems. pp. 483:1–483:13. CHI ’18, ACM, New York, NY,USA (2018). https://doi.org/10.1145/3173574.317405713. Kope´c, W., Wichrowski, M., Kalinowski, K., Jaskulska, A., Skorupska, K.,Cnotkowski, D., Tyszka, J., Popieluch, A., Voitenkova, A., Mas(cid:32)lyk, R., Gago, P.,Krzywicki, M., Kornacka, M., Biele, C., Kobyli´nski, P., Kowalski, J., Abramczuk,K., Zdrodowska, A., Pochwatko, G., Mo˙zaryn, J., Marasek, K.: Vr with older adults:Participatory design of a virtual atm training simulation. IFAC-PapersOnLine (19), 277–281 (2019). https://doi.org/10.1016/j.ifacol.2019.12.11014. Kornacka, M., Douilliez, C.: Steer away from rumination? the impact of repetitivenegative thinking on attentional disengagement. International Journal of Psychol-ogy , 632 (2016)15. Lee, M., Kim, K., Daher, S., Raij, A., Schubert, R., Bailenson, J., Welch, G.:The wobbly table: Increased social presence via subtle incidental movement of areal-virtual table (03 2016)16. Liao, Y., Skelton, K., Dunton, G., Bruening, M.: A Systematic Review of Methodsand Procedures Used in Ecological Momentary Assessments of Diet and PhysicalActivity Research in Youth: An Adapted STROBE Checklist for Reporting EMAStudies (CREMAS). Journal of Medical Internet Research (6), e151 (Jun 2016).https://doi.org/10.2196/jmir.4954,
17. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: A classof displays on the reality-virtuality continuum. Telemanipulator and TelepresenceTechnologies (01 1994). https://doi.org/10.1117/12.19732118. O’Sullivan, B., Alam, F., Matava, C.T.: Creating low-cost 360-degree virtual realityvideos for hospitals: A technical paper on the dos and don’ts (2018)19. Poeschl, S., Wall, K., Doering, N.: Integration of spatial sound in immersive virtualenvironments an experimental study on effects of spatial sound on presence. In:2013 IEEE Virtual Reality (VR). pp. 129–130 (2013)20. Waltemate, T., Gall, D., Roth, D., Botsch, M., Latoschik, M.: The Impact ofAvatar Personalization and Immersion on Virtual Body Ownership, Presence, andEmotional Response. IEEE Transactions on Visualization and Computer Graphics PP , 1–1 (2018). https://doi.org/10.1109/TVCG.2018.279462921. Wieringa, R., Condori-Fernandez, N., Daneva, M., Mutschler, B., Pastor, O.:Lessons learned from evaluating a checklist for reporting experimental and ob-servational research. In: Proceedings of the ACM-IEEE international symposiumon Empirical software engineering and measurement - ESEM ’12. p. 157. ACMPress, Lund, Sweden (2012). https://doi.org/10.1145/2372251.2372279,, 1–1 (2018). https://doi.org/10.1109/TVCG.2018.279462921. Wieringa, R., Condori-Fernandez, N., Daneva, M., Mutschler, B., Pastor, O.:Lessons learned from evaluating a checklist for reporting experimental and ob-servational research. In: Proceedings of the ACM-IEEE international symposiumon Empirical software engineering and measurement - ESEM ’12. p. 157. ACMPress, Lund, Sweden (2012). https://doi.org/10.1145/2372251.2372279,