A Survey on Synchronous Augmented, Virtual and Mixed Reality Remote Collaboration Systems
AA Survey on Synchronous Augmented, Virtual and MixedReality Remote Collaboration Systems
ALEXANDER SCHÄFER,
TU Kaiserslautern
GERD REIS,
German Research Center for Artificial Intelligence
DIDIER STRICKER,
German Research Center for Artificial Intelligence, TU KaiserslauternRemote collaboration systems have become increasingly important in today’s society, especially duringtimes where physical distancing is advised. Industry, research and individuals face the challenging task ofcollaborating and networking over long distances. While video and teleconferencing are already widespread,collaboration systems in augmented, virtual, and mixed reality are still a niche technology. We provide anoverview of recent developments of synchronous remote collaboration systems and create a taxonomy bydividing them into three main components that form such systems:
Environment , Avatars , and
Interaction .A thorough overview of existing systems is given, categorising their main contributions in order to helpresearchers working in different fields by providing concise information about specific topics such as avatars,virtual environment, visualisation styles and interaction. The focus of this work is clearly on synchronisedcollaboration from a distance. A total of 82 unique systems for remote collaboration are discussed, includingmore than 100 publications and 25 commercial systems.CCS Concepts: •
General and reference → Surveys and overviews ; •
Human-centered computing → Computer supported cooperative work ; Mixed / augmented reality ; Virtual reality ; Collaborativeinteraction .Additional Key Words and Phrases: virtual reality, augmented reality, mixed reality, collaboration, remoteassistance, distant cooperation, literature review
Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR) technologies are becomingmore mature and open new ways for remote collaboration. Video and teleconferencing systemsare already in extensive use in today’s society, enabling a focus on more novel alternatives whichutilize virtual, augmented and mixed reality technology.Systems for remote collaboration in AR, VR and MR are developing slowly but steadily, yet theyare not well established. Such a system consists of many different facets and requires expertisein various fields such as 3D modeling, animation, development of interactive systems, creationof avatars and dynamic content, multi-user communication and more. In addition, mixed realitysystems are often implemented using a combination of AR and VR hardware, which requires acertain expertise in both technologies.In this paper we give a thorough summary and discussion of synchronous remote collaborationsystems which utilize AR/VR/MR technology. The importance of such systems was emphasizedduring the COVID-19 outbreak in December 2019. People all over the world were put into quaran-tine, cities and local communities forbid traveling and even stepping outside. During this time, thescientific community was forced to find novel ways to network and communicate. Most scientificconferences where either cancelled or held completely virtual, some even using immersive 3Dexperiences utilizing VR HMD’s. The IEEE VR 2020 conference as an example, used virtual roomswhere conference participants could join and interact with each other. Paper and poster presenta-tions where done within a virtual environment which was streamed online for a broader audience.Although video and teleconferencing systems in particular experienced a significantly increaseduse as a result of this global crisis, the AR, VR and MR community received a major awarenesspush as well. a r X i v : . [ c s . H C ] F e b chäfer et al. A well sophisticated AR/VR/MR system could help to reduce travel costs, office space, time,and carbon emissions by creating shared immersive spaces with believable person embodimentand interaction. To compete with each other in this crisis, many companies and researchers haverecently invested in creating novel systems, which makes a recent review of existing systems andresearch even more interesting.We identify and classify the individual parts which are necessary for a remote collaborationsystem and provide an overview of existing systems for research and professional work. Remotecollaboration systems utilizing AR/VR/MR technology are used in many different fields such ashuman computer interaction, computer graphics, medicine, training, cognitive sciences and manymore. We define three components that each remote collaboration system needs to implement,namely
Environment , Avatars and
Interaction . A detailed explanation about this taxonomy is de-scribed in section 2.4. By providing condensed information on certain key topics, we want to helpresearchers assess the state of the art in a particular subject. As an example, a researcher focused oninteraction in multi-user collaborative environments will be intersted in inspecting Table 3 whichcategorises important works in regard to interaction types which where found during the survey.A researcher focused on novel environments for remote collaboration can use Table 1 which liststhe discussed work with respect to their technology, use case, visualisation style and the stimulatedsensory inputs. The representation of other users, in regard to their visualisation and animationstyle is shown in Table 2.
Related work was conducted by Phon et al. [62] which reviewed the state of the art in collaborativeAR systems with focus on education in 2014. This work has a clear focus on collaborative learningin AR and does not differentiate between remote and local collaboration experiences. Anothersurvey was done by Wang et al [113] with focus on AR and MR based collaborative design inmanufacturing. Ens et al. [11] review published work in collaboration through mixed reality upto the year 2018. Although the focus of mentioned work is not remote collaboration explicitely,the authors differentiate between remote and physically co-located systems. Another Survey wasconducted by de Belen et al. [8] in which the authors provide a systematic review of collaborativemixed reality technologies.
In our work we provide a concise focus on synchronous remote collaboration systems and wecategorise the results to assist scientists in different fields to cover their specific research interest.We emphasize the term remote , which means that physical co-location of users is not requiredand the term synchronous which allows users to collaborate in real-time. The focus is on virtual,augmented and mixed reality systems. Traditional video and teleconferencing systems are omitted.We define a remote collaboration system as a way of communicating, interacting and sharinga space beyond the boundaries of physical space exclusively through technological channelswith distributed users. We include work which uses or implements a combination of AR/VR/MRtechnology and synchronous remote collaboration. Systems which allow multiple users in a systembut require users to be in physical co-location are excluded. Exceptions are systems which whereused in a physical co-location scenario but could easily be extended for remote collaborationpurposes. Asynchronous systems that do not allow real-time communication between users arealso excluded. Survey on Synchronous AR, VR and MR Rem. Collab. Systems
The survey was conducted through an iterative process by integrating the most relevant papers first,identifying specific similarities and differences with subsequent categorisation. By incrementallyadding new relevant research work we evolved the categorisation process and therefore separatedthe relevant work into three main contribution categories:
Environment , Interaction and
Avatars remote , collaboration , social and more (as shown in Figure 1). Fig. 1. The used search methodology.
The tables in the following chapters summarise works from the same author if it is a continuationor extension of the previous work. Furthermore, the focus is on the general implementation of theproposed systems rather than on their specific research questions.
In this section we briefly introduce how we perceive AR, VR and MR systems. In general, AR isachieved with two approaches:
Video-Seethrough and
Optical-Seethrough . In both cases, the realworld is augmented to the user. In Video-Seethrough the world is captured with a camera andvirtual objects are placed onto the captured images. In Optical-Seethrough systems, users perceivethe outside world with their own eyes through a transparent projection surface which displaysthe AR content. Regarding VR, most literature couples the term VR with a Head Mounted Display chäfer et al. (HMD) which is placed on the head of a user. In this survey we also consider systems withoutHMD’s as VR, independent of the specific display device, as long as it is possible to immerse usersinto a virtual 3D environment. While AR and VR systems are often quite clear in their separation,the distinction between MR systems often leads to confusion. For this survey, we use Milgram etal.’s [53] definition of the Reality-Virtuality continuum which describes mixed reality as the areawhere both, the real and virtual world are mixed (see Figure 2). Fig. 2. The Reality-Virtuality continuum according to Milgram et al. [53].
More precisely, a system is referred to as MR in this survey if at least one of the following pointsapplies:(1) There is a mix betweeen AR and VR hardware (this also includes projector based systems)(2) Real world objects are used for interaction with either AR or VR hardwareAlthough the recently coined term Extended Reality XR is very popular, we do not use it in thiscontext because we explicitly exclude "pure reality" and the remaining parts are covered by theterms AR, VR and MR.
We create a taxonomy and categorise the relevant work in a logical manner. Many systems havedifferent aspects of novelty which cannot be described by assigning them to a specific category.E.g. one system might excel in the novelty of avatars while another introduced a new kind ofinteraction technique for remote collaboration. One goal of this survey is to help researchers froma wide range of fields who are interested in the area of remote collaboration systems which utilizeAR, VR and MR technology. To illustrate this: A researcher who is interested in the topic remotecollaboration using AR/VR/MR might ask "How are users represented in virtual environments?","What kind of interaction is possible in a shared virtual space?" or "Are there collaboration systemswhich enable shared gaze awareness?". With our survey we want to provide condensed informationto different research questions such as virtual representation of users, different types of interactionand the virtual environment. To achieve this, we elaborated a concept that enables the possibilityto view each of these systems from different viewpoints:
Environment , Avatars and
Interaction (see Figure 3) which we call the three pillars of remote collaboration systems . In the nextchapters we explain each of these components more detailed and present important and highlycited publications in each category. Additionally we provide tables for each category to allow quickaccess to the desired work: Table 1 is summarizing remote collaboration systems with focus onvirtual environment, Table 2 focuses on user representation and Table 3 identifies and categorisesdifferent interaction possibilities. Survey on Synchronous AR, VR and MR Rem. Collab. Systems
Fig. 3. The three pillars for remote collaboration systems.
Virtual environment refers to a simulated environment that stimulates the sensory impressionsof a user. One of the first virtual environments was
Sensorama , created by Morton Heilig in theearly sixties [27]. It featured a simulated motorcycle ride with 3D visuals, stereo sound, olfactorycues (aromas) and tactile cues (seat vibration and wind from fans). In recent literature, most virtualenvironments are not as comprehensive and complete as the prototype created by Heilig, but ratherfocus on specific areas that are mostly visual or acoustic stimuli. Exceptions are augmented realitysystems which utilize markers, where tangible interfaces with haptic feedback are still popular. Asan example, Wang et al. used tangible interfaces such as a regular table [115] or tabletop [117, 118].Other marker based systems used turntables, such as Shen et al. [85, 86] or additional interactiontools such as a pen in [87]. With increasing maturity of AR technology, marker based systemsbecame obsolete and such systems are not further developed.In case of VR, there is usually a 3D modeled scene which is rendered, while in AR, the virtualenvironment refers to the augmented virtual objects superimposed onto the real world. Some ARsystems do not include any 3D object rendering but use shared annotations and virtual pointersinstead [18–20, 23–25, 46, 79, 80]. Sense of presence, often called telepresence, is highly affected bythe quality and consistency of the virtual environment [122]. In early work, studies suggested thatthe overall sense of presence is increased by adding tactile and auditory cues [9]. The more sensoryimpressions are added, the greater the feeling of presence according to the studies of Dinh et al.[9]. In VR as example, telepresence is not only achieved by highly realistic 3D environments butalso with consistency i.e. avatars should blend in with the environment and interaction methodsshould be adequate [122]. The work of Yoon et al. [122] compares different types of avatars withdifferent styles. The authors’ findings include that a virtual environment in cartoon style shouldalso use avatars in cartoon style to achieve a higher sense of presence for the users. In Table 1we present research works, summarised by their respective virtual environment properties andordered according to their respective technology (AR/VR/MR). Furthermore, we categorised remotecollaboration systems in three main use cases:
Meeting, Design and
Remote Expert since they wheremost popular and consistent throughout the literature. The category
Meeting can also be seenas a means of sharing a workspace with other users . Some systems which are used for training orsocializing fit also in this category. Note that the category
Event has been added for VR-basedsystems, as there were three systems that could not otherwise be meaningfully categorised. Thesesystems are used solely for event purposes. Bigscreen [34] focuses on virtual cinemas, allowingpeople to buy tickets and then watch movies together in a collaborative virtual environment. Sansar chäfer et al. System / Authors Techn. Use Case Visualisation Style Sensory InputsBreakroom [91] VR Meeting Cartoon Audio, VisualEngageVR [67] VR Meeting Realistic Audio, VisualGlue Collab [31] VR Meeting Realistic Audio, VisualMeetInVR [51] VR Meeting Realistic Audio, VisualMozilla Hubs [55] VR Meeting Cartoon Audio, VisualNvidia Holodeck [58] VR Meeting Realistic Audio, VisualStage VR [109] VR Meeting Realistic Audio, VisualTechViz VR [100] VR Meeting Realistic Audio, VisualTheWild [105] VR Meeting Realistic Audio, VisualVive Sync [96] VR Meeting Cartoon Audio, VisualWorldViz [121] VR Meeting Cartoon Audio, VisualRegenbrecht et al. [75] VR Meeting Realistic Audio, VisualGu et al. [22] VR Meeting Realistic Audio, VisualSchäfer et al. [83] VR Meeting Realistic Audio, VisualVRChat [3] VR Meeting Cartoon Audio, VisualNeosVR [52] VR Meeting Cartoon Audio, VisualAcadicus [108] VR Meeting Cartoon Audio, VisualRumii [90] VR Meeting Realistic Audio, VisualVirBELA [107] VR Meeting Realistic Audio, VisualGarou [35] VR Meeting Realistic Audio, VisualMeetingRoom [50] VR Meeting Cartoon Audio, VisualFacebook Horizon [45] VR Meeting Cartoon Audio, VisualSecond Life [77] VR Meeting Realistic Audio, VisualTan et al. [99] VR Meeting Cartoon Audio, VisualWeissker et al. [120] VR Meeting Cartoon Audio, VisualIrisVR [36] VR Design Cartoon Audio, VisualHsu et al. [32] VR Design Realistic Audio, VisualLehner et al. [44] VR Design Realistic Audio, VisualBigScreen [34] VR Event Realistic Audio, VisualWave [119] VR Event Realistic Audio, VisualSansar [7] VR Event Realistic Audio, VisualOrts et al. [59] AR Meeting Realistic Audio, VisualRegenbrecht et al. [76] AR Meeting Realistic Audio, VisualShen et al. [85–88] AR Design Cartoon Audio, Visual, TactilePoppe et al. [68, 69] AR Design Cartoon Audio, VisualSodhi et al. [92] AR Remote Expert Cartoon VisualGurevich et al. [24, 25] AR Remote Expert Annotations Audio, VisualMasai and Lee et al. [43, 48] AR Remote Expert Cartoon Audio, VisualTait et al. [98] AR Remote Expert Cartoon Audio, VisualKurata et al. [40] AR Remote Expert Cartoon Audio, VisualOu et al. [60] AR Remote Expert Cartoon Audio, VisualIzadi et al. [37] AR Remote Expert Cartoon Audio, VisualLukosch et al. [46] AR Remote Expert Annotations Audio, Visual Survey on Synchronous AR, VR and MR Rem. Collab. Systems
System / Authors Techn. Use Case Visualisation Style Sensory InputsGauglitz et al. [18–20] AR Remote Expert Annotations Audio, VisualGupta et al. [23] AR Remote Expert Annotations Audio, VisualZillner et al. [126] AR Remote Expert Annotations VisualUtzig et al. [106] AR Remote Expert Annotations Audio, VisualZenati et al. [123–125] AR Remote Expert Annotations Audio, VisualHiguchi et al. [29] MR Meeting Realistic Audio, VisualSpeicher et al. [93] MR Meeting Annotations Audio, VisualRyskeldiev et al. [79, 80] MR Meeting Annotations VisualSpatial [97] MR Meeting Realistic Audio, VisualRegenbrecht et al. [74] MR Meeting Realistic Audio, Visual, TactileHaller et al. [26] MR Meeting Realistic Audio, Visual, TactileNorman et al. [57] MR Meeting Annotations Audio, VisualMatthes et al. [49] MR Meeting Realistic Audio, VisualGalambos et al. [13, 14] MR Meeting Realistic Audio, VisualBai et al. [2] MR Meeting Cartoon Audio, VisualLuxenburger et al. [47] MR Meeting Realistic Audio, VisualvTime [110] MR Meeting Realistic Audio, VisualPoseMMR [61] MR Meeting Annotations Audio, VisualGrønbæk et al. [21] MR Design Realistic Audio, VisualTeleAR [118] MR Design Cartoon Audio, Visual, TactileWang et al. [114–117] MR Design Realistic Audio, Visual, TactileSakong et al. [81] MR Design Realistic Audio, Visual, TactileSidharta et al. [89] MR Design Realistic Audio, Visual, TactileIbayashi et al. [33] MR Design Realistic Audio, Visual, TactileSasikumar et al. [82] MR Remote Expert Cartoon Audio, VisualLee et al. [41, 42] MR Remote Expert Realistic Audio, VisualTeo et al. [101–104] MR Remote Expert Realistic Audio, VisualKim et al. [38, 39] MR Remote Expert Annotations Audio, VisualRae et al. [71] MR Remote Expert Realistic Audio, VisualPiumsomboon et al. [65, 66] MR Remote Expert Cartoon VisualGao et al. [15, 16] MR Remote Expert Realistic Audio, VisualWang et al. [112] MR Remote Expert Annotations Audio, VisualElvezio et al. [10] MR Remote Expert Realistic VisualPouliquen-Lardy et al. [70] MR Remote Expert Realistic Audio, VisualAlem et al. [1] MR Remote Expert Cartoon Audio, VisualHiguch et al. [28] MR Remote Expert Realistic Audio, VisualChen et al. [4] MR Remote Expert Annotations Audio, VisualNittala et al. [56] MR Remote Expert Annotations Audio, Visual, TactileSun et al. [94, 95] MR Remote Expert Cartoon Audio, Visual
Table 1. Remote collaboration systems sorted by their respective technology and classified in differentcategories. chäfer et al. [7] and Wave [119] focus on virtual live events such as concerts. To the best of our knowledge,there is no existing system purely designed for events based on AR/MR technology. This category is for remote collaboration systems where users sharea common workspace or environment for collaboration. These systems usually support mediasharing, involve avatars to increase the sense of co-presence, and have interactive elements such asdrawing on a whiteboard. In addition, we include use cases with knowledge transfer i.e. educationaland learning scenarios in this category.A focus on transferring and obtaining knowledge through augmented and virtual reality remotecollaboration systems is shown by Monahan et al. [54], where a web-based VR system for managingand providing educattional content online was implemented. The system features an immersive3D environment, allowing the lecturer to add media and virtual objects. Avatars are able to usegestures e.g. they can raise hands to indicate a question.Chen et al. [5] created
ImmerTai , a system which is designed for remote motion training. Theparticipants are able to learn Chinese Taichi in an immersive collaborative environment. Studentand a teacher are physically separated and resembled as a full body avatar in the virtual environment.This system includes a motion capturing module utilizing a Microsoft Kinect, transferring the realworld motion to their avatars. A motion assessment module is used to rate the movements of thestudent and give hints for improvement during and after a Taichi session.Wang et al. [111] use a combination of camera, projector, VR HMD and hand tracker to create aremote collaboration system for knowledge transfer in a manufacturing scenario. A local workerassembles a water pump while a camera is recording and transmitting video footage of the worker’sassembling progress to a remote expert. The remote expert views the video material through aVR HMD and transmits visual cues back to the local worker. A projector on the side of the localworker projects the hand movements of the remote expert onto his working surface.Schäfer et al. [83] used panorama images to create a shared photorealistic virtual environmentin a meeting scenario. Users are able to hold virtual presentations with media sharing and handgestures for interacting with the augmented virtual objects.Weissker et al. [120] investigated group navigation in virtual immersive environments. Theauthors implemented a system which allowed users to navigate inside virtual environments togetheras a group or as individual. In their system, users can attach themselves to others and then organizeteleportation movement through the virtual world together. Their results showed advantages incollaborative work when a switch between individual and group navigation is implemented.
The categery design combines remote collaboration systems such asproduct design [85, 87, 88] and architectural design [6, 30, 32].One of the earlier works was done by Lehner and DeFanti [44], who used a CAVE system in1997. CAVE is a 10-foot by 10-foot by 9-foot surround screen which uses projections on the wallsand floor. The authors implemented a system which enabled multiple users to share the sameenvironment and discuss vehicle design remotely. The visual representation of other users wasachieved by streaming 2D video inside the virtual environment.Hsu et al. [32] developed an architectural design discussion system with interactive and immersiveelements. It features voice communcation, object manipulation, mid-air sketching and on-surfacesketching. Overall, this tool was implemented to help architects to better understand architecturemodeling and to discuss design decisions, even changing models during a remote collaborationsession.The work of Chowdhury et al. [6] implements a collaboration system with an immersive virtualenvironment specifically created for urban design ideation and generation. The work is intendedto be used by non-experts and concludes that even laypeople can take part in the design process Survey on Synchronous AR, VR and MR Rem. Collab. Systems of early stage urban design. While one user uses a VR HMD to view, interact and change virtualobjects, other participants are able to perceive the changes on a display screen while giving feedbackto the VR user.Hong et al. [30] utilizes the multi-user virtual environment Second Life [77] for creative col-laboration with focus on architectural design. They compared the effectiveness of collaborativearchitectural design between multi user virtual environments and a commercial architectural designsoftware which allows 2D sketching and communication through audio. The authors argue that aremote collaboration system with avatars is more effective than two-dimensional approaches dueto shared spatial informations.Ibayashi et al. [33] created a MR collaboration system which connects users on a tabletop devicewith a user wearing a VR HMD. The authors use a role based system with designers and occupants.Designers are able to view a 3D environment using a tabletop device. The environment can bechanged with a touch interface provided to the designers. The occupant is immersed in this shared3D environment with a VR HMD and is able to see the changes made by the designers in real-time.A see-through ceiling allows the occupant to see the designers by looking at the ceiling of the 3Denvironment while the designers are able to see the VR user moving around from the top-view.A petroleum well planning application was developed by Nittala et al. [56], using hand helddevices to augment the surroundings of a remote worker. A local user used a 3D printout, stylus andtablet as an interface to communicate with a remote worker who is on-site coordinating drillingoperations. The 3D printout was combined with AR visualisations to provide the local user withan overview of earth’s composition near the remote worker. The remote worker is able to see ARannotations made by the local user to plan drilling operations.
We identified several systems which include remote collaboration,and use a scenario with a local and a remote user. In such systems, the local user typically executesa predefined task, being physically present at the target location, while the remote user is generallyfar away and provides support with instructions or hints. In this type of collaboration scenario, theremote user is often called the remote expert. Many systems are based on a combination of AR andVR [17, 20, 41, 102], while the remote expert typically uses a VR HMD or 2D screen and the localuser transmits his surroundings with the help of an AR HMD or a mounted camera.A mixed realiy collaboration system was developed by Piumsomboon et al.[63]. The systemenables an AR user to share his local environment with a remote user. It provides collaborative,natural interaction with gaze and hand gesture data transmitted over a network to each user.Another MR collaboration system was developed by Lee et al. [41]. The authors developed asystem in which a host works with an AR HMD mounted with a 360° camera and a guest with a VRHMD. Nonverbal communication cues are transmitted via hand tracking and view awareness. Bothusers have visual feedback where the counterpart is currently looking at and are able to exchangehand gestures.Teo et al. [102] introduced a MR remote collaboration system which combines reconstructedscenes obtained through an AR HMD with 360 panorama images. A remote user who receivesfootage from the AR user can move through the transmitted visual information, without relyingon the local user to move. The system has been extended with the functionality that the remoteuser can trigger a 360 camera with the help of his VR controller to save spherical images that canbe accessed independently [104]. The authors propose to add more functionality such as mid-airdrawing in 3D to improve the usability of the prototype.Mixed reality remote collaboration systems supporting local and remote users are especiallyuseful in repairing tasks as Gauglitz et al. [20] suggest. In their work, a remote user is able to seethe local user’s current view and to annotate the view which is then visible in AR. chäfer et al. Avatar Type References
Cartoon [91] [31] [51] [55] [109] [96] [68, 69] [83] [52] [36] [108] [90] [78] [107][35] [50] [34] [119] [45] [110] [104] [82] [3] [13, 14] [2] [120] [70] [1][106]Realistic [67] [58] [97] [100] [59] [29] [7] [77] [99] [3] [52]Full Body [91] [67] [58] [109] [96] [3] [52] [7] [119] [110] [77] [99]Head & Hands [31] [51] [100] [121] [65] [83] [36] [108] [90] [50] [34] [104] [13, 14][2] [70]Upper Body [55] [97] [105] [68, 69] [78] [35] [45] [120]Reconstructed Model [67] [97] [100] [59]Video [74] [75] [22] [79, 80] [49] [44] [4] [33]AR annotations [92] [24, 25] [115] [23] [38] [112] [57] [126] [28] [106] [56] [123, 124]Hands [118] [92] [81] [37] [15] [41, 42] [82] [1] [39] [28] [33] [94, 95]Audio Avatar [85–88] [89] [18–20] [43, 48] [98] [40] [60] [46] [21] [61] [10] [47]
Table 2. Remote Collaboration Systems classified in avatar categories.
An interesting remote collaboration approach is e, a framework by Speicher et al[93]. It allows ad-hoc remote collaboration in AR via 360-degree live input. Users are able to adddigital annotations by drawing on a 360-degree video stream, either by means of a normal desktopapplication or mobile devices. The annotations made by remote participants are then visualised atthe local physical space through a projector.Gao et al. [17] implements a mixed reality collaboration system by mounting an RGB and RGB-Depth camera on top of a local user’s VR HMD. The VR HMD is used as a Video-Seethrough devicewhile it captures and transmits its view to the remote counterpart. A RGB-Depth camera is usedto obtain a point cloud which is streamed to the remote user. This point cloud is then stitched,enabling an independent view control of the local workspace.Bai et al. [2] developed a system which supports real-time 3D reconstruction by assemblingeight RGB-Depth cameras into one sensor cluster. The remote VR user is able to see the local userssurroundings through the transmission of the aligned pointclouds obtained through the RBG-Depthcameras at the local users space. The authors research focus is a shared virtual environment whichsupports gaze and gesture as visual cues in a remote expert scenario. Although the system supportsone-way transmission of natural communication cues only, the results demonstrate advantages byproviding natural gaze and gesture cues during collaboration.Overall, we found that the virtual environment of remote collaboration systems focus on audio-visual stimuli. Some work did not even implement audio, focusing completely on visual feedback[65, 66, 79, 80]. While other work included some tactile feedback, these systems where mostlymarker based AR systems and rely markers attached to physical objects [81, 89, 114, 117].
Avatars represent entities in virtual environments. We classify the most commonly used avatars inscientific literature as well as in commercial AR/VR/MR software in categories and differentiatethem by descriptive terms (see Table 2): • Realistic and Cartoon graphics
We distinguish avatars through visualisation style, i.e. how the 3D model of the avatar isrendered (cartoon or realistic style). One of the reasons we use this as a descriptor is that we Survey on Synchronous AR, VR and MR Rem. Collab. Systems want to help researches who are focused on the appearance of avatars. Additionally, there isexisting work which addresses certain research questions concerned with avatar visualisationstyles. E.g. according to Yoon et al. [122] there is no statistical difference in regards to socialpresence with different visualisation styles but that user perception differs between cartoonand realistic avatars. A cartoon avatar allows for a more playful atmosphere, whereas realisticavatars tend to represent a professional environment. • Avatar Type
We divide avatar types in subcategories: Full Body, Upper Body, Head & Handsand Hands only. Full body avatar refers to humanoid avatars where all limbs are attachedto it (e.g. hands, arms, legs etc.). An Upper Body avatar consists of a head, hands and torsobut no legs. The Head & Hands type of avatar is composed of a floating head combined with(detached) hands. Hands only means that a user is only represented by virtual hands. • Reconstructed Model Avatar
If a system is capable of creating an avatar that resemblesthe respective user we categorise it as a system that uses
Reconstructed Model avatars. Thiscategory includes avatars that are created from face reconstruction in any form and ex-cludes avatars that do not have a realistic face (e.g. reconstructed/personalized hands only isexcluded). Non-reconstructed avatar means in general choosing from existing 3D models,without significant customization options. • Video Avatar
Some systems implement avatars as video projections, similar to typicalvideoconferencing systems. Systems fall under this category if a user is seen as a video feedin an immersive virtual environment. Additionally, we categorise systems which use multiplecameras to reconstruct 3D video avatars here. As an example, the work of Matthes et al. [49]implements such a system based on multiple depth cameras. • Audio Avatar
Although avatars are often represented as a humanoid 3D model, the termavatar is in general used for any kind of user representation in virtual worlds, even includinginvisible forms. In this work,
Audio Avatar represents an entity in a system which enablescommunication with other users, regardless of visual appearance. Systems with this type donot rely on any visual form for users in remote collaboration systems. Users in such systemsuse audio for communicating with each other. • AR Annotations
Systems which use no 3D model for other users but have annotationsinstead are in this category. This differs from
Audio Avatar in the sense that
Audio Avatar uses audio communication only, whereas in
AR Annotations the remote expert communicateswith the local user with annotations, i.e. the other users presence is perceived through visualannotations. Unique avatars are usually not necessary in this scenario, because the roles areclearly separated and the users can distinguish each other by actions. In many systems using
AR Annotation avatars, audio communication is not implemented. As an example, the workof Zillner et al. [126] uses visual annotations such as text, pictures and freehand drawings togive precise instructions to the local worker without relying on audio communication.A typical avatar configuration for VR based systems consists of the head (position of VR HMD)and hands (position of controllers) and is usually the most minimalistic avatar in VR scenarios. Anavatar consisting solely of virtual hands tends to be used in AR based systems which use handtracking or gesture techniques [15, 92]. We identified a majority of
Audio Avatars in AR systems(see Table 2). While VR systems seem to always rely on 3D model representation of other users,AR based approaches often omit visual representation when using remote expert scenarios. Insuch cases, the local and remote user communicate via audio with each other and share theirview [18–20]. Furthermore, some systems rely mainly on visual annotations [24, 24, 92], which wemarked as
AR Annotations in Table 2. chäfer et al. Piumsomboon et al. [64] developed a system with an adaptive avatar
Mini-Me which usesredirected gaze and gestures to enhance remote collaboration with improved social presence. Toassess the usefulness of the avatar, a scenario where a remote expert in VR assists a local workerin AR was used. The remote expert was shown to the AR user as a miniature avatar which wasable to sucessfully transmit nonverbal communication cues according to the authors. Althoughfocusing on the novelty of the proposed avatar, the system proved to be useful for overall remotecollaboration.Elvezio et al. [10] developed a system with virtual twins of physical objects. A remote expertuses virtual replicas of physically existing objects to guide a local user performing certain taskswith such objects. In this case, communication with both users only takes place by transmitting thepose of the mentioned objects. In the work of Luxenburger et al. [47] the communication betweenusers takes place through media sharing. A user is filling out a report on a mobile device which isthen visible to a remote user by means of a VR HMD.In the commercial VR remote collaboration system EngageVR [67], users can create their own fullbody avatar with reconstructed face, by uploading a single picture. Machine learning techniquesin the backend of the system reconstruct a fully textured 3D mesh of the head and attaches itautomatically to a predefined body models. Some other commercial systems are not as sophisticatedand use cartoon like avatars [31, 51, 91]. Other popular systems such as VRChat [3] or NeosVR[52] allow users to create and upload their own avatars. By means of an SDK, they can uploadfully animated humanoid avatars regardless of their appearance. The seamless integration of thesearbitrary avatars is achieved by applying a specific skeletal structure to the model.
We identified common interactive elements in remote collaboration systems that are found invarious works and literature. In Table 3 we provide an overview of literature and work which iscategorised in multiple different interaction categories. It is to note that a category is not mutuallyexclusive to another, e.g. a system which uses media sharing might also use hand gestures. Thissection explains each interaction technique with a few examples. The common features we extractedare as follows:(1) Shared 3D Object Manipulation(2) Media Sharing(3) AR Annotations(4) 2D Drawing(5) AR Viewport Sharing(6) Mid-Air Drawing in 3D(7) Hand Gestures(8) Shared Gaze Awareness(9) Convey Facial ExpressionTable 3 guides the reader to interesting and major publications which use the mentioned interactiontechniques.
The most commonly shared feature between remote collabo-ration systems is the possibility to interact and manipulate shared 3D objects in a virtual space.The type of interaction differs between systems, but the focus is on manipulating one or many3D objects. AR technology is used by Shen et al. [86–88] where multiple users interact with 3Dobjects in a collaborative AR environment. A stylus with two markers attached is used as additionalinteraction tool which enables feature highlighting and 3d object manipulation. More recent work Survey on Synchronous AR, VR and MR Rem. Collab. Systems
Interactive Feature ReferencesShared 3D Object Manipulation [91] [67] [31] [51] [55] [58] [97] [109] [100] [105] [96] [121][74] [75] [26] [118] [115] [81] [89] [87] [86] [76] [59] [3] [52][36] [108] [90] [35] [50] [34] [110] [77] [61] [57] [99] [13, 14][10] [44] [70] [33]Media Sharing [91] [67] [31] [51] [55] [58][97] [109] [100] [105] [96] [121][74] [75] [26] [118] [37] [21] [93] [83] [3] [52] [90] [107] [35][50] [34] [119] [110] [77] [99] [13, 14] [47]2D Drawing [67] [31] [51] [55] [97] [109] [121] [74] [26] [22] [118] [60] [37][21] [93] [29] [52] [90] [50] [13, 14]Mid-Air Drawing in 3D [31] [51] [55] [97] [109] [121] [52] [36] [126]Shared Gaze Awareness [68, 69] [118] [48] [23] [65, 66] [93] [29] [112] [57] [2] [28][106]Convey Facial Expression [48] [43] [52] [99]Hand Gestures [97] [92] [69] [118] [115] [81] [37] [46] [65, 66] [15, 16] [41, 42][101] [83] [59] [52] [82] [99] [2] [1] [39] [28] [33] [94, 95]AR annotations [97] [92] [69] [68] [26] [118] [89] [86–88] [43, 48] [98] [40] [60][46] [76] [93] [79] [41, 42] [101] [38, 39] [82] [1] [126] [28] [4][56] [123, 124]AR viewport sharing [98] [40] [65, 66] [15, 16] [93] [79, 80] [41, 42] [101, 104] [38][82] [56] [94, 95] [123, 124]
Table 3. Shared interaction possibilities among remote collaboration systems. uses hand tracking/gestures to interact with objects [97]. Schlünsen et al. [84] compared free-hand-manipulation with widget-based manipulation techniques. Their study shows that free-handinteraction is preferred over widget-based interaction by users.
Systems which are able to share documents, images, videos and other form ofmedia are categorised here. Haller et al. [26] created a system with a tangible interface, a table withtouchscreen for media sharing. It additionally featured sharing media from desktop applications tothe tabletop. A web-based VR solution was developed by Monahan et al. [54] which implementsmedia sharing such as videos and images in an educational context.
One of the most commonly used tool for communication and interaction inAR based systems is annotations. Annotation types include 2D drawing, text, or simple pointers.The work of Speicher et al. [93] utilizes a 360 camera to capture the surroundings of one user, whileother users are able to draw and annotate on the input stream. The annotations and drawingsare then visualised by a projector to the local user’s physical space. Kurata et al. [40] present awearable HMD which receives remotely annotated input in form of drawings. A special feature ofthis system is a laser pointer that enables the wearer of an HMD to draw the attention of remoteusers towards a certain object by pointing on it.
Especially in remote collaboration systems with focus on replicating a virtualmeeting scenario, drawing on surfaces is a widely used feature. In more sophisticated systems suchas Glue Collaboration Platform [31], users are able to place a virtual whiteboard. This whiteboardcan be re-positioned and resized allowing multiple users to draw with virtual pens in many sizesand colors. Mimicking real world objects, it is also possible to use an eraser. chäfer et al. This category includes systems which implemented sharing a user’sview perspective. Tait et al. [98] created a system which reconstructs a local user’s environmentby using depth sensors attached to an HMD. Reconstructing the environment from a local scene,remote users can move independently through the virtual environment. The local user is thenrepresented as a frustum in the reconstructed scene, allowing the remote user to see where thelocal counterpart is looking at. The study of Tait et al. [98] suggests that implementing viewindependence between local and remote user improves task completion time. Sasikumar et al. [82]combined AR and VR users together and enabled view frustum sharing which is visible to the localuser as a grey cuboid. The goal of their work was to convey nonverbal communication cues suchas eye gaze and hand gestures.
Mid-Air drawing allows users to create 3D paintings, which can thenbe observed from multiple users in different angles. This interaction method seems to be mostlyavailable in commercial systems such as Glue Collab [31] or [55]. A user is able to draw in theair of the virtual world by utilizing a VR controller as virtual pen. Other systems such as the oneproposed by Zillner et al. [126] implement a remote expert scenario, where one user is streaminghis surroundings with an RGB-D camera and a remote expert is observing and annotating forassistance. The remote expert is able to segment objects, to create animations, to draw on geometryand to place annotations which can be viewed by the AR user.
Systems which utilize hand gestures and convey hand movements throughthe remote collaborative space are included in this category. Sophistated systems such as
Spatial [97]use AR HMD’s to enable a full interaction with the 3D environment via a hand tracker. Tan et al. [99]implemented a VR telepresence system which allowed multiple users to interact with objects andwatch videos together. The authors used motion capture gloves to animate arms, hands and fingersof a VR avatar. Kim et al. [39] implemented a MR collaboration system to evaluate combinationsof visual communication cues using gestures. The authors found that certain combinations ofcommunication cues such as hand visualisation together with finger pointing direction does notprovide any significant benefit for remote collaboration.
Systems which allow users to share gaze awareness belong into thiscategory. We include systems that allow precise tracking and transmission of gaze awareness andexclude systems that indicate gaze perception by head rotation only. E.g. Galombos et al. [14] is notincluded since the gaze directon of a user is only indicated by the direction the avatar is facing. Asan example, Speicher et al. [93] created a system that allows the participants to show exactly whichposition they are looking at in a 360° video feed. The work of Poppe et al. [68, 69] uses avatarsaround a virtually augmented table and positions them according to the gaze information of thecorresponding user. Billinghurst et al. developed
Emapthy Glasses [43, 48] which is a HMD thatenables streaming a live video feed with accurate gaze information. Norman et al. [57] implementeda system which gives direct visual feedback of other user’s gaze behavior. Using a system thatcombines multiple AR HMD’s with a desktop PC, participants are asked to place virtual furnitureon a regular table in a collaborative manner.
This category addresses work which is able to transmit facialexpressions to other users. Systems which use video transmission of other user’s faces are notincluded. Lee [43] and Masai[48] et al. use
Empathy Glasses to transmit facial expressions bidirec-tionally. A local user’s face is analyzed by the built in modules of the glasses, while a remote user’sexpression is tracked via webcam. The authors of [99] integrated lip syncing into their VR remotetelepresence system in order to enable a more immersive communication experience. Survey on Synchronous AR, VR and MR Rem. Collab. Systems
Currently, there are already many tools and collaboration systems available which utilize virtualor augmented reality. In this section we consider commercial and professional systems whichsupport more than 10 users simultaneously [31, 51, 55, 58, 67, 91, 97, 100, 105, 109, 121]. Duringthe COVID-19 outbreak in early 2020, one of the biggest scientific conferences for virtual realityresearch was held completely online and virtual. To this date, it was the first major conferenceheld completely virtually. During this time, the organizing committee faced a very challenging taskto provide a pleasant conference experience for all participants without the need for a physicalpresence. During this precedent case, the whole conference featured livestreams for each track,where each presenter had the opportunity to either give a presentation with pre-recorded video,live video transmission or presenting with a VR HMD. Additionally, there where multiple virtualmeeting rooms in which participants could join and then interact and network with other usersusing VR HMD’s. The virtual meeting rooms utilized during this conference where built on MozillaHubs [55]. It features a web-based meeting room creation software, which enables users to developand maintain their own meeting experience. Utilizing a cartoon graphics style, this system workswith a desktop application, internet browser and even mobile devices, covering a broad possibleuser audience. Avatars are chosen from existing, pre-defined models. They use a mix between anabstract representation and Upper Body avatars by using a robot-like representation for users.While this solution uses an open source approach, there are several commercial products available[31, 51, 91, 105, 109] which are similar in terms of remote collaboration utlizing VR technology.
GluePlatform [31] is a platform for business professionals offering immersive 3D graphics in cartoonstyle. It is built to be used with virtual reality HMD’s and claims to be an extension to the everydayworking life. Main features include spatial audio, 3D avatars, interactive and persistent objects. Theavatars use a simple Head & Hands approach.Another commercial software available is called Breakroom [91]. It supports VR HMD’s, isavailable for multiple platforms, and features full body avatars.
In this section, we compare existing commercial/professional remote collaboration systems whichare already in use by industry and individuals. We found several VR remote collaboration systemswhich are used in a professional context [31, 58, 67, 97]. These systems have many common aspects,such as allowing many users to participate, or the collaboration tools available in the virtualenvironment. Commercial systems seem to differ mostly in terms of avatars, environments andvisualisation styles. Especially in interaction, the systems share many common collaboration toolsinside the virtual world. A typical way to collaborate in these systems is using some drawingmechanism or mid-air drawing in 3D space. Copying real world interaction methods, users cansketch on virtual black- or whiteboards and share their results in real time with each other. Someof the mentioned systems also implement mid-air drawing, and allow sketches to be observed frommultiple angles. Usually there is also a name indicator, displaying the name of participants. Thisseems to be necessary even in systems with completely reconstructed or personalised avatars suchas EngageVR [67] and Spatial [97]. A common use case in the aforementioned systems is sharingand observing virtual objects together. Systems such as EngageVR [67] or Neos VR [52] allow usersto place any 3D object previously added to a catalogue. In some cases it is also possible to showother users information by floating markers i.e. annotations. The virtual environment in thesesystems usually have a table and multiple seats to copy the physical space of real world meetings.A common scenario involves users to sit on virtual chairs and to present on a virtual tv or projector. chäfer et al. Some more advanced systems (e.g. Mozilla Hubs [52, 55]) allow screen sharing to the virtual world.Even more sophisticated systems (e.g. EngageVR [67]) extends screen sharing functionality withfull control over normal desktop applications in the virtual world. Additionally, most developersare implementing platform independence, supporting devices such as desktop, VR HMD’s, tabletsand smartphones. Mozilla Hubs [55] is fully available in a web browser.
As part of this survey, we identified strengths and weaknesses of popular commercial virtualmeeting systems. This survey includes a total of 25 commercial and professional systems (24 VRand one MR based system). The combined strength of these systems include:(1) Many users are able to join and participate in virtual meetings simultaneously [51, 55, 91],usually allowing about 20-50 people to share a virtual space simultaneously(2) Intuitive interaction possibilities such as mid-air drawing [55], drawing on white- or black-boards [51], media sharing and screen sharing [67](3) Spatial audio which enables localisation of an audio source during meetings more naturally[31](4) Persistent virtual objects which exist through multiple sessions (e.g. a drawing from a sessionbefore is still present in the next session) [31](5) Placing arbitrary 3D objects in a shared virtual environment [52, 67](6) Reconstructed, personalised avatars [67] and user created avatars through an API which isprovided by the developers [3, 52](7) Availability on multiple platforms: desktop, mobile devices and web browser [31, 55]Some systems are enterprise solutions [31, 58], tailored to the specific needs of companies, whatrenders them unattractive or even inaccessible to the general public. One major issue with thesesystems is the missing transmission of nonverbal communication cues to the virtual environmentby means of an avatar, which is an important feature of traditional face-to-face collaboration.Another weak point of the current systems is dynamic content creation for the virtual environ-ment. These systems are limited to the choice of virtual environments provided by the developeror need expert knowledge to create them [31, 51, 55, 67].Some systems implement a whole Metaverse that acts as a virtual space for many scenariossuch as training, collaboration and other social activities inside a virtual world. NeosVR [52] andVRChat [3] allow experienced users to create custom environments, avatars and interactive objectsby providing an API within the game engine in which the platform is implemented.Many systems excel in certain aspects but lack novelty in others. As an example, the immersiveremote collaboration software VirBELA [107] allows hundreds of users to participate in a virtualworld simultaneously but the avatars lack personalisation.
In this section we provide an overview of the insights and statistical data we gathered throughoutthe survey. Especially, we want to emphasize on our taxonomy consisting of
Environment , Avatars and
Interaction and important findings in each category. It is to note that we included professionaland commercial systems in our survey (24 VR based systems and one MR based system). Thisimplies that some of the discussed applications are not published in scientific articles and theimplementation of interactive features, virtual environment and the audiovisual representationof users is subject to change in the future. E.g. a discussed system does not implement full bodyavatars at the time of writing, but could implement it later on. Survey on Synchronous AR, VR and MR Rem. Collab. Systems
We categorised remote collaboration systems according to their technology: AR/VR/MR, use case,visualisation styles and sensory inputs. The technology distribution of included systems is shownin Figure 4.
In the category of VR based synchronous remote collaboration systems weincluded 24 commercial and 7 research oriented systems. The majority of systems with VR tech-nology are commercial systems, which could be an indicator that VR based systems are currentlymore under development in the industrial sector rather than the research community and thereforecould be placed on the plateau of productivity. 17 purely AR based systems where found in whichusers are able to communicate and collaborate in real-time by means of Video-Seethrough, Optical-Seethrough AR or a combination of both. To the best of our knowledge there is no commercialsystem which is solely based on AR allowing real-time synchronous remote collaboration. MRbased systems form the majority of the discussed systems. 33 research oriented and one commercialsystem are included in this category.
MR41.5% VR37.8%AR20.7%
Fig. 4. Distribution of used technology in the discussed systems.
We divided systems into three different main use cases:
Meeting, Design and
Remote Expert as they are most popular in the literature. The distribution of mentioned use caseswith respect to their technology is shown as a graph in Figure 5.Systems based on VR technlogy often involve many users (more than two), with a focus onMeeting and Design use cases. The virtual environments used in VR technology tend to involve allparticipants equally, i.e. they can see the same things and have the same input modalities, whereasMR based systems often have asymmetric inputs. For example, an asymmetric input method wouldbe when two users share a virtual environment and one has a keyboard and the other has a VRheadset with controllers as input device. There was no purely VR based system which implementeda
Remote Expert scenario although it is by far the most popular use case in AR based systems. WhileVR systems have a focus on
Meeting scenarios and AR systems a focus on
Remote Expert scenarios,MR systems are more distributed throughout use cases. This indicates a strong correlation betweenthe hardware and its respective benefits in certain use cases. For example, VR HMD’s are morebeneficial in meeting scenarios with an immersive, shared virtual environment, while AR HMDs chäfer et al. have more advantages in supporting a user with a remote expert. Since MR is a mix between ARand VR, the distribution of cases is more equally. MeetingDesignRemote Expert 0 5 10 15 20 25VR AR MR
Fig. 5. Distribution of use cases in the discussed systems.
We did not find any hints that there is a clear deviation in respect tovisualisation styles of synchronous remote collaboration systems. We differentiated between car-toon, realistic and annotation style. A system was labeled cartoon style when there was an obvioussimplification of 3D objects with stylized visualisation. Older systems with realistic visualisationare also counted here, as long as no stylized visualisation technique was used. Many AR basedsystems use annotated video only while some use rendered 3D objects with a cartoon style.
The recent literature seems to focus more on audiovisual systems anddoes not support other sensory impressions such as olfactory and tactile cues. Although tangibleinterfaces where popular especially in AR technology, the focus is drifting towards audiovisualsystems. One of the reasons might be that markers are no longer required to be placed on physicalobjects which often already implied a tangible system, if markers are placed on non-stationaryobjects. Additionally, the tracking accuracy of AR systems is constantly being improved. An exampleis Rambach et al. [72, 73], where the authors use SLAM technology for accurate object trackingwithout markers.
Examining the different types of interaction in remote collaboration systems, a general majorityof the interaction type
Shared Object Manipulation is found. A possible explanation for this isthe ease of implementation and general versatility of a task involving the manipulation of 3Dobjects. Another popular feature is
Media Sharing , i.e. possibility to share images, videos andother form of media. Some interaction types are more popular in VR technology, such as
SharedObject Manipulation, Media Sharing, 2D Drawing and
Mid-Air Drawing in 3D . The interaction types Survey on Synchronous AR, VR and MR Rem. Collab. Systems
Viewport Sharing, AR annotations and
Shared Gaze Awareness had no implementations at all in apure VR scenario. Transmitting facial expressions by using avatars is by far the least prominentfeature in remote collaboration systems, even though it is an important step towards more naturalconversations over distance. A comparative graph about the interaction types and their presencein the discussed work is shown in Figure 6.
Shared Object Manipulation Media Sharing AR annotations Hand Gestures 2D Drawing AR viewport sharing Shared Gaze Awareness Mid-Air Drawing in 3D Convey Facial Expression
VR AR MR
Fig. 6. Interaction Types in respect to the used technology.
Additionally, we found that most of the commercial remote collaboration systems rely solely onVR technology (24 VR systems and one MR system). Since the purchase of commercially availableVR HMD’s and a reasonable PC is required to operate these systems, their actual use is still limited.Therefore most professional and commercial systems tend to implement a desktop and mobileversion of the VR application. Only a few companies, such as Spatial [97] focus on integrating AR,VR, desktop and mobile together. Deploying the same application on multiple hardware devices,each with different input modalities, raises new issues such as asymmetric input. For example, amobile device user can most likely collaborate only by voice or minimal interaction within a virtualenvironment, compared to a user with a VR HMD and full body tracking. In an attempt to solvethe asymmetric input problem, Fleury et al. [12] investigated how the same interaction possibilitiescan be realized in virtual space, in a CAVE environment, and a high-resolution 2D display in wallformat. Another approach is taken by Pouliquen-Lardy et al. [70], which implemented a multi-userremote collaboration MR system to study the asymmetric effects of different input modalities. Theauthors used an approach with two different roles, a guide who could observe and communicatevia audio, and a manipulator who was able to manipulate a 3D object. The overall result of theirstudy suggests that it is not necessary to develop symmetric interaction for all users during remotecollaboration, but rather the same interaction possibilities for roles. For example, a user in the roleof a guide should be able to observe and communicate via audio regardless of the hardware theyare using. chäfer et al. The systems were analysed with respect to their specific avatar implementation. An explanationof our avatar categorisation is found in section 3.2. We found that there is not a single most usedavatar type for AR, VR or MR systems. However, we have seen that certain types of avatars arenot used in combination with particular technologies. An overall distribution of avatars in thediscussed systems is shown in Figure 7.
Video9.5%Reconstructed Model4.8%Hands14.3%AR annotations15.5%Upper Body10.7% Full Body14.3%Head & Hands17.9%Audio Avatar13.1%
Fig. 7. Different avatar types in presented sytems.
Filtering the avatar types by technology gives interesting insights into the spectrum of personalembodiment in synchronous remote collaboration systems. Looking at Figure 8 we found that
FullBody, Head & Hands and
Upper Body are most prominent in VR based systems. To the best of ourknowledge, there is no AR system that uses a full body 3D model for representation of other users.The closest approach to a full body 3D model representation in an AR system is Holoportation [59],but since several RGB-Depth cameras are involved that transmit video in real-time, we classify thisapproach as a video-based avatar system. The
Hands approach for avatars is mostly used in AR andMR systems. This is due to the fact that hand gestures provide a natural and easyily understoodway to convey visual communication cues in a
Remote Expert scenario, which is used in AR and MRbased systems. Overall we found a lack of systems which utilizes
Reconstructed Models , i.e. an avatarthat is created through face reconstruction in the virtual environment. A reason for this is thatthe technology for easy reconstruction of humans is not yet widely available for researchers andindustry implementing such systems. Not surprisingly,
Audio Avatars are mostly used in AR and MRbased systems. One of the reasons is that the shared virtual environment is formed through videotransmission and the communication with other users happens through audio. VR systems use arendered virtual 3D scene that simplifies finding other users in a virtual environment, allowing fora more sophisticated visualisation approach such as
Full Body or other types of visual avatars. Survey on Synchronous AR, VR and MR Rem. Collab. Systems
CartoonRealisticFull BodyHead & HandsAudio AvatarUpper BodyAR annotationsHandsReconstructed ModelVideo
VR AR MR
Fig. 8. Different avatar types in respect to their used technology.
During this survey, we identified shared interactive elements, various types of avatars, and differentkinds of virtual environments for synchronous remote collaboration systems. To help researchersin various research fields we created a taxonomy consisting of
Environment, Interaction and
Avatars .This approach aims to provide condensed information to specific topics which needs to be addressedwhile designing and implementing a synchronous remote collaboration system. It appears thatAR based systems focus on sharing a person’s surroundings with emphasis on stimulating theaudiovisual senses. This has recently led to a focus on remote and local user scenarios in AR/MRsystems. In such scenarios, typically a user is physically present at a certain location and shareshis/her environment, which is then perceived by other users as a virtual world that can be augmentedwith virtual objects. This means however, that each type of user (local user or remote expert) hasdifferent input and output options in their respective virtual environment. As an example, a localuser streams the surroundings with an AR HMD, while a remote user can add annotations by usinge.g. a VR HMD or tablet device, which defines separate roles during remote collaboration.In contrast, VR based systems tend to involve all participants equally, allowing each user toparticipate in the same virtual environment and the same communication and collaboration possi-bilities. We also noticed a trend in VR technology to focus more on design and meetings rather thanremote expert scenarios, as is mainly the case with AR and MR systems. Independently betweenAR, VR and MR the research focus in remote collaboration software drifts towards integratingnon-verbal communication cues. Some researchers focus on developing solutions for nonverbalcommunication transmission, but this has not yet been integrated in professional and commercialremote collaboration systems. This research includes Wang et al. [118] focusing on eye-contact,Masai et al. [48] conveying facial expressions and Lei et al. [17] implementing mutual hand gesturesto name a few.
ACKNOWLEDGMENTS
This work was partially funded by Offene Digitalisierungsallianz Pfalz, BMBF, grant number03IHS075B. This work was also supperted by the EU Research and Innovation programme Horizon2020 under the grant agreement ID: 883293. chäfer et al. REFERENCES [1] Leila Alem, Franco Tecchia, and Weidong Huang. 2011. HandsOnVideo: Towards a gesture based mobile AR systemfor remote collaboration. In
Recent trends of mobile collaborative augmented reality systems . Springer, 135–148.[2] Huidong Bai, Prasanth Sasikumar, Jing Yang, and Mark Billinghurst. 2020. A User Study on Mixed Reality RemoteCollaboration with Eye Gaze and Hand Gesture Sharing. In
Proceedings of the 2020 CHI Conference on Human Factorsin Computing Systems . 1–13.[3] VR Chat. 2018. Create and Play in Virtual Worlds.
VRChat Inc (2018).[4] Henry Chen, Austin S Lee, Mark Swift, and John C Tang. 2015. 3D collaboration method over HoloLens™ and Skype™end points. In
Proceedings of the 3rd International Workshop on Immersive Media Experiences . 27–30.[5] Xiaoming Chen, Zhibo Chen, Ye Li, Tianyu He, Junhui Hou, Sen Liu, and Ying He. 2019. ImmerTai: Immersive MotionLearning in VR Environments.
Journal of Visual Communication and Image Representation
58 (2019), 416–427.[6] Shuva Chowdhury and Marc Aurel Schnabel. 2019. Laypeople’s Collaborative Immersive Virtual Reality DesignDiscourse in Neighborhood Design.
Frontiers in Robotics and AI
AIMS Electronics and ElectricalEngineering
3, 2 (2019), 181.[9] Huong Q Dinh, Neff Walker, Larry F Hodges, Chang Song, and Akira Kobayashi. 1999. Evaluating the importanceof multi-sensory input on memory and the sense of presence in virtual environments. In
Proceedings IEEE VirtualReality (Cat. No. 99CB36316) . IEEE, 222–228.[10] Carmine Elvezio, Mengu Sukan, Ohan Oda, Steven Feiner, and Barbara Tversky. 2017. Remote collaboration in ARand VR using virtual replicas. In
ACM SIGGRAPH 2017 VR Village . 1–2.[11] Barrett Ens, Joel Lanir, Anthony Tang, Scott Bateman, Gun Lee, Thammathip Piumsomboon, and Mark Billinghurst.2019. Revisiting collaboration through mixed reality: The evolution of groupware.
International Journal of Human-Computer Studies
131 (2019), 81–98.[12] Cédric Fleury, Nicolas Ferey, Jean-Marc Vézien, and Patrick Bourdot. 2015. Remote collaboration across heterogeneouslarge interactive spaces. In .IEEE, 9–10.[13] Péter Galambos, Péter Baranyi, and Imre J Rudas. 2014. Merged physical and virtual reality in collaborative virtualworkspaces: The VirCA approach. In
IECON 2014-40th Annual Conference of the IEEE Industrial Electronics Society .IEEE, 2585–2590.[14] Péter Galambos, Ádám Csapó, Péter Zentay, István Marcell Fülöp, Tamás Haidegger, Péter Baranyi, and Imre J Rudas.2015. Design, programming and orchestration of heterogeneous manufacturing systems through VR-powered remotecollaboration.
Robotics and Computer-Integrated Manufacturing
33 (2015), 68–77.[15] Lei Gao, Huidong Bai, Gun Lee, and Mark Billinghurst. 2016. An oriented point-cloud view for MR remote collaboration.In
SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications . 1–4.[16] Lei Gao, Huidong Bai, Rob Lindeman, and Mark Billinghurst. 2017. Static local environment capturing and sharingfor MR remote collaboration. In
SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications . 1–6.[17] Lei Gao, Huidong Bai, Thammathip Piumsomboon, G Lee, Robert W Lindeman, and Mark Billinghurst. 2017. Real-timevisual representations for mixed reality remote collaboration. (2017).[18] Steffen Gauglitz, Cha Lee, Matthew Turk, and Tobias Höllerer. 2012. Integrating the physical environment into mobileremote collaboration. In
Proceedings of the 14th international conference on Human-computer interaction with mobiledevices and services . 241–250.[19] Steffen Gauglitz, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. 2014. In touch with the remote world:Remote collaboration with augmented reality drawings and virtual navigation. In
Proceedings of the 20th ACMSymposium on Virtual Reality Software and Technology . 197–205.[20] Steffen Gauglitz, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. 2014. World-stabilized annotations andvirtual scene navigation for remote collaboration. In
Proceedings of the 27th annual ACM symposium on User interfacesoftware and technology . 449–459.[21] Kaj Grønbæk, Kristian Kroyer Gundersen, Preben Mogensen, and Peter Ørbæk. 2001. Interactive Room Support forComplex and Distributed Design Projects.. In
INTERACT . 407–414.[22] Ning Gu, Mi Jeong Kim, and Mary Lou Maher. 2011. Technological advancements in synchronous collaboration: Theeffect of 3D virtual worlds and tangible user interfaces on architectural design.
Automation in Construction
20, 3(2011), 270–278.[23] Kunal Gupta, Gun A Lee, and Mark Billinghurst. 2016. Do you see what I see? The effect of gaze tracking on taskspace remote collaboration.
IEEE transactions on visualization and computer graphics
22, 11 (2016), 2413–2422.22
Survey on Synchronous AR, VR and MR Rem. Collab. Systems [24] Pavel Gurevich, Joel Lanir, and Benjamin Cohen. 2015. Design and implementation of teleadvisor: a projection-basedaugmented reality system for remote collaboration.
Computer Supported Cooperative Work (CSCW)
24, 6 (2015),527–562.[25] Pavel Gurevich, Joel Lanir, Benjamin Cohen, and Ran Stone. 2012. TeleAdvisor: a versatile augmented reality tool forremote assistance. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 619–622.[26] Michael Haller, Mark Billinghurst, J Leithinger, D Leitner, and Thomas Seifried. 2005. Coeno: enhancing face-to-facecollaboration. In
Proceedings of the 2005 international conference on Augmented tele-existence . 40–47.[27] Morton L Heilig. 1962. Sensorama simulator. US Patent 3,050,870.[28] Keita Higuch, Ryo Yonetani, and Yoichi Sato. 2016. Can eye help you? Effects of visualizing eye fixations on remotecollaboration scenarios for physical tasks. In
Proceedings of the 2016 CHI Conference on Human Factors in ComputingSystems . 5180–5190.[29] Keita Higuchi, Yinpeng Chen, Philip A Chou, Zhengyou Zhang, and Zicheng Liu. 2015. Immerseboard: Immersivetelepresence experience using a digital whiteboard. In
Proceedings of the 33rd Annual ACM Conference on HumanFactors in Computing Systems . 2383–2392.[30] Seung Wan Hong, Ahmed El Antably, and Yehuda E Kalay. 2019. Architectural design creativity in Multi-User VirtualEnvironment: A comparative analysis between remote collaboration media.
Environment and Planning B: UrbanAnalytics and City Science
46, 5 (2019), 826–844.[31] Glue HQ. 2020. Glue Universal Collaboration Platform. https://glue.work/[32] Ting-Wei Hsu, Ming-Han Tsai, Sabarish V Babu, Pei-Hsien Hsu, Hsuan-Ming Chang, Wen-Chieh Lin, and Jung-HongChuang. [n.d.]. Design and Initial Evaluation of a VR based Immersive and Interactive Architectural Design DiscussionSystem. ([n. d.]).[33] Hikaru Ibayashi, Yuta Sugiura, Daisuke Sakamoto, Natsuki Miyata, Mitsunori Tada, Takashi Okuma, Takeshi Kurata,Masaaki Mochimaru, and Takeo Igarashi. 2015. Dollhouse vr: a multi-view, multi-user collaborative design workspacewith vr technology. In
SIGGRAPH Asia 2015 Emerging Technologies
Second AnnualIEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP’07) . IEEE, 3–10.[38] Seungwon Kim, Mark Billinghurst, and Gun Lee. 2018. The effect of collaboration styles and view independence onvideo-mediated remote collaboration.
Computer Supported Cooperative Work (CSCW)
27, 3-6 (2018), 569–607.[39] Seungwon Kim, Gun Lee, Weidong Huang, Hayun Kim, Woontack Woo, and Mark Billinghurst. 2019. Evaluating thecombination of visual communication cues for HMD-based mixed reality remote collaboration. In
Proceedings of the2019 CHI conference on human factors in computing systems . 1–13.[40] Takeshi Kurata, Nobuchika Sakata, Masakatsu Kourogi, Hideaki Kuzuoka, and Mark Billinghurst. 2004. Remotecollaboration using a shoulder-worn active camera/laser. In
Eighth international symposium on wearable computers ,Vol. 1. IEEE, 62–69.[41] Gun A Lee, Theophilus Teo, Seungwon Kim, and Mark Billinghurst. 2017. Sharedsphere: MR collaboration throughshared live panorama. In
SIGGRAPH Asia 2017 Emerging Technologies . 1–2.[42] Gun A Lee, Theophilus Teo, Seungwon Kim, and Mark Billinghurst. 2018. A user study on mr remote collaborationusing live 360 video. In . IEEE, 153–164.[43] Youngho Lee, Katsutoshi Masai, Kai Kunze, Maki Sugimoto, and Mark Billinghurst. 2016. A remote collaborationsystem with empathy glasses. In .IEEE, 342–343.[44] Valerie D Lehner and Thomas A DeFanti. 1997. Distributed virtual reality: Supporting remote collaboration in vehicledesign.
IEEE Computer Graphics and Applications
Workshop at CSCW 2012: Exploring collaboration in challenges environments: Fromthe car to the factory and beyond, 11-15 februari 2012, Seattle, Washington, USA . ACM.[47] Andreas Luxenburger, Alexander Prange, Mohammad Mehdi Moniri, and Daniel Sonntag. 2016. MedicaLVR: towardsmedical remote collaboration using virtual reality. In
Proceedings of the 2016 ACM International Joint Conference onPervasive and Ubiquitous Computing: Adjunct . 321–324.[48] Katsutoshi Masai, Kai Kunze, Maki Sugimoto, and Mark Billinghurst. 2016. Empathy glasses. In
Proceedings of the2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems . 1257–1263.23 chäfer et al. [49] Carl Matthes, Tim Weissker, Emmanouil Angelidis, Alexander Kulik, Stephan Beck, Andre Kunert, Anton Frolov,Sandro Weber, Adrian Kreskowski, and Bernd Froehlich. 2019. The Collaborative Virtual Reality Neurorobotics Lab.In . IEEE, 1671–1674.[50] MeetingRoom.io. 2020. MeetingRoom. https://meetingroom.io/[51] MeetInVR. 2020. MeetInVR. https://meetinvr.net/[52] Neos VR Metaverse. 2020. NeosVR. https://neosvr.com/[53] Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. 1995. Augmented reality: A class of displays onthe reality-virtuality continuum. In
Telemanipulator and telepresence technologies , Vol. 2351. International Society forOptics and Photonics, 282–292.[54] Teresa Monahan, Gavin McArdle, and Michela Bertolotto. 2008. Virtual reality for collaborative e-learning.
Computers& Education
50, 4 (2008), 1339–1353.[55] Mozilla. 2020. Hubs by Mozilla. https://hubs.mozilla.com/[56] Aditya Shekhar Nittala, Nico Li, Stephen Cartwright, Kazuki Takashima, Ehud Sharlin, and Mario Costa Sousa. 2015.PLANWELL: spatial user interface for collaborative petroleum well-planning. In
SIGGRAPH Asia 2015 Mobile Graphicsand Interactive Applications . 1–8.[57] Mitchell Norman, Gun A Lee, Ross T Smith, and Mark Billingurst. 2019. The impact of remote user’s role in a mixedreality mixed presence system. In
The 17th International Conference on Virtual-Reality Continuum and its Applicationsin Industry . 1–9.[58] NVIDIA. 2020. NVIDIA Holodeck. https://nvidia.com/holodeck/[59] Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim,Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016. Holoportation: Virtual 3d teleportation in real-time. In
Proceedings of the 29th Annual Symposium on User Interface Software and Technology . 741–754.[60] Jiazhi Ou, Xilin Chen, Susan R Fussell, and Jie Yang. 2003. DOVE: Drawing over video environment. In
Proceedings ofthe eleventh ACM international conference on Multimedia . 100–101.[61] Ye Pan and Kenny Mitchell. 2020. PoseMMR: A Collaborative Mixed Reality Authoring Tool for Character Animation.In . IEEE, 759–760.[62] Danakorn Nincarean Eh Phon, Mohamad Bilal Ali, and Noor Dayana Abd Halim. 2014. Collaborative augmentedreality in education: A review. In .IEEE, 78–83.[63] Thammathip Piumsomboon, Arindam Day, Barrett Ens, Youngho Lee, Gun Lee, and Mark Billinghurst. 2017. Ex-ploring enhancements for remote mixed reality collaboration. In
SIGGRAPH Asia 2017 Mobile Graphics & InteractiveApplications . 1–5.[64] Thammathip Piumsomboon, Gun A Lee, Jonathon D Hart, Barrett Ens, Robert W Lindeman, Bruce H Thomas, andMark Billinghurst. 2018. Mini-me: An adaptive avatar for mixed reality remote collaboration. In
Proceedings of the2018 CHI conference on human factors in computing systems . 1–13.[65] Thammathip Piumsomboon, Youngho Lee, Gun Lee, and Mark Billinghurst. 2017. CoVAR: a collaborative virtual andaugmented reality system for remote collaboration. In
SIGGRAPH Asia 2017 Emerging Technologies . 1–2.[66] Thammathip Piumsomboon, Youngho Lee, Gun A Lee, Arindam Dey, and Mark Billinghurst. 2017. Empathic mixedreality: Sharing what you feel and interacting with what you see. In . IEEE, 38–41.[67] VR Education Holdings PLC. 2020. EngageVR: Communicate, Teach, Learn. https://engagevr.io/[68] Erik Poppe, Ross Brown, Daniel Johnson, and Jan Recker. 2012. Preliminary evaluation of an augmented realitycollaborative process modelling system. In . IEEE, 77–84.[69] Erik Poppe, Ross A Brown, Jan C Recker, and Daniel M Johnson. 2011. A prototype augmented reality collaborativeprocess modelling tool. (2011).[70] Lauriane Pouliquen-Lardy, Isabelle Milleville-Pennel, Francois Guillaume, and Franck Mars. 2016. Remote collaborationin virtual reality: asymmetrical effects of task distribution on spatial processing and mental workload.
Virtual Reality
20, 4 (2016), 213–220.[71] Irene Rae, Bilge Mutlu, and Leila Takayama. 2014. Bodies in motion: mobility, presence, and task awareness intelepresence. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 2153–2162.[72] Jason Rambach, Paul Lesur, Alain Pagani, and Didier Stricker. 2019. SlamCraft: Dense Planar RGB Monocular SLAM.In . IEEE, 1–6.[73] Jason Rambach, Alain Pagani, and Didier Stricker. 2017. [poster] augmented things: Enhancing ar applicationsleveraging the internet of things and universal 3d object tracking. In . IEEE, 103–108.[74] Holger Regenbrecht, Michael Haller, Joerg Hauber, and Mark Billinghurst. 2006. Carpeno: interfacing remotecollaborative virtual environments with table-top interaction.
Virtual Reality
10, 2 (2006), 95–107.24
Survey on Synchronous AR, VR and MR Rem. Collab. Systems [75] Holger Regenbrecht, Tim Lum, Petra Kohler, Claudia Ott, M Wagner, Wilhelm Wilke, and Erich Mueller. 2004. Usingaugmented virtuality for remote collaboration.
Presence: Teleoperators & Virtual Environments
13, 3 (2004), 338–354.[76] Holger T Regenbrecht, Michael Wagner, and Gregory Baratoff. 2002. Magicmeeting: A collaborative tangibleaugmented reality system.
Virtual Reality
6, 3 (2002), 151–166.[77] Linden Research. 2020. Second Life. https://secondlife.com/[78] Rec Room. 2020. Rec Room. https://recroom.com/[79] Bektur Ryskeldiev. 2018. Spatial Social Media: Towards Collaborative Mixed Reality Telepresence “On The Go”. In
Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems . 1–4.[80] Bektur Ryskeldiev, Michael Cohen, and Jens Herder. 2018. Streamspace: Pervasive mixed reality telepresence forremote collaboration on mobile devices.
Journal of Information Processing
26 (2018), 177–185.[81] Kyung Sakong and Tek-jin Nam. 2006. Supporting telepresence by visual and physical cues in distributed 3Dcollaborative design environments. In
CHI’06 Extended Abstracts on Human Factors in Computing Systems . 1283–1288.[82] Prasanth Sasikumar, Lei Gao, Huidong Bai, and Mark Billinghurst. 2019. Wearable RemoteFusion: A Mixed RealityRemote Collaboration System with Local Eye Gaze and Remote Hand Gesture Sharing. In . IEEE, 393–394.[83] Alexander Schäfer, Gerd Reis, and Didier Stricker. 2019. Towards Collaborative Photorealistic VR Meeting Rooms. In
Proceedings of Mensch und Computer 2019 . 599–603.[84] Robin Schlünsen, Oscar Ariza, and Frank Steinicke. 2019. A VR Study on Freehand vs. Widgets for 3D ManipulationTasks. In
Proceedings of Mensch und Computer 2019 . 223–233.[85] Yan Shen, SK Ong, and AYC Nee. 2006. A framework for multiple-view product representation using AugmentedReality. In . IEEE, 157–164.[86] Yan Shen, Soh-Khim Ong, and Andrew YC Nee. 2008. Collaborative design in 3D space. In
Proceedings of The 7thACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry . 1–6.[87] Yan Shen, Soh-Khim Ong, and Andrew YC Nee. 2008. Product information visualization and augmentation incollaborative design.
Computer-Aided Design
40, 9 (2008), 963–974.[88] Yan Shen, Soh Khim Ong, and Andrew YC Nee. 2010. Augmented reality for collaborative product design anddevelopment.
Design Studies
31, 2 (2010), 118–145.[89] Ronald Sidharta, James Oliver, and Adrian Sannier. 2006. Augmented reality tangible interface for distributed designreview. In
International Conference on Computer Graphics, Imaging and Visualisation (CGIV’06)
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems .179–188.[93] Maximilian Speicher, Jingchen Cao, Ao Yu, Haihua Zhang, and Michael Nebeling. 2018. 360anywhere: Mobile ad-hoccollaboration in any environment using 360 video and augmented reality.
Proceedings of the ACM on Human-ComputerInteraction
2, EICS (2018), 1–20.[94] Hongling Sun, Yue Liu, Zhenliang Zhang, Xiaoxu Liu, and Yongtian Wang. 2018. Employing different viewpoints forremote guidance in a collaborative augmented environment. In
Proceedings of the sixth international symposium ofChinese CHI . 64–70.[95] Hongling Sun, Zhenliang Zhang, Yue Liu, and Henry BL Duh. 2016. OptoBridge: assisting skill acquisition in theremote experimental collaboration. In
Proceedings of the 28th Australian Conference on Computer-Human Interaction .195–199.[96] Vive Sync. 2020. Vive Sync. https://blog.vive.com/us/2018/11/09/vive-sync-announced-mainstream-enterprise-vr-collaboration/[97] Spatial Systems. 2020. Spatial. https://spatial.io/[98] Matthew Tait and Mark Billinghurst. 2015. The effect of view independence in a collaborative AR system.
ComputerSupported Cooperative Work (CSCW)
24, 6 (2015), 563–589.[99] Zhipeng Tan, Yuning Hu, and Kun Xu. 2017. Virtual reality based immersive telepresence system for remoteconversation and collaboration. In
International Workshop on Next Generation Computer Animation Techniques .Springer, 234–247.[100] TechViz. 2020. TechViz VR. https://techviz.net/[101] Theophilus Teo, Gun A. Lee, Mark Billinghurst, and Matt Adcock. 2019. 360Drops: Mixed Reality Remote Collaborationusing 360 Panoramas within the 3D Scene. In
SIGGRAPH Asia 2019 Emerging Technologies . 1–2.[102] Theophilus Teo, Ashkan F. Hayati, Gun A. Lee, Mark Billinghurst, and Matt Adcock. 2019. A Technique for MixedReality Remote Collaboration using 360 Panoramas in 3D Reconstructed Scenes. In . 1–11. 25 chäfer et al. [103] Theophilus Teo, Louise Lawrence, Gun A Lee, Mark Billinghurst, and Matt Adcock. 2019. Mixed Reality RemoteCollaboration Combining 360 Video and 3D Reconstruction. In
Proceedings of the 2019 CHI Conference on HumanFactors in Computing Systems . 1–14.[104] Theophilus Teo, Gun A Lee, Mark Billinghurst, and Matt Adcock. 2019. Merging Live and Static 360 Panoramas Insidea 3D Scene for Mixed Reality Remote Collaboration. In . IEEE, 22–25.[105] TheWild. 2020. TheWild: Immersive Collaboration for Teams. https://thewild.com/[106] Sebastian Utzig, Robert Kaps, Syed Muhammad Azeem, and Andreas Gerndt. 2019. Augmented reality for remotecollaboration in aircraft maintenance tasks. In
The International Journalof Advanced Manufacturing Technology . IEEE, 1219–1220.[113] Peng Wang, Shusheng Zhang, Mark Billinghurst, Xiaoliang Bai, Weiping He, Shuxia Wang, Mengmeng Sun, and XuZhang. 2019. A comprehensive survey of AR/MR-based co-design in manufacturing.
Engineering with Computers (2019), 1–24.[114] Xiangyu Wang. 2007. Exploring an innovative collaborative design space through mixed reality boundaries. In . IEEE, 264–269.[115] Xiangyu Wang and Rui Chen. 2009. An experimental study on collaborative effectiveness of augmented realitypotentials in urban design.
CoDesign
5, 4 (2009), 229–244.[116] Xiangyu Wang and Phillip S Dunston. 2006. Potential of augmented reality as an assistant viewer for computer-aideddrawing.
Journal of computing in civil engineering
20, 6 (2006), 437–441.[117] Xiangyu Wang and Phillip S Dunston. 2008. User perspectives on mixed reality tabletop visualization for face-to-facecollaborative design review.
Automation in construction
17, 4 (2008), 399–412.[118] Xiangyu Wang, Peter ED Love, Mi Jeong Kim, and Wei Wang. 2014. Mutual awareness in collaborative design: AnAugmented Reality integrated telepresence system.
Computers in Industry
65, 2 (2014), 314–324.[119] Wave. 2020. WaveXR. https://wavexr.com/[120] Tim Weissker, Pauline Bimberg, and Bernd Froehlich. 2020. Getting There Together: Group Navigation in DistributedVirtual Environments.
IEEE Transactions on Visualization and Computer Graphics . IEEE, 547–556.[123] Nadia Zenati, S Benbelkacem, M Belhocine, and A Bellarbi. 2013. A new AR interaction for collaborative E-maintenancesystem.
IFAC Proceedings Volumes
46, 9 (2013), 619–624.[124] Nadia Zenati, Mahfoud Hamidia, Abdelkader Bellarbi, and Samir Benbelkacem. 2015. E-maintenance for photovoltaicpower system in Algeria. In . IEEE, 2594–2599.[125] Nadia Zenati-Henda, Abdelkader Bellarbi, Samir Benbelkacem, and Mahmoud Belhocine. 2014. Augmented realitysystem based on hand gestures for remote maintenance. In . IEEE, 5–8.[126] Jakob Zillner, Erick Mendez, and Daniel Wagner. 2018. Augmented reality remote collaboration with dense recon-struction. In2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)