Towards augmented reality for corporate training
TTowards augmented reality for corporate training
B. R. Martins a,b , J. A. Jorge c and E. R. Zorzal a,c a Instituto de Ciˆencia e Tecnologia, Universidade Federal de S˜ao Paulo; b Embraer S.A.,Brazil; c Instituto Superior T´ecnico, Universidade de Lisboa, INESC-ID Lisboa, Portugal
ARTICLE HISTORY
Compiled February 19, 2021
ABSTRACT
Corporate training relates to employees acquiring essential skills to operate equip-ment or effectively performing required tasks both competently and safely. Unlikeformal education, training can be incorporated in the task workflow and performedduring working hours. Increasingly, organizations adopt different technologies de-velop both individual skills and improve their organization. Studies indicate thatAugmented Reality (AR) is quickly becoming an effective technology for trainingprograms. This systematic literature review (SLR) aims to screen works publishedon AR for corporate training. We describe AR training applications, discuss currentchallenges, literature gaps, opportunities, and tendencies of corporate AR solutions.We structured a protocol to define keywords, semantics of research, and databasesused as sources of this SLR. From a primary analysis, we considered 1952 articlesin the review for qualitative synthesis. We selected 60 among the selected articlesfor this study. The survey shows a large number of 41.7% of applications focusedon automotive and medical training. Additionally, 20% of selected publications usea camera-display with a tablet device, while 40% refer to head-mounted-displays,and many surveyed approaches (45%) adopt marker-based tracking. Results indi-cate that publications on AR for corporate training increased significantly in recentyears. AR has been used in many areas, exhibiting high quality and provides viableapproaches to On-The-Job training. Finally, we discuss future research issues relatedto increase relevance regarding AR for corporate training.
KEYWORDS augmented reality; mixed reality; corporate training; on-the-job training;systematic literature review
1. Introduction
Manufacturing companies face serious challenges related to ever-changing demands bycustomers and suppliers alike. New technological changes and suggested interventionsaim to exploit the economic potential resulting from rapidly advancing informationand communication technology in the industry. Training the workforce is not a one-time issue to be addressed. Instead, it is an ongoing effort that must be nurtured andintegrated into the corporate culture.Indeed, it must continuously be on the managers’ table. Certainly, on-the-job train-ing has the potential to both improve worker productivity and save resources. Todaywe witness what could be the dawn of the fourth industrial revolution coming on the
CONTACT E. R. Zorzal. Email: [email protected] a r X i v : . [ c s . H C ] F e b eels of information technologies (IT). This revolution is characterised by Internettechnologies becoming pervasive. Indeed, IT is becoming easier to use and permeateour everyday lives via intelligent components, robots, the internet of things (IoT), andinteractive technologies.These developments lie at the core of emerging smart factories where physical anddigital systems become integrated to enable both mass customisation and faster prod-uct development (Demartini et al., 2017). Also, industries are investing in new tech-nologies to improve complex processes (Blanco-Novoa et al., 2018). According to Car-doso et al. de Souza Cardoso et al. (2020), Augmented Reality (AR) is one of theleading technologies in this context. This AR lead happens because it can be appliedin different industrial environments to improve process flexibility, providing informa-tion to manufacturing, improving product inspection, and providing more efficientlogistics while supporting maintenance process monitoring.AR is currently used in many different niche applications and diverse goals suchas emulating critical situations with low human risks, creating promising opportuni-ties to train medical professionals in a safe environment, providing general academicresources, serving as maintenance facilitator devices, in the manufacturing industriessuch as automotive and aeronautical as well.AR is known as a technology that allows computer-generated virtual imagery tooverlay physical objects in real-time accurately (Zhou et al., 2015). In AR most visualsensations and sensory stimuli come from the real world, and the virtual elementscontribute less, the virtual object is set as if it were part of the real world. In Vir-tual Reality (VR), most sensory information is computer-generated where the virtualenvironment is presented as if the user were part of it.AR applications are becoming even more affordable, thanks to more powerful hard-ware, including processors, head-mounted-displays (HMD) and smaller form-factorssuch as smartphones. Thanks to these progresses and even more sophisticated userinterfaces AR is mounting to new levels of usability (Wanderley et al., 2006).AR, as an interaction paradigm, is predicted to be one of the enabling technologiesthat will power the transformation supported by the Industry 4.0 initiative (Davies,2015), which is expected to revolutionise the current production systems. Indeed, ARis welcome in manufacturing (Damiani et al., 2018) as it can help humans to:(1) Speed up reconfiguration of production lines;(2) Support shop-floor operations;(3) Implement virtual training for assembling parts;(4) Manage the warehouse efficiently;(5) Support advanced diagnostics integrated into modules with the working envi-ronment (Damiani et al., 2015).This SLR focuses on the topic: Implementing virtual training for assembling parts.It is not circumscribed to industry settings but also delves on the medical, service, mil-itary and many other corporate training applications. Our goal is to elaborate a com-prehensive SLR on AR as a training paradigm focused providing just-in-time Rentroia-Bonito et al. (2005) instructions to people so that they can perform activities moreefficiently with minimum supervision from senior staff and with online assessment soas to become a reliable foundation for On-the-Job Training (OJT).In the context of this survey we look for training suites with the potential to beused for OJT. This requires both the ability to update pedagogical content on shortnotice and, more importantly, to integrate the module in the actual task workflow.2JT makes it possible to offer spontaneous explanations or demonstrations related to aperson’s job responsibilities and performance requirements. Proper OJT enables peopleto hone their skills either by trial-and-error learning or by observing and imitating thebehaviours of others (Jacobs and Osman-Gani, 1999). Although not all the studiesselected in this survey target OJT, all show potential to be used in that context.As both service providers and manufacturing systems switch from mass productionto mass customisation (Burger et al., 2017), this leads to more client-specific, tailor-made products and services delivered to an even diverse and larger customer base. Thisrequires even more skilled workers to perform many different tasks in widely diversecontexts to meet market requirements. The human workforce is integrated with themanufacturing systems, and people too need to be flexible and adaptive (Yew et al.,2016). AR technology, as highlighted in this paper, provides remarkable tools andfeatures to support the needs of this new workforce, by leveraging on and enhancingthe human cognitive abilities towards augmenting their abilities and productivity.In this paper, we survey studies related to corporate training, including training pro-grams provided by organisations to empower the workforce to meet client expectationsand to contribute to both the profitability and the mission of the company.AR technologies are scrutinised to identify the main benefits as measured due totheir application. Furthermore, we try and expose the disadvantages noted by authors.Moreover, to verify the current trend regarding the number of articles published, andcheck on business acceptance of AR for training purposes, we classify surveyed articlesregarding AR configurations, quality of results, tracking methods used and the displayhardware employed in experiments.
2. Methodology
The first step when conducting a systematic literature review is to define the researchobjectives and circumscribe the problem being addressed (Ten´orio et al., 2016). Thissystematic literature review assesses AR as used to promote work performance. Thispaper aims to answer the following research questions:(1) Is AR attracting more interest to be used as a corporate training recently?(2) What are the main challenges to adopting AR in OJT?(3) What are the main benefits achieved by AR in OJT?(4) Is AR a potencial tool to be used for OJT?Before starting the systematic study we need to define a search strategy for primarypublications, documenting empirical studies germane to our questions (Kitchenhamand Charters, 2007).We adopted a two-stage strategy: first we defined the keywords and the semanticsof research and then we selected which digital libraries, journals, and conferences tosearch for studies considering the following factors: • Availability of articles in bibliographic databases; • Search availability from keywords to define a search string • Relevance of bibliographic database (Kitchenham and Charters, 2007; Ten´orioet al., 2016).To define the search strings to be explored, the authors discussed and agreed on (1)specifying a keyword string and (2) analyse, observe and evaluate the search results3rom the same databases for both its relevance and its representativeness.Based on the possible results and to structure the systematic review, the selectedstring should feature a sufficiently large corpus of articles and their content shouldmake it possible to answer the proposed research questions. After discussing the possi-ble results within the authors and theirs limitations, the selected result string includesall articles and journals that feature the words ((“AUGMENTED REALITY”) OR(“MIXED REALITY”)) AND (TRAINING) as keywords or in their titles. We chosefive different databases, considering the previously discussed criteria: ACM DigitalLibrary, IEEE Xplore, Elsevier (Science Direct), Elsevier Scopus, and SpringerLink.Aiming to improve the results, we defined different selection criteria (inclusion, ex-clusion, and quality) based on the research question, String search and bibliographicaldatabases. Our objective was to identify primary papers that would provide directevidence about the research questions, also to reduce the likelihood of bias (Kitchen-ham and Charters, 2007). Specifically, we used the guidelines for performing system-atic literature reviews in software engineering (Kitchenham and Charters, 2007) andPRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)(Liberati et al., 2009; Moher et al., 2010) as a methodological foundation for theselection, evaluation, and exclusion phases.As a means to facilitate this understanding, we summarise our findings in Table 1.We read the paper titles to reduce article count. After ward, we evaluated the abstractsfor each initially retained article. Last, we read the introduction and conclusions ofeach paper. After applying these filters, we read the remaining papers integrally. Toassess the quality of the finally chosen articles, we used ten criteria which we presentin Table 2. We describe them succinctly below:
RAT
Is there a rationale and discussion on the assumptions upon which the studywas based? Does the study feature a specific goal to be achieved in the ARscenario? (Ten´orio et al., 2016) LL Are the study conclusions grounded on empirical research or does it feature a“lessons learned” report based on expert opinion? (Ten´orio et al., 2016)
OBJ
Is there a clear statement of the goals of the research? This is important toascertain to what extent they were met (Ten´orio et al., 2016)
MAT
Are the materials and research methods clearly described? A clear and detaileddescription would make it easier to better assess and reproduce the study results.
CTXT
Is there an adequate description of the context (such as industry, laboratorysetting, and products used) in which the research was carried out? Again, a cleardescription of context would support third party verification and reproductionof the study.
EMP
Was the study empirically evaluated and its results quantified? An empiricalevaluation would strengthen the conclusions of a given study.
DISC
Is there a discussion about the results of the study and its impacts? A clearand reasoned argument on the validity and generalisability of the conclusionssignificantly adds to the contribution of the report.
LIM
Are the limitations of this study explicitly discussed? Such a discussion providesgreater insights on the follow-up research and greatly adds to the contributionof the study as reported (Ten´orio et al., 2016).
OUTC
Does the research also add value to the industrial community? Does thepaper conclusions apply to, or identify process improvements? Can these beextrapolated beyond the original scope? AR Are the display and tracking techniques clearly described? A clear description4 able 1.
Summary of inclusion and exclusion criteria.
Inclusion ExclusionFeatures a training scope based on a tool that uses ARas its main characteristic Academic trainingPublished from January 2014 to August 2020 Presents less than 5 pagesEnglish is the primary language Duplicated articlesFull articles Augmented virtuality training focused allows for better contextualisation and assessment of outcomes.Each paper received one point for each criterion duly addressed. During this reading,we collected data to provide the information needed for this classification. Furthermore,we only retained articles that scored a five minimum point grade for this SLR.We searched each of the databases separately seeking articles from January 2014 toAugust 2020 that would match our seed terms. We thus collected 1952 candidate pub-lications, including duplicates. It is important to note that we restricted our search tomanuscripts written in English. Articles written in other languages did not contributeto those initial 1952 texts.We removed 85 duplicate articles, reducing the total number to 1867. After that, allthe scanning through titles and keywords have been done, and then articles identifiedas irrelevant ones were excluded. This step was performed by one of us and double-checked by another to avoid biased information or any misinterpretation trimming thetotal candidate texts from 1867 to 310.At least one author read the abstract and classified each of the the remaining essays.This classification and any doubts that arose were discussed within the group. Weexcluded pieces not compliant with the scope of work, reducing the SLR to 180 entries.From this point on, we retrieved each complete paper from the corpus and one ofus read both its introduction and conclusions to eliminate non-relevant studies. Thereader shared the classifications and we discussed possible doubts reducing the corpusto 60 papers. Figure 1 depicts the selection process.Finally, the remaining 60 papers were read in their entirety by two authors. Weretrieved the data assessing the quality of the 60 selected publications to verify andillustrate their relevance according to the selected parameters. Any doubts or disagree-ments were shared within the group to guarantee consistency in the final information.The parameters identified and a summary of studies included in the SLR are presentedin Table 2. As the papers were read, all the relevant information was registered, inorder to fulfil this systematic review goal and to structure the data to be analysed.
3. Results and Discussion
After reading and analysing each of the remaining 60 papers, we extracted and trans-lated the data into information in a process guided by answer based on the researchquestionnaires described previously. This will be detailed in the following subsections.Figure 2(a) shows for each paper its country of origin. Notably, only twelve paperscome from outside Europe or North America. USA and Italy are responsible for 15manuscripts. European researchers alone produced 34 studies as can be seen on Fig-ure 2(b). This analysis is relevant to identify the locations of research centres focusingon AR for corporate training. Thus, it becomes easier to highlight regions where AR incorporate training is most relevant. Therefore, we can conclude that developed coun-5 igure 1.
PRISMA flow diagram: study selection process. tries contribute a larger share than developing countries, suggesting that investmentsto perform AR research are more prone to be made in locations with more resourcesand more educated workers.Figure 3 identifies industries where AR is applied. Indeed, the automotive industryhas the potential to become the principal application for AR training in manufactur-ing environments, featuring 11 papers. Another issue of note is that AR applicationssupporting blended learning for medical training have garnered both public and sci-entific interest as they are the subject of 13 publications. The remaining studies aredistributed more evenly among diverse economic sectors.Figure 4 shows display configurations used in research as well as the tracking meth-ods applied. As we can see, HMDs are most commonly used, as they free both handsto execute required tasks during the training activities. A tablet is the second mostused display both due its flexibility and its low cost. Finally, projectors and moni-tors make for a smaller share of display technologies. Although the use of HMD leadsthe applications, the sum of non-immersive displays is superior than immersive onesFurther analysis of display technologies can be found later in this section.Marker-based tracking methods are seemingly the most popular. Indeed, marker-based tracking is a well-established mechanism in AR (Schmalstieg et al., 2011) andits popularity may be due to the reduced cost (Pinto et al., 2008) besides that camerascan detect fiducial markers in real time with no difficulty (Pfeiffer and Renner, 2014).Although 12 publications do not clearly describe the tracking method adopted, the6 able 2.
Summary of studies included in the SLR and the parameters used.
Reference RAT LL OBJ MAT CTXT EMP DISC LIM OUTC AR Score(Wright et al., 2017) + + + + + + + + + 9(Martino et al., 2017) + + + + + + + + + 9(Rogado et al., 2017) + + + + + + + + + + 10(Torres-Jim´enez et al., 2018) + + + + + + + + 8(Pena-Rios et al., 2018) + + + + + + + + 8(Abhari et al., 2014) + + + + + + + + 8(Li et al., 2018) + + + + + + + 7(Hou et al., 2017) + + + + + + + + + + 10(Ram´ırez et al., 2015) + + + + + + + + + + 10(Ullo et al., 2019) + + + + + + + 7(Mendoza et al., 2015) + + + + + + + + 8(Mourtzis et al., 2018) + + + + + + + + + + 10(Hoˇrejˇs´ı, 2015) + + + + + + + + + + 10(Jetter et al., 2018) + + + + + + + + + 9(Syberfeldt et al., 2016) + + + + + + + + + + 10(Borsci et al., 2015) + + + + + + + 7(Quandt et al., 2018) + + + + + + + + + 9(Aebersold et al., 2018) + + + + + + + + 8(Perdikakis et al., 2015) + + + + + + + + + 9(Segovia et al., 2015) + + + + + + + 7(Sorkoa and Brunnhofera, 2019) + + + + + 5(Longo et al., 2017) + + + + + + + + + 9(Tati´c and Teˇsi´c, 2017) + + + + + + 6(Barsom et al., 2016) + + + + + 5(Sebillo et al., 2016) + + + + + + + 7(Westerfield et al., 2015) + + + + + + + + + + 10(Uva et al., 2018) + + + + + + + + + + 10(Doshi et al., 2017a) + + + + + + + + + 9(Piedimonte and Ullo, 2018) + + + + + + 6(Lee, 2019) + + + + + + + + + + 10(Wang et al., 2018) + + + + + 5(Stefan et al., 2018) + + + + + + + + 8(Bacca et al., 2018) + + + + + + + + + 9(Kobayashi et al., 2018) + + + + + + + + 8(Limbu et al., 2018) + + + + + + + + + + 10(Stone et al., 2017) + + + + + + + + 8(Rochlen et al., 2017) + + + + + + + + + 9(Okazaki and Takaseki, 2017) + + + + + + + + + + 10(Mitsuhara et al., 2017) + + + + + + + + + 9(Tamaazousti et al., 2016) + + + + + + + + 8(Herron, 2016) + + + + + 5(Wang et al., 2016) + + + + + + + + 8(Bifulco et al., 2014) + + + + + + + + + 9(Leitritz et al., 2014) + + + + + + + + + 9(Kwon and Kim, 2019) + + + + + + + + 8(Yang et al., 2019) + + + + + + + + + + 10(B¨uttner et al., 2020) + + + + + + + + + 9(Koo et al., 2019) + + + + + + + + + 9(Ferrati et al., 2019) + + + + + + + + + + 10(van Lopik et al., 2020) + + + + + + + + + + 10(Catal et al., 2019) + + + + + + + + + 9(Balian et al., 2019) + + + + + + + + + 9(Wang et al., 2020) + + + + + + + + + + 10(Boonbrahm et al., 2019) + + + + + + + + 8(Pilati et al., 2020) + + + + + + + + + 9(Aziz et al., 2020) + + + + + + + + + 9(Eder et al., 2020) + + + + + + + + + + 10(Koutitas et al., 2020) + + + + + + + + + 9(Romero et al., 2019) + + + + + + + + + 9(Gabajov´a et al., 2019) + + + + + + + + 8 data are still valid to list 3D recognition in second place, with an increasing trendfor the near future. 15 papers feature a more computationally demanding technology,markerless tracking, that combines 2D and 3D recognition Sensor-based approachesare less popular and include electromagnetic and inertial tracking.The following subsections discuss the four Research Questions (RQ) in detail, dis-cussing the relevant findings in the surveyed literature.7 a) Final publication distribution by country.(b) Final publication distribution by continent.
Figure 2.
Final publication distribution by continent and by country.
As it is observed in Figure 5, there is a steady increase in publications on corporatetraining over the past few years. Indeed, from the initial year considered in this SLRuntil the entire last year, their number has grown significantly. Also worth noting,the number of publications featuring AR and training in either their title or keywordshas steadily increased over the years, illustrating a growing interest of the scientificcommunity.Putting into perspective, the percentage of studies regarding corporate training hasbeen stable from 2.43% to 1,75% in recent year , following the increasing trend in theAR articles published. While in 2014 the selection rate was 2/82, in 2019 this numberreached 13/737. Thus AR is increasing as the main tool in training, both causingand reflecting the growing number of publications studying its effectiveness. Thus wecan answer positively Research Question 1: Is AR attracting more interest to be usedas a corporate training recently. Moreover, training is a growing area of application8 igure 3.
Final area distribution. for AR, taking into account the share of training related papers among AR scientificpublications.
Only 16 of the selected articles do not address difficulties in developing training appli-cations. On the other hand, the remaining 60, identify at least one challenge. In general,the following issues have been reported on using AR as a training tool: adapting non-tech savvy users to AR technology (Wright et al., 2017), training engagement (Rogadoet al., 2017), field of vision (Wang et al., 2018), visual occlusion limitations (Herron,2016), ergonomics (Stone et al., 2017), environment interference (Bifulco et al., 2014),dependence on internet connection (Ullo et al., 2019), dependence on batteries, fear ofchanges (Pena-Rios et al., 2018), handling perspective and depth (Abhari et al., 2014),management engagement (Li et al., 2018), high cost of customization (Longo et al.,2017), synchronizing reality and virtuality (Hou et al., 2017), choosing the trainingscope (Ram´ırez et al., 2015), resource costs when updating AR training content (Ulloet al., 2019), acquiring trainers for interpersonal interaction (Mourtzis et al., 2018),camera focus (Hoˇrejˇs´ı, 2015), software issues (Ullo et al., 2019) and ergonomic issuesof wearables when used continuously (Jetter et al., 2018).Training people non-familiar with the technology was a challenge identified in eightof the studies including Perdikakis et al. (2015). Since AR training systems commonlyuse software and require minimum digital expertise, people experiencing difficultieswith IT are either prone to reject AR, be too ashamed to ask for help, or do notengage in the experience (Wright et al., 2017). The interest in and value perception ofAR technologies are likely to be higher for both students and technology-savvy peopleand may also vary according to gender (Habig, 2019).Another issue regarding engagement is percentage of studies that highlighted either9 a) Final display classification.(b) Final tracking method classification.
Figure 4.
Final display and tracking method classification. low enthusiasm or a low perception of benefits by trainers. Indeed, 8.3% of the studiesdiscussed this finding. Notably, several studies indicate that either the managers andthe corporate directors actively support and endorse the new training technology orthe trainers/operators will tend to reject it.Another aspect covered by the analysis is the hardware required. Displays, if notchosen carefully, can pose significant obstacles to training. In effect, 12 studies men-tioned troubles with the displays selected including Herron (2016). The problems eitherrelated to the wrong choice of display (ex: instead of using a handheld display, a wear-able would fit the function better) or troubles pertaining to the hardware proper, orreported discomfort with wearable devices during continued use.Other issues mentioned in the literature include display price point. Since manyexpensive displays are exclusively used for specialised niche training applications, theymight not be cost-effective.Ten studies reported issues affecting AR markers. Both complexity of marker tech-nology and interference from the environment are the leading causes of problems re-ported in Westerfield et al. (2015). Placing markers in corners or in open environmentscan cause misreadings (Bifulco et al., 2014). Also, the cost and the environmental con-straints cameras to establish reliable tracking were also discussed. On the other hand,software can also be a source of complications. Eleven studies mentioned the softwarepresenting at least one problem, as discussed by Hoˇrejˇs´ı (2015). In this case, the mainissue was due to inflexible configurations or applications designed for on rails exe-10 igure 5.
Final publication year distribution. cution and not being able to handle unusual conditions or inputs in a graceful way.Other problems reported off-the-shelf software lacking customisation for AR, such asvideo calling applications. Customisation was pointed out as a strong feature in 18articles e.g. (Torres-Jim´enez et al., 2018) given the added flexibility. However, eightpapers identify room for improvement as seen in the Longo et al. (2017). Customisingsoftware can be difficult mainly due to the deep knowledge required to edit the train-ing software, or the preparation need to change training sequences or dialogues. Eventhough when inside a training session it may be possible to insert additional informa-tion or change feedback triggered from markers relatively easy and cheap, changingthe business logic of the training software or physically moving tracking devices for adifferent training module, can be much harder and expensive.Internet access is flagged as an issue in two studies (Pena-Rios et al., 2018; Ullo et al.,2019), mainly due to unstable connections, or insufficient bandwidth as AR requiresvideo streaming when deployed remotely, or when it requires external processing.The 3D perception and perspective are mentioned as challenges by 14 studiesnamely Kobayashi et al. (2018). Properly matching virtual content with real settingsto meet training requirements is essential to avoiding perception troubles. Simulator(3D) sickness (Lee, 2019) needs to be avoided, as the distortion between real objectsand virtual content minimises breaks in the flow of work and increases the trainee’sattention to the instructional content.Table 3 synthesises the main points extracted during our survey regarding challengesto the adoption of AR. We also describe them succinctly below:
LELP
Low enthusiasm or a low perception of benefits by trainers; in many contextsthis may mean scepticism towards unproven technologies or a steep learningcurve characteristic of some early systems;
EIFC
Engagement issues or fear of changes; Indeed in many corporate environments,introducing a new technology can be met with resistance by people who perceivetheir job security being threatened;
3D perception regarding the perspective and deepness; These maybe related todisplay quality, registrationb problems or both;
HWI
Hardware issues; Some of the early hardware could be quirky and prone tofailure;
DMR
Difficulties regarding the marker readings; These might either be caused bypoor illumination or unfortunate choice of surfaces;
DBC
Difficulties with broadband connection and data transfer; Perceived lag cancause simulator sickness, poor registration or even missing key animations; net-work latency can exacerbate these issues;11 I Customisation issues; As AR training environments can be complex, budgetaryissues can lead to under-featured poorly adapted software;
SOFT
Software issues; Bugs or poorly documented features can lead to unantici-pated behavior;
TPNF
Training people non-familiar with the technology; Some more complex sys-tems may have a steep learning curve, which prevents non-specialised people totake full advantage of AR for training of job-specific tasks;
NDNC
Not described or not clear.
From examining the 60 articles we conclude that AR, although with different descrip-tions, provides the following advantages: reduced training costs (Ullo et al., 2019),facilitated customization (Uva et al., 2018), raised effectiveness (Doshi et al., 2017b),low-risk when exercising critical safety issues (Rogado et al., 2017), attractive tospecific groups (Wright et al., 2017), flexible information display (Pena-Rios et al.,2018), improved worker confidence (Torres-Jim´enez et al., 2018), fast access to in-formation (Sebillo et al., 2016), support to decision-making (Kobayashi et al., 2018),improved skill transfer process (Pena-Rios et al., 2018), real-time interaction (Sebilloet al., 2016), empowering operators (Syberfeldt et al., 2016), displaying immersive en-vironments (Li et al., 2018), familiarisation with the work routine (Abhari et al., 2014),allowing non-specialised staff to perform specific tasks (Ullo et al., 2019), manpowersavings (Hou et al., 2017), decreased training time (Wang et al., 2016), decreasedperceived distances (Hou et al., 2017), decreased error rates (Leitritz et al., 2014),reduced cognitive workload (Okazaki and Takaseki, 2017), easy to store and trans-port (Perdikakis et al., 2015), decreased set-up time (Quandt et al., 2018), welcomeby users (Mitsuhara et al., 2017), increased motivation (Bacca et al., 2018), friendlyremote assistance and better long term retention of information (Kobayashi et al.,2018) among others. In this subsection, we will discuss how the previously mentionedbenefits are documented throughout the articles and analyse how these affect theperceived suitability of AR to training in the working environment. Although 15 ar-ticles presented at least a quantified benefit, 59 mentioned qualitatively perceived orimmeasurable gains.Tallying the information from (Ferrati et al., 2019; Hoˇrejˇs´ı, 2015; Koutitas et al.,2020; Lee, 2019; Longo et al., 2017; Mourtzis et al., 2018; Ram´ırez et al., 2015; Uvaet al., 2018; Westerfield et al., 2015) time savings with respect to traditional ap-proaches average 28.48% (considering only the final number provided in the respectivepaper). An additional 27 articles mentioned unquantified gains related to time spentas compared to traditional methods. The main issue regarding time measurements isthat the training suite can advance stages as the operator progresses, as all modulesare geared to individual learning and the information presented is focused, in order tonarrow attention to the training goals. Another characteristic observed is that trainingmodules can use display resources to provide unsolicited information at any moment.Effectively, there is no need to wait for human-generated events to happen, as thesimulation can evolve in real time. Another key benefit highlighted by studies is theperceived lower error rate during task execution when using AR. In effect, Balian et al.(2019); Doshi et al. (2017b); Ferrati et al. (2019); Koutitas et al. (2020); Leitritz et al.(2014); Mourtzis et al. (2018); Uva et al. (2018) measured the operator errors to be, on12 able 3.
Summary of challenges identified in the SLRReference LELP EIFC 3DP HWI DMR DBC CI SOFT TPNF NDNC(Wright et al., 2017) +(Martino et al., 2017) +(Rogado et al., 2017) +(Torres-Jim´enez et al., 2018) +(Pena-Rios et al., 2018) + + + + +(Abhari et al., 2014) +(Li et al., 2018) +(Hou et al., 2017) + +(Ram´ırez et al., 2015) + +(Ullo et al., 2019) +(Mendoza et al., 2015) +(Mourtzis et al., 2018) +(Hoˇrejˇs´ı, 2015) + +(Jetter et al., 2018) + + + + +(Syberfeldt et al., 2016) + + +(Borsci et al., 2015) +(Quandt et al., 2018) +(Aebersold et al., 2018) +(Perdikakis et al., 2015) + + +(Segovia et al., 2015) +(Sorkoa and Brunnhofera, 2019) +(Longo et al., 2017) +(Tati´c and Teˇsi´c, 2017) +(Barsom et al., 2016) + + +(Sebillo et al., 2016) +(Westerfield et al., 2015) + +(Uva et al., 2018) +(Doshi et al., 2017a) +(Piedimonte and Ullo, 2018) + +(Lee, 2019) +(Wang et al., 2018) + +(Stefan et al., 2018) +(Bacca et al., 2018) +(Kobayashi et al., 2018) +(Limbu et al., 2018) + + +(Stone et al., 2017) + +(Rochlen et al., 2017) + +(Okazaki and Takaseki, 2017) +(Mitsuhara et al., 2017) +(Tamaazousti et al., 2016) +(Herron, 2016) + +(Wang et al., 2016) +(Bifulco et al., 2014) + + +(Leitritz et al., 2014) +(Kwon and Kim, 2019) +(Yang et al., 2019) + +(B¨uttner et al., 2020) + +(Koo et al., 2019) +(Ferrati et al., 2019) + +(van Lopik et al., 2020) + + + + +(Catal et al., 2019) +(Balian et al., 2019) +(Wang et al., 2020) +(Boonbrahm et al., 2019) +(Pilati et al., 2020) +(Aziz et al., 2020) +(Eder et al., 2020) +(Koutitas et al., 2020) + +(Romero et al., 2019) + +(Gabajov´a et al., 2019) + average, about 47.85% lesser than conventional settings during the task execution. 20articles mentioned a decrease in related mistakes in comparison with traditional meth-ods. AR can provide specific, relevant and customised information about every stepof the training, which can explain why AR-based training tends to yield less mistakes.Another corroborating characteristic is that AR presents information graphically andthus makes it easier to understand than text-based delivery.Cost-related issues are also relevant. Besides the inherent money savings due to13earning time reduction and decreased error rates, the costs regarding learning contentproduction can be addressed. One study Ram´ırez et al. (2015) measured these to be41% lesser as compared to traditional methods. Besides that, 25 studies mentioned,without measuring, that total cost of training was lower, namely Stone et al. (2017).An advantage that is immeasurable at this point but worth mentioning is timesavings by senior operators cited by 4 essays (Boonbrahm et al., 2019; Ferrati et al.,2019; Ullo et al., 2019; Wang et al., 2020) in the production environment who wouldotherwise be engaged to help and provide information to new employees.The flexibility afforded by AR was both welcome and noticed in 19 articles, includ-ing Torres-Jim´enez et al. (2018), which stated that AR makes content customisationeasier. Once the main body and training methodology are defined, it becomes simplerto change environments (Sebillo et al., 2016), change the information given, or evenupdate these in real-time depending on perceived requirements.The possibility to simulate a given environment is quoted by 25 studies e.g. (Barsomet al., 2016). Also, improvements in safety are quoted by 7 papers e.g. (Barsom et al.,2016; Li et al., 2018; Rogado et al., 2017; Torres-Jim´enez et al., 2018; Wang et al.,2020). These two characteristics are strongly related. As it becomes easier to simulatean environment, specific conditions can be simulated too (Aebersold et al., 2018).Hence it is possible to train the operator using comparatively inexpensive equipmentwithout incurring health or safety risks in case of misguidances (Stone et al., 2017). Itis possible to simulate events and manage stressful situations without incurring extracosts beyond content production. Dangerous environments or unsafe activities canalso be simulated without major safety issues to the operator, as discussed in Stefanet al. (2018), it is not necessary to expose novice operators to radiation, but trainingis required on how to manage emergency situations in hazardous environments e.g.nuclear power plants.The cognitive advantages of AR are discussed in 29 articles. Indeed, many operatorswelcome and experience positive attitudes towards AR-based learning as it requiresless cognitive resources to achieve results comparable to traditional methods (Okazakiand Takaseki, 2017). A key advantage is that learning scenarios can be repeated asmany times as needed without embarrassment. Since the delivery is strongly visualthe scene can be placed exactly at the time and place relevant to training, avoidingpossible confusion. The same situation can also be played in different ways, as a poka-yoke to overcome any misunderstanding of the message the training is supposed toimpart (Jetter et al., 2018). Table 4 synthesises the main points extracted during ourliterature survey regarding the main benefits outlined above. We also describe thembriefly below: RTC
Reduced training cost;
ECE
Easy customisation and editing;
LER
Lower error rate during task execution;
LTT
Lower training time;
LCL
Lower cognitive load;
ARFT
AR provided environment facilities for training;
SAFTI
Safety improvements;
BNDNC
Benefits not described or not clear. Poka-yoke is a Japanese term that means ”mistake-proofing” or ”inadvertent error prevention”. A poka-yokeis any mechanism in any process that helps an equipment operator avoid (yokeru) mistakes (poka). able 4. Summary of benefits of AR-based learning and training as identified in our literature reviewReference RTC ECE LER LTT LCL ARFT SAFTI BNDNC(Wright et al., 2017) + +(Martino et al., 2017) + + + + +(Rogado et al., 2017) + +(Torres-Jim´enez et al., 2018) + + + +(Pena-Rios et al., 2018) + + +(Abhari et al., 2014) + + +(Li et al., 2018) + + + +(Hou et al., 2017) + + +(Ram´ırez et al., 2015) + +(Ullo et al., 2019) + +(Mendoza et al., 2015) + + +(Mourtzis et al., 2018) + +(Hoˇrejˇs´ı, 2015) + + +(Jetter et al., 2018) + + + +(Syberfeldt et al., 2016) + + + + +(Borsci et al., 2015) +(Quandt et al., 2018) + + + + +(Aebersold et al., 2018) + + +(Perdikakis et al., 2015) + + + +(Segovia et al., 2015) +(Sorkoa and Brunnhofera, 2019) + + + +(Longo et al., 2017) + + +(Tati´c and Teˇsi´c, 2017) +(Barsom et al., 2016) + + + +(Sebillo et al., 2016) + + +(Westerfield et al., 2015) + +(Uva et al., 2018) + + + + +(Doshi et al., 2017a) + +(Piedimonte and Ullo, 2018) + + + +(Lee, 2019) + + +(Wang et al., 2018) + + +(Stefan et al., 2018) + + +(Bacca et al., 2018) + +(Kobayashi et al., 2018) + + + +(Limbu et al., 2018) + +(Stone et al., 2017) + +(Rochlen et al., 2017) + +(Okazaki and Takaseki, 2017) +(Mitsuhara et al., 2017) + +(Tamaazousti et al., 2016) + +(Herron, 2016) + + +(Wang et al., 2016) + + + +(Bifulco et al., 2014) + +(Leitritz et al., 2014) +(Kwon and Kim, 2019) +(Yang et al., 2019) + + + +(B¨uttner et al., 2020) + +(Koo et al., 2019) + +(Ferrati et al., 2019) + + + +(van Lopik et al., 2020) + +(Catal et al., 2019) + + +(Balian et al., 2019) + + + +(Wang et al., 2020) +(Boonbrahm et al., 2019) + + +(Pilati et al., 2020) +(Aziz et al., 2020) +(Eder et al., 2020) + +(Koutitas et al., 2020) + + + + +(Romero et al., 2019) + +(Gabajov´a et al., 2019) + + +
Based on the answers to the three previous research questions, it is possible to concludeAR has potential to become a useful tool for OJT in many settings. Its features andbenefits seemingly mesh well with OJT requirements, and we can see that once devel-opment environments support enough flexibility (Tamaazousti et al., 2016), trainingsuites are geared towards the workplace experience (Barsom et al., 2016), the learn-ing experience becomes job-oriented, learning module contents can be updated with15elative ease and training packages can be used without specialised supervision (Ulloet al., 2019). The just-in-time based e-learning can be supported by AR technologysince the training modules just require the operator to be available and the trainingstation prepared (Mendoza et al., 2015).The advantages highlighted in the RQ3 fit well with most objectives of OJT. Indeed,59 studies report strong benefits as exemplified by Quandt et al. (2018), and only 46described difficulties, notably in Syberfeldt et al. (2016). Nonetheless, as far as theresult we analysed, 15 of these studies could quantify productivity gains e.g. (Leitritzet al., 2014). On the other hand, none of the reported disadvantages could be preciselymeasured during the published experiments. While current trends show AR as beingapplied to individual training tasks, comprehensive OJT can be the next applicationto enjoy widespread AR deployment.
Even though the authors intended to cast as wide a net as possible, the articles re-viewed in this research are limited to the journals indexed in the databases: ACMDigital Library, IEEE Xplore, Elsevier (Science Direct), Elsevier Scopus, and Springer-Link. This research is limited to experimental studies presented in article format, thuspossibly missing more anecdotal or corporate proprietary evidence. Although all se-lected studies were integrally read, some info could be missing or misunderstood duringthe analysis. As the field evolves and given the rapid pace of AR development, futureiterations of this study could shed more light on our findings. In addition to that, thisSLR is limited to academic papers in a review intended to assess the imminent impactof a technology in the industry.
4. Conclusion
From the analysis of the benefits and disadvantages presented in this paper, it is clearthat AR is a maturing approach to corporate training scenarios as the technology over-comes its growing pains and associated challenges. Indeed, at the current developmentstage, well-thought-out planning stages before training programs and learning contentare deployed can maximise both skill improvement and workforce quality. This plan-ning should include the correct evaluation of the target audience. Indeed, dependingon the digital skill levels of the target audience, AR applications can be complicatedand feature unnecessary layers or be straight and simple to address less sophisticatedor casual audiences.The display selection can be critical to success and should consider both the trainingscope, environmental, safety, and ergonomics issues. If the activity requires a hands-freeoperation, we recommend either an HMD or a projector-based setting. If the trainingis time-intensive, wearable devices can present ergonomic issues. If the budget doesnot include specific hardware for each worker, or if the training has to be conducted si-multaneously by many trainees, it should be desirable to adopt smartphones or tabletsas primary displays.Tracking methods need attention while considering the environment. Variations inlighting can interfere with the reading of markers depending on the camera used. Thesurface on which to place the markers should be studied. If irregular surfaces are likelyto cause misreadings, other tracking devices such as a sensor-based or a markerlesstracker could be chosen. 16he training environment should be correctly prepared to support any flow of ac-tion that the trainees shall undergo. Avoiding inflexible trainee-”on rails” experiencesshould improve the learning experience.Software development should take into account the ultimate training objectives andaccommodate the stakeholders mentioned above.If developers are to heed these recommendations, they can increase the training’sbenefits and minimise the challenges of using AR as an OJT useful tool.Following current trends, design thinking should contribute better results to devel-oping AR training software. Another issue to take into account is gamificationBarataet al. (2013) as a device to optimise and to engage trainees. Moreover, just-in-timesolutions can facilitate software development regarding both the game engine andtraining structure to decrease efforts required to develop the software and increase theapplication range of the training and its access to more developers.Although the studies identified different benefits and some of them performed mea-surements, there is still a strong need for a comprehensive survey that combines allmeasurements and compares them with the traditional OJT training methods. Thechallenge consists of AR-based training in an industrial OJT application for the mas-tering of one activity, in which the following critical performance indicators shall bedirectly compared with traditional training programs: (1) the cost, (2) the trainingpace, (3) the time required, (4) the learning curve, (5) the cognitive workload, (6) theperception regarding AR and (7) the technology acceptance.As a means to develop future work relevance, AR will increasingly use smartphonesor other wearable devices to accommodate the growing adoption of AR everyday set-tings and improve people’s familiarity with the technology. However, AR can stillevolve in fundamental ways to improve both the user and learning experiences. Fur-ther Artificial Intelligence developments can make for more natural and sophisticateddialogues and better coaching of trainees to identify possible pitfalls and proactivelysuggest better ways to accomplish tasks. Further advances in HMDs and wearabledevices will surely make AR applications usable for extended periods, a typical workercomplaint since the inception of the field. Finally, ever more advanced user interfacetechniques, coupled with more natural interaction modalities going beyond speechand gestures, should make for ever more natural user experiences, increasing workerengagement, and productivity of AR-based learning environments.
Acknowledgements
The last author would like to thank the S˜ao Paulo Research Foundation (FAPESP)for support of this research: grant
Funda¸c˜ao para a Ciˆencia e a Tecnologia , under projectUIDB/50021/2020.
Notes on contributors
Bruno R. Martins is a student at the Professional Master in Technological Innovation,Federal University of Sao Paulo, where he is researching about the augmented realityapplications on training. He is graduated in industrial engineering and simultaneouslyworks as an engineer at EMBRAER S.A on the manufacturing system. His main17esearch interests are Virtual Reality, Augmented Reality and On-The-Job Training.
Joaquim A. Jorge is a full professor of computer graphics and multimedia at theDepartment of Computer Science and Engineering of the IST, University of Lisbon,Portugal, and scientific coordinator of the Research Group on Visualisation and Multi-modal Interfaces of the Institute for Computer and Systems Engineering - INESC-ID.Since 2007 he is editor in chief of Computers & Graphics Journal and serves or hasserved on the board for six other international journals.
Ezequiel R. Zorzal is an associate professor of computer science & engineering at In-stitute of Science and Technology, Federal University of Sao Paulo. He is an integratedresearcher at Professional Master in Technological Innovation. He is simultaneously anassociate researcher of the Visualisation and Intelligent Multimodal Interfaces Groupat INESC-ID, IST, University of Lisbon, Portugal. His main research interests arevirtual reality, augmented reality, educational and medical user interfaces.
References
Abhari, K., Baxter, J. S., Chen, E. C., Khan, A. R., Peters, T. M., de Ribaupierre, S., andEagleson, R. (2014). Training for planning tumour resection: augmented reality and humanfactors.
IEEE Transactions on Biomedical Engineering , 62(6):1466–1477.Aebersold, M., Voepel-Lewis, T., Cherara, L., Weber, M., Khouri, C., Levine, R., and Tait,A. R. (2018). Interactive anatomy-augmented virtual simulation training.
Clinical simula-tion in nursing , 15:34–41.Aziz, F. A., Alsaeed, A. S., Sulaiman, S., Ariffin, M. K. A. M., and Al-Hakim, M. F. (2020).Mixed reality improves education and training in assembly processes.
Journal of Engineering& Technological Sciences , 52(4).Bacca, J., Baldiris, S., Fabregat, R., and Kinshuk (2018). Insights into the factors influencingstudent motivation in Augmented Reality learning experiences in Vocational Education andTraining.
Frontiers in Psychology , 9(AUG).Balian, S., McGovern, S. K., Abella, B. S., Blewer, A. L., and Leary, M. (2019). Feasibility of anaugmented reality cardiopulmonary resuscitation training system for health care providers.
Heliyon , 5(8):e02205.Barata, G., Gama, S., Jorge, J., and Gon¸calves, D. (2013). So fun it hurts – gamifying anengineering course. In Schmorrow, D. D. and Fidopiastis, C. M., editors,
Foundations ofAugmented Cognition , pages 639–648, Berlin, Heidelberg. Springer Berlin Heidelberg.Barsom, E., Graafland, M., and Schijven, M. (2016). Systematic review on the effectiveness ofaugmented reality applications in medical training.
Surgical endoscopy , 30(10):4174–4183.Bifulco, P., Narducci, F., Vertucci, R., Ambruosi, P., Cesarelli, M., and Romano, M. (2014).Telemedicine supported by Augmented Reality: An interactive guide for untrained peoplein performing an ECG test.
BioMedical Engineering Online , 13(1):1–16.Blanco-Novoa, ´O., Fern´andez-Caram´es, T. M., Fraga-Lamas, P., and Vilar-Montesinos, M.(2018). A practical evaluation of commercial industrial augmented reality systems in anindustry 4.0 shipyard.
IEEE Access , 6:8201–8218.Boonbrahm, P., Kaewrat, C., and Boonbrahm, S. (2019). Interactive marker-based augmentedreality for CPR training.
International Journal of Technology , 10(7):1326.Borsci, S., Lawson, G., and Broome, S. (2015). Empirical evidence, evaluation criteria andchallenges for the effectiveness of virtual and mixed reality tools for training operators ofcar service maintenance.
Computers in Industry , 67:17–26.Burger, N., Demartini, M., Tonelli, F., Bodendorf, F., and Testa, C. (2017). InvestigatingFlexibility as a Performance Dimension of a Manufacturing Value Modeling Methodology(MVMM): A Framework for Identifying Flexibility Types in Manufacturing Systems.
Pro-cedia CIRP , 63:33–38.B¨uttner, S., Prilla, M., and R¨ocker, C. (2020). Augmented reality training for industrial ssembly work-are projection-based ar assistive systems an appropriate tool for assemblytraining? In Proceedings of the 2020 CHI Conference on Human Factors in ComputingSystems , pages 1–12.Catal, C., Akbulut, A., Tunali, B., Ulug, E., and Ozturk, E. (2019). Evaluation of augmentedreality technology for the design of an evacuation training game.
Virtual Reality , pages1–10.Damiani, L., Demartini, M., Guizzi, G., Revetria, R., and Tonelli, F. (2018). Augmented andvirtual reality applications in industrial systems: A qualitative review towards the industry4.0 era.
IFAC-PapersOnLine , 51(11):624–630.Damiani, L., Revetria, R., and Volpe, A. (2015). Augmented reality and simulation overdistributed platforms to support workers. In
Proceedings of the 2015 Winter SimulationConference , WSC ’15, pages 3214–3215, Piscataway, NJ, USA. IEEE Press.Davies, R. (2015). Industry 4.0. Digitalisation for productivity and growth.
European Parlia-mentary Research Service , (September):10.de Souza Cardoso, L. F., Mariano, F. C. M. Q., and Zorzal, E. R. (2020). A survey of industrialaugmented reality.
Computers & Industrial Engineering , 139:106159.Demartini, M., Tonelli, F., Damiani, L., Revetria, R., and Cassettari, L. (2017). Digitaliza-tion of manufacturing execution systems: The core technology for realizing future smartfactories. ,2017-Septe:326–333.Doshi, A., Smith, R. T., Thomas, B. H., and Bouras, C. (2017a). Use of projector basedaugmented reality to improve manual spot-welding precision and accuracy for automotivemanufacturing.
The International Journal of Advanced Manufacturing Technology , 89(5-8):1279–1293.Doshi, A., Smith, R. T., Thomas, B. H., and Bouras, C. (2017b). Use of projector basedaugmented reality to improve manual spot-welding precision and accuracy for automotivemanufacturing.
International Journal of Advanced Manufacturing Technology , 89(5-8):1279–1293.Eder, M., Hulla, M., Mast, F., and Ramsauer, C. (2020). On the application of augmentedreality in a learning factory working environment.
Procedia Manufacturing , 45:7–12.Ferrati, F., Erkoyuncu, J. A., and Court, S. (2019). Developing an augmented reality basedtraining demonstrator for manufacturing cherry pickers.
Procedia CIRP , 81:803–808.Gabajov´a, G., Furmannov´a, B., Medveck´a, I., Grzn´ar, P., Krajˇcoviˇc, M., and Furmann, R.(2019). Virtual training application by use of augmented and virtual reality under universitytechnology enhanced learning in slovakia.
Sustainability , 11(23):6677.Habig, S. (2019). Who can benefit from augmented reality in chemistry? sex differences insolving stereochemistry problems using augmented reality.
British Journal of EducationalTechnology , 52:629–644.Herron, J. (2016). Augmented Reality in Medical Education and Training.
Journal of Elec-tronic Resources in Medical Libraries , 13(2):51–55.Hoˇrejˇs´ı, P. (2015). Augmented reality system for virtual training of parts assembly.
ProcediaEngineering , 100:699–706.Hou, L., Chi, H.-L., Tarng, W., Chai, J., Panuwatwanich, K., and Wang, X. (2017). A frame-work of innovative learning for skill development in complex operational tasks.
Automationin Construction , 83:29–40.Jacobs, R. L. and Osman-Gani, A. A. (1999). Status, impacts and implementation issuesof structured on-the-job training: a study of singapore-based companies.
Human ResourceDevelopment International , 2:17–24.Jetter, J., Eimecke, J., and Rese, A. (2018). Augmented reality tools for industrial applications:What are potential key performance indicators and who benefits?
Computers in HumanBehavior , 87:18–33.Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature re-views in software engineering.Kobayashi, L., Zhang, X., Collins, S., Karim, N., and Merck, D. (2018). Exploratory Appli- ation of Augmented Reality/Mixed Reality Devices for Acute Care Procedure Training. Western Journal of Emergency Medicine , pages 158–164.Koo, G., Lee, N., and Kwon, O. (2019). Combining object detection and causality miningfor efficient development of augmented reality-based on-the-job training systems in hotelmanagement.
New Review of Hypermedia and Multimedia , 25(3):112–136.Koutitas, G., Smith, S., and Lawrence, G. (2020). Performance evaluation of ar/vr trainingtechnologies for ems first responders.
VIRTUAL REALITY .Kwon, C. and Kim, D. H. (2019). A comparative study on the effectiveness of the trainingsystem of an augmented reality-based automated external defibrillator with an added tactilesense.
International Journal of Innovative Technology and Exploring Engineering , 8.Lee, I. J. (2019). Using augmented reality to train students to visualize three-dimensionaldrawings of mortise–tenon joints in furniture carpentry.
Interactive Learning Environments ,0(0):1–15.Leitritz, M. A., Ziemssen, F., Suesskind, D., Partsch, M., Voykov, B., Bartz-Schmidt, K. U.,and Szurman, G. B. (2014). Critical evaluation of the usability of augmented reality oph-thalmoscopy for the training of inexperienced examiners.
Retina , 34(4):785–791.Li, X., Yi, W., Chi, H. L., Wang, X., and Chan, A. P. (2018). A critical review of virtual andaugmented reality (VR/AR) applications in construction safety.
Automation in Construc-tion , 86(November 2017):150–162.Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A.,Clarke, M., Devereaux, P. J., Kleijnen, J., and Moher, D. (2009). The prisma statementfor reporting systematic reviews and meta-analyses of studies that evaluate health careinterventions: Explanation and elaboration.
PLoS Medicine , 6(7):e1000100.Limbu, B. H., Jarodzka, H., Klemke, R., Wild, F., and Specht, M. (2018). From ar to expertise:A user study of an augmented reality training to support expertise development.
Journalof Universal Computer Science , 24(2):108–128.Longo, F., Nicoletti, L., and Padovano, A. (2017). Smart operators in industry 4.0: A human-centered approach to enhance operators’ capabilities and competencies within the new smartfactory context.
Computers & industrial engineering , 113:144–159.Martino, S., Gonbeau, O., Boisseau, C., Recapet, J.-P., Blanc, F., and Augustin, B. (2017).Enedis field experience of augmented and virtual-reality technologies at the service of net-work operators.
CIRED-Open Access Proceedings Journal , 2017(1):1081–1084.Mendoza, M., Mendoza, M., Mendoza, E., and Gonz´alez, E. (2015). Augmented reality as atool of training for data collection on torque auditing.
Procedia Computer Science , 75:5–11.Mitsuhara, H., Iguchi, K., and Shishibori, M. (2017). Using digital game, augmented reality,and head mounted displays for immediate-action commander training.
International Journalof Emerging Technologies in Learning , 12(2).Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G. (2010). Preferred reporting itemsfor systematic reviews and meta-analyses: The prisma statement.
International Journal ofSurgery , 8(5):336–341.Mourtzis, D., Zogopoulos, V., and Vlachou, E. (2018). Augmented reality supported productdesign towards industry 4.0: A teaching factory paradigm.
Procedia Manufacturing , 23:207–212.Okazaki, T. and Takaseki, R. (2017). Override ship maneuvering simulator using ar toolkit.
Intelligent Automation & Soft Computing , 23(1):167–174.Pena-Rios, A., Hagras, H., Owusu, G., and Gardner, M. (2018). Furthering service 4.0: Harness-ing intelligent immersive environments and systems.
IEEE Systems, Man, and CyberneticsMagazine , 4(1):20–31.Perdikakis, A., Araya, A., and Kiritsis, D. (2015). Introducing augmented reality in nextgeneration industrial learning tools: a case study on electric and hybrid vehicles.
Procediaengineering , 132:251–258.Pfeiffer, T. and Renner, P. (2014). Eyesee3d: a low-cost approach for analysing mobile 3d eyetracking data using augmented reality technology. In
Proceedings of the Symposium on EyeTracking Research and Applications . iedimonte, P. and Ullo, S. L. (2018). Applicability of the mixed reality to maintenanceand training processes of c4i systems in italian air force. In , pages 559–564. IEEE.Pilati, F., Faccio, M., Gamberi, M., and Regattieri, A. (2020). Learning manual assemblythrough real-time motion capture for operator training with augmented reality. ProcediaManufacturing , 45:189–195.Pinto, F., Buaes, A., Francio, D., Binotto, A. P. D., and Santos, P. (2008). Bratrack: a low-costmarker-based optical stereo tracking system. In
SIGGRAPH Posters , page 131.Quandt, M., Knoke, B., Gorldt, C., Freitag, M., and Thoben, K.-D. (2018). General require-ments for industrial augmented reality applications.
Procedia CIRP , 72(1):1130–1135.Ram´ırez, H., Mendoza, E., Mendoza, M., and Gonz´alez, E. (2015). Application of augmentedreality in statistical process control, to increment the productivity in manufacture.
ProcediaComputer Science , 75:213–220.Rentroia-Bonito, M., Tribolet, J., Jorge, J., and Ghaoui, C. (2005). Just-in-time approach tolearning: Arguing the case for cost-effective knowledge dissemination. In Ghaoui, C., Jain,M., Bannore, V., and Jain, C., editors,
Knowledge-Based Virtual Education , volume 178 of
Studies in Fuzziness and Soft Computing . Springer.Rochlen, L. R., Levine, R., and Tait, A. R. (2017). First person point of view augmentedreality for central line insertion training: A usability and feasibility study.
Simulation inhealthcare: journal of the Society for Simulation in Healthcare , 12(1):57.Rogado, A. B. G., Quintana, A. M. V., and Mayo, L. L. (2017). Evaluation of the use oftechnology to improve safety in the teaching laboratory.
IEEE Revista Iberoamericana deTecnologias del Aprendizaje , 12(1):17–23.Romero, J. A., Quero, W. D., S´anchez, J. S., and Andaluz, V. H. (2019). Training assistant forindustrial processes through augmented reality. In
Proceedings of the 2019 11th InternationalConference on Education Technology and Computers , pages 308–315.Schmalstieg, D., Langlotz, T., and Billinghurst, M. (2011). Augmented reality 2.0. In
Virtualrealities , pages 13–37. Springer.Sebillo, M., Vitiello, G., Paolino, L., and Ginige, A. (2016). Training emergency respon-ders through augmented reality mobile interfaces.
Multimedia Tools and Applications ,75(16):9609–9622.Segovia, D., Ram´ırez, H., Mendoza, M., Mendoza, M., Mendoza, E., and Gonz´alez, E. (2015).Machining and dimensional validation training using augmented reality for a lean process.
Procedia Computer Science , 75:195–204.Sorkoa, S. R. and Brunnhofera, M. (2019). Potentials of augmented reality in training.
ProcediaManufacturing , 31:85–90.Stefan, P., Habert, S., Winkler, A., Lazarovici, M., F¨urmetz, J., Eck, U., and Navab, N.(2018). A radiation-free mixed-reality training environment and assessment concept forC-arm-based surgery.
International Journal of Computer Assisted Radiology and Surgery ,13(9):1335–1344.Stone, R. J., Guest, R., Mahoney, P., Lamb, D., and Gibson, C. (2017). A ‘mixed reality’simulator concept for future medical emergency response team training.
Journal of theRoyal Army Medical Corps , 163(4):280–287.Syberfeldt, A., Danielsson, O., Holm, M., and Wang, L. (2016). Dynamic operator instructionsbased on augmented reality and rule-based expert systems.
Procedia CIRP , 41:346–351.Tamaazousti, M., Naudet-Collette, S., Gay-Bellile, V., Bourgeois, S., Besbes, B., and Dhome,M. (2016). The constrained SLAM framework for non-instrumented augmented reality:Application to industrial training.
Multimedia Tools and Applications , 75(16):9511–9547.Tati´c, D. and Teˇsi´c, B. (2017). The application of augmented reality technologies for theimprovement of occupational safety in an industrial environment.
Computers in Industry ,85:1–10.Ten´orio, T., Bittencourt, I. I., Isotani, S., and Silva, A. P. (2016). Does peer assessment inon-line learning environments work? a systematic review of the literature.
Computers inHuman Behavior , 64:94–107. orres-Jim´enez, E., Rus-Casas, C., Dorado, R., and Jim´enez-Torres, M. (2018). Experiencesusing qr codes for improving the teaching-learning process in industrial engineering subjects. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje , 13(2):56–62.Ullo, S. L., Piedimonte, P., Leccese, F., and De Francesco, E. (2019). A step toward thestandardization of maintenance and training services in C4I military systems with MixedReality application.
Measurement: Journal of the International Measurement Confederation ,138:149–156.Uva, A. E., Gattullo, M., Manghisi, V. M., Spagnulo, D., Cascella, G. L., and Fiorentino, M.(2018). Evaluating the effectiveness of spatial augmented reality in smart manufacturing:a solution for manual working stations.
International Journal of Advanced ManufacturingTechnology , 94(1-4):509–521.van Lopik, K., Sinclair, M., Sharpe, R., Conway, P., and West, A. (2020). Developing aug-mented reality capabilities for industry 4.0 small enterprises: Lessons learnt from a contentauthoring case study.
Computers in Industry , 117:103208.Wanderley, I., Kelner, J., Costa, N., and Teichrieb, V. (2006). A survey of interaction in mixedreality systems. In
Symposium on Virtual Reality , pages 1–4.Wang, M., Callaghan, V., Bernhardt, J., White, K., and Pe˜na-Rios, A. (2018). Augmented re-ality in education and training: pedagogical approaches and illustrative case studies.
Journalof Ambient Intelligence and Humanized Computing , 9(5):1391–1402.Wang, Y., Anne, A., and Ropp, T. (2016). Applying the Technology Acceptance Model to Un-derstand Aviation Students’ Perceptions toward Augmented Reality Maintenance TrainingInstruction.
International Journal of Aviation, Aeronautics, and Aerospace , 3(4).Wang, Z., Bai, X., Zhang, S., He, W., Zhang, X., Yan, Y., and Han, D. (2020). Information-levelreal-time ar instruction: a novel dynamic assembly guidance information representation as-sisting human cognition.
The International Journal of Advanced Manufacturing Technology ,pages 1–19.Westerfield, G., Mitrovic, A., and Billinghurst, M. (2015). Intelligent augmented reality train-ing for motherboard assembly.
International Journal of Artificial Intelligence in Education ,25(1):157–172.Wright, T., de Ribaupierre, S., and Eagleson, R. (2017). Design and evaluation of an augmentedreality simulator using leap motion.
Healthcare technology letters , 4(5):210–215.Yang, C.-K., Chen, Y.-H., Chuang, T.-J., Shankhwar, K., and Smith, S. (2019). An augmentedreality-based training system with a natural user interface for manual milling operations.
Virtual Reality , pages 1–13.Yew, A. W., Ong, S. K., and Nee, A. Y. (2016). Towards a griddable distributed manufacturingsystem with augmented reality interfaces.
Robotics and Computer-Integrated Manufacturing ,39:43–55.Zhou, J., Lee, I., Thomas, B., Menassa, R., Farrant, A., and Sansome, A. (2015). In-SituSupport for Automotive Manufacturing Using Spatial Augmented Reality.
InternationalJournal of Virtual Reality , 11(1):33–41., 11(1):33–41.