Virtual Environments for Training: From Individual Learning to Collaboration with Humanoids
aa r X i v : . [ c s . G R ] A ug Virtual Environments for Training: fromindividual learning to collaboration withhumanoids
St´ephanie Gerbaud and Nicolas Mollet and Bruno Arnaldi IRISA/INRIA/INSA Rennes, France. [email protected] IRISA/INRIA Rennes, France. [email protected] IRISA-INSA Rennes, France. [email protected]
Abstract.
The next generation of virtual environments for training isoriented towards collaborative aspects. Therefore, we have decided toenhance our platform for virtual training environments, adding collab-oration opportunities and integrating humanoids. In this paper we putforward a model of humanoid that suits both virtual humans and rep-resentations of real users, according to collaborative training activities.We suggest adaptations to the scenario model of our platform making itpossible to write collaborative procedures. We introduce a mechanism ofaction selection made up of a global repartition and an individual choice.These models are currently being integrated and validated in GVT , avirtual training tool for maintenance of military equipments, developedin collaboration with the French company NEXTER-Group. Key words: collaborative virtual environments, virtual reality, training,collaboration, virtual humans
The need for virtual environments for training is not to be demonstrated any-more: lower costs and risks, no need for available equipments to train on andcontrol of pedagogical situations are examples of the numerous benefits broughtby the use of Virtual Environments (VEs). Among the existing VEs for train-ing, only a few provide collaborative training opportunities. However, VEs canbring other assets for collaborative training: the possibility to collaborate withdistant people via the Internet and the opportunity to train with virtual hu-mans, so that the trainee can train not only everywhere but also at any time.Therefore, we have decided to enhance our platform for virtual training environ-ments, by adding collaboration opportunities and integrating humanoids. Thisplatform has been specifically designed for training on procedures (especiallyindustrial maintenance procedures) rather than on technical gestures. We arenow interested in collaborative procedures. GVT is a trademark of NEXTER-Group he work presented in this paper is both the integration and the continuationof previous research activities performed in our team (BUNRAKU, at IRISA)dedicated to humanoid animation [1], humanoid behavior [2] and collaboration[3]. In OpenMASK , our Virtual Reality platform, existing features are alreadydeployed to enable collaboration, but at the interaction level. We rely on thesefacilities to model a higher level of collaboration: collaboration of activity thatoccurs at the scenario level.After analyzing the existing Collaborative Virtual Environments (CVEs) fortraining and the functionalities they must offer, we will first expose the modelsthat we put forward and then finish presenting GVT , the industrial applicationin which they are being integrated (Fig.1 and 2). Fig. 1. a humanoid in GVT
Fig. 2. a humanoid interacting inGVT
The next generation of VEs for training is oriented towards collaborative as-pects. Collaboration in a VE for training implies sharing: sharing of a commonenvironment, sharing of the others’ vision, sharing of activity (the procedure).Those concepts will be reviewed in the following state-of-the-art. We will firstpresent existing CVEs for training, then we will expose the awareness issue,followed by the requirements for collaborative scenarios and last of all we willdescribe the use of virtual humans in CVEs for training.
Among existing CVEs for training, we will only describe the most relevant ones.COVET (COllaborative Virtual Environment for Training) [4,5,6] is a prototype f multi-user teletraining application which allows users, represented by avatars,to learn how to replace a faulty card on an ATM switch. It remains a very basicCVE for training since only one user can interact with the objects in the scene(for example the trainer who demonstrates the procedure). The others can onlywatch this user’s avatar acting, change their points of view, move and chat.Steve[7,8] has been extended to support team training [9,10]: students mustlearn their individual roles in the team and how to coordinate their actions withtheir teammates. Steve agents can play two pedagogical roles: either tutor orsubstitute for missing teammates.The MRE (Mission Rehearsal Exercise)[11,12] system is a VE for training whichhas been designed to teach critical decision-making skills to small-unit leadersin the U.S. Army. In this application, there is only one trainee but he mustcollaborate with virtual humans able to converse, show emotions and reason.S ecu R e V i (Security and Virtual Reality) [13] is a CVE to train firemen officersfor operational management and commandment. This application is based onthe MASCARET [14] model that organizes the interactions between agents andprovide them reactive, cognitive and social abilities. The goal is to train teamsto collaborative and procedural work. The first important need in a CVE is awareness: a user must be aware of theother users populating the environment and their actions. A common solution isto embody the user into an avatar, which can be humanoid. Benford et al. [15]identified issues that the use of an avatar can solve in a CVE: presence, locationand identity being the most fundamental ones. Fraser et al. [16,17] introducedmeans of representing the user activity in a CVE. For example, they proposed touse a wire-frame or transparent pyramid of vision to visualize the field of view ofone user. They also proposed to extend the avatar’s arms to touch the artefact(wire-framed) when portraying the grasping/moving of that object. As we haveseen, several possibilities exist to deal with awareness in a CVE but the avatarappears to be the most meaningful one: ”without sufficient embodiment, usersonly become known to one another through their (disembodied) actions; onemight draw an analogy between such users and poltergeists, only visible throughparanormal activity” [15]. Several types of avatars can be used to achieve thesegoals, from the most symbolic one, such as a head and two hands to symbolize areal trainee in Steve or a simplified body in COVET, to the most realistic one,such as the humanoids in S ecu R e V i and MRE. Existing ways of writing procedures can be found in [18], but for collaborativeprocedures an essential notion remains missing: the notion of role. Indeed, wemust specify who can do each action and this assignment is generally achievedby associating one role to each action of the scenario, like in MASCARET orSteve. The concept of role (for example the lieutenant role, played by the trainee,n MRE) accounts for the agent’s responsibilities in the team. The role that astudent plays determines the actions that this student can or must perform inthe procedure. Nevertheless, a limitation can be noticed in these environments:only a single role can be associated with a given action of the scenario. Moreover,even though it is not forbidden that more than one person play the same role,the repartition of the actions between several persons playing the same role isnot specified and will thus be uncertain: it could always be the same person whoperforms all the actions, the first person that has the will to do it, etc. In short,there will be no logic in the task repartition. As a result, we can say that theseenvironments are designed for the case where one action of the scenario can bedone only by one person, so that each person can always know whether it’s upto him to perform this action or not.
Finally, a useful functionality that is often offered in CVEs for training is thepresence of virtual humans that can play specific roles (in MRE) or replace amissing teammate (in Steve or S ecu R e V i ). Virtual humans enable the traineeto realize a collaborative procedure even if nobody else is available. In most ofthe VEs offering this facility, the virtual human who is a substitute for a missingteammate respects his role and performs the actions of the procedure he has to(Steve, S ecu R e V i ). In addition, independently of his role in the team, a virtualhuman could play various pedagogical roles, such as those proposed by Buche etal. [14]: tutor, companion or troublemaker. As we have observed in this state-of-the-art, three major functionalities can beidentified in a CVE for training: the perception of the others and their actions,the writing of collaborative scenarios with identified roles and the presence ofvirtual humans. However, some limitations can be seen in the existing CVEsfor training. These environments only allow to specify one role for each actionin the scenario, so that the repartition of actions between the team members isfixed. Furthermore, virtual humans are predictable because they always followthe procedure, without making errors.
To palliate the limitations of existing CVEs while preserving their requirements,we propose scenarios that have a strict scheduling of key actions but a flexiblerepartition of these actions between the team members. Moreover, our virtualhumans have a pedagogical profile and, depending on it, may choose to do thecorrect action in the scenario or another action. Our proposed models are cur-rently being developed under our industrial project, GVT, which is detailed inthe next section.e will begin with an overview of the actual kernel of our platform forvirtual training environments, then we will present the global humanoid module,afterwards we will focus on adaptations of the scenario model and we will finishwith detailing the mechanism of action selection.
The actual kernel of our platform is divided into four elements, which can beseen on Fig.3: – a reactive and behavioral environment.
It is an informed 3D world,made of behavioral objects which can interact with each other. The model ofbehavioral objects and interaction used is STORM (Simulation and TrainingObject-Relation Model) [19]. – an interaction engine.
This engine, the STORM engine, aims at managingcomplex interactions between STORM objects. These interactions depend onthe abilities of the STORM objects. This engine is used to determine whatcould be done in the environment, what are the possible interactions. – a scenario engine.
This engine is used to know what are the next steps ofthe procedure that the trainee can do, and its state evolves as the trainee doesachieve actions. The scenario is written in the LORA language (Languagefor Object-Relation Application), whose graphical version is inspired fromGraphcet, and which makes the writing of complex procedures possible. Moreinformation on LORA can be found in [18]. – a pedagogical engine.
This engine, employed to assist the trainer, uses thetwo engines above to decide what the trainee is allowed to do. It adapts itsreactions to the trainee, and to the pedagogical strategy used. This engineis developed in the CERV laboratory.These elements have been designed to be generic and reusable. In order to builda VE for training, we must: design STORM objects (or re-use existing ones) andspecify their abilities, describe the procedure in LORA and define a pedagogicalstrategy. Fig. 3.
Global vision of our platform .2 Modelling of humanoids A humanoid, used to reinforce users’ awareness, is either the representationof a real user in the virtual world (an avatar) or a virtual human (controlledby the system). Moreover, the modelling of a humanoid can be divided intotwo parts: a bodily part and a behavioral part. The physical part consists inthe visualization and the animation of the humanoid avatar. We use MKM (Manageable Kinematic Motion) [1] for the appearance of the humanoid and itsmovements. MKM is a real-time animation engine for synthetic humans whichautomatically synchronizes, blends and adapts captured motions. This physicalpart is encapsulated into the behavioral part, which drives it. A humanoid isa STORM object so that it is possible to interact with it in the same way aswith any other object, depending on this humanoid’s abilities. For example, onFig.2 we can see a humanoid interacting with a cable. A humanoid has two kindsof abilities: general abilities owned by every humanoid and specific additionalabilities due to the roles he plays. The behavioral aspect of a humanoid drivenby a real human is limited to his roles and his abilities. It is up to the real userto decide what to do. A virtual human has, additionally, a pedagogical profileand a mechanism of decision-making described in section 3.4. In order to make possible the writing of a collaborative scenario, we must adaptthe LORA language [18], making modifications that concern:
Roles allowed for an action
We must specify what are the roles allowed toperform an action. Therefore we associate the roles allowed and their prioritieswith each action of the procedure. Contrary to other existing CVEs, we allowseveral roles for each action, which makes some interesting properties appear: – the task repartition is not fixed anymore: the scenarios are more flexible. – we can use GVT in order to design or perfect procedures. Indeed, thanks tothe mechanism of action selection that will be presented in section 3.4, itis possible to let virtual humans realize the procedure. The observation ofvirtual humans adapting to unexpected situations, the problems they haveto cope with or simply the division of work could help to perfect a procedure(change the repartition of tasks, the number of people needed, etc ...). – implicit collaboration can emerge, such as a virtual human who could helpa real user even if it is not explicitly written in the scenario. Indeed, we canspecify that an action can be done by different roles or even by anyone thathave the suitable abilities. As a result, different persons can be allowed toperform some actions, even if a particular role is expected in priority. Thus,if the user with the priority role for an action finds himself in a situationwhere it is impossible or inconvenient to achieve it, another user may help im and perform this action. Let us take an example: a screw that maintainsan object must be unscrewed. The procedure is to hold the object and withthe other hand to unscrew. This sequence of actions must be done in priorityby worker1 or secondary by anyone. We want to realize this scenario in thedark. Therefore, the trainee that plays the role of worker1 has to hold a lightall along the scenario, so that he has only one available hand. He beginsto unscrew the screw and a virtual human that is nearby come to hold theobject maintained by the screw because this virtual human has noticed thatthe trainee wasn’t able to perform this action. This is what we call implicitcollaboration which is made possible by the mechanism of action selection. New types of actions: actions of communication and collaborative ac-tions
We must add new types of actions such as actions of communication, tomodel a communication act that has an important impact on the procedure, andcollaborative actions, to model actions made by several users at the same time: – The communication is sometimes so important in a procedure that this ac-tion is an integral part of the procedure (to give an order, to report, etc). Forexample, when a guide must help his partner to manoeuvre, the guide mustcommunicate information to his partner, in a codified manner. The commu-nication act is transcripted in the scenario by an action of communication.This action can be done by selecting the user to communicate with and themessage to transmit (for example: ”turn on the right”). – A collaborative action is an action that must be done by several users atthe same time. Such an action, for example holding a heavy object, requiresa high synchronization level between the collaborators. The first step is,for each user, to declare his intention to collaborate. In the real life thisdeclaration is done by saying that we are waiting for someone to help usor by beginning the action to show our need for help. This mechanism willbe transcripted in the scenario thanks to a special action: notification ofintention to collaborate . This action, one for each collaborator, will serve for ahumanoid to declare that he is ready to collaborate. Once every collaboratorhave done this declaration, the collaborative action automatically begins aswe can see on Fig.4. Nevertheless, we must set up a timeout in order to cancelthe effective notification of intention to collaborate if another notification ofintention to collaborate is still missing after this time-limit.
Action of grasping and humanoid’s hands
The maintenance activity isbased on actions made thanks to tools that the worker holds in his hands. Themanagement of the state of the worker’s hands and the actions of grasping andlaying of tools are thus omnipresent in maintenance procedures. – In collaborative scenarios and especially adaptive scenarios where we don’tknow who will realize some actions, it is very constraining to specify everyactions in the scenario like taking and putting a tool. In order to make the ig. 4. representation of a collaborative action in the LORA languagescenarios more flexible and easier to write, the actions of catch and laying oftools are not written in the scenario anymore. A resource manager deducesthe catch of a tool from an action in the scenario that requires this tool,and proposes the laying of an object in order to free the humanoid’s hands.Furthermore, in reality, such actions are implicit in the specification of amaintenance procedure. Thus, making it implicit also in the virtual scenariowould make it fit better to the real written procedure. – Information that we must make explicit for each action is the state of hu-manoid’s hands required and their states after the action. Four states areavailable: free, holding an object, busy and indifferent. A resource managerreasons on these states in order to predict if a humanoid is entering a se-quence of actions that he won’t be able to finish alone (for example because itwill monopolize his two hands), so that an implicit collaboration is required.
When a virtual human takes part in a collaborative procedure, he must regularlychoose the next action he will do, taking into account a lot of parameters. Tomanage the behavior of the entities that populate a VE, two approaches exist:a centralized one (example:[20]), where a global mechanism manages the behav-ior of each entity and gives them orders, and a local one (example: oRis [21]),where each entity is responsible for its own behavior and the global behavior ofthe environment is emergent (for example multi-agent system and autonomousvirtual humans). In our applicative domain (industrial maintenance training),we must deal with a procedure to respect and virtual humans with a part of au-tonomy. That is why we have chosen to mix these two approaches, by modellingautonomous virtual humans that are incited to follow the scenario. Furthermore,this approach tends to reduce the combinatorics of possible actions.he module of action repartition makes a global repartition of the actionsamong the humanoids, respecting the scenario requirements. But it is only apropositional repartition and it is up to the virtual human to make an individualchoice thanks to its own decisional module. These two modules form the wholeprocess of ”action selection”, illustrated on Fig.5, that we describe now.
Fig. 5.
The mechanism of action selection
Action repartition
Several roles can be associated with one action, but thisleads to a problem: how to evaluate who is the best candidate for this action. Thisevaluation will be done by this module of action repartition. This module aimsat sorting the candidates for each possible action giving them a score accordingto various criteria such as the role priority, the proximity with the object tointeract with, the easiness for the humanoid to do the action, etc. This lastcriterion will be determined thanks to a resource manager which has planificationfunctionalities and which is able to predict blocking situations. The criteria totake into account can be various but are easy to parametrize: we must attributeto each one a weight, possible values and a coefficient for each value. A positivecoefficient indicates that this value tends to favour the choice of the associatedhumanoid. The goal of this module is dual: the first one is to help a virtualhuman in his decisional process because, for each action allowed in the scenario,this module gives the best candidate from the scenario point of view. The secondone is to propose a pedagogical feedback to the real user about the best actionto choose. To illustrate these goals, let us take an example: a virtual human thatcan do two actions in the scenario. The first, action 1, is easy for him becausehe is near the object to interact with, but a lot of people can do it. The second,action 2, is less easy, but in theory a second person could do it. Let us imaginethat this second person can not do it, because he is busy for example. Withoutthis module the virtual human would not have enough data to choose betweenthese two actions, but with this module, he will see that for action 2 he is thenly one who can do it, so he will deduce that it would be better from a globalpoint of view to choose this one. As a result, this module makes propositions ofrepartition that is coherent with the scenario in order to help the humanoid totake a decision about the next action to do.
Decision-making
The last step of the mechanism of action selection is the de-cisional module of a virtual human. Considering the results of the action repar-tition module, and the pedagogical profile of the virtual human, this modulechooses the next action to do. The first step is to collect the actions providedby other modules. The interaction engine provides the actions that are possible.The action repartition module provides the actions allowed in the scenario, withinformation about the best actions to do. The pedagogy engine can provide spe-cific demands, for pedagogical reasons. The second step is to tag these actions inorder to identify specific ones like important actions (collaborative ones, urgentones), hindering actions, etc. The final step is to respect the pedagogical profileof the virtual human to select the next action to do. The pedagogical profile isa set of propensities (for example to respect the procedure, to make mistakes,etc), it can correspond to the pedagogical roles proposed by Buche et al. [14].
Synthesis
In short, allowing several roles for one action leads to some inter-esting properties: flexibility, implicit collaboration, perfecting of scenarios. Butin return, the global process of action selection must be divided into two parts:a global repartition which is optimized from the scenario viewpoint, and anindividual choice made by each humanoid.
Our platform has been developed within the GVT project (Giat Virtual Train-ing). This project is a Research/Industry collaboration, with three partners:IRISA and CERV laboratories, and NEXTER-Group (previously Giat Indus-tries). This last partner is an important French company, specialized in militaryequipments such as the Leclerc tank. The main current application of our plat-form (called GVT) is virtual training on NEXTER’s maintenance procedures.GVT could be used, not only for maintenance procedures, but also for train-ing on various kinds of procedures, such as diagnosis procedures, procedures ofvehicle starting, etc. And nowadays, collaborative procedures are also possible.We are currently integrating our propositional models to support collabora-tion in GVT in order to validate them. As GVT leans on OpenMASK platform,we took advantage of its distribution possibility to distribute GVT. Now sev-eral distant visualizations can be created, allowing different users to connecton the same scene. For example the trainer can freely navigate in the scenewhile a trainee is training on a procedure. The integration of humanoids is un-der progress: we can move avatars in the environment, and a first collaborativeprocedure with a virtual human has been created (Fig.1). This procedure is theaking down of the auxiliary winch on a vehicle. Two roles are needed: an opera-tor who realizes the main part of the procedure and an assistant who helps himto manoeuvre the winch. The trainee plays the role of the operator and a basicvirtual human plays the role of the assistant. The virtual human have specificactions to do like pulling the cable (Fig.2). This action is automatically triggeredwhen the trainee realizes the correct action that precedes in the procedure.
In our propositional models, we allow the association of several roles with onescenario action, and let the repartition module make propositions about the bestcandidates for each action. The repartition of the tasks is not fixed any more, andwe believe that this will extend the possibilities offered by a CVE for training:flexibility of the scenarios, emergence of implicit collaboration between virtualhumans and real users, possibility of perfecting procedures by letting virtualhumans realize the procedure and by seeing the actions repartition. Furthermore,we will be able to parametrize the behavior of our virtual humans thanks totheir pedagogical profiles. Thus it will be possible to create virtual humans thatsimply assist the trainee performing the actions he has to, but also to createvirtual humans who can make mistakes or who want to hinder the trainee. Thesepossibilities will increase the pedagogical opportunities.
Acknowledgement.
The authors would like to thank all the members of theGVT project as well as Fabien Lotte, St´ephane Donikian, St´ephane Le Foll andGr´egoire Jacob for their helpful comments on this paper.
References
1. S. Menardais, R. Kulpa, F. Multon, and B. Arnaldi. Synchronization for dy-namic blending of motions. In
SCA ’04: Proceedings of the 2004 ACM SIG-GRAPH/Eurographics symposium on Computer animation , pages 325–335, NewYork, NY, USA, 2004. ACM Press.2. F. Lamarche and S. Donikian. Automatic orchestration of behaviours throughthe management of resources and priority levels. In
AAMAS’02 : Proceedings ofAutonomous Agents and Multi Agent Systems , Bologna, Italy, July 15-19 2002.3. T. Duval, A. Lecuyer, and S. Thomas. Skewer: a 3d interaction technique for 2-usercollaborative manipulation of objects in virtual environments. , 0:69–72, 2006.4. J.C. Oliveira, M. Hosseini, S. Shirmohammadi, M. Cordea, E. Petriu, D. Petriu,and N.D. Georganas. Virtual theater for industrial training: A collaborative virtualenvironment. In
CSCC 2000: Proceedings of 4th WORLD MULTICONFERENCEon Circuits, Systems, Communications & Computers , Greece, July 2000.5. J. C. Oliveira, S. Shirmohammadi, and N. D. Georganas. A collaborative virtualenvironment for industrial training. In
VR ’00: Proceedings of the IEEE VirtualReality 2000 Conference , page 288, Washington, DC, USA, 2000.. M. Hosseini and N. D. Georganas. Collaborative virtual environments for training.In
MULTIMEDIA ’01: Proceedings of the ninth ACM international conference onMultimedia , pages 621–622, New York, NY, USA, 2001. ACM Press.7. W. L. Johnson and J. Rickel. Steve: an animated pedagogical agent for proceduraltraining in virtual environments.
SIGART Bull. , 8(1-4):16–21, 1997.8. J. Rickel and W. L. Johnson. Animated agents for procedural training in virtualreality: Perception, cognition, and motor control.
Applied Artificial Intelligence ,13(4 - 5):343–382, May 1999.9. J. Rickel and W. L. Johnson. Virtual humans for team training in virtual reality.In
Proceedings of the Ninth International Conference on Artificial Intelligence inEducation , 1999.10. J. Rickel and W. L. Johnson. Extending virtual humans to support team trainingin virtual reality.
Exploring artificial intelligence in the new millennium , pages217–238, 2003.11. D. Traum, J. Rickel, J. Gratch, and S. Marsella. Negotiation over tasks in hybridhuman-agent teams for simulation-based training. In
AAMAS ’03: Proceedings ofthe second international joint conference on Autonomous agents and multiagentsystems , pages 441–448, New York, NY, USA, 2003. ACM Press.12. W. Swartout, J. Gratch, R. Hill, E. Hovy, S. Lindheim, R.and Marsella, J. Rickel,and D Traum. Simulation meets hollywood: Integrating graphics, sound, story andcharacter for immersive simulation. Multimodal Intelligent Information Presenta-tion, 2005.13. R. Querrec, C. Buche, E. Maffre, and P. Chevaillier. S´ecuR´eVi : virtual envi-ronments for fire-fighting training. In Simon Richir, Paul Richard, and BernardTaravel, editors, th virtual reality international conference (VRIC’03) , pages 169–175, Laval, France, May 2003.14. C. Buche, R. Querrec, P. De Loor, and P. Chevaillier. Mascaret: Pedagogical multi-agents system for virtual environment for training. In CW ’03: Proceedings of the2003 International Conference on Cyberworlds , pages 423–430, Washington, DC,USA, 2003. IEEE Computer Society.15. S. Benford, J. Bowers, L. E. Fahlen, C. Greenhalgh, and D. Snowdon. User em-bodiment in collaborative virtual environments. In
CHI ’95: Proceedings of theSIGCHI conference on Human factors in computing systems , pages 242–249, NewYork, NY, USA, 1995. ACM Press/Addison-Wesley Publishing Co.16. M. Fraser, S. Benford, J. Hindmarsh, and C. Heath. Supporting awareness andinteraction through collaborative virtual interfaces. In
UIST ’99: Proceedings ofthe 12th annual ACM symposium on User interface software and technology , pages27–36, New York, NY, USA, 1999. ACM Press.17. M. Fraser, T. Glover, I. Vaghi, S. Benford, C. Greenhalgh, J. Hindmarsh, andC. Heath. Revealing the realities of collaborative virtual reality. In
CVE ’00:Proceedings of the third international conference on Collaborative virtual environ-ments , pages 29–37, New York, NY, USA, 2000. ACM Press.18. N. Mollet and B. Arnaldi. Storytelling in virtual reality for training. In
Edutain-ment , pages 334–347, 2006.19. N. Mollet.
De l’Objet-Relation au Construire en Faisant: Application a la spec-ification de scenarios de formation a la maintenance en Realite Virtuelle . PhDthesis, INSA Rennes, 2005.20. G. Ridsdale and T. Calvert. Animating microworlds from scripts and relationalconstraints.
Computer Animation ’90 , pages 107–118, 1990.1. V. Rodin and A. N´ed´elec. oris: an agents communication language for distributedvirtual environment. In