A Task Allocation Approach for Human-Robot Collaboration in Product Defects Inspection Scenarios
AA Task Allocation Approach for Human-Robot Collaboration inProduct Defects Inspection Scenarios
Hossein Karami , Kourosh Darvish , Fulvio Mastrogiovanni Abstract — The presence and coexistence of human operatorsand collaborative robots in shop-floor environments raises theneed for assigning tasks to either operators or robots, or both.Depending on task characteristics, operator capabilities and theinvolved robot functionalities, it is of the utmost importance todesign strategies allowing for the concurrent and/or sequentialallocation of tasks related to object manipulation and assembly.In this paper, we extend the F
LEX
HRC framework presentedin [1] to allow a human operator to interact with multiple,heterogeneous robots at the same time in order to jointly carryout a given task. The extended F
LEX
HRC framework leveragesa concurrent and sequential task representation framework toallocate tasks to either operators or robots as part of a dynamiccollaboration process. In particular, we focus on a use caserelated to the inspection of product defects, which involves ahuman operator, a dual-arm Baxter manipulator from RethinkRobotics and a Kuka youBot mobile manipulator.
I. I
NTRODUCTION
Robots are increasingly adopted in industrial environmentsto carry out dangerous, repetitive, or stressful tasks. Theintroduction of robots in production lines has improved anumber of key performance indicators, and has addressed amarket-driven goods growing demand with quality products[2]. However, due to well-known limitations of robot per-ceptual, cognitive, and reasoning capabilities, certain tasks,which are difficult to model or require a higher-level ofawareness because they cannot be easily modelled nor for-malised, are still better handled by human operators. Theintroduction of collaborative robots (nowadays referred to as cobots ) in recent years has contributed to relax those limi-tations, and implicitly promoted human working conditions[3]. Among the tasks typically considered stressful, qualitycontrol and defects inspection play a key role in definingthe quality of a finished or semi-finished product. Currently,trained and expert personnel is tasked with establishingbenchmarks and examining products quality, which requireprolonged focus and continuous attention. In this work, weargue that the collaboration between an experienced humanoperator and a robot may lead to higher rates in defectsspotting, overall productivity, and safety [4], [5].Human-robot collaboration (HRC) is defined as the pur-poseful interaction among humans and robots in a sharedspace, and it is aimed at a common goal. A natural collabo-ration requires a robot to perceive and correctly interpret theactions (as well as the intentions) of other humans or robots Department of Informatics, Bioengineering, Robotics, and SystemsEngineering, University of Genoa, Via Opera Pia 13, 16145, Genoa, Italy. Dynamic Interaction Control, Istituto Italiano di Tecnologia, Genova,Italy.Corresponding authors email: [email protected]. [6], [7]. The main goal of this paper is to extend the human-Fig. 1: A human operator and and two robots collaborating ina product defect inspection scenario: the mobile manipulatorsupplies a human operator and the dual-arm manipulator withobjects to inspect.robot collaboration model proposed in [8], referred to asF
LEX
HRC, along two directions. On the one hand, to allowfor a collaboration model taking multiple, heterogeneousrobots into account, while the original work in [8] consideredmodels with one human operator and one robot. On the otherhand, introduce a use case whereby a human operator anda robot must collaboratively perform a defects inspection,whereas the original work focused on assembly tasks.The scenario we consider is shown in Figure 1. A mobilemanipulator (in our case, a Kuka youBot) picks-up objectsto be inspected (wooden pieces) from a warehouse area(a marked region in the workspace), and carries them outto deliver them to human operators or another robot (inour case, a dual-arm Baxter manipulator) for inspection [9].When the object to inspect is delivered to human operators,these undertake the foreman task [10], [11], and then theobject is passed to the manipulator for a further vision-basedinspection. Afterwards, the manipulator sorts the object outas faulty or non faulty in two different boxes. Scenariosmodelling defects inspection impose functional requirementswhich are partially in overlap with the ones considered in[8] for the assembly of semi-finished products. The mainfunctional requirement in quality control is the validationof products quality with a reliable estimation. In an HRC a r X i v : . [ c s . R O ] S e p rocess, such a requirement can be met by a double-checkcarried out by an expert operator in case the defects classi-fication accuracy as provided by the robot is below a pre-specified threshold. However, differently from the work in[12], [13], [14], whereby a visual inspection is carried outby a robot, in order to validate the quality of products anintegration of auditory, tactile, and visual perception is likelyto be needed [15], [16]. Such an integration is still an openissue and it is not considered in this paper.This paper introduces and discusses C ONC
HRC, a frame-work extending F
LEX
HRC that addresses the need forconcurrent, multi human-robot collaboration in industrialenvironments, and validates the models in an inspectionuse case. The novelty of the approach is two-fold: (i) thedesign and development of an AND/OR graph based multihuman-robot collaboration model that allows for concurrent,modelled, operations in a team made up of multiple humanoperators and/or robots; (ii) the description of a particularinstance of such a cooperation model, implemented within anexisting human-robot collaboration architecture, and extend-ing it whereby a human operator, a mobile manipulator, anda dual-arm manipulator collaborate for a defect inspectionpurpose. In the paper, the focus is on the concurrent HRCmodel for the quality control task, and therefore we decidedto simplify the robot perception system.The paper is organized as follows. Section II discussesrelated work. Section III introduces the C
ONC
HRC architec-ture, and Section IV formalises the concurrent model. SectionV lays the experimental scenario and the related discussion.Conclusions follow.II. R
ELATED W ORK
For a natural human-robot collaboration, different aspectssuch as safety, robot perception, task representation, andaction execution must be considered when designing acollaborative-friendly workspace [17], [5]. This paper fo-cuses on task representation when multiple human operatorsand/or robots group as a team to reach a common goal,which is a priori known to all collaborators, either humansor robots. The uncertainties in perception, task representa-tion and reasoning that a robot must face increase whencollaborating with humans, because a natural cooperation,i.e., a one in a way similar to human-human teams [18], mayrequire the robot to make sense of or even anticipate humanintentions. The need arises to provide robots with reasoningcapabilities about the state of the human-robot cooperationprocess, suitable to be executed online.Although approaches based on offline planning and taskallocation fulfil a requirement related to the effectivenessof the collaboration [19], [20], they neither ensure such anatural collaboration nor address its intrinsic uncertainties.Differently, the approaches described in [21], [22], [1], [8]are aimed at enhancing the naturalness and the flexibilityof the collaboration based on online task allocation and/orcontingency plans, such that the robot is able to adaptto human decisions on the spot and uncertainties. Such flexibility requires a rich perception for recognising humanactions as well as the collaboration state [8].Some of the methods applied for robot action planning incollaboration scenarios include
Markov Decision Processes [23], [24],
Task Networks [22], [20],
AND/OR graphs [25],[19], [1], and STRIPS-based planners [26]. Among thesemethods, finding the priors and the reward function forMarkov Decision Processes and the exponential growth ofthe computational load of STRIPS-based planners make themvery difficult to be adopted in practice. Task Networks andAND/OR graphs ensure that the generated collaborationmodels are in accordance with domain expert desiderata,hence guaranteeing shared mental models between humanoperators and robots. In order to allocate tasks to human op-erators or robots, and to meet such collaboration constraintsas limited resources, a common approach in the literatureis to maximise the overall utility value of the collaboration[27] on the basis of multi-objective optimisation criteria.However, in these examples the number of human operatorsor robots is limited.In order to enhance the efficiency of the collaboration,and to face the inherent limitations owing to workspace con-straints, human skills, and robot capabilities, an approach canbe to raise the number of human operators or heterogeneousrobots involved in the collaboration. To this aim, humanoperators and robots must schedule their actions according toresources, timings, and skill constraints. An example can befound in [28] whereby concurrent cooperation models areformalised according to relational activity processes. Theauthors in that study model the cooperation and predictfuture actions using a Monte Carlo method along withlearning by demonstration. A similar approach is adopted in[29], whereby a temporal graph plan with the considerationof action durations has been applied. Another illustrationof concurrent HRC, with a probabilistic formulation dueto uncertainties, is presented in [30], where a concurrentMarkov Decision Process is adopted.In previous work [1], [8] where we demonstrated a flexiblecollaboration between human operators and robots, this paperextends the notion of AND/OR graph to a concurrent model,and adopts it to model multi human-robot collaborationscenarios. This is further detailed in Section IV.III. S
YSTEM ’ S A RCHITECTURE
Figure 2 depicts the overall architecture of the C
ONC
HRCframework. The architecture is made up of three layers,including a perception layer in green, a representation layerin blue, and an action layer in red. The perception layerprovides information regarding the activities carried out byhuman operators, a part’s defect status, and object locationsin the robot workspace. The representation layer forms theconcurrency model, stores the necessary knowledge, andmanages task execution to reach the collaboration goal. Theaction level simulates and executes robot actions.The perception layer encapsulates three modules, whichare called
Human Activity Recognition , Product Defect De-tection , and
Scene Perception . The latter two modules pro-ig. 2: System’s architecture for a multi human-robot collaboration model in a defects detection scenario.vide the
Knowledge Base module with information aboutthe status of the workspace, human operators, and robots,whereas the former communicates detected human activitiesto the
Task Planner . Human Activity Recogntion obtainsinertial data originating from wearable sensors worn byhuman operators, and run a series of algorithms to detectand classify performed actions. Those are modelled using
Gaussian Mixture Modelling (GMM) and
Regression [31],[1]. In our setup, defects detection is considered as a classifi-cation problem.
Product Defect Detection exploits the imagescoming from a robot-centric camera to detect defects.The action layer is made up of three modules, namely
Robot Execution Manager , Simulator , and
Controller . The
Robot Execution Manager module receives discrete, sym-bolic commands from the
Task Planner , maps them toactual values, and drives the behaviour of the
Controller or the
Simulator . This module retrieves information aboutthe workspace, human operators and robots from the
Knowl-edge Base . The
Robot Execution Manager is in charge ofsequencing robot behaviours, on the basis of the plan asprovided by the
Task Representation module. It also providesan acknowledgement to the
Task Planner upon the executionof a command by the robots. The
Simulator module is aimedat predicting the outcome of robot behaviours before theiractual execution. It simulates a closed-loop model of therobot and the controller, by solving the ordinary differentialequations online. The
Controller receives the configuration(in joint space) or the task space command (in the Cartesianspace) from the
Robot Execution Manager . It computes thejoints velocity reference values at each control time step tothe robot, while receiving feedback from it [32].The representation layer embeds
Task Representation , Task Planner , and the
Knowledge Base module. In theC
ONC
HRC, an AND/OR graph with several layers repre-sents the collaborative task [1]. In order to model concur-rency in a multi-agent collaboration scenario, the AND/ORgraph based F
LEX
HRC framework has been extended, as de-scribed in the next Section. Along with the AND/OR graph,the
Task Planner module is in charge of decision makingand the adaptation of the ongoing parallel tasks. To do so,the
Task Planner provides a set of achieved cooperation states or transitions between states to the
Task Representation module, and receives the set of allowed cooperation statesand transitions with the associated costs to follow. Later,it associates each state or state transition with an orderedset of actions, and according to the workspace’s, humanoperator’s, and robot’s status, along with online simulationresults, it assigns actions to the either human operators orrobots. Finally, it informs each human operator or robotinvolved in the cooperation about the action to follow. Oncean action is carried out, it receives the acknowledgementfrom the action level and updates its internal structure. The
Knowledge Base stores all relevant information to make thecooperation progress, as better described in [1].IV. A C
ONCURRENT M ODEL FOR M ULTI - AGENT C OOPERATION
In this Section, we describe first a multi human-robotcooperation model based on a -layer AND/OR graph,then we consider an extended n -layer AND/OR graph, andfinally a concurrent model based on a constrained n -layerconfiguration, which we refer to as a c -layer AND/OR graph. A. -layer AND/OR Graphs In order to formalise the multi human-robot cooperationprocess in C
ONC
HRC we adopt AND/OR graphs [33], [34],[35], as discussed above. An AND/OR graph allows forrepresenting procedures to follow, which can be decomposedin subproblems as parts of the graph, as well as the logic relationships among them, i.e., the graph interconnectivity.The root node conventionally represents the goal state ofthe process being modelled, and achieving the goal meanstraversing the graph from leaf nodes to the root node viaintermediate nodes and hyper-arcs according to its structure.A -layer AND/OR graph G can be formally defined asa -ple (cid:104) N, H (cid:105) where N is a set of | N | nodes, and H is aset of | H | hyper-arcs. An hyper-arc h ∈ H induces the set N c ( h ) ⊂ N of its child nodes, and the singleton N p ( h ) ⊂ N made up of a parent node, such that h : N c ( h ) → N p ( h ) . (1)Furthermore, we define n ∈ N as a leaf node if n is not aparent node for any hyper-arc, i.e., if h ∈ H does not existuch that n ∈ N p ( h ) , or as a root node if it is the only nodethat is not a child node for any hyper-arc, i.e., if h ∈ H doesnot exist such that n ∈ N c ( h ) .In a multi human-robot cooperation scenario, each node n ∈ N represents a cooperation state , e.g., faulty objectinside box , whereas each hyper-arc h ∈ H represents a(possibly) many-to-one transition among states, i.e., activitiesperformed by human operators and/or robots, which makethe cooperation move forward, such as the robot puts thefaulty object into the box . The relation among child nodes inhyper-arcs is the logical and , whereas the relation betweendifferent hyper-arcs inducing on the same parent node is thelogical or , i.e., different hyper-arcs inducing on the sameparent node represent alternative ways for a cooperationprocess to move on. Each hyper-arc h ∈ H implementsthe transition in (1) by checking the requirements definedby nodes in N c ( h ) , executing actions associated with h ,and generating effects compatible with the parent node. Eachhyper-arc h ∈ H executes an ordered set A ( h ) of actions ,such that A ( h ) = ( a , . . . , a | A | ; (cid:22) ) , (2)where the precedence operator (cid:22) defines the pairwise ex-pected order of action execution. The sequence can bescripted or planned online [26]. Before an hyper-arc h isexecuted, all actions a ∈ A ( h ) are marked as undone , i.e., done ( a ) ← f alse . When one action a is executed by anyagent, its status changes to done ( a ) ← true . An hyper-arc h ∈ H is marked as solved , i.e., solved ( h ) ← true iff allactions a ∈ A ( h ) are done in the expected order. In a similarway, nodes n ∈ N may be associated with a (possiblyordered) set of processes P ( n ) , which are typically robot behaviours activated in a cooperation state but not leading toa state transition.It is possible to introduce the notion of feasibility. Anode n ∈ N is feasible , i.e., feasible ( n ) ← true , iff asolved hyper-arc h ∈ H exists, for which n ∈ N p ( h ) , and met ( n ) ← f alse , i.e., ∃ h ∈ H. ( solved ( h ) ∩ n ∈ N p ( h ) ∩ ¬ met ( n )) . (3)All leaf nodes in an AND/OR graph are usually feasible atthe beginning of the multi human-robot cooperation process,which means that the cooperation can be performed in manyways. An hyper-arc h ∈ H is feasible , i.e., feasible ( h ) ← true , iff for each node n ∈ N c ( h ) , met ( n ) ← true and solved ( h ) ← f alse , i.e., ∀ n ∈ N c ( h ) . ( met ( n ) ∩ ¬ solved ( h )) . (4)Once an hyper-arc h i ∈ H is solved, all other feasiblehyper-arcs h j ∈ H \ { h i } , which share with h i at leastone child node, i.e., N c ( h i ) ∩ N c ( h j ) (cid:54) = ∅ , are marked asunfeasible, in order to prevent the cooperation process toconsider alternative ways to cooperation that have becomeirrelevant.Given and AND/OR graph, the multi human-robot coop-eration process is modelled as a graph traversal procedurewhich, starting from a set of leaf nodes, must reach the root node by selecting hyper-arcs and reaching states inone of the available cooperation paths , depending on thefeasibility statuses of nodes and hyper-arcs. According tothe graph structure, multiple cooperation paths may exist,meaning that multiple ways to solve the task may be equallylegitimate. The traversal procedure dynamically follows thecooperation path that at any time is characterised by thelowest cost. The entire algorithm has been described in [1],[8]. The traversal procedure suggests to human operatorsagents actions in the hyper-arcs that are part of the path,and sends to robots the actions they must execute. Humanoperators can override the suggestions at any time, executingdifferent actions, which may cause the graph to reach a statenot part of the current path. When this happens, C ONC
HRCtries to progress from that state onwards [1], [8]. Thismechanism enables C
ONC
HRC to pursue an optimal pathleading to the solution, while it allows human operatorsto choose alternative paths. As long as the multi human-robot cooperation process unfolds, and the AND/OR graphis traversed, we refer with N f and H f to the sets of currently feasible nodes and hyper-arcs, respectively. We say thatan AND/OR graph G is solved , i.e., solved ( G ) ← true , iff its root node r ∈ N is met, i.e., met ( r ) ← true .Otherwise, if the condition N f ∪ H f = ∅ , i.e., there areno feasible nodes nor hyper-arcs, then the multi human-robot cooperation process fails, because there is no feasiblecooperation path leading to the root node. B. n -layer AND/OR Graphs A n -layer AND/OR graph G n can be recursively definedas a -ple (cid:104) Γ , Θ (cid:105) where Γ is an ordered set of | Γ | up to ( n − -layer AND/OR graphs, such that: Γ = (cid:0) G , . . . , G | Γ | ; (cid:22) (cid:1) , (5)and Θ is a set of | Θ | pairwise transitions between them.In (5), the AND/OR graphs are ordered according to theirlayer. Lower-layer AND/OR graphs are characterised bya decreasing level of abstraction, i.e., they are aimed atmodelling the HRC process more accurately. Transitions in Θ define how different AND/OR graphs in Γ are connected,and in particular model the relationship between graphsbelonging to different layers.If we recall (1) and we contextualise it for an AND/ORgraph G n = (cid:104) N n , H n (cid:105) , we observe that a given hyper-arcin H n represents a mapping between the set of its childnodes and the singleton parent node. We can think of ageneralised version of such a mapping to encompass a wholeAND/OR graph G n − = (cid:104) N n − , H n − (cid:105) , where the set ofchild nodes is constituted by the set N n − L of leaf nodes,and the singleton parent node by the graph’s root node r n − ∈ N n − . As a consequence, a transition T ∈ Θ can bedefined between a hyper-arc h ∈ H n and an entire AND/ORgraph G n − , such that T : h → G n − , (6)subject to the fact that appropriate mappings can be definedbetween the set of child nodes of h and the set of leaf nodesf the deeper graph, i.e., M : N c ( h ) → N L ∈ N n − , (7)and between the singleton set of parent nodes of h n and theroot node of the deeper graph, i.e., M : N p ( h ) → r n − ∈ N n − . (8)Mappings M and M must be such that the correspond-ing information in different layers should be semanticallyequivalent , i.e., it should represent the same informationwith a different representation granularity. The same appliesfor N p ( h ) and the root of G n − . Once these mappings aredefined, it easy to see that G n has a tree-like structure, wheregraphs in Γ are nodes and transitions in Θ are edges.An AND/OR graph G n is feasible, i.e., feasible ( G n ) ← true iff it has at least one feasible node or hyper-arc. If atransition T ∈ Θ exists in the form (6), a hyper-arc h ∈ H n is feasible iff the associated AND/OR graph G n − , isfeasible, i.e., ∀ T. (cid:0) feasible ( h ) ↔ feasible ( G n − ) (cid:1) . (9)As a consequence, when the nodes in N n − L of G n − becomes feasible, the hyper-arc h in G n becomes feasible aswell. Furthermore, the hyper-arc h is solved iff the associatedAND/OR graph G n − is solved, i.e., ∀ T. (cid:0) solved ( h ) ↔ solved ( G n − ) (cid:1) . (10) C. c -layer AND/OR Graphs A concurrent AND/OR graph is modelled as a restrictionof a n -layer AND/OR graph whereby the n -th layer isaimed at modelling the termination condition for the wholehierarchy of ( n − -layer graphs, and the latter modeldifferent, concurrent activities part of the HRC process. A c -layer AND/OR graph must also specify if and how nodesbelonging to separate lower-layer graphs are synchronised.Analogously to an n -layer graph, a c -layer AND/OR graph G c can be defined as a -ple (cid:104) Γ c , Θ c (cid:105) where Γ c is an orderedset of | Γ c | up to ( n − -layer AND/OR graphs, such that: Γ c = (cid:0) G , . . . , G | Γ c | ; (cid:22) (cid:1) , (11)and Θ c is a set of | Θ c | pairwise transitions between them.Whilst the considerations related to n -layer AND/ORgraphs apply for c -layer AND/OR graphs, the compositionof the constituting sets of nodes and hyper-arcs may differ.Let us recall that for a generic AND/OR graph G we refer to N as its set of nodes, and with H as its set of hyper-arcs, andlet us consider two AND/OR graphs G i and G j ∈ Γ c . Let uslimit ourselves to a weak notion of independence betweengraphs. We consider G i and G j as mutually independent iff there is no node in G i (respectively, G j ) that needs to bemet before another node of G j (respectively, G i ). If this isthe case, G i (respectively, G j ) can be modelled as a generic n -layer AND/OR graphs (cid:104) N i , H i (cid:105) (respectively, (cid:104) N j , H j (cid:105) ).Otherwise, if G i is dependent on G j , i.e., a node n j in G j must be met before another node n i in G i can be met , weneed to formally model it as an external dependence. To this aim, and in general terms, we augment the set ofnodes G i with a set of dependence nodes, whose associatedlogic predicates met are entangled with the correspondingnodes in G j , such that their truth values always correspond.A node n e of an AND/OR graph G i is said to be entangled with a node n j of an AND/OR graph G j , with i (cid:54) = j , iff forthat node met ( n e ) ↔ met ( n j ) (12)and n e is a leaf node for G i , i.e., N c ( n e ) = ∅ . Then, a dependent AND/OR graph G i is defined as a -ple (cid:104) N ci , H ci (cid:105) ,such that N ci = N i ∪ { n e , . . . , n eη } , i.e., the union betweenthe set of nodes N i as if the graph were not dependent on anyother graph, plus the set of the entangled nodes, and H ci = H i ∪ { h e , . . . , h eλ } , i.e., the union between the set of hyper-arcs H i as if the graph were not dependent on any othergraph, plus the set of the hyper-arcs reliant on entanglednodes. V. E XPERIMENTAL V ALIDATION
A. Implementation of the Multi Human-Robot CollaborationProcess for Defects Inspection
In order to validate the effectiveness of C
ONC
HRC, weimplemented an abstract defects inspection scenario. Thescenario has been briefly described in the Introduction, andis represented in Figure 1. A Kuka youBot omni-directionalmobile manipulator is used to pick up objects from a ware-house area , and brings them close to the defects inspection cell, where a human operator and a dual-arm Baxter robotare expected to collaborate. The youBot and the objects to bemanipulated are localised in the workspace using an externalmotion capture system based on passive markers, i.e., asystem composed of OptiTrack-Flex motion capturecameras. Baxter is provided with the standard grippers, andis equipped also with a RGB-D camera mounted on therobot head and pointing downward, which is used to acquireimages for defects inspection. Since, in our case, the focus ison the multi human-robot collaboration process, we decidedto over-simplify the inspection, which is surrogated usingQR tags corresponding to faulty , non faulty , Na , respectively.Actions carried out by human operators are perceived viatheir inertial blueprint via an LG G Watch R (W110) smart-watch, worn at the right wrist. Data are transmitted througha standard WiFi link to a workstation. The workstation isequipped with an Intel(R) core i7-8700 @ 3.2 GHz × . rad/s . Limits on the youBot’s linear and angular velocitiesare . m/s and . rad/s , respectively. These limits areapplied to both simulated and real robots. Action modelsforeseen for human activity recognition are simply pick up and put down . Instead, actions used for Baxter arms include approach , grasp , ungrasp , hold on , stop , check object status ,whereas for the youBot arm we considered only approach .ig. 3: The collaboration graph for defects inspection.Our scenario includes three physical agents, i.e., a humanoperator, Baxter and youBot, but five logical agents, i.e.,the operator, the Baxter left arm, the Baxter right arm, theyouBot base, and the youBot arm. However, one plannermanages both Baxter arms, and likewise one planner man-ages the youBot base and arm, so they are used sequentially.In the scenario, objects are randomly placed in the warehousearea. Objects are cylinders labeled with three different QRcode types (Figure 4). The youBot must find each object,move towards it, pick it, take it to the area where thehuman operator and the Baxter are located, and hand it overthe operator. This sequence is repeated until all objects aredelivered. On the other side of the collaboration scenario,the Baxter starts its operations when the human operatorputs down an object on the table in front of the robot. Bydefault, its right arm is used to pick the object up, and tocheck whether it is faulty, non-faulty or the defect cannot beassessed. If the object is faulty, it is placed in a faulty boxclose to the right arm, or in case of a non-faulty object, theobject is handed over to the left arm to be placed in a non-faulty box. If the object level of defects cannot be assessed,then it is handed back to the human operator for an ad hoc assessment. This process is repeated for all objects. B. Description of the Experiment
Figure 3 shows a c -layer concurrent AND/OR graph,which is composed of two -layer AND/OR graphs, for theyouBot ( G ) and the Baxter ( G ), respectively. Entanglednodes of both graphs are depicted in red, which makes graph G dependent on graph G . In order for the leaf node of G (i.e., new object ) to be feasible, the root node of G (i.e., obj on table ) must be met.During the HRC process, the human operator is typicallyclose to the Baxter, as shown in Figure 5. When the youBotapproaches, the operator executes a discrete gesture movingan arm upward in order to announce a pick up action. Oncethe gesture is detected, youBot releases the object opening Fig. 4: Four tagged cylinders used in our scenario.the end-effector to hand it over. Afterwards, the operatorannounces via a put down gesture the fact that the object toinspect has been located on the table for the Baxter to startinspection.Figure 5 shows a typical run of the collaboration process.In the initial configuration, shown in Figure 5a, both thehuman operator and the robots are in stand by mode. TheyouBot moves towards the next object to inspect ( obj in ws state), according to graph G . The object is selected on thebasis of the time it takes to perform the whole operationin simulation. After approching the object ( youbot+obj ), theyouBot’s arm attempts grasping (Figure 5b), and then pickingit up ( obj picked ). In the meantime, the Baxter is waiting forhuman operator actions to start collaboration. The youBotmoves towards the human operator (Figure 5c), and waitsfor a command to release the object ( youbot+obj+human state). This is done by the operator by moving an arm upward( human ready ), which implies the youBot to open the gripper.The operator takes the object ( human+obj ) and puts it down( obj on table ) on the table. The operator, then, can keepmoving downward one arm, therefore notifying to the Baxterthat an object is on the table (Figure 5d). An entangled node( new object ) becomes feasible after the root node of G ismet. It is noteworthy that in some cases the youBot was notable to grasp objects properly, or dropped it actually beforehandover could occur. Furthermore, it happened that humanactions were not recognised, which required the operator torepeat them. In these cases it is the operator’s responsibilityto handle the situation by taking appropriate actions in orderto make the collaboration fluent. Upon the notification of theappropriate operator gesture, the Baxter starts grasping theobject (Figure 5e) and moves it in order to place it in frontof the head-mounted camera, rotating it ( obj checked ) fordefects inspection (Figure 5f). In Figure 5g, it is shown howthe object is recognised as faulty , and therefore the right armplaces it in the faulty box ( obj at box ). While the Baxter isinspecting the object, the youBot continues to look for otherobjects (Figure 5h). After a while, as shown in Figure 5i, oneof the objects is classified as non faulty . Since the related boxcannot be reached by the Baxter right arm, an handover in-between the two arms is executed (Figure 5j). In case theassessment cannot be done (this is simulated with a specificQR tag), the graph reaches a NA state, which implies thatthe Baxter requires the human operator to inspect the objectdirectly (Figure 5k). After all objects are inspected ( inspected a) (b) (c) (d)(e) (f) (g) (h)(i) (j) (k) (l) Fig. 5: A typical sequence of tasks in a defects inspection experiment.TABLE I: Execution times.
Module Avg. time [s] Avg. time [%]
Std. dev. [s]Task Representation 0.52 0.21 0.01Task Planner 0.02 0.008 0.003Simulator 3.69 1.49 0.24Baxter actions 203.00 82.00 5.00Human actions 39.00 15.80 6.00Total 246.75 100.00 11.253 (a) Baxter-related activities.
Module Avg. time [s] Avg. time [%]
Std. dev. [s]Task Representation 0.43 0.13 0.02Task Planner 0.02 0.00 0.004Simulator 2.74 0.79 0.40youBot actions 268.00 86.00 14.00Human actions 39.00 12.50 6.00Total 310.19 100.00 20.424 (b) youBot-related activities. state), the human operator performs a check (Figure 5l).In order to perform a realistic computational assessmentof the architecture, the whole scenario has been tested fivetimes. Results can be seen in Table I, where times are relatedto the whole experiments . Statistics presented in Table Iaand Table Ib seem to indicate that the representation andplanning modules together require less than 1% of the overallexecution time, whereas the major portion of collaborationtime is related to human or robot actions. The standarddeviation related to task planners and the representationmodules for both robots are low enough to be neglected,and imposes no latency in collaboration proccess. C. Discussion
On the basis of the experiments we carried out, it ispossible to make two different remarks. A video is available at https://youtu.be/0aOOeqCL2So.
The first is related to the robustness associated with theoverall process. In spite of such faults as unsuccessful robotgrasps, or issues related to false positives or negatives whenmonitoring the activities carried out by human operators, theinherent flexibility of C
ONC
HRC allows human operators tointervene and manage these issues. This is even more relevantconsidering that our current setup does not focus on such arobustness level.The second is the insight that using parallel instancesof AND/OR graph representation layers seems to be moreefficient with respect to an equivalent, common, single in-stance model. We observed that the adoption of C
ONC
HRCreduces the overall idle time considerably. This is an obviousconsequence of the fact that the total time needed for a multihuman-robot collaboration process to conclude is determinedby the maximum one associated with the longest executionbranch in the graph. On the contrary, if the HRC processwere implemented as a single, non concurrent, model, thenthe total time would correspond to the sum of all timesassociated with single cooperation paths. As an example, inour scenario C
ONC
HRC allows for a total collaboration timeequal to . s , whereas an equivalent implementationusing F LEX
HRC the total collaboration time can be up to . s . VI. C ONCLUSIONS
In this paper we present and discuss C
ONC
HRC, a frame-work aimed at modelling multi human-robot collaborationprocesses. The framework builds upon F
LEX
HRC, whichdid not consider concurrent task allocation and execution.C
ONC
HRC has been preliminary analysed in a use caserelated to defects inspection, where one human operator andfour robot agents are present. Two general remarks can bedone. The first is a general robustness of the human-robotcooperation flow with respect to issues related to objectrasping and manipulation, as well as the recognition ofhuman actions. The second, which is related to best practicesin modelling the cooperation scenario, is a tendency towardsminimising idle times.Obviously enough, the work can be improved along manydirections: (i) evaluating the use of a scheduler instead of aset of concurrent planners, especially considering approachesbased on Answer Set Programming or metaheuristics; (ii)the gesture recognition module, used to detect and classifyhuman activities, may be improved allowing for models ableto predict them. These two aspects are subject of currentwork. R
EFERENCES[1] K. Darvish, F. Wanderlingh, B. Bruno, E. Simetti, F. Mastrogiovanni,and G. Casalino, “Flexible human–robot cooperation models forassisted shop-floor tasks,”
Mechatronics , vol. 51, pp. 97–114, 2018.[2] B. Esmaeilian, S. Behdad, and B. Wang, “The evolution and future ofmanufacturing: A review,”
Journal of Manufacturing Systems , vol. 39,pp. 79–100, 2016.[3] S. Kock, T. Vittor, B. Matthias, H. Jerregard, M. K¨allman, I. Lundberg,R. Mellander, and M. Hedelind, “Robot concept for scalable, flexibleassembly automation: A technology study on a harmless dual-armedrobot,” in . IEEE, 2011, pp. 1–5.[4] A. De Santis, B. Siciliano, A. De Luca, and A. Bicchi, “An atlas ofphysical human–robot interaction,”
Mechanism and Machine Theory ,vol. 43, no. 3, pp. 253–270, 2008.[5] P. A. Lasota, T. Fong, J. A. Shah et al. , “A survey of methods forsafe human-robot interaction,”
Foundations and Trends R (cid:13) in Robotics ,vol. 5, no. 4, pp. 261–349, 2017.[6] J. A. Adams, “Human-robot interaction design: Understanding userneeds and requirements,” in Proceedings of the Human Factors andErgonomics Society Annual Meeting , vol. 49, no. 3. SAGE Publica-tions Sage CA: Los Angeles, CA, 2005, pp. 447–451.[7] A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz,and M. Goodrich, “Common metrics for human-robot interaction,” in
Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction , 2006, pp. 33–40.[8] K. Darvish, B. Bruno, E. Simetti, F. Mastrogiovanni, and G. Casalino,“Interleaved online task planning, simulation, task allocation andmotion control for flexible human-robot cooperation,” in . IEEE, 2018, pp. 58–65.[9] A. Kattepur, “Workflow composition and analysis in industry 4.0warehouse automation,”
IET Collaborative Intelligent Manufacturing ,vol. 1, no. 3, pp. 78–89, 2019.[10] M. Huber, A. Knoll, T. Brandt, and S. Glasauer, “When to assist?-modelling human behaviour for hybrid assembly systems,” in
ISR 2010(41st International Symposium on Robotics) and ROBOTIK 2010 (6thGerman Conference on Robotics) . VDE, 2010, pp. 1–6.[11] S. Glasauer, M. Huber, P. Basili, A. Knoll, and T. Brandt, “Interactingin time and space: Investigating human-human and human-robotjoint action,” in . IEEE, 2010, pp. 252–257.[12] Y. Kawaguchi, I. Yoshida, H. Kurumatani, T. Kikuta, and Y. Yamada,“Internal pipe inspection robot,” in
Proceedings of 1995 IEEE Interna-tional Conference on Robotics and Automation , vol. 1. IEEE, 1995,pp. 857–862.[13] J.-K. Oh, G. Jang, S. Oh, J. H. Lee, B.-J. Yi, Y. S. Moon, J. S. Lee,and Y. Choi, “Bridge inspection robot system with machine vision,”
Automation in Construction , vol. 18, no. 7, pp. 929–941, 2009.[14] K. H. Cho, H. M. Kim, Y. H. Jin, F. Liu, H. Moon, J. C. Koo,and H. R. Choi, “Inspection robot for hanger cable of suspensionbridge: Mechanism design and analysis,”
IEEE/ASME Transactionson mechatronics , vol. 18, no. 6, pp. 1665–1674, 2013.[15] C. Spence and M. Zampini, “Auditory contributions to multisensoryproduct perception,”
Acta Acustica united with Acustica , vol. 92, no. 6,pp. 1009–1025, 2006. [16] S. K. Garrett, B. J. Melloy, and A. K. Gramopadhye, “The effects ofper-lot and per-item pacing on inspection performance,”
InternationalJournal of Industrial Ergonomics , vol. 27, no. 5, pp. 291–302, 2001.[17] M. A. Goodrich, A. C. Schultz et al. , “Human–robot interaction: asurvey,”
Foundations and Trends R (cid:13) in Human–Computer Interaction ,vol. 1, no. 3, pp. 203–275, 2008.[18] T. Meneweger, D. Wurhofer, V. Fuchsberger, and M. Tscheligi, “Work-ing together with industrial robots: Experiencing robots in a productionenvironment,” in . IEEE, 2015, pp.833–838.[19] L. Johannsmeier and S. Haddadin, “A hierarchical human-robotinteraction-planning framework for task allocation in collaborativeindustrial assembly processes,” IEEE Robotics and Automation Letters ,vol. 2, no. 1, pp. 41–48, 2016.[20] S. Lemaignan, M. Warnier, E. A. Sisbot, A. Clodic, and R. Alami,“Artificial cognition for social human–robot interaction: An imple-mentation,”
Artificial Intelligence , vol. 247, pp. 45–69, 2017.[21] K. P. Hawkins, S. Bansal, N. N. Vo, and A. F. Bobick, “Anticipatinghuman actions for collaboration in the presence of task and sensoruncertainty,” in . IEEE, 2014, pp. 2215–2222.[22] S. J. Levine and B. C. Williams, “Concurrent plan recognition andexecution for human-robot teams,” in
Twenty-Fourth InternationalConference on Automated Planning and Scheduling , 2014.[23] D. Claes and K. Tuyls, “Human robot-team interaction,” in
ArtificialLife and Intelligent Agents Symposium . Springer, 2014, pp. 61–72.[24] J. W. Crandall, M. Oudah, F. Ishowo-Oloko, S. Abdallah, J.-F. Bon-nefon, M. Cebrian, A. Shariff, M. A. Goodrich, I. Rahwan et al. ,“Cooperating with machines,”
Nature communications , vol. 9, no. 1,pp. 1–12, 2018.[25] G. Xie, J. Pegn, X. Zhang, and K.-C. Lin, “A dynamic task planningbased on task subcontracting and dynamic re-decomposition,” in
Proceedings of the 29th Chinese Control Conference . IEEE, 2010,pp. 4518–4523.[26] A. Capitanelli, M. Maratea, F. Mastrogiovanni, and M. Vallati, “Onthe manipulation of articulated objects in human–robot cooperationscenarios,”
Robotics and Autonomous Systems , vol. 109, pp. 139–155,2018.[27] P. Tsarouchi, G. Michalos, S. Makris, T. Athanasatos, K. Dimoulas,and G. Chryssolouris, “On a human–robot workplace design andtask allocation system,”
International Journal of Computer IntegratedManufacturing , vol. 30, no. 12, pp. 1272–1279, 2017.[28] M. Toussaint, T. Munzer, Y. Mollard, L. Y. Wu, N. A. Vien, andM. Lopes, “Relational activity processes for modeling concurrentcooperation,” in . IEEE, 2016, pp. 5505–5511.[29] D. E. Smith and D. S. Weld, “Temporal planning with mutualexclusion reasoning,” in
IJCAI , vol. 99. Citeseer, 1999, pp. 326–337.[30] D. S. Weld et al. , “Planning with durative actions in stochasticdomains,”
Journal of Artificial Intelligence Research , vol. 31, pp. 33–82, 2008.[31] B. Bruno, F. Mastrogiovanni, A. Saffiotti, and A. Sgorbissa, “Usingfuzzy logic to enhance classification of human motion primitives,” in
International Conference on Information Processing and Managementof Uncertainty in Knowledge-Based Systems . Springer, 2014, pp.596–605.[32] E. Simetti and G. Casalino, “Whole body control of a dual armunderwater vehicle manipulator system,”
Annual Reviews in Control ,vol. 40, pp. 191–200, 2015.[33] L. H. De Mello and A. C. Sanderson, “And/or graph representationof assembly plans,”
IEEE Transactions on robotics and automation ,vol. 6, no. 2, pp. 188–199, 1990.[34] G. Luger,
Artificial Intelligence: Structures and Strategies forComplex Problem Solving . Pearson Addison-Wesley, 2009. [Online].Available: https://books.google.it/books?id=bZwdAQAAIAAJ[35] S. Russell, S. Russell, P. Norvig, and E. Davis,