Adaptive Workload Allocation for Multi-human Multi-robot Teams for Independent and Homogeneous Tasks
Tamzidul Mina, Shyam Sundar Kannan, Wonse Jo, Byung-Cheol Min
11 Adaptive Workload Allocation for Multi-humanMulti-robot Teams for Independent andHomogeneous Tasks
Tamzidul Mina , Shyam Sundar Kannan , Wonse Jo , and Byung-Cheol Min Abstract —Multi-human multi-robot (MH-MR) systems havethe ability to combine the potential advantages of robotic systemswith those of having humans in the loop. Robotic systems con-tribute precision performance and long operation on repetitivetasks without tiring, while humans in the loop improve situationalawareness and enhance decision-making abilities. A system’sability to adapt allocated workload to changing conditions andthe performance of each individual (human and robot) during themission is vital to maintaining overall system performance. Previ-ous works from literature including market-based and optimiza-tion approaches have attempted to address the task/workloadallocation problem with focus on maximizing the system outputwithout regarding individual agent conditions, lacking in real-time processing and have mostly focused exclusively on multi-robot systems. Given the variety of possible combination of teams(autonomous robots and human-operated robots: any number ofhuman operators operating any number of robots at a time)and the operational scale of MH-MR systems, development ofa generalized framework of workload allocation has been aparticularly challenging task. In this paper, we present sucha framework for independent homogeneous missions, capableof adaptively allocating the system workload in relation tohealth conditions and work performances of human-operatedand autonomous robots in real-time. The framework consists ofremovable modular function blocks ensuring its applicability todifferent MH-MR scenarios. A new workload transition functionblock ensures smooth transition without the workload changehaving adverse effects on individual agents. The effectivenessand scalability of the system’s workload adaptability is validatedby experiments applying the proposed framework in a MH-MRpatrolling scenario with changing human and robot condition,and failing robots.
Index Terms —Adaptive Workload Allocation, Agent-BasedSystems, Cognitive Human-Robot Interaction, Human-RobotTeam, Multi-Robot Systems, Workload Transition.
I. I
NTRODUCTION M ULTI-HUMAN MULTI-ROBOT (MH-MR) systemshave an immense potential for applicability in variousindependent and non-sequential tasks such as coverage prob-lems of surveillance, patrolling, search and rescue, inspectionor assembly of items in an industrial conveyor belt by roboticmanipulators, and various other multi-agent scenarios. Robots Tamzidul Mina is with the SMART Lab, Department of Computer andInformation Technology, Purdue University, and with the School of Me-chanical Engineering, Purdue University, West Lafayette, IN 47907, USA [email protected] Shyam Sundar Kannan, Wonse Jo, and Byung-Cheol Min arewith the SMART Lab, Department of Computer and InformationTechnology, Purdue University, West Lafayette, IN 47907, USA { kannan9,jow,minb } @purdue.edu This work was supported in part by NSF CAREER Award IIS-1846221.
Emotional stateCognitive state
Human condition assessment
Robot 2 (A) Robot 1 (A)Robot 3 (O)
Allocation transition
Adaptive workload allocation New workload allocation
Robot 1 (A) Robot 2 (A)Robot 3 (O)
Robot condition assessment
Robot 1 (A):Robot 2 (A):
Wi-Fi Bat. Temp. Error
Robot 3 (O):Performance Evaluation …………
Robot 1 (A) Robot 2 (A) Robot 3 (O) …… BeforeAfter
Robot 1 (A) Robot 2 (A)Robot 3 (O)
Robot 1 (A) Robot 2 (A) Robot 3 (O)
Robot 1 (A) Robot 2 (A)Robot 3 (O)
Fig. 1: Conceptual illustration of the proposed multi-humanmulti-mobile-robot (MH-MR) system with adaptive workloadallocation to human and robot conditions and performancewith workload transitional considerations. Potential applica-tion includes autonomous and multi-human operated multi-mobile robot patrolling, surveillance, multi-robot manipulatortasks on a moving conveyor belt in an industrial settingetc. The dynamically allocated workspace in the differentapplications change with human and robot operator conditionand performance in real-time.allow long operation hours on repetitive tasks and provideconsistent and precise performance beyond human capabil-ity, while human operators contribute improved situationalawareness, experienced and intuitive decision making, and theability to work around unexpected situations. While researchon human-robot interaction has gained a lot of momentum inrecent years [1]–[3], MH-MR systems are a relatively newarea involving interaction and collaboration between multiplehumans and robots.Task/workload allocation is an important problem in MH-MR systems. Previous works have investigated team organiza-tion [4], a number of operator-mediated robot control methods[5], awareness studies in human-robot systems [6], and variousclassifications of human-robot systems [7] for task/workload a r X i v : . [ c s . M A ] J u l allocation. Tsarouchi et al. introduced a system for designingand assigning tasks to operators and human workplaces [8].Automation adaptation based on human perceived workloadhas also been studied in [9]. Physiological measurements ofhumans have been used as triggers in the control of unmannedaerial vehicles (UAVs) to initiate different workload states andadapt operator performance [10], [11]. Error rates and taskdifficulty as perceived by operators have also been used astriggers to re-allocate or automate workload [12]. However,task allocation in a multi-agent system increases in complexityif the triggers are less than perfect; sudden or unpredictablechanges in workload or mission may have a negative impacton the operator’s performance pertaining to the understandingof the automation behaviors and the system functions theycontrol [13]. Sudden and/or drastic changes may overwhelmor momentarily catch human operators off guard while tryingto cope with their allocated work.Musi´c and Hirche have proposed an architecture for plan-ning human roles in robot team control [14] to optimizecollaboration and teaming mechanisms across a wide rangeof human operators and robots. Task allocation in multi-robotscenarios have been widely studied in [15]–[19], consideringresource constraints and robot performance. Task allocationwith unknown robot capabilities have also been studied in[20]. Optimal task allocation with multi-humans in the loophas also been proposed in [21], where task allocation is per-formed over multiple-levels (group and individual) comprisingof high-risk and low-risk information in order to maximizeeffectiveness of the entire system minimizing processing costand time, considering human factors given limited resources.In market-based approaches for multi-agent task allocation,the team seeks to optimize an objective function based uponrobots utilities for performing particular tasks [22], [23];desirable features of these approaches include efficiency insatisfying the objective function, robustness and scalabilityof the system. However, in systems where fully centralizedapproaches are feasible, market-based approaches can be morecomplex to implement and can produce poorer solutions; whenfully distributed approaches suffice, market-approaches can beunnecessarily complex in design and can require excessivecommunication and computation [24]. Mixed integer linearprogramming optimization approaches have also been used fortask allocation [25]–[27]. Population based approaches such asthe genetic algorithm was also proposed for task allocation indisaster scenarios [28]. Ant colony optimization has also beenproposed for task allocation of multi-agent systems in [29].Most of the task/workload allocation methods proposed inliterature have focused on maximizing the system’s work out-put without considering individual agent conditions. Moreover,most of the research work on task allocation has remainedconfined to multi-robot systems only. In contrast, we presenta task/workload allocation method considering quantified bothhuman and robot condition and performance equally, i.e.prioritizing the ability of all agents to work in an MH-MRsystem maintaining agent level work efficiency, while ensuringfull coverage of the application workspace.In this paper, we present a generalized MH-MR frameworkcapable of workload allocation for independent, non-sequential homogeneous tasks, consisting of independent modular func-tion blocks assessing human and robot conditions and theperformances of human-operated and autonomous robots. Thesystem is designed to be compatible with previously estab-lished normalized quantitative human and/or robot health andperformance assessment tests. The framework also incorpo-rates a re-allocated workload transition model to minimizethe effects of sudden changes in workload or mission thatmay have negative impacts on operator or robot performance.We demonstrate the applicability, effectiveness, and scalabilityof the framework through various scenarios of a MH-MRpatrolling application as validation of the proposed concept.An overview of our MH-MR work allocation concept appliedin example scenarios of mobile robot coverage problems androbotic manipulators in an assembly line conveyor belt isshown in Fig. 1.II. P RELIMINARIES AND A SSUMPTIONS
We consider a homogeneous group of m robots capable ofcarrying out autonomous missions, each denoted as R i , for i ∈ I R = { , , .., m } with state definition of q i ∈ R w , where w represents the dimension of the system workspace. R i may beteleoperated by any number of human operators at any time,each denoted as O j for j ∈ I O = { , , .., h } , modeled as anedge E in an undirected graph G = ( V , E ) without any self-connectivity, where V represents the set of nodes ( R i , O j ) , i ∈ I R , j ∈ I O , such that { R i , O j } ∈ E . We denote the set of indicesof human-operated robots as I H and the set of indices forautonomous robots in the system as I A . For the convenienceof the reader, we summarize the terminology usage in thissection as: r / R for robots, o / O for human Operators, c / C and p / P for condition and performance (human and robot).Each human operator may control multiple robots andassume/relinquish control of any robot in the system at anytime, triggering a change in the robot operation mode. We donot limit robots operated by humans to be only teleoperated;some level of autonomy might exist while the human operatoracts as a supervisor. Regardless, the performance of such arobot is dependent on the state of the human operator as well.The condition or health status of each robot R i , for i ∈ I R inthe MH-MR system can be monitored at all times as a set ofrobot health states denoted as C R i ∈ R w r , where w r equals thenumber of robots in the system m . Physiological measurementsand the emotional state of each human operator O j , for j ∈ I O in the MH-MR system can be monitored at all times as a setof human operator health states denoted as C O j ∈ R w o , where w o is the number of human operators in the system h . Definition II.1.
The performance of each robot (autonomousand human-operated) R i , i ∈ I R on their respective allottedmission/task can be evaluated based on a predefined evalu-ation metric relevant to the mission/task using observation set P R i ∈ R w r , i ∈ I R .We define the constraints of the system workspace (work-load) W ∈ R w as finite, known apriori, and covered by m robots without any overlap. We assume each robot is equippedwith appropriate low level velocity/position controllers with Indiv. autonomous mission planning/coordinating
Adaptive workload allocation Allocation transition E n v i r o n m e n t Robot performance evaluation metricRobot condition evaluation metric M H - M R S y s t e m T a s k Indiv. autonomous mission planning/coordinating
Human condition evaluation metric H u m a n o p e r a t e d r o b o t s A u t o n o m o u s r o b o t s 𝑐̂ (cid:3039)(cid:3045) , ∀𝑙 ∈ 𝐼 (cid:3019) 𝑐̂ (cid:3038)(cid:3042) , ∀𝑘 ∈ 𝐼 (cid:3016) |{𝑅 (cid:3036) , 𝑂 (cid:3038) } ∈ 𝐸, ∀𝑖 ∈ 𝐼 (cid:3009) 𝑝̂ (cid:3036)(cid:3045) , ∀𝑖 ∈ 𝐼 (cid:3019) 𝐶 (cid:3016) (cid:3286) , ∀𝑘 ∈ 𝐼 (cid:3016) 𝐶 (cid:3019) (cid:3287) , ∀𝑙 ∈ 𝐼 (cid:3019) 𝜎 (cid:3036) ′, ∀𝑖 ∈ 𝐼 (cid:3019) 𝜎 (cid:3036) , ∀𝑖 ∈ 𝐼 (cid:3019) Module A.1Module RModule P Module A.2Module H 𝑃 (cid:3019) (cid:3284) , ∀𝑖 ∈ 𝐼 (cid:3019) Robot performance feedbackRobot condition feedback Human condition feedback
Fig. 2: Adaptive MH-MR workload allocation system modular framework based on individual human and robot conditionand performance. The MH-MR system may consists of autonomous robots and combinations of single human-operated singlerobot, multi-human operated single robots and/or single human operated multi-robots. The workload allocation module takesequal weighted metric inputs from each modular condition and performance evaluation module to allocate new workload. Theallocation transition module ensures a smooth transition to the new workload.collision avoidance relevant to the MH-MR application and iscapable of fully autonomous behavior when required [30]–[32]. Once a mission is assigned to a specific robot, anautonomous robot uses its own individual mission plan-ning/coordinating algorithms to conduct the mission. Individ-ual mission/workload assigned to human-operated robots iscoordinated by their human counterparts.
Definition II.2.
We define workload on R i , i ∈ I R (eitherautonomous or human-operated) at time t as σ i ( t ) and thecorresponding workspace as W i ∈ R w regardless of its taskdepending on the application. Upon mission assignment to theMH-MR system, the initial workload for each robot σ i ( ) mayor may not be equally distributed.The objective is to provide a systematic approach to anadaptive workload allocation in MH-MR systems based on:(a) robot and human operator state monitoring ( C R i , i ∈ I R and C O j , j ∈ I O ), and (b) autonomous robot and human-operatedrobot work performance states ( P R i , i ∈ I R ). The transition pro-cess between workload changes for a robot or operator mustconsider the effect of the prescribed change. The frameworkmust maintain generality for applicability in any MH-MRsystem.At the core of the proposed adaptive MH-MR systemframework, the adaptive workload allocation system and aworkload transition system, designated as module A.1 andmodule A.2 respectively, provide a workload distribution so-lution on the assigned mission based on the condition andperformance of each human and robot unit operating in thesystem. The state of each human and robot in the system,and the performance of each autonomous and human-operated robot are assessed to adaptively re-allocate the total missionworkload for continuous performance.We realize that the relevance of such evaluations or as-sessments are application specific and may be irrelevant incertain systems. Therefore, to maintain generality of ouradaptive MH-MR system framework, we propose a modulardesign consisting of robot and human state and performanceassessment function blocks. Each module provides a real-timemetric of its unit system adhering to its own procedure basedupper and lower bounds. The modules and their metrics arelisted as follows: • Module R: Robot state monitoring and evaluation metric c rl ∈ [ u R , v R ] from C R l , l ∈ I R , normalized as ˆ c rl • Module H: Human operator state monitoring and evalua-tion metric c ok ∈ [ u O , v O ] from C O k , k ∈ I O , normalized asˆ c ok • Module P: Robot (human-operated or autonomous) per-formance assessment metric p ri ∈ [ u P , v P ] from P R i , i ∈ I R ,normalized as ˆ p ri ,where u and v represent the lower and upper bounds of thecorresponding metric respectively.The aforementioned human and robot states and perfor-mance metrics from the modular function blocks are fed intothe adaptive workload allocation module A.1 for workload re-allocation. The workload is re-allocated to maximize overallsystem performance at all times. The modular design ensuresthat any module may be added or removed from the systemdepending on the application requirement, pertaining to thegeneralization of the framework. Fig. 2 illustrates the proposedmodules of the MH-MR system framework. III. A
DAPTIVE
MH-MR S
YSTEM F RAMEWORK
A. Module A.1 Adaptive Workload Allocation
We design the workload allocation of the system based onthe maximum outcome of combining the incoming human androbot states and performance metrics. A variant of the softmaxfunction, also known as the normalized exponential functionis proposed to determine the workload allocation for each ofthe m robots. We define a vector S of m normalized inputssuch that, [ S ] i = (cid:40) γ i | Λ i | + (cid:0) ˆ c ri + ∑ k ∈ Λ i ˆ c ok + ˆ p ri (cid:1) if i ∈ I H γ i ( ˆ c ri + ˆ p ri ) if i ∈ I A (1)where Λ i is a vector of λ ∈ I O |{ R i , O λ } ∈ E , γ i = min ( ˆ c ri , ˆ c ok , ˆ p ri ) , ∀ k ∈ Λ i , and γ i = min ( ˆ c ri , ˆ p ri ) . By this design,the γ terms ensure that the system allocates zero workload toa robot (autonomous or human-operated), if the correspondingrobot and/or human operator is detected to have voluntar-ily/involuntarily stopped working ( ˆ p r would equal to zero),completely failed, incapacitated and/or may have suffered fromany discontinuity or disconnectedness in the teleoperation andcommunication graph structure ( ˆ c r and/or ˆ c o would equalto zero). It also ensures that the workload allocated is pro-portional to the worst human/robot condition/performance insituations where one increases and another decreases equally.We calculate the share of the total workload for robot R i , i ∈ I R at current time t as, σ (cid:48) i ( t ) = S ( i ) ∑ ml = S ( l ) for i = , ..., m . (2)The normalization ensures that the sum of all σ (cid:48) is 1, pertain-ing to the total workload of the system. B. Module A.2 Workload Allocation Transition
Sudden changes in workload allocation may have over-whelming effects on a host. The transitions must be smoothand manageable without any drastic changes. We model sucha transition process for the workload change from the currentactual workload allocation σ i ( t ) , i ∈ I R to the proposed work-load allocation σ (cid:48) i ( t ) , i ∈ I R considering the effect of the changeto the highest affected agent in the system as follows.We model the workload transition process as, σ i ( t + ) = σ i ( t ) + K e ∆ σ i ( t ) (3)where ∆ σ i ( t ) = σ (cid:48) i ( t ) − σ i ( t ) , and K e ∈ [ , ] is a transitionmodel coefficient dependent on the highest effect of theproposed change on the system.Denoting the proposed 2-D workspace for R i , i ∈ I R corre-sponding to proposed workload σ (cid:48) i ( t ) as W (cid:48) i , we determinethe highest effect on the system as q f = min ( q c ) , where q c denotes the shortest Euclidean distance between the boundaryof workspace W (cid:48) i ( t ) and q i ( t ) , ∀ i ∈ I R . In situations wherea complete robot failure occurs or a human operator isincapacitated, the failed robot is ignored in q f determination.The transition coefficient K e can therefore be modeled asan exponential function of q f , K e = − e − Kq f (4) where K is a positive scaling constant. The exponential natureof the transition allows for a smooth change in workload where K may be tuned to control the rate of transition depending onparticular application scenarios.The workload allocation cycle must be synced with thecontributing modules in the framework. The workload al-location update cycle time constant can therefore be setas τ = max ( τ p r , τ c o , τ c r ) where τ p r , τ c o and τ c r denote therequired operation cycle time constants for function modulesP, H, and R respectively. C. Human and Robot, Condition and Performance1) Module H: Human Condition Evaluation Metric:
We de-fine human operator condition as their ability to perform theirtask of teleoperating robots as a function of stress, emotion,and/or direct physiological measurements depending on theMH-MR application. For human operator condition evaluation,we refer to previous studies in literature for quantitative andqualitative techniques. Primary approaches include predictingstress or emotion from audio signals [33], gestures [34], [35],facial expressions [36], body gestures [37] or physiologicalsignals such as heart rate, skin conductance, and respiration[38]–[43]. The measurements and predictions are used toevaluate stress and psychological dynamics in the interestof creating effective working conditions [44]. Individual ora combination of a number of emotional responses may bemeasured and used as human operator condition for ModuleH, but at this stage we focus on human operator stress levelsthat have been shown to have a direct negative impact on workperformance [45].Galvanic skin response (GSR) or skin conductance is areliable indicator of stress [46]. Under stress, skin conductanceof an individual is increased [47] due to increase in moistureon the surface of the skin, which increases the flow of electric-ity. Healey et al. proposed a continuous stress measurementmetric in [46] that can be normalized and used as a measureof the human operator condition directly for our proposedframework. Also, a number of other such human operator con-dition measurement metrics based on facial expressions, bodygestures, heart rate and respiration have been summarized inthe stress recognition literature survey [48] that may be usedas Module H in our proposed framework.Stress detection using a combination of multiple noninvasivephysiological variables such as galvanic skin responses, bloodvolume pulse, pupil diameter and skin temperature have beenproposed in [49]. A support vector machine is used to per-form the supervised classification of effective states between”stress” and ”relaxed”. Stress levels may also be furtherdiscretized as ”low”, ”medium” and ”high”; such discretestates may be quantified as discrete human operator conditionvalues simply as 0 .
75, 0 . .
25 respectively, or a movingaverage may also be applied depending on the application.We stress here that our proposed method is designed forcontinuous human operator condition values, but may still beadapted with a discrete human operator condition evaluationsystem as well appropriate of the application.Heart rate variability (HRV) in terms of the length betweenheartbeat intervals, also called an R-R interval or inter-beat
Time (minute) RR i n t e r v a l s ( m s ) RR intervals (a) Raw data of R-R intervals from [50]. R-R interval was measuredfrom participants for a walking task between 5 −
20 mins followedby a small recovery period, and then a task to watch a horror filmbetween 30 −
70 mins. H u m an c ond i t i on Time (minute)
FalseTrue S t r e ss de t e c t i on Stress detected areas Curve-fitted stress Estimated human condition (b) Stress detection using the TPOT algorithm [51]. A possible mov-ing average filter implementation for continuous stress quantification(blue) is shown followed by its human condition metric assessment(red).
Fig. 3: An example implementation of a machine learning-based stress detection algorithm using HRV signals and R-Rintervals; (a) raw data of R-R intervals from the existing dataset [50], and (b) the stress detection and subsequent humancondition assessment.interval (IBI) plays an important role in predicting human con-dition in neurosciences and medical fields [52]. HRV has beenutilized to predict stress in various works of literature [53]–[56], due to its responses to physiological and environmentalstimulus. Ottesen proposed a stress detection algorithm usingwearable devices and machine learning technology [57] usingboth heart rate (HR) and HRV as a training dataset from [50]as shown in Fig. 3a; a machine learning model was proposedfor automated machine learning and the evolutionary algorithmcalled TPOT [51]. The model had a stress detection accuracyof 79 .
9% using an existing dataset from a user study, whereparticipants watched a horror movie after a 15-minute walkingtask to differentiate between physical and mental stress, suchas lowering the R-R intervals. An implementation of the TPOTalgorithm for stress detection is shown in Fig. 3b.An example implementation of the moving average filter asa possible continuous stress quantification method is shown onthis discrete assessment of stress from the machine learning-based stress detection algorithm as the blue line denoted as s ( t ) . This moving average filter smooths the rapidly changingbinary output of the stress detection algorithm in the timedomain [58]. The human condition metric is defined to bein the closed range [ , ] for worst to perfect; therefore, thecontinuous stress plot s ( t ) ∈ [ , ] is mapped to the estimatedhuman condition as 1 − s ( t ) to obtain a continuous humancondition metric required in the MH-MR workload allocationframework. We chose this specific dataset in our study to showstress and subsequent human condition assessment because ofthe following properties: drastically changing human condi-tion between 0 −
15 mins, slow change between 15 −
50 minsand sudden changes between 25 −
40 mins. We validate theeffectiveness of our proposed framework on simplified casesof these rates of change of human condition in Section IV-A3.We deem noise reduction and disturbance rejection in mea-suring human operator condition as beyond the scope of ourcurrent work and included within the above presented humancondition measurement metrics; a few specific works on signalprocessing and noise filtering of physiological measurementshave been proposed in [59], [60]. Therefore, we assume that human operator condition can be measured and quantified withenough certainty and noise rejection for application in ourproposed MH-MR workload allocation framework.To establish the generality of the modular human conditionassessment function block in the proposed work allocationframework, we stress the following notes on possible human-operated robot scenarios. In cases where one human operatesone robot or one human operates multiple robots, the humanoperator’s health condition would be independently used inModule H for each of the operated robot’s work allocation inModule A. However, if multiple humans control a single robotfor an MH-MR application, the condition assessments of allthe human operators of this particular robot would have tobe considered for its workload allocation. In such a scenariowhere one robot is operated by more than one human operator,we assume that each human operator of the robot has exclusiveon-board tasks: one operates navigation, one operates surveil-lance etc.; if one operator’s condition deteriorates, one on-board task is affected. i.e. as a whole, this specific multi-humanrobot team’s working ability is also affected. The definition of γ in Eq. (1) ensures that the system allocates zero workloadto a robot for any of its operators becoming incapacitated.The proposed MH-MR workload allocation system is thereforegeneral to any number of operators controlling any numberof robots in the system. With these generality notes, Eq. (3)assigns workload to individual robots reflecting its individual ability to work considering the conditions of all its humanoperators.
2) Module R: Robot Condition Evaluation Metric:
On-board quantitative measurements of robot health may includebattery level, communication signal strength, internal tem-perature and a variety of other factors [61]. Detecting sub-nominal characteristics and isolating problems through self-checking have also been considered in different autonomousrobot platforms currently available. Qualitative evaluationsmay also be included for robot condition evaluation based onthe robot’s physical state. We refer to the Neglect Tolerancemetric [62] for autonomous robots as a measure of how arobot’s effectiveness declines in autonomous mode without any human supervision or control. It includes task complexityand robot capability among various other factors to provide anoverall measure of a robot’s condition of autonomy.
3) Module P: Robot Performance Evaluation Metric:
Robot (either autonomous or human-operated) performancemetrics such as percentage area coverage or distance travelledproposed in [63] may be used to asses robot performancedepending on the MH-MR application. Performance of robotsmay be determined in terms of task completion time, path fol-lowing cross-track error [64] etc. depending on the applicationof the proposed framework. Human Robot Interaction (HRI)metrics recommended by Steinfeld et al. [65] and reviewed byMurphy [66] in terms of navigation (e.g. localization, effectivepath determination around objects), perception (e.g. surveil-lance, target identification, sensor area coverage), manipula-tion, and management at the human, robot, and system levelperspectives may be used to evaluate human-operated robotperformance using an arbitrary evaluation function plugged inas Module P. We leave performance assessments for ModuleP at the discretion of their relevance to particular applications.IV. V
ALIDATION & R
ESULTS
As validation of the effectiveness of the adaptive taskallocation mechanism, we present our experimental findingsof applying the proposed framework to a MH-MR patrollingapplication, where human operator and robot conditions affecttheir patrolling ability.Four experiment scenarios were independently investigated.In the first scenario (S1), we simulate temporary and per-manent deteriorated conditions for a human operator and anautonomous robot in sequence, while in the second scenario(S2) we simulate complete failure of a robot, and analyze thesystem’s workload adaption in each scenario. In two furtherscenarios (S3 and S4) we present workload transitioning andscalability analysis with similar conditions as S1 and S2respectively.Before moving on to including real human operators in theexperiments, it is vital that controllable evaluation scenariosare used to validate our proposed work. Therefore, in thispaper we present the investigation results of our proposedmethod using simulated human operator conditions of differentcharacteristics.
A. S1 and S2: Adaptive Workload Allocation in Patrolling1) Patrolling Application:
Machado et al. broadly definedpatrolling as “the act of walking or traveling around an area,at regular intervals, in order to protect or supervise it” [67].Therefore, we set up our representative patrolling applicationwith a given number of robots traveling around allocatedrectangular regions on a plane, where the sum of the area ofall rectangular regions represents the total workload. The allo-cated workload from our proposed framework may be directlyused in more complex region allocations for the patrollingscenario following capacity-constrained Voronoi tessellationworks proposed in literature [68]. Applications in multi-robotcoverage problems include [69]. However, for simplicity andease of analysis we validate our system using rectangular Fig. 4: Experiment setup with m = R and R human-operated, and R autonomous. Robots patrol rectangular regions on the plane, defined as boundaryfollowing its allocated area. Patrolling velocities are modeleddependent on human operator and robot conditions.patrolling regions, and we define patrolling for each robot asboundary following its allocated area within a specified time τ ∗ within its ability.A patrolling performance metric is defined for comparisonstudy with and without the proposed workload allocationmethod. Patrolling performance of the complete MH-MRsystem is measured as the maximum time taken to patrol theentire area once by the MH-MR system expressed as, T L = max ( t l , t l , .., t l m ) (5)where t l i denotes the patrolling lap time of R i for i ∈ I R duringone cycle of full area patrolling, given that the entire area iscovered by all robots.
2) Experiment Setup:
We consider a MH-MR system of h = m = q i = [ x i , y i ] , for i ∈ I R on a level plane as shown in Fig. 4.Robot position data was recorded using a VICON system.True velocity estimation of the Jackals were made from thecollected position data with time. We simulate robots R and R as being controlled by human operators while R remainsautonomous in patrolling. To simulate the human operators,human operator condition assessment inputs are provided for R and R ; all robots utilize the same low level line-of-sight path following controller for consistency. The effect ofworkload change in the system at time t depends on theminimum distance from q i , ∀ i ∈ I R to the changing rectangularregion boundaries using the allocation transition coefficient K e model in Eq. (4).At initial time, the patrolling area was distributed equallyamong all robots as rectangular regions with a specified safetydistance between rectangular boundaries to prevent inter-robot collisions while patrolling, and the human operator androbot conditions were considered optimal. In course of theexperiments, the rectangular region areas were re-allocatedbased on the proposed workload allocation framework. Weacknowledge that increasing workload on an agent due to re-allocation, may reduce performance or in turn cause conditiondeterioration. Nevertheless, such effects on agents were ig-nored for validation purposes of the proposed framework. We model robot patrolling ability v ablei dependent on currenthuman operator and robot condition, v ablei = κ v max (6)for, κ = (cid:40) min ( c ok , c ri ) i ∈ I H , ∀ k ∈ Λ i c ri i ∈ I A (7)where Λ i is a vector of λ ∈ I O |{ R i , O λ } ∈ E , assuming c ok and c ri are bounded within [ , ] , and v max denotes the maximumallowed velocity of R i .The required patrolling velocity of R i is modeled as, v reqi = Perimeter ( W i ) τ ∗ for i = , ..., m . (8)where the patrolling time threshold is set as τ ∗ = ±
10 s and v reqi = [ , v max ] . Velocity of R i is therefore modeled as, v i = min ( v ablei , v reqi ) . (9)The initial value of τ ∗ is arbitrarily set large enough forexperimental analysis purposes with v max = . τ =
500 ms,workload transition scaling constant K = . ψ . In reality,the performance measure would also include v i − v actuali corre-sponding to deteriorated performance of the robot. However,we intentionally do not consider velocity differences in ourrobot performance assessment in this validation setup, sincewe focus on independent analysis and assessment of theproposed workload allocation based on human operator androbot condition only. Performance measure of all robots isassumed to be unity at all times.
3) S1: Adaptation to Deteriorated Conditions:
We modelthe conditions for the human operator O of R denoted as c o to deteriorate drastically at time τ S and then subsequentlyrecover slowly back to 1 after a sudden further small de-terioration at time τ S as shown in Fig. 5a; this is basedon simplified, observed and analyzed condition patterns ofquantified human stress from Fig. 3 (drastically changinghuman condition between 0 −
15 mins, slow change between15 −
50 mins and sudden changes between 25 −
30 mins).Here we stress the design of the simulated events havingdrastic and different rates of changes on the two separatetime instances (abrupt and slow), to show their effects on theworkload allocation. The minor further deterioration beforerecovery after τ S is simulated to investigate the sensitivityof the proposed workload allocation framework to suddensmall changes in operator condition. Deteriorated conditionof the autonomous robot R denoted as c r is simulated asshown in 5b. c r is simulated to deteriorate permanently at τ S . The experiment S1 is repeated with and without theproposed adaptive workload allocation framework to comparetheir effects on the patrolling application using the definedpatrolling performance metric. The results are presented inFig. 5, 6 and 7. Time(s) S i m u l a t ed hu m an c ond i t i on c c = S11 = S12 = S13 (a) Simulated human conditions over time. c o deteriorates drasticallyat τ S , and then subsequently recovers back to 1 after a further smalldeterioration at τ S ; c o remains at 1 at all times. Time(s)0 100 200 300 400 500 600 S i m u l a t ed r obo t c ond i t i on c c c = S11 = S12 = S13 (b) Simulated robot conditions over time. c r deteriorates perma-nently at τ S ; c r , c r remains at 1 at all times. Time(s) W o r k l oad a ll o c a t i on , < < < < = S12 = S11 = S13 (c) Workload allocation of patrolling robots change according to thesimulated human and robot conditions: allocated workload of robotswith deteriorated human and/or robot condition is reduced andcompensated by robots with better human and/or robot condition.The workload transition function ensures that drastic changes aresmoothened for a manageable effect on the host; yet remainssensitive enough to capture sudden changes in agent condition.
Time(s) R obo t v e l o c i t y ( m / s ) (O)R w/ workload allocation(O)R w/ workload allocation(A)R w/ workload allocation(O)R w/o workload allocation(O)R w/o workload allocation(A)R w/o workload allocation (d) Translation velocity profiles of patrolling robots with and withoutworkload allocation ignoring region corner rotations. Translationvelocity of robots with deteriorated human and/or robot conditions isobserved to have lowered velocity ( R , R after τ S and τ S respec-tively); robots with increased workload after re-allocation havingbetter conditions observed to increase their translation velocity ( R , R after τ S and R after τ S ). Fig. 5: S1: Adaptive workload allocation for temporary andpermanent human and robot condition deterioration.With initially set equal workload, all robots patrol theirequally allocated rectangular boundaries until time event τ S .At τ S where c o shows drastically falling conditions, the (a) Patrolling trajectory of robots with workload allocation (b) Patrolling trajectory of robots without workload allocation Fig. 6: S1: Patrolling trajectory following comparison with and without workload allocation for temporary and permanenthuman and robot condition deterioration.
Patrol Lap P a t r o l Lap T i m e ( s ) R w/ workload allocationR w/ workload allocationR w/ workload allocationTotal Patrolling time w/ workload allocation R w/o workload allocationR w/o workload allocationR w/o workload allocationTotal Patrolling time w/o workload allocation Fig. 7: S1: Total and individual patrolling time requiredcomparison with and without workload allocation.workload is re-allocated to reduce load on the human O operated robot R and increased equally among R and R having better conditions as seen in Fig. 5c; the re-allocationis reflected as a smaller patrolling region for R and equallarger regions for R and R in Fig. 6a. R and R were bothpositioned roughly equally close to the changing boundaryof their rectangular regions during the first event at τ S ,and thus both robots transition to their allocated workloadat the same time of around 300 s shown in Fig. 5c. Thecorresponding changes in the velocity profiles for each robotwith workload allocation is shown in Fig. 5d. The velocityof R is seen drastically reduced with v able < v req ; and withincreased allocations of patrolling regions, the other two robotsat this point still in good condition are observed to slightlyincrease their velocities ( v req < v able and v req < v able ). Thepatrolling time for R is observed as increasing above the τ ∗ tolerance at lap 5 and eventually levelling at lap 6 due tothe slow workload transition process as shown in Fig. 7; andremained high over laps 6 and 7 due to more frequent slower turning at corners. Patrolling times for R and R with optimalconditions remained steady within defined τ ∗ tolerance aftertime event τ S .Similar observations are made after time event τ S , whererobot R suffers a sudden condition deterioration. The work-load of R is reduced and re-distributed among the othertwo as seen in Fig. 5c and 6a. Velocity of R decreasespermanently due to its deteriorated condition. R is left withpatrolling a larger region in comparison to the others; itsvelocity increases to maintain the patrolling time requirement.However, the velocity of R remains the same due to itspreviously deteriorated condition. Thus, the patrolling time for R remains considerably higher than R and R for laps 7 and8 after time event τ S as shown in Fig. 7.Compared to the drastic change of c o at τ S and c r at τ S , c o gradually returns to 1 after a sudden drop at τ S . The workloadallocation is seen to change relatively slowly as well for thistime event over time period 540 s to 630 s as observed in Fig.5c. Right before the recovery, the workload allocation plotshows a slight dip in allocated workload for R and smallincreases for R and R validating the effective sensitivityof the proposed workload allocation method. Upon conditionimprovements of O at τ S , the workload is redistributedagain to equal portions among R and R with correspondingrectangular block patrolling trajectories shown in Fig. 6a. Thevelocity of R and R equalize to a larger value than R tocompensate for the re-allocated patrolling regions with lowerworkload for R . Patrolling lap times for all robots returnwithin the defined τ ∗ tolerance after event τ S at patrol lap9 with the highest time taken by R .To validate the effectiveness of the proposed method, theexperiment scenario is repeated without using the adaptiveworkload allocation framework. The robot patrolling trajec-tories followed the initial equal rectangular region allocationthroughout the experiment as presented in Fig. 6b. With Time(s) S i m u l a t ed hu m an c ond i t i on c c = S21 (a) Simulated human conditions over time. c o and c o remains at 1at all times. Time(s) S i m u l a t ed r obo t c ond i t i on c c c = S21 (b) Simulated robot conditions over time. c r and c r remains at 1 atall times; c r deteriorates permanently at τ S . Time(s) W o r k l oad a ll o c a t i on , < < < < = S21 (c) Workload allocation of patrolling robots change according tothe simulated human and robot conditions: allocated workload ofincapacitated robots is zero and compensated by robots with betterhuman and/or robot condition. The workload transition functionmodule ensures that drastic changes are smoothened for a man-ageable effect on the host.
Time(s) R obo t v e l o c i t y ( m / s ) (O)R w/ workload allocation(O)R w/ workload allocation(A)R w/ workload allocation (d) Translation velocity profiles of patrolling robots with and withoutworkload allocation ignoring region corner rotations. Translationvelocity of incapacitated robots (deteriorated ability ) observed tobe zero ( R after τ S ), while robots with increased workload afterre-allocation having better conditions observed to increase theirtranslation velocity to compensate ( R , R after τ S ). Fig. 8: S2: Adaptive workload allocation for complete robotfailure condition.equal rectangular region allocation over the entire experimentduration, the robot velocities only reflected the temporary andpermanent deteriorating conditions of O ( v able < v req ) and R ( v able < v req ) showing slower patrolling speed as observedin Fig. 5d and higher patrolling times after τ S and τ S respectively. Referring to the previously defined patrollingperformance metric, the total area patrolling time was recorded T i m e ( s ) Y(m) X(m) (O) R (O) R (A) R Fig. 9: S2: Patrolling trajectory of robots with workloadallocation for complete robot failure condition.to be 7 s higher on lap 5 (right after τ S ) without workloadallocation. With the initial dip in c o after τ S , the areapatrolling time was initially recorded to be 14 s higher onlap 8 without workload allocation in comparison, that reducedwithin the set τ ∗ tolerance on lap 9, when the simulated c o gradually returned to 1.The events τ S , τ S and τ S triggered changes in workloadon immediate neighbors of R , allowing it to quickly adjust itsvelocity to meet the required patrolling lap time. The workloadchange after event τ S , was slower for R in comparison as itadjusted to the change following transitioning of R , resultingin increased patrolling lap times in laps 5 and 6. The event τ S triggered in between patrolling laps 6 and 7 of R permanentlykept its patrolling velocity at 70 s with workload allocation. Incomparison, its patrolling lap time without workload allocationis observed to increase on lap 7 and permanently stay 3 shigher for the rest of the experiment. Although insignificantcompared to R , the total area patrol time remained 3 s lessdue to R with the proposed workload allocation after event τ S on lap 9.
4) S2: Adaptation to Robot Failure:
Experimental cases ofcomplete robot failures have also been investigated, where R is completely incapacitated by setting c r = τ ina separate experiment. Fig. 8a and 8b shows the simulatedhuman and robot conditions for S2.We refer to Fig. 8c to present the resulting allocatedworkloads after event τ . At event τ , the initial area of R is equally allocated amongst R and R for continuous fullpatrolling area coverage; i.e. at any time instant, the total al-located workload was always unity with the proposed adaptiveworkload allocation framework. This verifies that the workloadwas always re-allocated to ensure total area coverage by theMH-MR patrolling system. The resulting robot trajectory plotsare shown in Fig. 9. The modeled velocity plots shown in Fig.8d confirm the increased patrolling velocities for R and R tocompensate for their allocated larger equal areas. For S2, weomit comparison of patrolling performance with and without Time step (t) W o r k l oad t r an s i t i on e rr o r , K R R R R R R R R R R X Y X Y X Y t=0t=88 t=0 yy yy x x t=140 (a) Initial and final workload al-located regions for m =
10 sta-tionary robot MH-MR system.
Time step (t) A ll o c a t ed W o r k l oad , R , R R , R , R , R (b) Allocated workload conver-gence for K = Time step (t) W o r k l oad t r an s i t i on e rr o r , R , R , R , R R , R (c) Allocated workload conver-gence error over time for K = Fig. 10: S3: Transition analysis for m =
10 stationary robotsequidistant from their region boundaries along the horizontalaxis, with equal initial workload allocation. Human and robotconditions are simulated as c o = . c r = . c o = . c o = .
75 with the rest as 1 from t =
0, and the system adaptivelyconverges to the new workload depending on q f . Green dotsrepresent robots. Zoomed sections of plots shown in insets.using adaptive workload allocation, since total area patrollingcould only be achieved with the proposed adaptive workloadallocation framework. B. S3 and S4: Workload Allocation Transition Analysis
The workload transitioning module of the proposed MH-MR workload allocation framework is a function of q f . Wepresent the effects of different q f on workload transitionwith simulation results of m =
10 robots stationary at alltimes. Scenario S3 simulates deteriorating human and robotconditions with all robots initially placed at the center of theirregions along the horizontal axis; scenario S4 simulates failedrobot cases with all robots initially placed closer to the leftboundary of their rectangular regions. Odd-indexed robots areassumed to be human-operated while even-indexed robots areassumed autonomous. At t =
0, the agent conditions are set to c o = . c o = . c r = .
75 in both scenarios; R is set to c r = . c r = R converged to the lowest allocated workload followed by R and R , while the other robots compensated with increasedallocated workload. As such, the workload convergence ratewas highest for R followed by R and R with increasingly Time step (t) W o r k l oad t r an s i t i on e rr o r , K R R R R R R R R R R X Y X Y X Y t=0t=88 t=0 yy yy x x t=140 (a) Initial and final workload al-located regions for m =
10 sta-tionary robot MH-MR system.
Time step (t) A ll o c a t ed W o r k l oad , R , R , R (b) Allocated workload conver-gence for K = Time step (t) W o r k l oad t r an s i t i on e rr o r , R , R R , R , R (c) Allocated workload conver-gence error over time for K = Fig. 11: S4: Transition analysis for m =
10 stationary robotscloser to their left region boundary along the horizontal axis,with equal initial workload allocation. R is simulated tocompletely fail with c r =
0, along with human and robotconditions c o = . c o = . c o = .
75 from t =
0, and thesystem adaptively converges to the new workload dependingon q f . Green dots represent working robots and red dotsrepresent failed robots. Zoomed sections of plots shown ininsets.slower rates respectively following smaller ∆ σ . The rest of therobots showed the smallest rate of convergence to increasedallocated workload with small and equal change in ∆ σ .In contrast, S4 where R is simulated to fail completely,converges to the zero allocated workload much faster giventhe larger ∆ σ as shown in Fig. 11b. The actual workloadconvergence rate for R was followed by R and R withincreasingly slower rates respectively following smaller ∆ σ in comparison. R and R are both observed to gain a higherworkload initially at around t =
15 due to their close proximityto the largest changing workload allocation in the system for c r =
0, before reaching an equilibrium workload with the otherrobots. Between 0 < t <
25 with σ shrinking to zero fasterthan the other robots, R and R compensate with a larger shareof actual workload temporarily experiencing a faster conver-gence rate compared to the other agents before the adjustmentsare propagated to the rest of the robots reaching an equilibriumin the group. The convergence rate of workload allocationfor R is initially observed slightly higher than R for t < R initially placed further away from the changing boundarywith R . R then shows a higher workload convergence ratebetween 10 < t <
15 as the changing boundary moves awayfrom R and closer to R . The rest of the robots in the groupreach an equilibrium workload fairly slowly in comparison, as R and R adjust over time.In Fig. 11c, the workload transition error of R shows aninitial error of 0 . c r = t =
0. Over thenext few time steps, the error is observed to rise sharply a little Time step (t) T o t a l w o r k l oad t r an s i t i on e rr o r , ' | " < | K=1K=3K=5K=10
Fig. 12: Effect of K on the total workload transition timefor m =
50 robots following scenario S3: human and robotconditions set to c o = . c o = . c r = .
75 and c r = . K =
10 and increasingly slower with lower K as expected.over 0 .
15 as the effect of the deteriorated conditions of R and R are propagated to the transitioning workloads of the rest ofthe robots including R ; i.e. the error for R was compoundedwith the compensating errors of R and R before reachinga transitional error of zero. With a complete robot failurein S4, the amount of workload to be re-allocated was largerwhile considering transitional effects on all agents; hence theworkload convergence time for S4 was recorded higher thanS3. The workload convergence and the convergence error plotfor S4 are shown in Fig. 11b and 11c respectively.As scalability analysis of the proposed MH-MR workloadallocation and transition framework, scenario S3 was repeatedfor m =
50 with varying K . Fig. 12 plots the total workloadtransition error along a logarithmic time scale for K = K = K = K =
10. The total workload transition error for allcases of K reach zero in finite time. K =
10 yielded the fastestconvergence of the error to zero with increasingly slower ratesfor lower values of K as expected. Similar observations aremade for m = m =
100 and m =
500 robot cases each with K = K =
10 as shown in Fig. 13; a larger K yielded afaster convergence of the total workload convergence error tozero. The effect of larger K gets smaller with larger m ; a minordifference is observed for the two K cases for m = m due to the initiallyequal distribution of workload assumption of the scenario.V. D ISCUSSION
The workload-allocation and transition problem is addressedfrom a high-level abstraction to maintain generality of itsapplication. The proposed MH-MR framework is suitablefor homogeneous and heterogeneous robots (ground, aerialetc.) on homogeneous tasks, given that all robots in thegroup are capable of completing the homogeneous task ofthe system independently, and each robot is equipped withall appropriate and relevant low-level controllers. The systemis robust to addition and removal (varying m ) of autonomousand teleoperated robots alike at any time during the missionsince each update cycle of the workload allocation process isindependent of the previous; the system will simply reallocate Time step (t) T o t a l w o r k l oad t r an s i t i on e rr o r , ' | " < | m=20, K=5m=20, K=10m=50, K=5m=50, K=10m=100, K=5m=100, K=10m=500, K=5m=500, K=10 Fig. 13: Scalability analysis with m = m = m = m =
500 robots following scenario S3: human and robotconditions set to c o = . c o = . c r = .
75 and c r = . K is consistent with larger K yielding faster totalworkload transition time. Increasingly larger m resulted inincreasingly smaller total workload allocation error and longertransition times as expected.the workload accordingly in the next update cycle followingEq. (1). Workload transition considerations of added robots inthe next update cycle are also made with σ ( t + ) determinedwith σ ( t ) = τ . Any discontinuity or dis-connectedness in the teleoperation or communication graphstructure defined in Section II is treated as a failed robot withzero health .Our current work limits the update cycle to the slowest fre-quency of the individual modules. However, we acknowledgethat it may be improved by considering the highest frequencyof all the individual modules and relying on current estimatesfor the slower modules; an implementation of the Kalmanfilter for the slower modules may also be used for bettercurrent estimates. We identify this potential improvement inthe system update cycle as future work on our proposedframework.We acknowledge that if a large number of agents suffer fromdeteriorated conditions at once and the rest of the agents areasked to compensate, since the system is designed to ensurethat the entire workspace is allocated at all times, it mayoverwhelm them as well and in turn affect their health / ability and performance as well. The current framework is unableto consider how much of the total workload can actually beallocated to the given number of agents such that certain agentsare not overwhelmed even if their health / ability are optimal.We identify this as a limitation to the proposed frameworkand hope to address this issue in our future work. The cur-rent framework is therefore only applicable assuming enoughagents are in optimal conditions in terms of health / ability , suchthat the total workspace could be covered at all times without affecting compensating agents.The current MH-MR framework assigns workload relyingonly on agent condition and performance. However, humanoperators in the system may have different levels of skill,experience and responsiveness despite the measured humancondition metric. As such, we acknowledge that with thecurrent design for workload allocation the full potential ofthe human robot team may not be utilized. Different humanoperators may also have different working capabilities evenunder stress or different emotional states which have not beenconsidered in the current system. A number of other com-plexities also exist on measuring human operator health andcondition in the real world in terms of applicability of sensors,the variable calibration requirements and environmental effectsthat contribute to the huge variety in recorded human behavior[70]. Therefore, as future work of our MH-MR workloadallocation framework, we intend to investigate independenthuman condition assessment and incorporate further humanoperator attributes in the workload allocation process.The proposed framework allows multi-human and multi-robots to work together in a given application; robots arefree to work autonomously and may also be teleoperated byhuman operators all the while ensuring that the total workalways sums to unity. Therefore, autonomy of the system onthe application is shared amongst all individual agents. Withthe lap time comparison for scenario S1, and failed robotcase in scenario S2 presented in the validation section of themanuscript, we established the effectiveness of our proposedworkload allocation framework. Therefore, we believe thatthe proposed MH-MR workload allocation framework is aneffective shared-autonomy tool. S3 and S4 presented theeffects of q f on workload allocation followed by the scalabilityof the system established for various K .The proposed workload allocation and transition functionmodules are designed to reflect the condition and performanceof the autonomous robots, and the humans and robots inteleoperated robots focusing on the working ability of eachindividual rather than overall optimal system performance.This approach is important for operators (any agents in thesystem) to believe that the system will consider any deteriora-tion in their health / ability to work and adjust their workloadaccordingly, such that they are never overwhelmed. We believethat this function could potentially instill trust in operators (anyagents in the system) on the shared-autonomy of the systemwhile working to ensure that they are never overwhelmed withtheir currently allocated workload.VI. C ONCLUSION
An adaptive multi-human multi-robot system framework hasbeen proposed that performs real-time workload allocationbased on both human operator and robot conditions andon performance, with workload transitional considerations.The design allows compatibility with previously establishedquantitative human and robot health assessments tests; as-suming human and/or robot conditions and/or performanceare measured with enough accuracy, the modular design ofthe framework can be used for a wide variety of multi-agent applications, including search and rescue, exploration, surveillance and monitoring based on specific requirements.The system functions independent of the number of humansor robots and is therefore scalable to hosting any number ofagents.The applicability, effectiveness and scalability of the pro-posed framework was validated experimentally with a MH-MR patrolling application, demonstrating system adaptationto maintain performance despite simulated temporary and per-manent, deteriorating human and robot conditions, includingcomplete robot failures. Further work on incorporating modu-lar function blocks on human experience, skill, responsivenessand safety protocols within the work allocation module inthe presence of sub-nominal human and/or robot conditionsis currently underway, along with field deployment studies ofthe proposed framework.R
EFERENCES[1] M. Khoramshahi and A. Billard, “A dynamical system approach to task-adaptation in physical human–robot interaction,”
Autonomous Robots ,vol. 43, no. 4, pp. 927–946, 2019.[2] V. Villani, F. Pini, F. Leali, C. Secchi, and C. Fantuzzi, “Survey onhuman-robot interaction for robot programming in industrial applica-tions,”
IFAC-PapersOnLine , vol. 51, no. 11, pp. 66–71, 2018.[3] M. Shiomi, K. Shatani, T. Minato, and H. Ishiguro, “How should a robotreact before people’s touch?: Modeling a pre-touch reaction distance fora robot’s face,”
IEEE Robotics and Automation Letters , vol. 3, no. 4,pp. 3773–3780, 2018.[4] M. Lewis, H. Wang, S.-Y. Chien, P. Scerri, P. Velagapudi, K. Sycara, andB. Kane, “Teams organization and performance in multi-human/multi-robot teams,” in . IEEE, 2010, pp. 1617–1623.[5] S.-Y. Chien, M. Lewis, S. Mehrotra, N. Brooks, and K. Sycara,“Scheduling operator attention for multi-robot control,” in .IEEE, 2012, pp. 473–479.[6] J. L. Drury, J. Scholtz, and H. A. Yanco, “Awareness in human-robot interactions,” in
SMC’03 Conference Proceedings. 2003 IEEEInternational Conference on Systems, Man and Cybernetics. ConferenceTheme-System Security and Assurance (Cat. No. 03CH37483) , vol. 1.IEEE, 2003, pp. 912–918.[7] H. A. Yanco and J. Drury, “Classifying human-robot interaction: anupdated taxonomy,” in , vol. 3. IEEE, 2004,pp. 2841–2846.[8] P. Tsarouchi, G. Michalos, S. Makris, T. Athanasatos, K. Dimoulas,and G. Chryssolouris, “On a human–robot workplace design and taskallocation system,”
International Journal of Computer Integrated Man-ufacturing , vol. 30, no. 12, pp. 1272–1279, 2017.[9] D. B. Kaber, C. M. Perry, N. Segall, C. K. McClernon, and L. J.Prinzel III, “Situation awareness implications of adaptive automationfor information processing in an air traffic control-related task,”
Inter-national Journal of Industrial Ergonomics , vol. 36, no. 5, pp. 447–462,2006.[10] L. J. Prinzel, F. G. Freeman, M. W. Scerbo, P. J. Mikulka, and A. T. Pope,“A closed-loop system for examining psychophysiological measuresfor adaptive task allocation,”
The International Journal of AviationPsychology , vol. 10, no. 4, pp. 393–410, 2000.[11] G. F. Wilson and C. A. Russell, “Operator functional state classificationusing multiple psychophysiological features in an air traffic control task,”
Human Factors , vol. 45, no. 3, pp. 381–389, 2003.[12] R. Parasuraman, M. Barnes, K. Cosenzo, and S. Mulgund, “Adaptiveautomation for human-robot teaming in future command and controlsystems,”
International Journal of Command and Control , vol. 1, no. 2,pp. 43–68, 2007.[13] C. A. Miller and R. Parasuraman, “Designing for flexible interactionbetween humans and automation: Delegation interfaces for supervisorycontrol,”
Human factors , vol. 49, no. 1, pp. 57–75, 2007.[14] S. Musi´c and S. Hirche, “Control sharing in human-robot team interac-tion,”
Annual Reviews in Control , vol. 44, pp. 342–354, 2017.[15] D.-H. Lee, “Resource-based task allocation for multi-robot systems,”
Robotics and Autonomous Systems , vol. 103, pp. 151–161, 2018. [16] T. Setter and M. Egerstedt, “Energy-constrained coordination of multi-robot teams,” IEEE Transactions on Control Systems Technology ,vol. 25, no. 4, pp. 1257–1263, 2016.[17] A. Koubaa, H. Bennaceur, I. Chaari, S. Trigui, A. Ammar, M.-F. Sriti,M. Alajlan, O. Cheikhrouhou, and Y. Javed, “General background onmulti-robot task allocation,” in
Robot Path Planning and Cooperation .Springer, 2018, pp. 129–144.[18] K. E. Booth, G. Nejat, and J. C. Beck, “A constraint programmingapproach to multi-robot task allocation and scheduling in retirementhomes,” in
International conference on principles and practice ofconstraint programming . Springer, 2016, pp. 539–555.[19] M. Otte, M. J. Kuhlman, and D. Sofge, “Auctions for multi-robot taskallocation in communication limited environments,”
Autonomous Robots ,vol. 44, no. 3, pp. 547–584, 2020.[20] Y. Emam, S. Mayya, G. Notomista, A. Bohannon, and M. Egerst-edt, “Adaptive task allocation for heterogeneous multi-robot teamswith evolving and unknown robot capabilities,” arXiv preprintarXiv:2003.03344 , 2020.[21] M. S. Malvankar-Mehta and S. S. Mehta, “Optimal task allocation inmulti-human multi-robot interaction,”
Optimization Letters , vol. 9, no. 8,pp. 1787–1803, 2015.[22] F. Tang and L. E. Parker, “A complete methodology for generating multi-robot task solutions using asymtre-d and market-based task allocation,”in
Proceedings 2007 IEEE international conference on robotics andautomation . IEEE, 2007, pp. 3351–3358.[23] R. Zlot and A. Stentz, “Market-based multirobot coordination forcomplex tasks,”
The International Journal of Robotics Research , vol. 25,no. 1, pp. 73–101, 2006.[24] M. B. Dias, R. Zlot, N. Kalra, and A. Stentz, “Market-based multirobotcoordination: A survey and analysis,”
Proceedings of the IEEE , vol. 94,no. 7, pp. 1257–1270, 2006.[25] N. Atay and B. Bayazit, “Mixed-integer linear programming solution tomulti-robot task allocation problem,” 2006.[26] M. Darrah, W. Niland, and B. Stolarik, “Multiple uav dynamic taskallocation using mixed integer linear programming in a sead mission,”in
Infotech@ Aerospace , 2005, p. 7164.[27] A. R. Mosteo and L. Montano, “Simulated annealing for multi-robothierarchical task allocation with flexible constraints and objective func-tions,” in
Workshop on Network Robot Systems: Toward IntelligentRobotic Systems Integrated with Environments. IROS , 2006.[28] E. G. Jones, M. B. Dias, and A. Stentz, “Time-extended multi-robot co-ordination for domains with intra-path constraints,”
Autonomous robots ,vol. 30, no. 1, pp. 41–56, 2011.[29] J. Wang, Y. Gu, and X. Li, “Multi-robot task allocation based on antcolony algorithm,”
Journal of Computers , vol. 7, no. 9, pp. 2160–2167,2012.[30] D. Fox, W. Burgard, S. Thrun, and A. B. Cremers, “A hybrid col-lision avoidance method for mobile robots,” in
Proceedings. 1998IEEE International Conference on Robotics and Automation (Cat. No.98CH36146) , vol. 2. IEEE, 1998, pp. 1238–1243.[31] A. Pandey, S. Pandey, and D. Parhi, “Mobile robot navigation andobstacle avoidance techniques: A review,”
Int Rob Auto J , vol. 2, no. 3,p. 00022, 2017.[32] A. Martinez, E. Tunstel, and M. Jamshidi, “Fuzzy logic based collisionavoidance for a mobile robot,”
Robotica , vol. 12, no. 6, pp. 521–527,1994.[33] H. Lu, D. Frauendorfer, M. Rabbi, M. S. Mast, G. T. Chittaranjan, A. T.Campbell, D. Gatica-Perez, and T. Choudhury, “Stresssense: Detectingstress in unconstrained acoustic environments using smartphones,” in
Proceedings of the 2012 ACM Conference on Ubiquitous Computing .ACM, 2012, pp. 351–360.[34] F. Noroozi, D. Kaminska, C. Corneanu, T. Sapinski, S. Escalera, andG. Anbarjafari, “Survey on emotional body gesture recognition,”
IEEEtransactions on affective computing , 2018.[35] E. Garcia-Ceja, V. Osmani, and O. Mayora, “Automatic stress detectionin working environments from smartphones accelerometer data: a firststep,”
IEEE journal of biomedical and health informatics , vol. 20, no. 4,pp. 1053–1060, 2015.[36] S. M. Lajevardi and H. R. Wu, “Facial expression recognition inperceptual color space,”
IEEE transactions on image processing , vol. 21,no. 8, pp. 3721–3733, 2012.[37] D. Giakoumis, A. Drosou, P. Cipresso, D. Tzovaras, G. Hassapis,A. Gaggioli, and G. Riva, “Using activity-related behavioural featurestowards more effective automatic stress detection,”
PloS one , vol. 7,no. 9, p. e43571, 2012. [38] M. Zhao, F. Adib, and D. Katabi, “Emotion recognition using wirelesssignals,” in
Proceedings of the 22nd Annual International Conferenceon Mobile Computing and Networking . ACM, 2016, pp. 95–108.[39] D. Carneiro, J. C. Castillo, P. Novais, A. Fern´andez-Caballero, andJ. Neves, “Multimodal behavioral analysis for non-invasive stress de-tection,”
Expert Systems with Applications , vol. 39, no. 18, pp. 13 376–13 389, 2012.[40] V. Villani, L. Sabattini, C. Secchi, and C. Fantuzzi, “Natural interactionbased on affective robotics for multi-robot systems,” in .IEEE, 2017, pp. 56–62.[41] A. Sano and R. W. Picard, “Stress recognition using wearable sensorsand mobile phones,” in . IEEE, 2013, pp. 671–676.[42] F.-T. Sun, C. Kuo, H.-T. Cheng, S. Buthpitiya, P. Collins, and M. Griss,“Activity-aware mental stress detection using physiological sensors,”in
International conference on Mobile computing, applications, andservices . Springer, 2010, pp. 282–301.[43] A. Muaremi, B. Arnrich, and G. Tr¨oster, “Towards measuring stresswith smartphones and wearable devices during workday and sleep,”
BioNanoScience , vol. 3, no. 2, pp. 172–183, 2013.[44] S. Greene, H. Thapliyal, and A. Caban-Holt, “A survey of affective com-puting for stress detection: Evaluating technologies in stress detectionfor better health,”
IEEE Consumer Electronics Magazine , vol. 5, no. 4,pp. 44–56, 2016.[45] M. Westman and D. Eden, “The inverted-u relationship between stressand performance: A field study,”
Work & Stress , vol. 10, no. 2, pp.165–173, 1996.[46] J. A. Healey and R. W. Picard, “Detecting stress during real-world driv-ing tasks using physiological sensors,”
IEEE Transactions on intelligenttransportation systems , vol. 6, no. 2, pp. 156–166, 2005.[47] W. Liao, W. Zhang, Z. Zhu, and Q. Ji, “A real-time human stressmonitoring system using dynamic bayesian network,” in . IEEE, 2005, pp. 70–70.[48] N. Sharma and T. Gedeon, “Objective measures, sensors and compu-tational techniques for stress recognition and classification: A survey,”
Computer methods and programs in biomedicine , vol. 108, no. 3, pp.1287–1301, 2012.[49] J. Zhai and A. Barreto, “Stress detection in computer users basedon digital signal processing of noninvasive physiological variables,” in . IEEE, 2006, pp. 1355–1358.[50] J. A. Healey, “Wearable and automotive systems for affect recognitionfrom physiology,” Ph.D. dissertation, Massachusetts Institute of Tech-nology, 2000.[51] T. T. Le, W. Fu, and J. H. Moore, “Scaling tree-based automatedmachine learning to biomedical big data with a feature set selector,”
Bioinformatics , vol. 36, no. 1, pp. 250–256, 2020.[52] H.-G. Kim, E.-J. Cheon, D.-S. Bai, Y. H. Lee, and B.-H. Koo, “Stressand heart rate variability: A meta-analysis and review of the literature,”
Psychiatry investigation , vol. 15, no. 3, p. 235, 2018.[53] N. Munla, M. Khalil, A. Shahin, and A. Mourad, “Driver stress leveldetection using hrv analysis,” in . IEEE, 2015, pp. 61–64.[54] D. McDuff, S. Gontarek, and R. Picard, “Remote measurement of cog-nitive stress via heart rate variability,” in .IEEE, 2014, pp. 2957–2960.[55] J.-P. Gouin, K. Wenzel, S. Boucetta, J. O’Byrne, A. Salimi, and T. T.Dang-Vu, “High-frequency heart rate variability during worry predictsstress-related increases in sleep disturbances,”
Sleep medicine , vol. 16,no. 5, pp. 659–664, 2015.[56] E. Jovanov, A. Lords, D. Raskovic, P. G. Cox, R. Adhami, andF. Andrasik, “Stress monitoring using a distributed wireless intelligentsensor system,”
IEEE Engineering in Medicine and Biology Magazine ,vol. 22, no. 3, pp. 49–55, 2003.[57] C. A. Ottesen, “Investigating heart rate variability: a machine learningapproach,” Master’s thesis, Queen Mary University of London, 8 2017.[58] H. Sarker, M. Tyburski, M. M. Rahman, K. Hovsepian, M. Sharmin,D. H. Epstein, K. L. Preston, C. D. Furr-Holden, A. Milam, I. Nahum-Shani et al. , “Finding significant stress episodes in a discontinuous timeseries of rapidly varying mobile sensor data,” in
Proceedings of the2016 CHI conference on human factors in computing systems , 2016,pp. 4489–4501. [59] S. Lahmiri and M. Boukadoum, “Physiological signal denoising withvariational mode decomposition and weighted reconstruction after dwtthresholding,” in . IEEE, 2015, pp. 806–809.[60] Q. Li, R. G. Mark, and G. D. Clifford, “Robust heart rate estimationfrom multiple asynchronous noisy sources using signal quality indicesand a kalman filter,” Physiological measurement , vol. 29, no. 1, p. 15,2007.[61] K. M. Reichard, “Integrating self-health awareness in autonomoussystems,”
Robotics and Autonomous Systems , vol. 49, no. 1-2, pp. 105–112, 2004.[62] M. A. Goodrich and D. R. Olsen, “Seven principles of efficient humanrobot interaction,” in
SMC’03 Conference Proceedings. 2003 IEEEInternational Conference on Systems, Man and Cybernetics. ConferenceTheme-System Security and Assurance (Cat. No. 03CH37483) , vol. 4.IEEE, 2003, pp. 3942–3948.[63] S. C. Wong, L. Middleton, B. A. MacDonald, and N. Auckland,“Performance metrics for robot coverage tasks,” in
Proceedings ofAustralasian Conference on Robotics and Automation , vol. 27, 2002,p. 29.[64] N. A. Cruz and J. C. Alves, “Navigation performance of an autonomoussailing robot,” in . IEEE, 2014, pp. 1–7.[65] A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz,and M. Goodrich, “Common metrics for human-robot interaction,” in
Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction . ACM, 2006, pp. 33–40.[66] R. Murphy and D. Schreckenghost, “Survey of metrics for human-robot interaction,” in
Proceedings of the 8th ACM/IEEE internationalconference on Human-robot interaction . IEEE Press, 2013, pp. 197–198.[67] A. Machado, G. Ramalho, J.-D. Zucker, and A. Drogoul, “Multi-agentpatrolling: An empirical analysis of alternative architectures,” in
Inter-national workshop on multi-agent systems and agent-based simulation .Springer, 2002, pp. 155–170.[68] M. Balzer, “Capacity-constrained voronoi diagrams in continuousspaces,” in .IEEE, 2009, pp. 79–88.[69] N. Karapetyan, K. Benson, C. McKinney, P. Taslakian, and I. Rekleitis,“Efficient multi-robot coverage of a known environment,” in . IEEE, 2017, pp. 1846–1852.[70] R. Khusainov, D. Azzi, I. E. Achumba, and S. D. Bersch, “Real-timehuman ambulation, activity, and physiological monitoring: Taxonomyof issues, techniques, applications, challenges and limitations,”
Sensors ,vol. 13, no. 10, pp. 12 852–12 902, 2013.
Tamzidul Mina received the B.S degree in Me-chanical Engineering from Purdue University, WestLafayette, USA, in 2012. He is currently pursuing aPh.D. degree in Mechanical Engineering at PurdueUniversity, West Lafayette, IN.His research interests include multi-robot systems,control system design for bio-inspired robotic sys-tems with a focus on applications in social groupbehavior in robotic swarms.
Shyam Sundar Kannan received the B.E degreein Computer Science and Engineering from AnnaUniversity, Chennai, India, in 2016 and the M.S.degree in Computer and Information Technologyfrom Purdue University, West Lafayette, IN, USA,in 2019. He is currently pursuing the Ph.D. degreein Technology at Purdue University, West Lafayette,IN.From 2016 to 2017, he worked as a ResearchAssistant at Advanced Geometric Computing Lab,IIT-Madras, Chennai, India. His research interestsinclude SLAM, localization and path planning for multi-agent systems andcomputational geometry.
Wonse Jo received the B.S. in robotics engineeringfrom Hoseo University, South Korea in 2013 andM.S. degrees in electronic engineering from theKyung-Hee University, South Korea, in 2015. He iscurrently pursuing the Ph.D. degree in computer andinformation technology at Purdue University, WestLafayette, IN, USA.His research interests include human-robot in-teraction, environmental robotics, and assistiverobotics.