FASTSWARM: A Data-driven FrAmework for Real-time Flying InSecT SWARM Simulation
FFASTSWARM: A Data-driven F r A mework forReal-time Flying In S ec T SWARM
Simulation
Wei Xiang Xinran Yao He Wang Xiaogang Jin ∗ State Key Lab of CAD&CG, Zhejiang University, Hangzhou 310058, China School of Computing, University of Leeds, Leeds LS2 9JT, United Kingdom ZJU-Tencent Game and Intelligent Graphics Innovation Technology Joint Lab, Hangzhou 310058, China
Abstract
Insect swarms are common phenomena innature and therefore have been actively pur-sued in computer animation. Realistic insectswarm simulation is difficult due to twochallenges: high-fidelity behaviors and largescales, which make the simulation practicesubject to laborious manual work and excessivetrial-and-error processes. To address bothchallenges, we present a novel data-drivenframework, FASTSWARM, to model complexbehaviors of flying insects based on real-worlddata and simulate plausible animations of flyinginsect swarms. FASTSWARM has a linear timecomplexity and achieves real-time performancefor large swarms. The high-fidelity behaviormodel of FASTSWARM explicitly takes intoconsideration the most common behaviors offlying insects, including the interactions amonginsects such as repulsion and attraction, the self-propelled behaviors such as target following andobstacle avoidance, and other characteristicssuch as the random movements. To achievescalability, an energy minimization problemis formed with different behaviors modelledas energy terms, where the minimizer is thedesired behavior. The minimizer is computedfrom the real-world data, which ensures theplausibility of the simulation results. Extensivesimulation results and evaluations show thatFASTSWARM is versatile in simulating var-ious swarm behaviors, high fidelity measuredby various metrics, easily controllable ininducing user controls and highly scalable . ∗ Corresponding author. E-mail: [email protected]
Keywords: insect swarm simulation, data-driven, optimization, collective behavior, realtime
Insects are ubiquitous in both the real and virtualworlds, and many of them present collective be-haviors for efficient and collaborative work. Inthe real world, flying insect swarms can exhibita great variety of behaviors such as aggrega-tion, mating, migration and escaping [1], wherethe individual behaviors are often correlated invarious ways, from collaborative to competitiveor even adversarial. Simulating realistic insectswarms are in the interest of many areas. Inrobotics, research of insect swarms has led tonew algorithms for robots collective jobs on in-formation transfer, decision-making, task par-titioning or transport [2, 3]. In computer ani-mation, insect swarms have been used to createwondrous natural phenomena and interesting vi-sual effects [4, 5]. However, simulating scalablecollective behaviors of insect swarms with high-fidelity remains challenging.Existing methods for simulating flying insectswarms mainly fall into two categories: empir-ical and data-driven. Empirical methods aimto abstract swarm behaviors into mathematicalmodels and deterministic systems, such as thefield-based methods [4], or a combination ofthe field-based with the force-based methods[5]. One limitation of such methods is thatthe simulated trajectories are often too regular a r X i v : . [ c s . G R ] J u l nd lack of visual diversity, due to their deter-minism nature. In contrast, data-driven meth-ods tend to rely on real-world data, such as us-ing computer vision techniques to capture 3Dtrajectories of swarms [6, 7, 8] for simulation[9, 10, 11]. However, due to the intrinsic limi-tations of optical sensors (e.g. occlusions), themotion capture is set up in massively simplifiedlaboratory environments, and there are still ex-cessive tracking errors where only short track-lets can be relatively reliably obtained. This cre-ates tremendous difficulties in simulating flyinginsect swarms with the desired high-fidelity andscalability. First, the captured trajectories can-not be relied upon to extract all behaviors of fly-ing insects. Second, the generalizability of themodel based on simple data is limited by boththe environment complexity and the swarm size.In this paper, we propose a novel data-drivenframework (FASTSWARM) to address the chal-lenges for simulating flying insect swarms. Ourframework models insects as agents, and theswarm behavior computation as an energy min-imization problem. A variety of important be-haviors identified in numerical analysis [12] andempirical observations are captured by differentenergy terms, including the interaction amongagents, the self-propulsion of agents and the mo-tion noise of agents, so that the minimizer leadsto realistic behaviors. Besides, our frameworkalso model user-defined behaviors by employ-ing user-control energy terms. The total energyfunction is constructed in the way that it canbe optimized quickly to achieve scalability andreal-time performance.During optimization, instead of seeking aminimizer by pure mathematical optimizationswhich would make the minimizer only ideal intheory, we seek the minimizer by referencinga motion characteristic dataset generated fromthe real-world data, so that the simulated behav-iors mimic the real data. However, this meansthat the motion characteristics we reply on inthe reference dataset has to be reliable. In thereal-world data, although excessive noises ex-ist and whole trajectories can rarely be obtained,velocity is much more reliable as it can be esti-mated from short tracklets [13]. In our simula-tion framework, both velocity and accelerationare regarded as the motion characteristics forgenerating the reference dataset, and we there- fore optimize for the velocity to update the mo-tion states of the agents. In addition, we use animplicit Euler scheme to improve the stability.Formally, the contributions of the paper in-clude: • A novel data-driven 3D swarm simulationframework which captures a variety of bio-logically important behaviors. • An optimization method that maximallymakes use of real-world data to ensure thesimulation fidelity for flying insect swarms. • A scalable model for large swarm simula-tion with straightforward user control.The remainder of this paper is organized asfollows. After briefly introducing related workin Section 2, we give a pipeline overview of ourapproach in Section 3. In Section 4, we explainour optimization-based data-driven model. Weshow simulation results and evaluations of ourmethod in Section 5, and discuss the limitationsand future work in Section 6.
Data-Driven Simulation . In graphics, data-driven methods have been proposed to simulatebehaviors of crowds and traffics. Given trajec-tories (or tracklets) extracted from crowd data,example-based methods can blend them to gen-erate new animations [14], use a “clone andpaste” technique to generate larger crowds [15],or cluster them into groups and update the mo-tion of an agent based on the actions of its near-est patch or associated group [16, 17, 18].In data-driven traffic simulation, Chao et al.[19] present a video-based approach to learn thespecific driving characteristics of drivers to re-construct or simulate traffic flows. By takingthe spatio-temporal information of traffic flowsas a 2D texture, a texture synthesis techniqueis developed to populate virtual road networkswith realistic traffic flows [20]. In addition,deep learning can also be used to learn the latentpatterns of vehicle trajectories for intersectionaltraffic simulation and editing [21]. Recently,an interactive data-driven optimization approach[13] has been proposed to simulate traffic sce-narios with heterogeneous agents. epulsion
Zone
Attraction-AgentZone
Motion ScenarioMotion Statesof Agents
Initialization Optimization
Real TrajectoriesReference Dataset
Data Processing
Attraction
Interaction among Agents Response to environment
Noise
Result
User-Defined Control
Danger
Figure 1: Overview of our data-driven approach for simulating flying insect swarms.Although these methods can generate plausi-ble crowd or traffic animations, they focus on2D simulation and cannot be easily extended to3D flying insect swarms because the motion dy-namics of insects are significantly different.
Insect Swarm Simulation . There has beenan interdisciplinary effort in the research of col-lective behaviors of insects. Researchers in agri-culture proposed an insect migration trajectorysimulation method to accurately predict the des-tinations of insect migration and achieve effec-tive early warning to reduce the impact of pestson agriculture [22]. Public health researchersproposed an indoor flight behavior model ofhost-seeking mosquitoes for selecting bed netsthat can effectively reduce the spread of thevirus from mosquitoes [23]. In swarm robotics,swarm behaviors can be used to control collab-orative robots, such as the insect-based biobotsfor search and rescue [24], unmanned aerial ve-hicle quadrotors emulating insect swarm behav-iors [25], and the task allocation algorithm basedon ant colonies [26].In graphics, methods have been proposed tosimulate collective behaviors of ants [27, 28].However, it is non-trivial to extend these meth-ods to simulate flying insect swarms because oftheir significant differences in motion dynam-ics. Data-driven methods have been developedto simulate flying insect swarms. Li et al. [9]present a framework to simulate flying insectswarms by sampling the statistical informationfrom real datasets. However, it is not generalenough because a new local steering model isneeded whenever the data changes. Differentfrom the prior work, we present a general frame-work which is maximally data-driven and auto-matically adaptive to data, and therefore is not tied to specific data. Recently, data-driven noisemodels and force-based models are introducedin [10, 11] to generate biologically plausible an-imations of flying insect swarms. However, theydo not generalize well to more complex scenar-ios because they are prone to numerical errorsduring generalization.Except for data-driven methods, a hybridmodel combining potential fields and curl-noise[29] is developed in [4] to simulate various be-haviors of flying insect swarms. However, sucha forced-based and field-based method oftengenerates trajectories that look too regular be-cause all agents share similar motion patterns.For the special effect simulation application,Chen et al. present a flock morphing methodof flying insect swarms with pre-defined shapeconstraints [5]. Instead of pre-defined morph-ing, our framework can generate plausible ani-mations based on real-world motions to satisfyuser-defined constraints.
As illustrated in Figure 1, FASTSWARM canbe conceptually described as a three-phase pro-cess: the data processing stage, the initializa-tion stage, and the real-time simulation stage.Our simulation algorithm makes full use of thedataset extracted from the data processing stageto compute the trajectories of the agents in theswarm. At the data processing stage, the rawdata are noisy tracklets [6, 8]. For each type ofinsects, we generate a reference dataset that con-tains the motion characteristics specific to thattype. Each data sample consists of two items:velocity and acceleration, estimated using for-ward differencing. We organize the data by thepeed, similar to [13], for fast indexing. Atthe initialization stage, we initialize the motionscenario, the number of insects, and the initialmotion state of each agent. The motion stateof an agent includes its position, velocity, mo-tion randomness, and control direction. Duringsimulation, we employ a data-driven approachto update the motion state for each agent. Forevery time step, our model selects a velocityfrom the reference dataset that minimizes an ob-jective function which models the interactions,self-propulsion, and user-specified controls.
The reference data is denoted as D = ∪ v d v ,where d v = [ v , a ] , v is the velocity, and a is theacceleration. For a swarm with N agents at time t , the state of agent i ( i = 1 , ..., N ) is denotedas s i,t = [ p i,t , v i,t , ˆ v ni,t , ˆ v cdi,t ] , s i,t ∈ R , where p i,t ∈ R is the current position, v i,t ∈ R is thecurrent velocity, ˆ v ni,t ∈ R is the noise direction,and ˆ v cdi,t is the user-defined control direction. Wefurther use S t = ∪ i s i,t to represent the motionstate of the whole swarm at t . Then, the motiondynamics of agent i is formulated as: v i,t +1 = arg min v i,t +1 ∈ d v ∈ D E ( i, v , a , S t ) , p i,t +1 = p i,t + v i,t +1 ∆ t, ˆ v ni,t +1 = f N ( s i,t ) , ˆ v cdi,t +1 = f CD ( s i,t , EN V t ) , (1)where the velocity v i,t +1 ∈ d v ∈ D minimizesthe objective function E , a is the accelerationin d v , ∆ t is a timestep, and p i,t +1 is the posi-tion at t + 1 . After updating the velocity andposition of each agent, we update the noise di-rection ˆ v ni,t +1 and the user-defined control direc-tion ˆ v cdi,t +1 using f N and f CD , respectively. Theobjective function E is defined as: E ( i, v , a , S t ) = E i,int + E i,sp + E i,n + E i,user , (2)where E i,int is an interaction term to modelthe interactions among agents, E i,sp is an self-propulsion term to control the motion of agents, E i,n is an noise-induced control term to modelthe randomness of agents’ movements, and E i,user is the user-control term to enforce user-defined constraints.For each agent i , the minimization of E aimsto update v i,t +1 by selecting a v ∈ d v ∈ D . Forsimplicity, we use the selected velocity v and ˆv to represent v i,t +1 and its direction in the fol-lowing definitions of the energy terms. Insects move in coordination with eachother and aggregate without collisions. InFASTSWARM, such coordination is modeledas interactions which include repulsion forcollision avoidance and attraction for aggrega-tion. The interaction energy E i,int is thereforedefined as: E i,int = E i,rep + E i,att , (3)where E i,rep and E i,att represent the repulsionenergy and the attraction energy, respectively. An agent avoids collisions with other agents thatare too close to it. Based on the zonal interactionmodel, which dictates that an insect has a short-range repulsion within a circular distance-basedzone [12], we describe the repulsion zone usinga spherical kernel function, centered at the agentwith a predefined cut-off radius d rep (see Fig-ure 1 for the 2D representation), and an agentwill avoid collisions with its neighboring agentswithin the repulsion zone . Assuming that agent i will move in a timestep using velocity v inthe data term d v and it’s repulsive neighbor j will hold its current velocity v j,t in a timestep,then the distance-based repulsion energy E i,rep is calculated as follows: p (cid:48) i,t +1 = p i,t + v ∆ t, p (cid:48) j,t +1 = p j,t + v j,t ∆ t,E i,rep = w rep · | RN | (cid:88) j ∈ RN e ρ (cid:18) − (cid:107) p (cid:48) i,t +1 − p (cid:48) j,t +1 (cid:107) drep (cid:19) , (4)where w rep ≥ is the weight of the repulsionenergy, RN is the set of neighbor agents, | RN | is the number of the repulsive neighbors, ρ > is a constant used for the scaling of the energy,nd p (cid:48) i,t +1 and p (cid:48) j,t +1 are the predicted positionsof agent i and its neighbor j . Attraction exists between agents, and also be-tween every agent and the center of the swarm[12]. The attraction energy E i,att therefore con-sists of both the attraction from the neighborsand the attraction from the swarm center: E i,att = E i,oa + E i,cos , (5)where E i,oa is the energy of attraction from theneighbors, and E i,cos is the attraction energyfrom the center of the swarm. As shown inFigure 1, the attraction zone of an agent to itsneighbors is also formulated as a spherical ker-nel function with two cut-off radii, one inner d att, and one outer d att, : The attraction energyfrom the other agents E i,oa is defined as: E i,oa = w oa · | AN | (cid:88) j ∈ AN e ρ (cid:18) (cid:107) p (cid:48) i,t +1 − p j,t (cid:107) datt, − (cid:19) , (6)where w oa ≥ is the weight of the energy term, AN is the set of the attraction neighbors of agent i , and | AN | is the number of attraction neigh-bors. The attraction energy on agent i is normal-ized by the total number of the neighbor agents.For the attraction to the swarm center on agent i , we use ˆ v i,center , the direction that points fromthe agent’s current position to the swarm center,as the desired direction of the chosen velocity ˆ v . The attraction from swarm center E cosi is de-fined as: E i,cos = w cos · e ρ (cid:107) ˆ v − ˆ v i,center (cid:107) , (7)where w cos ≥ is the weight of the energy term. Besides being reactive, insects are also self-propelled. We therefore define a self-propulsionenergy term, not only to drive the agents to keepmoving similar to the reference trajectories, butalso to response to external stimuli in environ-ments. For agent i , the self-propulsion energyincludes the internal propulsion term E i,vel todrive the agent to generate new trajectories sim-ilar to the data, and the reaction term E i,env to drive the agent to react to external stimuli: E i,sp = E i,vel + E i,env . (8) We assume that the internal drive ensures themotion smoothness, which is formulated con-sidering the first-order and second-order deriva-tives of motions: E i,vel = E i,dir + E i,acc , (9)where E i,dir = w dir · e ρ (cid:107) ˆ v − ˆ v i,t (cid:107) is the direc-tional continuity energy, and w dir ≥ is theweight. We also consider the continuity of itsacceleration, E i,acc = w acc · ( E i,adir + E i,amag ) , (10)where w acc ≥ is the weight. We use a (cid:48) = v − v i,t to compute the predicted accelera-tion so that if a d v is selected, and a is thecorresponding acceleration in d v . In Equation10, E i,adir = e ρ (cid:107) ˆ a (cid:48) − ˆ a (cid:107) is designed for thecontinuity of the direction of acceleration, and E i,amag = |(cid:107) a (cid:48) (cid:107) − (cid:107) a (cid:107)| is designed for the con-tinuity of the magnitude. Overall, E i,dir min-imizes the direction changes and E i,acc mini-mizes the acceleration changes compared to thereal data. The regularization on both motionderivatives leads to smooth motions. Insects can react quickly to external stimuli,e.g. following a target, or escaping from an ap-proaching predator. To model one stimulus, theenergy of response to it is defined as: E i,env = w env · e ρ Ψ , (11)where w env ≥ is the weight, and the function Ψ is defined as: Ψ = (cid:107) ˆ v − ˆ v i,sti (cid:107) , attracted, − (cid:107) p (cid:48) i,t +1 − p sti,t (cid:107) d danger , startled, (12)where p sti,t is the stimulus’ position, ˆ v i,sti is thedirection pointing from the agent’s current po-sition to the stimulus. In addition, as Figure 1shows, an agent is startled if a potential dangeris within a range defined by d danger . .3 Noise As the movements of swarms show strong ran-domness, we introduce a noise-induced controlterm to generate plausible trajectories. We in-troduce the curl noise [29], which has been usedin force-based models to generate more accuratesimulation results [10, 11], to model the motionrandomness. The function f N denotes the curlnoise function for generating a noise direction ˆ v ni,t . Similar to the calculation of the energy ofdirection continuity in Equation 9, the energy ofnoise-induced direction control is defined as: E i,n = w n · e ρ (cid:107) ˆ v − ˆ v ni,t (cid:107) , (13)where w n ≥ is the weight of the energy term. Besides the behavioral modeling of swarms, it isnecessary to induce user-control for the purposeof animation. We model user-control as a direc-tion control signal and introduce an energy termto constrain an agent to follow predefined user-defined trajectories. For agent i , the user-controlenergy is calculated as: E i,user = w user · e ρ || ˆ v − ˆ v cdi,t || , (14)where w user ≥ is the weight and ˆ v cdi,t is thecontrol direction of the agent for generating spe-cific trajectories. The implementation is done in C++ and theexperiments were run on a PC with an In-tel (R) Core (TM) i7 4.00 GHz CPU, 32 GBRAM, and an NVIDIA Geforce GTX 1060GPU. We provide both qualitative and quantita-tive evaluations to demonstrate the performanceof FASTSWARM. Due to the space limit, weonly show representative results and refer thereaders to the supplementary materials for moredetails. In all our experiments, we set ρ = 2 . ,and the weights of the energy terms for the testscenarios are shown in Table 1. We first show qualitative results. The results aredivided into two parts: natural behaviors and (a) 37 insects (b) 300 insects
Figure 2:
Aggregation behavior of insectswarms in different scales.user-controlled behaviors, the former showingthe high visual realism and the latter showingcontrollability.
We use the reference dataset from [6] to generatethe aggregation , mating , and escaping behaviorsof flying insect swarms. The results show thatFASTSWARM is capable of simulating a varietyof natural behaviors with good visual quality. Aggregation . Our approach can generate the aggregation behavior of insect swarms in differ-ent scales (see Figure 2). Figure 2(a) and Figure2(b) respectively shows two swarms of differentsizes in the aggregation scenario of the referencedataset with an unchanged swarm center.
Mating . Figure 3 shows the mating behaviorof insects generated by our approach. 100 maleagents (the cyan ones) are attracted by a female(the red one) insect and run after the female.
Escaping . Our approach can also generatethe escaping behavior of insect swarms. In Fig-ure 4, 100 flying insects are startled by a sud-den obstacle, and try to escape from the ob-stacle. During simulation, the obstacle willpass through the swarm, and the insects in theswarm escape from the danger and then aggre-gate again.
To evaluate the controllability, we ask the usersto draw different 3D shapes which are usedas constraints. Figure 5 shows two shape-constrained examples to constrain the agents tomove following different 3D curves. We regardthe 3D curve as a bidirectional curve with sev-eral keypoints (Figure 1: User-Defined Control).igure 3:
Mating behavior. Figure 4:
Escaping behavior.Each agent is randomly initialized near a keypoint with a random velocity selected from thereferenced dataset. Then, it is assigned a taskto traverse the keypoints. During simulation, weupdate the control direction ˆ v cdi,t of agent i as thedirection from the agent’s position p i,t to the po-sition of the goal keypoint p goali,t : V i,goal = p goali,t − p i,t , ˆ v cdi,t = ˆ V i,goal , (15)where ˆ V i,goal is the directional vector of V i,goal .The goal keypoint will be updated to the nextkeypoint when the agent reaches its current goal. (a) 500 flying insects forma heart shape. (b) 500 flying insects forma star shape. Figure 5: Two shape-constrained flocking ex-amples. There are 11 key points in (a)and 10 key points in (b).
To quantitatively test the scalability, we evalu-ate FASTSWARM with an increasing numberof insects in an aggregation scenario, similarto Section 5.1.1. Table 2 shows the time per-formance of the test scenarios in Section 5.1. Figure 6 shows the time performance of ourmethod against different swarm scales. Theo-retically, the time complexity is O ( kN ) where N is the number of insects and k is data sam-ple size in the reference data. In our experi-ments, k = 300 is enough to generate all be-haviors. The linear time complexity guaran-tees the high-performance of FASTSWARM. Inpractice, we further accelerate the computationin three ways: • Fast indexing. We discretize the spaceinto a 3D grid and index the neighbor-hood region of every agent, and approxi-mate the repulsion and attraction regionsusing cubes whose dimension is the dis-tance thresholds in their respective energyterms. During simulation, only local searchand computation is done. • Reduced search space. We sort the refer-ence dataset by the speed and divide thedataset into several groups. To choose anew velocity for an agent, our algorithmonly traverses the corresponding group andthe groups with similar speed. • Parallelization. As Equation 1 is com-puted for each agent and is thereforehighly parallelizable. We also parallelizeFASTSWARM to concurrently computethe updates for multiple agents.Figure 6: Time performance of our approach.The computation cost is linear w.r.t thenumber of insects
We compare our method with the method in[10] and the ground-truth under different met-rics. Here, we take the real tracklets [8] as theground-truth. In both comparisons, the scenes cenario w rep w oa w cos w dir w acc w env w n w user Aggregation( N = 37 / 300) 1 1 0.05 1 1 0 0.2 0Mating 1 1 0.05 1 1 1 0.5 0Escaping Aggregation 1 1 0.2 1 1 2 0.3 0In danger 0 0Shape Constraint(Heart / Star) 1 0.5 0 1 1 0 1 1 Table 1: The weights for our implementation in Section 5.1.
Scenario N Time (s/f)Aggregation 37 0.0003300 0.0025Mating 100 0.0159Escaping 100 0.0107Shape Constraint(Heart / Star) 500 0.0037
Table 2: Time performance of the simulation re-sults shown in Section 5.1are the same aggregation scenario in a box asthe ground-truth [8], and the swarms are initial-ized by randomly selecting one frame from thereal data. Since directly comparing individualtrajectories is not possible, the evaluation met-rics include distributions of density, velocity andacceleration, as they capture both the state andthe motion dynamics. Similar to previous work[13], we use minimum-distance to describe thedensity. Results are shown in Figure 7. Thedistributions of velocity, minimal distance andacceleration of FASTSWARM are much closerto those in the ground-truth than [10]. This istrue for both motion dynamics (velocity in Fig-ure 7(a) and acceleration in Figure 7(c-d)) andstates (density in Figure 7(b)). The better mo-tion dynamics shows that FASTSWARM is su-perior in capturing realistic behaviors.For comparing the density, in [10], the densityof the swarm is controlled simply by a distance-based attraction force, and the homogeneousdistance control results in the relatively uniformdensity of the swarm as the agents aggregatearound the attraction boundary. Addressing thisissue, we control the aggregation of agents bycombining the distance-based attraction amongagents with a direction control that drives theagents to move towards the swarm center, andFigure 7(b) shows that our method performs bet- ter in controlling the density of the swarm to besimilar to the reference dataset.
Given only trackelets (instead of whole trajec-tories) are avaiable, FASTSWARM provides apossible avenue to recover them for animationapplications. Although it is difficult to exactlyrecover the trajectories or even compute theirnumerical accuracy, due to the lack of ground-truth data, our recovered trajectories can achieveglobal visual similarities. As shown in Figure 8,we extract the trajectory segments of 37 agentsfrom the real-world dataset of [6] (Figure 8(a)).By taking these trajectory segments as the ini-tial trajectories of a swarm with 37 agents, ourmethod can be used to predict the subsequenttrajectories (Figure 8(c)).Our method can also synthesize plausible in-sect swarm animations by mixing the trajecto-ries from the real-world dataset with the syn-thetic trajectories. As Figure 8(d) shows, inthe real-world trajectories in 8(b), we add 100agents that are simulated by FASTSWARM.
We have presented a general and scalable data-driven optimization framework to simulate fly-ing insect swarms in real time. Our optimiza-tion method is capable of generating natural col-lective behaviors of flying insects by utilizinga motion characteristic dataset extracted fromreal data. The generated animations are plau-sible and have high visual realism. We havevalidated our approach using extensive experi-ments through qualitative and quantitative anal-ysis. Our method also provides a means to gen-erate interesting shape-constrained swarm be-
600 -400 -200 0 200 4000 (b)(a)
Real DataOur Method[Ren et al. 2016] P r o b a b ili t y Distance (pixel) (c)
Acceleration - x (pixel/s ) (d) Acceleration - y (pixel/s ) (e) Acceleration - z (pixel/s )Velocity (pixel/s) Figure 7: Comparisons of the distributions of the velocity (a), minimum distance (b), and acceleration(c-e for x , y and z direction) between our method, the real data, and [10]. (a) (b)(c) (d) Figure 8: Results of trajectory synthesis. (a)shows the captured intermittent trajec-tories of 37 agents. (b) shows thereal trajectories of the following 500frames where exist lots of intermittentsegments. (c) shows our predictionresult of (a) by extending the initialtrajectories to 500 frames. (d) showsthe blending trajectories of 100 virtualagents with (b).haviors controlled by users, which benefits com-puter animation. Moreover, our method can beused to predict the missing trajectories in thecaptured dataset and augment the dataset withdifferent number of insects while maintainingthe global visual similarity to the real data.
Limitations and Future Work . Our methodrelies on the quality of the reference dataset.However, as our reference dataset is extractedfrom imperfect trajectories captured using com-puter vision techniques, which contain noisesand erroneous data, our result may replicate thedefects in the input data. In the future, we areinterested in exploring a more general frame-work so that it can simulate any types of mo-tion (not restricted to insects) with limited cap- tured trajectories from real world or created byanimators. We are also planning to employ theLong Short-Term Memory (LSTM) networks toexplore the latent representations of trajectoriesto further enhance our results.
References [1] Michael Sinhuber and Nicholas T Ouellette.Phase coexistence in insect swarms.
Physicalreview letters , 119(17):178003, 2017.[2] Christian Blum and Xiaodong Li. Swarm intel-ligence in optimization. In
Swarm intelligence ,pages 43–85. Springer, 2008.[3] Jos´e Halloy, Gr´egory Sempo, Gilles Caprari,Colette Rivault, Masoud Asadpour, FabienTˆache, Imen Sa¨ıd, Virginie Durier, StephaneCanonge, Jean Marc Am´e, et al. Social inte-gration of robots into groups of cockroachesto control self-organized choices.
Science ,318(5853):1155–1158, 2007.[4] Xinjie Wang, Xiaogang Jin, Zhigang Deng,and Linling Zhou. Inherent noise-aware insectswarm simulation.
Computer Graphics Forum ,33(6):51–62, 2014.[5] Qiang Chen, Guoliang Luo, Yang Tong, Xi-aogang Jin, and Zhigang Deng. Shape-constrained flying insects animation.
ComputerAnimation and Virtual Worlds , 30(3-4):e1902,2019.[6] Douglas H. Kelley and Nicholas T. Ouel-lette. Emergent dynamics of laboratory insectswarms.
Scientific reports , 3:1073, 2013.[7] James G Puckett, Douglas H Kelley, andNicholas T Ouellette. Searching for effectiveforces in laboratory insect swarms.
Scientificreports , 4:4766, 2014.[8] Hai Shan Wu, Qi Zhao, Danping Zou, andYan Qiu Chen. Automated 3d trajectory mea-suring of large numbers of moving particles.
Optics express , 19(8):7646–7663, 2011.9] Weizi Li, David Wolinski, Julien Pettr´e, andMing C. Lin. Biologically-inspired visual sim-ulation of insect swarms.
Computer GraphicsForum , 34(2):425–434, 2015.[10] Xinjie Wang, Jiaping Ren, Xiaogang Jin, andDinesh Manocha. Bswarm: biologically-plausible dynamics model of insect swarms.In
Proceedings of the 14th ACM SIG-GRAPH/Eurographics Symposium on Com-puter Animation , pages 111–118, 2015.[11] Jiaping Ren, Xinjie Wang, Xiaogang Jin, andDinesh Manocha. Simulating flying insects us-ing dynamics and data-driven noise modelingto generate diverse collective behaviors.
PloSone , 11(5):e0155698, 2016.[12] Ryan Lukeman, Yue-Xian Li, and LeahEdelstein-Keshet. Inferring individualrules from collective behavior.
Proceed-ings of the National Academy of Sciences ,107(28):12576–12580, 2010.[13] Jiaping Ren, Wei Xiang, Yangxi Xiao, RuigangYang, Dinesh Manocha, and Xiaogang Jin.Heter-sim: Heterogeneous multi-agent systemssimulation by interactive data-driven optimiza-tion.
IEEE Transactions on Visualization andComputer Graphics , 2019.[14] Eunjung Ju, Myung Geol Choi, Minji Park, Je-hee Lee, Kang Hoon Lee, and Shigeo Taka-hashi. Morphable crowds.
ACM Transactionson Graphics (TOG) , 29(6):1–10, 2010.[15] Yi Li, Marc Christie, Orianne Siret, RichardKulpa, and Julien Pettr´e. Cloning crowdmotions. In
Proceedings of the ACM SIG-GRAPH/Eurographics Symposium on Com-puter Animation , pages 201–210, 2012.[16] Kang Hoon Lee, Myung Geol Choi, QyounHong, and Jehee Lee. Group behavior fromvideo: a data-driven approach to crowd sim-ulation. In
Proceedings of the 2007 ACMSIGGRAPH/Eurographics symposium on Com-puter animation , pages 109–118, 2007.[17] Mingbi Zhao, Stephen John Turner, and Wen-tong Cai. A data-driven crowd simulationmodel based on clustering and classification.In , pages 125–134, 2013.[18] Panayiotis Charalambous and YiorgosChrysanthou. The pag crowd: A graphbased approach for efficient data-driven crowdsimulation.
Computer Graphics Forum ,33(8):95–108, 2014. [19] Qianwen Chao, Jingjing Shen, and XiaogangJin. Video-based personalized traffic learning.
Graphical Models , 75(6):305–317, 2013.[20] Qianwen Chao, Zhigang Deng, Jiaping Ren,Qianqian Ye, and Xiaogang Jin. Realistic data-driven traffic flow animation using texture syn-thesis.
IEEE Transactions on Visualization andComputer Graphics , 24(2):1167–1178, 2018.[21] Huikun Bi, Tianlu Mao, Zhaoqi Wang, and Zhi-gang Deng. A deep learning-based frameworkfor intersectional traffic simulation and editing.
IEEE Transactions on Visualization and Com-puter Graphics , 2019.[22] Yixuan Wang, Rui Wang, Kai Cui, Weim-ing Tian, Dongli Wu, and Shuqing Ma. In-sect trajectory simulation method based onradar observation.
The Journal of Engineering ,2019(21):7365–7369, 2019.[23] Jeff Jones, Gregory D Murray, and Philip JMcCall. A minimal 3d model of mosquitoflight behavior around the human baited bednet. bioRxiv , 2020.[24] Alper Bozkurt, Edgar Lobaton, Mihail Sichi-tiu, Tyson Hedrick, Tahmid Latif, Alireza Di-rafzoon, Eric Whitmire, Alexander Verderber,Juan Marin, and Hong Xiong. Biobotic in-sect swarm based sensor networks for searchand rescue. In
Signal Processing, Sen-sor/Information Fusion, and Target Recogni-tion XXIII , volume 9091, pages 498–503, 2014.[25] Argel A Bandala, Elmer P Dadios, RyanRhay P Vicerra, and Laurence A Gan Lim.Swarming algorithm for unmanned aerial ve-hicle (uav) quadrotors–swarm behavior for ag-gregation, foraging, formation, and tracking.
Journal of Advanced Computational Intelli-gence and Intelligent Informatics , 18(5):745–751, 2014.[26] Yara Khaluf, Seppe Vanhee, and PieterSimoens. Local ant system for allocating robotswarms to time-constrained tasks.
Journal ofComputational Science , 31:33–44, 2019.[27] Shihui Guo, Meili Wang, Gabriel Notman, JianChang, Jianjun Zhang, and Minghong Liao.Simulating collective transport of virtual ants.
Computer Animation and Virtual Worlds , 28(3-4):e1779, 2017.[28] Wei Xiang, Jiaping Ren, Kuan Wang, ZhigangDeng, and Xiaogang Jin. Biologically inspiredant colony simulation.
Computer Animationand Virtual Worlds , 30(5):e1867, 2019.29] Robert Bridson, Jim Houriham, and MarcusNordenstam. Curl-noise for procedural fluidflow.