Example-based Real-time Clothing Synthesis for Virtual Agents
Nannan Wu, Qianwen Chao, Yanzhen Chen, Weiwei Xu, Chen Liu, Dinesh Manocha, Wenxin Sun, Yi Han, Xinran Yao, Xiaogang Jin
EEXAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 1
Example-based Real-time Clothing Synthesis forVirtual Agents
Nannan Wu, Qianwen Chao,
Member, IEEE,
Yanzhen Chen, Weiwei Xu,
Member, IEEE,
Chen Liu, Dinesh Manocha,
Fellow, IEEE,
Wenxin Sun, Yi Han, Xinran Yao, Xiaogang Jin*,
Member, IEEE
Abstract —We present a real-time cloth animation method for dressing virtual humans of various shapes and poses. Our approachformulates the clothing deformation as a high-dimensional function of body shape parameters and pose parameters. In order toaccelerate the computation, our formulation factorizes the clothing deformation into two independent components: the deformationintroduced by body pose variation (Clothing Pose Model) and the deformation from body shape variation (Clothing Shape Model).Furthermore, we sample and cluster the poses spanning the entire pose space and use those clusters to efficiently calculate theanchoring points. We also introduce a sensitivity-based distance measurement to both find nearby anchoring points and evaluate theircontributions to the final animation. Given a query shape and pose of the virtual agent, we synthesize the resulting clothingdeformation by blending the Taylor expansion results of nearby anchoring points. Compared to previous methods, our approach isgeneral and able to add the shape dimension to any clothing pose model. Furthermore, we can animate clothing represented with tensof thousands of vertices at 50+ FPS on a CPU. We also conduct a user evaluation and show that our method can improve a user’sperception of dressed virtual agents in an immersive virtual environment compared to a conventional linear blend skinning method.
Index Terms —clothing animation, virtual agents, social VR, virtual try on clothing shape models. (cid:70)
NTRODUCTION
There is considerable interest in generating human-like vir-tual agents for AR and VR applications. These agents areused to generate immersive social experiences for games,training, entertainment, virtual space visitations, or social-phobia treatments. In order to maintain the sense of pres-ence in virtual environments, it is essential that these virtualagents look realistic and interact in a plausible manner [1].There has been a great deal of work on improvingthe realism of virtual humans in terms of rendering, bodyshapes, facial expressions, hair, locomotion, etc. A key issueis related to dressing three-dimensional virtual humansusing garments and animating the cloth deformation cor-responding to draping and wrinkles. This is crucial becauseup to % of a human body can be covered by clothing. Asvirtual agents move, bend, or interact with the environment,the clothing folds, wrinkles, and stretches to conform to thevirtual agents’ poses. Moreover, the garments may stick tothemselves and/or other pieces of clothing. This problemis also important in the context of efficient virtual try-onsimulation. In these applications, the user can experiencethe fit of several garments on their 3D models or avatars. • N. Wu, Y. Chen, W. Xu, W. Sun, Y. Han, X. Yaoand X. Jin are with State Key Lab of CAD&CG, Zhe-jiang University, Hangzhou 310058, P. R. China. E-mail: { wunannanzju, choconutscyz } @gmail.com, [email protected],[email protected], [email protected], [email protected], [email protected]. • Q. Chao is with the Department of Computer Science, Xidian University,Xi’an 710038, P. R. China. E-mail: [email protected]. • C. Liu is with Zhejiang Linctex Digital Technology Co. Ltd., China. E-mail : [email protected]. • Dinesh Manocha is at University of Maryland, USA. E-mail :[email protected]. • X. Jin is the corresponding author.
The goal is to experience real-life clothes on virtual modelsand evaluate the garment behavior when a user movesaround or stands in different poses. Recently, many AR-based interfaces have been proposed for virtual try-on. Allthese applications must be able to simulate or animate thecloth and garments at interactive rates.Physics-based simulation methods directly model thenon-linear behavior of clothing and contacts to gener-ate realistic cloth simulations [2], [3], [4]. However, high-resolution clothing meshes and computationally expensivenonlinear solvers are frequently used, making it difficult tosimulate clothing at interactive rates (i.e. 30fps or more).Moreover, dressing bodies of different shapes requires aseparate simulation or setup for each body shape. Thismakes it difficult to use physics-based methods for real-timeVR or virtual try-on applications.Most of the data-driven clothing animation methodssynthesize the cloth deformation from pre-computed cloth-ing deformation samples for different body poses [5], [6].In order to achieve real-time performance, these methodscompromise the animation quality by simplifying the rela-tionship between the deformed clothing and the underlyingbody poses by assuming linearity or locality. To considerthe effect of body shape on cloth deformation, Peng etal. [7] model the cloth deformation as a function of bodymotion and shape and resize the garment to fit the body.Recently, [8] presented a learning-based model to predictthe deformation of a fixed-size garment under various posesand shapes. However, it uses an exhaustive database con-struction procedure that requires substantial computationalresources offline.
Main Results:
In this paper, we present a novel real-timealgorithm for cloth synthesis for VR and virtual try-on appli-cations. We use a data-driven approach and present a new a r X i v : . [ c s . G R ] J a n XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 2
Fig. 1. Our method can generate real-time clothing animation results with detailed wrinkles for various body poses and shapes with different clothingtypes on a commodity CPU. Our method can also be applied in VR scenarios. method to simulate plausible deformation effects at an inter-active rate. We factorize the clothing deformation into twoindependent components: the pose-dependent deformationand the shape-dependent deformation. We refer to the pose-dependent deformation as the clothing pose model , which isused to predict clothing deformations under various poses;we refer to the shape-dependent deformation as the clothingshape model , which can predict clothing deformations undervarious shapes. We also present a clothing synthesis schemethat combines these two components through Taylor ex-pansion. Our method adopts a pose- and shape-dependentskinning scheme for clothing synthesis to meet the needsof real-time virtual try-on on a CPU for synthetic bodies ofvarious shapes and poses (as shown in Figure 1). The threenovel components of our work are:
Clothing shape model:
We present a novel pose-independent clothing shape model. Given a set of clothinginstances simulated on bodies with a specific pose andvarious shapes, we first reduce the dimensionality withPrincipal Component Analysis (PCA). We use the first k principal component coefficients spanning the clothing de-formation space (we experimentally set k = 5 , as describedin Section 3.2). Next, we map the body shape parameters tothe coefficients of the PCA basis of the clothing deformationspace. Therefore, given a set of body shape parameters, wecan predict the corresponding clothing deformation result. Taylor expansion to approximate clothing deforma-tions:
We present a real-time clothing synthesis methodby considering both the pose-dependent deformation andthe shape-dependent deformation using Taylor expansions.We represent clothing deformation as a function of bodyshape parameters β and body pose parameters θ , f ( β, θ ) .The clothing deformation in the neighborhood of the givenanchoring point ( β , θ ) is approximated with a Taylor ex-pansion. The partial derivatives of the clothing deformationare calculated numerically using the clothing pose modeland the clothing shape model to predict f ( β , θ + ∆ θ ) and f ( β + ∆ β, θ ) , respectively. Given the new parameters ( β = β + ∆ β, θ = θ + ∆ θ ) , we synthesize the clothingdeformation by blending the Taylor expansion results fromthe nearby anchoring points. Accordingly, our approachcan add the shape dimension to any clothing pose model;this is in contrast to [6], which can only deal with pose-dependent deformations. Moreover, we use a sensitivity-based measurement to find nearby anchoring points and tocalculate the blending weights. Sampling scheme:
We present a pose space analysis method to generate a compact example database in parallel,which can significantly reduce the offline computationalcost. In order to generate plausible results, we cluster thepose space into a small number of clusters. We use thecluster centers to calculate the anchoring points and therebybuild a compact database. In our implementation, we onlysample data points along the θ axis, which enables us tonot have to train the input clothing pose model at var-ious shapes. This sampling scheme can also be used togenerate the database of the sensitivity-optimized rigging(SOR) method [6], and thereby significantly improving theirresults.Our approach is general and provides a universalscheme to add the shape dimension to any clothing posemodel with a compact database. We validate our methodwith SOR [6] and a sequence of simulated clothing instances.Our results show that, with a small number of anchorpoints (approximately 150), our method can generate real-istic clothing deformations with an extra time consumptionof 4ms per frame. On a commodity CPU, we obtain FPSfor a long-sleeved shirt with K vertices. We also performa preliminary user study in an immersive VR setting andhighlight the perceptual benefits of our approach comparedto prior interactive methods based on linear blend skinning.The rest of the paper is organized as follow: In Sec-tion 2, we detail relevant related work in clothing anima-tion. In Section 3, we describe Taylor expansion, clothingshape model, clothing pose model, runtime synthesis, anddatabase construction of our method. Section 4 details theresults of our method. Discussion and conclusion will begiven in Section 5 and Section 6.
ELATED W ORK
Physics-based clothing simulation.
In the past twodecades, following the seminal work by Baraff et al. [2],physics-based clothing simulation has become a hot topicin the computer graphics community. The following worksfocus on integration methods [9], [10], [11], strain limiting[12], [13], [14], [15], [16], and various clothing simulationmodels [4], [17], [18], [19], [20], [21], [22], [23], [24]. Whilethese methods can produce highly realistic clothing defor-mations, they are typically quite time-consuming, especiallywith high-resolution cloth meshes. Many acceleration meth-ods have been developed, including the projective dynam-ics method [25], [26], where integration is interpreted asan optimization problem; the Chebyshev semi-iterative ap-proach, which accelerates the projective dynamics method
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 3 [27], [28]; and position-based dynamics [29], where internalforces are replaced by position-based constraints to achieveboth efficiency and stability. Recently, parallel GPU-basedmethods have been developed for implicit integration andcontact handling [30], [31], [32], [33], [34], which performimplicit integration and accurate collision handling (in-cluding self-collisions). The underlying collision queries areperformed using CCD tests that involve use of algebraicsolvers and reliable computations [35], [36], [37], [38] for theelementary tests. The overall simulation algorithms exploitthe parallelism on one or more GPUs and can accuratelysimulate at − fps on high-end desktop GPUs. Theactual running time can vary based on mesh resolutionas well as the number of colliding configurations betweentriangles. Most VR applications require 30 fps (or higherperformance) and current physics-based methods cannotoffer such performance. Furthermore, in many applications,we need to perform cloth simulation on a mobile device(e.g., a smartphone) and we need methods that have alower computational overhead. The time-consuming natureof collision processing has also prompted many researchersto focus on developing accelerated data structures suchas bounding-volume hierarchies [39], distance fields [40],shape approximation with simple primitives [41], [42], andother spatial partitioning methods [43]. Combined withposition-based dynamics [29], these methods can achievereal-time clothing animation with plausible dynamics for amedium resolution mesh. However, the resulting clothinganimation lacks detailed wrinkles. Data-driven clothing animation.
Data-driven clothinganimation techniques have received considerable attentionin recent years for real-time applications that require highfidelity. De Aguitar et al. [44] reduced the dimension of clothspace and body space with PCA and then learned a con-ditional dynamical model of cloth in the low-dimensionallinear cloth space. Their method is fast and stable butcannot generate clothing deformations with highly detailedwrinkles. Wang et al. [45] regarded wrinkles as a functionof local joint angles and augmented coarse simulations withdetailed wrinkles from a pre-computed wrinkle database.Kim et al. [5] exhaustively searched a motion graph to gen-erate realistic secondary motion of clothing deformations.However, both the memory space and the computationalresources for the database are prohibitively high. Hahn etal. [46] simulated clothes in low-dimensional linear sub-spaces learned from performing PCA on clothing instancesin different poses. While they can reproduce detailed fold-ing patterns with only a few bases, the method is stilltoo costly to be used in real-time clothing animation. Tobalance speed and quality, Xu et al. [6] introduced real-time example-based clothing synthesis using sensitivity-optimized rigging to achieve physically plausible clothingdeformation. Given a set of pre-computed example clothingdeformations sampled at different poses, the method firstrigs the example clothing shapes to their poses with theunderlying body skeleton at each example pose in theoffline stage. At runtime, the method synthesizes a clothingdeformation of the input pose by blending skinned clothingdeformations computed from nearby examples. Jin et al. [47]represented clothing shapes as offsets from the underlyingbody surface, and performed convolutional neural networks to learn pose-dependent deformations in image space. Othermethods directly used CNNs on triangle meshes, therebypredicting high resolution clothing deformations based onlow resolution inputs [48].In addition to the significant advances achieved in pose-dependent clothing animation, variability in the humanbody shape has also been considered in clothing animationfor virtual try-on applications. Inspired by SCAPE [49],Peng et al. [7] introduced a model of clothing animationcalled DRAPE (DRessing Any PErson), which separatedclothing deformations due to body shape from those dueto pose variation. DRAPE can fit avatars of various posesand shapes with customized garments and change the cloth-ing model according to the body shape. Recently, manytechniques have been proposed for learning-based clothingsimulation. Wang et al. [50] learned a shared shape spacein which users are able to indicate desired fold patternssimply by sketching, and the system generates correspond-ing draped garment and body shape parameters. However,their approach cannot generate clothing deformations cor-responding to different poses. Inspired by SMPL [51], San-testeban et al. [8] introduced a learning model of cloth drapeand wrinkles. The key innovation of their method is thatthey added corrective displacements caused by body shapeand pose variations to the template cloth mesh and thendeformed the template mesh using a skinning function. Inmany ways, our approach is complimentary to this method.A limitation of their method is that the training dataset isprohibitively large, which is similar to other learning-basedmethods [52], [53]. By treating clothing as an extra offsetlayer from the body, Ma et al. [54] trained a conditionalMesh-VAE-GAN to learn the clothing deformation from theSMPL body model. This work has the limitation in thatthe level of geometric details they can achieve is upper-bounded by the mesh resolution of SMPL. Yang et al. [55]modeled the clothing layer as an offset from the body andperformed PCA to reduce the self-redundancies. In contrast,we perform PCA directly on the coordinates of clothingdeformations to compute our clothing shape model. Manyother works focused on cloth reconstruction from a singleimage or scan data [56], [57], [58], as well as cloth materialrecovery from video [59].
Sensitivity analysis.
Sensitivity analysis was originallyused to determine the impact of the input data on the outputresults in linear programming problems [60], and it has beenwidely used to solve optimization problems in the fieldof graphics, including shell design [61], composite siliconerubber design [62], and robotics design [63], [64]. Sensitivityanalysis was first introduced to the clothing simulationcommunity by Umetani et al. [65] to build up a mappingbetween variations of 2D patterns and 3D clothing deforma-tions to achieve interactive clothing editing. After that, Xuet al. [6] proposed a technique called sensitivity optimizedrigging (SOR) to perform real-time clothing animation. Theyuse sensitivity to both rig the clothing instances in exampleposes and find the example poses nearest to the input pose.In this paper, we use sensitivity to find the anchoring pointsnearest to the input pose and shape.
Taylor expansion.
A complex function can be approxi-mated by its first order Taylor expansion in a neighborhoodof the keypoint locations. Such a method has been applied
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 4 in the field of simulation and animation to solve vari-ous approximation problems. To generate real-time facialanimations, Barrielle et al. [66] applied first-order Taylorapproximation to the computations of the Singular ValueDecomposition, thereby significantly accelerating the sim-ulation of volumetric forces. Shen et al. [67] adopted afinite Taylor series approximation of the potential energyto avoid numerical singularity and instability during thesimulation of inextensible ribbon. In the cloth simulationcommunity, Taylor expansion is generally used to make thefirst order approximation of the internal force. To solve thenonlinear equation involved in the implicit backward Eulermethod, Baraff et al. [2] applied a Taylor series expansionto the force acting on the cloth and made the first orderapproximation, which leads to a linear system. As a result,their cloth simulation system can handle large time steps ina stable manner. Chen et al. [68] proposed a fully geometricapproach to simulate inextensible cloth that is subjected toa conservative force. They use Taylor expansion to linearizethe constraint that preserves isometric deformations. In thispaper, we use Taylor expansion to factor the clothing de-formation (which is a complex high-dimensional nonlinearfunction without an analytical expression) into two parts, asintroduced by body shape and pose variations.
ETHOD
Fig. 2 illustrates the pipeline of our approach. Our systemconsists of two stages: the offline training stage and the run-time clothing synthesis stage. To model the nonlinear defor-mation of clothing mesh under different poses and shapes,we precompute a pose training set of clothing meshes with asingle template body in multiple poses and develop a shapetraining set fit to different body shapes in each pose. Weuse the pose training data to train a clothing pose model topredict the pose-dependent clothing deformation, given atemplate body (e.g. SOR); for each pose in the shape trainingdata, we train a clothing shape model to predict the shape-dependent clothing deformation with a fixed pose. At run-time, the method separates deformations induced by poseand shape (i.e. a clothing pose model and a clothing shapemodel) through Taylor expansion. Given a query body poseand shape, we first find nearby clothing pose models andclothing shape models and then blend the Taylor expansionresult of each nearby anchoring point to synthesize anintermediate mesh. We then resolve penetrations and adddamping to get the final clothing mesh for the input body.Specifically, we represent the input body with shapeparameters β and pose parameters θ . The clothing defor-mation Y can be formulated as a high-dimensional function Y = f ( β, θ ) and approximated by the clothing on itsnearby example body with shape parameters β and poseparameters θ (we refer to ( β , θ ) as the anchoring point)using first-order Taylor expansion: f ( β, θ ) = f ( β , θ ) + ∆ βf β ( β , θ ) + ∆ θf θ ( β , θ ) , (1)where ∆ β and ∆ θ are the shape and pose difference, respec-tively, between input body and its nearby example body. We use forward difference to calculate the partial deriva-tives f β ( β , θ ) and f θ ( β , θ ) in Eq. 1 as follows: f ( β, θ ) = f ( β , θ ) + ∆ β · f ( β, θ ) − f ( β , θ )∆ β +∆ θ · f ( β , θ ) − f ( β , θ )∆ θ = f ( β , θ ) + ( f ( β, θ ) − f ( β , θ )) , (2)where f ( β , θ ) represents the clothing under a new posewith the body shape unchanged, which can be computedvia a clothing pose model . f ( β, θ ) denotes the clothing fora new body shape with the pose fixed, which can becalculated through a clothing shape model . Correspondingly, ( f ( β, θ ) − f ( β , θ )) represents the shape-dependent clothdeformation, which is the cloth mesh deformation duringthe body shape change from β to β under the anchoringpose θ .In this way, the cloth deformations induced by bodyshape and pose can be separately computed and then com-bined. However, such an approximation cannot accuratelyobtain the clothing deformation under ( β, θ ) since the termof shape-dependent cloth deformation in Eq. 2 should bemeasured under the new pose θ as ( f ( β, θ ) − f ( β , θ )) ,rather than under the anchoring pose θ . To improve the ap-proximation accuracy, we apply the Linear Blend Skinning(LBS) method to predict the cloth deformation for the newpose θ from the cloth mesh under its nearby sample pose θ . Therefore, the shape-and-pose dependent cloth meshdeformation f ( β, θ ) can be formulated as the AugmentedTaylor Expansion: f ( β, θ ) = f ( β , θ ) + (cid:0) LBS θθ ( f ( β, θ )) − LBS θθ ( f ( β , θ )) (cid:1) , (3)where LBS θθ stands for the linear blend skinning from pose θ to θ .Since Taylor expansion can only predict the functionvalue at points that are not far away from the expansionpoint, we apply Taylor expansion at multiple points, andthen blend these approximation results.Note that both training and running a clothing posemodel are time-consuming in practice. Taking SOR as anexample, it takes about 30 hours to construct the necessarydatabase and more than 10ms to predict the clothing defor-mation under a new pose, which is too time-consuming inour scenario since we need to run the clothing pose modelonce at each expansion point. To address this problem,we choose to apply Taylor expansion along the θ axis, i.e.expansion points have the same β value, such as ( β , θ ) , ( β , θ ) , ( β , θ ) · · · , etc. In this way, for each expansionpoint ( β , θ s ) , the only thing we need to do is to train aclothing shape model at θ s , and we use only one clothingpose model f ( β , θ ) trained at β , referred to as the originalshape. In our implementation, we set the shape parametersof the SMPL body model [51] to be all zeros to get a mediumstature body.At run-time, given a query pose and shape, our methodfirst finds the nearby clothing shape models through asensitivity-based distance measure. For each nearby an-choring point, we compute the approximation result forthe query pose and shape according to Eq. 3. After that,we blend the approximation results with blending weights XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 5
Clothing Pose Model Intermediate ResultPoseDimension Final ResultShapeDimension
Clothing Shape ModelsClothing Shape ModelsQueryQuery
Taylor ExpansionTaylor Expansion
LBS Synthesis for Original Shape LBS Synthesis for Original Shape LBS Synthesis for New Shape LBS Synthesis for New Shape Fig. 2. Overview of our clothing synthesis workflow. Given a query pose and shape, we first calculate the LBS synthesis results for the new shapeand the original shape, and we refer to these as the shape dimension. The “pose dimension” is obtained from an external clothing pose model withthe query pose and the original shape. Then we blend the “shape dimension” and the “pose dimension” using Taylor expansion to synthesize anintermediate result. Finally, we apply some refinement techniques such as penetration resolution to generate the final result. computed from the distance measure. In summary, ourmodel predicts the clothing under any given body shapeparameter β and pose parameter θ by f ( β, θ ) = f ( β , θ ) + N s (cid:88) s =1 w s LBS θθ s ( f ( β, θ s )) − N s (cid:88) s =1 w s LBS θθ s ( f ( β , θ s )) , (4)where N s is the number of clothing shape models and w s is the blending weight of the s -th data point. Figure2 illustrates the computation of this equation where thethree items in the right-hand side of Eq. 4 are referred toas the pose dimension , the LBS synthesis for new shape , and the
LBS synthesis for original shape . Details will be given in thefollowing sections.
The clothing shape model captures clothing deformationinduced only by body shape. The model is learned from theclothing shape examples that are simulated under variousbody shapes with a fixed pose (each column in Figure 7).Thus, the following procedure will be run independently forall anchoring poses, each to generate a clothing shape modelfor a specific pose. We use the SMPL parametric humanmodel to represent the variations in human body shape. Inthe offline database construction, we select 17 training bodyshapes in the same pose. For each of the first 4 principalcomponents of the body shape parameters β in the SMPLmodel, we generate 4 body shapes ( β k = − , − , , ),keeping the remaining parameters in β as 0. To these 16body shapes, we add the nominal shape with β = 0 .For each generated body shape, we perform a one-second simulation to completely drape the clothing on the avatar. All simulations are produced using a clothing modelthat combines the StVK membrane model [20] and the iso-metric bending model [69] to simulate a garment example. Itis worth noting that in the clothing simulation, the patternsand characteristics of the garment mesh do not change forall body shapes.For the k -th generated clothing instances, we concatenatethe clothing coordinates of all vertices into a single columnvector (cid:126)c k . Then all 17 clothing samples are collected into amatrix S = [ (cid:126)c , ..., (cid:126)c k , ..., (cid:126)c ] . Principal component analysis(PCA) is used to find a low-dimensional subspace so that (cid:126)c k can be approximated by the following equation: (cid:126)c k = U (cid:126)φ k + (cid:126)u, (5)where (cid:126)u is the mean coordinates of clothing meshes and (cid:126)φ k is the clothing shape coefficients to represent the clothingshape (cid:126)c k . The matrix U represents the first few principalcomponents of the shape deformation space. As shownin Figure 10, the variance converges around 5 principalcomponents, which is 98.6% of the total variance. Moreover,we find that the remaining principal components have littleeffect on the final result. Therefore, we use the first 5 prin-cipal components in our clothing shape model to balanceefficiency and accuracy. Figure 3 illustrates the mean andfirst three principal components for a men’s long-sleevedshirt.Given 17 body and clothing training pairs, we learna linear mapping, W , between body shape parametersand clothing shape parameters using L2-regularized leastsquares with the weight of the regularized term being 0.2.For an input body shape β , the corresponding clothingshape coefficients (cid:126)φ can be predicted by: (cid:126)φ = W · (cid:16) (cid:126)β, (cid:126)β , (cid:17) T . (6) XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 6 (a) Mean (b) PC1 (c) PC2 (d) PC3
Fig. 3. Clothing shape model. Deviations from the mean shape: (a) the average coordinates of the training data, referred to as the mean shape;(b-d) mean shape deformed along the first three principal component directions ( ± Thereafter, the clothing shape deformation under thebody shape β can be predicted through (cid:126)φ using Eq. 5. Inpractice, we find that there might be some minor interpene-trations between the predicted clothing mesh and the bodymesh. We resolve this by pushing the implicated clothingvertex in the direction of the normal of its closest bodyvertex until the interpenetration is settled. This should notbe confused with the penetration handling step in Sec. 3.5.2. The clothing pose model captures clothing deformationintroduced only by pose changes. Many excellent clothingpose models have been presented over the last decade [5],[6], [46]. Our animation scheme does not limit the type ofclothing pose model; in extreme cases, we apply a simulatedsequence as the clothing pose model (see Sec. 4.5). For real-time virtual try-on applications, we adopt the sensitivity-optimized rigging method proposed in [6] since it has agood balance of accuracy and speed.The system consists of two phases: the offline riggingstage and the run-time clothing synthesis stage. Given a setof pre-computed example clothing deformations sampled atdifferent poses, we first rig the example clothing shapes totheir poses with the underlying body skeleton bones at eachexample pose in the offline stage. At run-time, a clothingdeformation of the input pose is synthesized by blendingskinned clothing deformations computed from its nearbyexamples.
Given an input body with shape parameters β and poseparameters θ , our method first finds nearby example posesthrough a sensitivity-based distance measure and then ap-proximates the clothing deformation using Taylor expan-sion, as shown in Eq. 4. Specifically, the first term f ( β , θ ) computes the clothing under the input pose with the origi-nal body shape, which can be predicted through the clothingpose model described in Sec. 3.3.The computation of the second term (cid:80) N s s =1 w s LBS θθ s ( f ( β, θ s )) consists of three steps: predictingnew clothing instances f ( β, θ s ) for each clothing shapemodel, applying LBS to f ( β, θ s ), and blending the LBSresults.For each clothing shape model, we first calculate theclothing shape coefficients through Eq. 6 using the inputshape β . Then we compute the new clothing instances f ( β, θ s ) by Eq. 5.To apply LBS to new clothing instances f ( β, θ s ) , we firstfind the closest body vertex for each clothing vertex and setthe bone weights, w sb , of a clothing vertex to that of its closestbody vertex. We refer to this step as the binding information updating step . Then we deform each nearby clothing mesh f ( β, θ s ) towards the query pose as ¯ y s = (cid:80) N b b =1 w sb (cid:0) R θ s ,θb y s + T θ s ,θb (cid:1) , where R θ s ,θb and T θ s ,θb are the relative rotation andtranslation of bone b from example pose θ s to input pose θ , respectively, and w sb is the bone weight defined on theclothing vertex y s of f ( β, θ s ) . We denote this equation as LBS θθ s ( f ( β, θ s )) , as shown in Eq. 4.We use a sensitivity-based distance measurement bothto find nearby clothing shape models and to compute theblending weights w s . To reduce the required clothing shapemodels in the database, we divide the clothing mesh intoseveral regions, as in [6]. To this end, we manually partitionthe bones into N g = 7 regions (shown on the top left ofFigure 4). A region weight w g,y for a clothing vertex y iscomputed by summing the bone weights w b for the bonesof the current region. w b is computed in the T-pose s = 0 for β through the binding information updating step . In this way,our method synthesizes the result for each region separately.For each region g , we compute the sensitivity-baseddistance D sg ( θ ) between the input pose θ and the pose ofthe s -th data point as the weighted sum of differences ofjoint angles: D sg ( θ ) = (cid:88) y ∈ Y w g,y N L (cid:88) m =1 || ¯ s y,m · Θ m ( θ, θ s ) || = N L (cid:88) m =1 Q g,m || Θ m ( θ, θ s ) || , (7)where superscript s indicates values for the s -th cloth-ing shape model, N L is the number of joints, and Θ m ( θ, θ s ) calculates the m -th joint angle difference. ¯ s y,m = (cid:80) k =1 || s ky,m || is the average sensitivity of 17 trainingshapes, where s ky,m indicates the three-dimensional coordi-nate differences of a clothing vertex y under a small jointrotation of m -th joint angle, calculated under T-pose and the k -th training shape. Q g,m = (cid:80) y ∈ Y w g,y || ¯ s y,m || reflects theinfluence of the m -th joint angle on region g . ¯ s y,m indicatesthe influence of the m -th joint angle on clothing vertex y ,which is computed in the database construction stage. Eachtime we change the body shape, we update Q g,m for eachregion once according to the new region weights of clothingvertices. At run-time, we can compute the sensitivity-baseddistance D sg ( θ ) efficiently since we only have to calculate thejoint angle differences Θ m ( θ, θ s ) .Given the distance D sg ( θ ) , the weight for the region iscalculated as W sg ( θ ) = 1 / ( D sg ( θ ) + (cid:15) ) k , where (cid:15) is a smallnumber in case of zero division and k regulates the influenceof closer examples. A small k tends to smooth the animationand lose fine wrinkles, while a large k tends to preserve finedetails but results in discontinuity. In our implementation,we set k = 3 . In practice, we use the first five nearby XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 7
Body RegionsBody Regions
LBS Synthesis
Region 1Region N
Clothing Shape Models
QueryQuery
Near Clothing Shape ModelNear Clothing Shape Model Change ShapeChange Shape RiggingRigging BlendBlendChange ShapeChange Shape RiggingRigging
Fig. 4. The LBS synthesizing process. Given a query pose and shape, we first choose nearby clothing shape models. For each chosen clothingshape model, we change its shape to the query shape (the original shape), and deform the clothing mesh to the query pose using a linear blendskinning. Finally, we blend these deformations to get what we call the
LBS synthesis for the query shape (the original shape). optimal damping ratio
Fig. 5. Relationship between the damping ratio and the norm of theresidual. We choose the maximum damping ratio that keeps the normof the residual low. clothing shape models to synthesize the final result, i.e. weset W sg ( θ ) = 0 except for the top five largest ones.Finally, the blending weight for each clothing vertex inexample s is calculated as: w s = N G (cid:88) g =1 (cid:0) w g W sg ( θ ) / N s (cid:88) s =1 W sg ( θ ) (cid:1) . (8)The computation pipeline of the third term (cid:80) N s s =1 w s LBS θθ s ( f ( β , θ s )) is basically the same as thesecond term except for the body shape. In other words, wefirst compute clothing instances under the original shape β for each clothing shape model, denoted as f ( β , θ s ) .Then we blend the LBS results of f ( β , θ s ) using the sameweights as in the second term. Note that f ( β , θ s ) andtheir binding information are computed only once for anapplication. In practice, we find our clothing synthesis result may ex-perience sudden changes for sudden input pose changes.Similar to [6], we prevent such a problem by blending thedistance at the current time step D sg with that of the previoustime step D (cid:48) sg : D sg = ηD (cid:48) sg + (1 − η ) D sg , (9) Fig. 6. The result before (left) and after (right) our penetration handlingprocess. where η is the damping ratio, ranging from 0 to 1. Todetermine the damping ratio, we first calculate the relation-ship between the damping ratio and the norm of residual,computed under the original body shape. Then we choosethe maximum value of the damping ratio while the normof the residual is still low (see Figure 5). The rationale isthat the higher the value of the damping ratio, the strongerthe decaying effect and the less flickering. We believe thissimple technique will introduce some hysteresis while main-taining a natural result. After we get the synthesized result through Eq. 4, theremight be interpenetrations between the clothing mesh andthe body mesh (as shown in Figure 6). Like SOR [6], weuse a re-projection technique with three steps to resolve thisproblem.First, every time we change the shape of the avatar,for each clothing shape model we re-calculate the initialdistance of a clothing vertex from its closest body vertex.We refer to this as the initial clearance. Second, in the run-time stage, if the clearance between a clothing vertex and itsclosest body vertex is less than the initial clearance, for eachnearby clothing shape model, we re-project the clothingvertex towards the direction of the normal of its closest body
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 8
Fig. 7. Part of our training data. Each column is the training data for aclothing shape model. vertex as: (cid:40) ˆ y s = y + d · (cid:126)nd s = max (0 , d s − h s ) , (10)where y is a clothing vertex synthesized by Eq. 4, (cid:126)n is thenormal of the closest body vertex of y , and h s is currentclearance. We set d s = min { h s , (cid:15) p } , where h s is the initialclearance, and (cid:15) p is to mimic the penetration depth marginin cloth simulation (in our implementation, we empiricallyset (cid:15) p = 5 mm ). Finally, we blend the re-projection results ofall nearby examples as ¯ y = (cid:80) N s s =1 w s ˆ y s . To generate our example database, we first select 32 motionsequences from the CMU motion capture library [70] andsample a pose for every four frames. In total, we obtain17k different poses representing the whole pose space. Thenwe use weighted K-means to classify these poses into acertain number of clusters, which will be used to generateour example database of the clothing shape model. In ourimplementation, it takes about one hour to classify theseposes into a typical value of 150 clusters.The weight of a joint should reflect its importance inclothing animation. For instance, while the rotation of theknee joint has little, if any, influence on the animation resultof a T-shirt, it plays a crucial role in the deformation of pants.To this end, we use the sum of the norm of the sensitivity ofa joint, s L , as its weight in the clustering process, calculatedas s L = (cid:88) y ∈ Y (cid:88) m =1 || ¯ s y,m || , (11)where m represents the degree of freedom of joint L and ¯ s y,m is the same as in Eq. 7 (i.e. s L is computed over all thetraining shapes). The poses in each row of Figure 7 are the clustering resultof a male wearing a long-sleeved shirt. The cluster centersare essentially the anchoring points for Taylor expansionin Eq. 4. For each anchoring point, we generate a clothingshape model, which we elaborate on in Sec. 3.2. XPERIMENTS
We implemented our approach in C++ and reported its per-formance on an off-the-shelf computer with an Intel Core i7-7700K CPU 4.20GHz and 16GB memory. The same materialparameters are applied for all types of clothing: bendingstiffness is − N/m , stretching stiffness is N/m , areadensity is . kg/m , and dynamics and static coefficient offriction is . . Clothing Long-sleeved shirt T-shirt Pantsnumber of vertices 12.2k 12.0k 11.7knumber of triangles 23.4k 23.8k 22.4knumber of data points 150 / 150 150 / 150 150 / 170database size (MB) 208.5 / 56.7 206.3 / 56.0 199.5 / 61.7construction time (hrs) 15 / 7 10 / 6 14 / 8runtime frame rate (FPS) 56 55 71
TABLE 1Statistics for three different clothing databases. The number to the leftof “/” is the data for our method while the number to the right of “/”indicates the data for the external clothing pose model we use, i.e.SOR. Note that we have discarded the translation item in SOR andreplaced the sampling method with ours. These measures contribute tothe high speed of the database generation process of SOR.
We create databases for three clothing models: a long-sleeved shirt and pants for a male body and a T-shirt for afemale body (see Figures 1 and 12). For each clothing model,we first calculate its joint weights using Eq. 11; then we useweighted K-means to obtain anchoring points; finally, wegenerate a clothing shape model for each anchoring point.Based on the independence of the anchoring points, ourdata points can be generated in parallel. Take T-shirt as anexample, we generate ×
17 + 150 (the data used for ourmethod + the data for SOR) clothing instances to generateour database. On the other hand, [8] simulated × clothing instances. That is, the size of their database is morethan 40 times larger than ours. Table 1 shows the details ofdatabases constructed for these clothing models. In this section, we validate the local linear separability ofpose-dependent deformation and shape-dependent defor-mation. For an anchoring point, we uniformly sample m neighboring data points (we set m = 100 in our experi-ment), all of which are a distance r from the anchoring point.Specifically, we randomly generate N L + 4 (number ofjoint angles plus 4 shape parameters) numbers that are thenconcatenated into a vector (cid:126)e . We normalize (cid:126)e in terms of 2-norm and then multiply it by r . The first N L components ofthe final (cid:126)e are the displacement of joint angles from the an-choring point while the last components are the displace-ment of the shape parameters. Then, for each neighboringpoint, we calculate the average Euclidean vertex distance XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 9
Anchoring point: anchoring pose & the original shape simulation result on new pose and shape practical result error: 0.78cm theoretical result error: 0.27cm
Fig. 8. Validation of the local linear separability of pose-dependent deformation and shape-dependent deformation. Practically, we use Eq. 3 tocompute the synthesized clothing and, theoretically, we replace each term on the right-hand side of Eq. 3 with the corresponding simulation result.As shown in the graph, both approximation errors converge to zero when the distance to the anchoring point approaches zero. The images on theright are the results for a randomly generated pose and shape with r = 0 . . Both results (the middle two) can realistically recover the folds andwrinkles (the right-most figure is the ground truth) while the theoretical result has a smaller error. (a) (b) (c) (d) Fig. 9. Qualitative comparison between the original SOR and ourmethod: (a) original SOR result; (b) error between original SOR resultand the simulation result; (c) our result; (d) error between our result andthe simulation result. The errors are marked in green. We can see thatour result is closer to the ground truth and looks more natural, especiallyin the sleeve area. between the simulation result and the synthesized result asthe approximation error. The average approximation errorof all neighboring points is regarded as the approximationerror of the anchoring point at distance r .We run this procedure both practically and theoretically.Practically, we apply Eq. 3 to compute the synthesizedclothing; theoretically, we replace each term on the right-hand side of Eq. 3 (i.e. f ( β , θ ); f ( β, θ ) ; and f ( β , θ ) ) withthe corresponding simulation result. Hence, in the practicalexperiment, the error comes from three sources: the cloth-ing pose model, the clothing shape model, and the linearapproximation. In the theoretical experiment, the error iscaused only by the linear approximation. As shown in Fig-ure 8, for a randomly chosen anchoring point, both resultscan recover the folds and wrinkles with good fidelity; theerror of the theoretical experiment is smaller than that of thepractical experiment given the same distance r . Meanwhile,when r is small, both approximation errors are relativelysmall. The errors converge to zero when r approaches zero.This demonstrates the approximate separability of pose-dependent deformation and shape-dependent deformation. Each step of MCMC sampling in SOR aims to find the posethat maximizes the norm of the residuals of the synthesized
Fig. 10. Convergence of variance during PCA of the training data (leftmost column in Figure 7). Our method applies PCA to coordinates ofclothing vertices while DRAPE applies PCA to deformation gradients ofclothing triangles. We can see that our method converges faster andcaptures more variance with a given number of principal components. clothing, i.e. they try to minimize the maximum error.However, a pose with the biggest error is sometime nota natural human pose. On the other hand, our samplingmethod can generate poses that are representative of reallife. Therefore, we believe our sampling method is betterthan the one used in SOR. To demonstrate this, we replacethe sampling scheme in SOR with our sampling methodand regenerate the database for SOR. The weights of jointsin the clustering process are computed with the sensitivityunder the original body shape from SOR. Qualitatively, ourmethod can generate more natural clothing deformation,as shown in Figure 9. Quantitatively, as shown in Table 2,our method reduces the norm of the residual force by over22%, as computed over the 32 motion sequences describedin Sec. 4.1. Furthermore, the sampling points generated byour sampling method are independent of each other, whichenables the database to be generated in parallel.
We compare our clothing shape model to that of DRAPE[7]. Instead of using the SCAPE body [49], we map thefirst four shape parameters of SMPL [51] to their clothingshape parameters and use our training data to train this
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 10 n dp TABLE 2Quantitative comparison between the original SOR and our method. n dp stands for the number of data points and “Imp” is an abbreviation of“improvement”. We can see that our sample scheme can reduce the norm of the residual by more than 22%, meaning that the synthesizedclothing deformations of our method are closer to the equilibrium states. mapping function. We take a men’s long-sleeved shirt as therepresentative clothing type for quantitative experiments ofclothing shape models. The results are similar for otherclothing types. To better evaluate the performance of theclothing shape model, we use the average Euclidean vertexdistance to measure the prediction error and compute theaverage prediction error for 100 randomly generated bodyshapes.Figure 11 (left) illustrates the average errors for 20 differ-ent clothing shape models (20 different poses). Comparedto DRAPE, our method reduces the error for each clothingshape model by 60%. This can also be seen in the predictionresults of the second clothing shape model for a randombody shape; please see the right part of Figure 11. Asshown in Figure 10, our method captures more variancethan DRAPE given the same number of principal compo-nents (98.6% vs. 87.2% when using the first 5 principalcomponents), which contributes to the higher performanceof our clothing shape model. It is also worth noting that thesquare term β reduces the prediction error by 3%.Furthermore, our mapping function from body shapeto clothing deformation only involves multiplication andaddition, and thus is more efficient to calculate than DRAPE,where an optimization problem needs to be solved. Inpractice, it takes 22ms for our clothing shape model topredict the new clothing information, compared to 300msfor DRAPE. We use SMPL [51] to generate template body meshes underdifferent shapes and apply dual quaternion blending [71] toanimate the body mesh. The skinning weights are calculatedusing Pinnochio [72] and the motion sequences are from theCMU motion capture library [70].At run-time, given a new input pose, we employ Eq.4 to obtain a coarse result and then add the decayingeffect and resolve interpenetrations to get the final result.Given a new input shape, we re-compute the new clothinginstance f ( β, θ s ) and update the binding information foreach clothing shape model (See Sec. 3.4), which takes 0.022seach time. In our implementation, all clothing shape modelscan be handled in parallel, and it takes 0.7s to handle 150data points for the men’s long-sleeved shirt.Theoretically, our algorithm can add the shape dimen-sion to any clothing pose model. To demonstrate this, werun our method with both SOR and a sequence of simu-lated clothing instances. First, we use SOR as our clothingpose model. We turn off the penetration handling processin SOR and leave it for our method to address. We alsodiscard the translation item (rightmost item in Eq. 1 of[6]) in SOR since we find it has little impact on the final result, especially when using our sampling method. Thesemeasures contribute to the high speed of our method. Weuse our sampling method to generate both the SOR databaseand our database. As shown in Figure 12 and Figure 1, ourmethod can predict clothing instances with fine wrinkles.Please refer to Table 1 for detailed statistics.Second, we use simulated clothing instances as ourclothing pose model. We simulate the garment mesh under arandomly chosen motion sequence while keeping the samebody shape parameters. Then we use the resulting clothinginstances as the clothing pose model in Eq. 4. Figure 13(b)shows that our method can recover realistic fold patterns.We can see that the quality of our result relies on the externalclothing pose model, i.e. if the clothing pose model is moreaccurate, our result is closer to the ground truth (Figure13(c)). This can also be seen in Figure 14. We will elaborateon this in the following section. We use the average Euclidean vertex distance between thesynthesized clothing and the simulation clothing as the errorto measure the physical accuracy of our result. The error iscalculated over 16 ×
400 randomly chosen shape and posepairs (16 shapes and 400 poses for each shape). As shown inFigure 14, the error of the results with SOR drops to 1.06 cmat 150 data points, which we believe is acceptable in mostscenarios. Given the number of data points, the error of the results with simulation is much less than that of results withSOR , which demonstrates that our method can be improvedif a more accurate clothing pose model is employed.
Our method can be applied to VR scenarios. As shownin Figure 15, we provide the user with an immersive VRexperience from a first-person perspective with an HTCVive (left most). The clothing deformations for the agentsare generated by our method.
We conducted a user study to demonstrate the perceptualbenefit of our method compared to the prior technique ingenerating clothing deformations for the agent in immersivesettings.
Experiment Goals & Expectations:
We hypothesize thatthe clothing deformations generated by our method will ex-hibit more detailed wrinkles and more dynamics comparedto prior methods and that participants will strongly preferour results to those of prior methods.
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 11
Fig. 11. Comparison of our clothing shape model and that of DRAPE. The bar graph on the left shows the errors for 20 clothing shape models. Theprediction results for the second clothing shape model for a random body shape are shown on the right. The errors are marked in green.Fig. 12. Synthesis results. Clothing patterns are shown in blue.
Fig. 13. Comparison between the synthesis results achieved using SOR as the clothing pose model and those achieved using simulation as theclothing pose model. (a) Synthesis result (left) using SOR (right); (b) synthesis result (left) using simulation (right); (c) simulation result; (d) Euclideanvertex distance between the synthesis results and the simulation clothing. The errors are marked in green. We can see that the synthesis resultachieved using simulation (right) has fewer errors than the result achieved using SOR (left).
Question (Which...) 1 2 3 4 5 6 7 mean SDclothing animation looks more realistic? 4 / 4 / 4 5 / 6 / 9 6 / 7 / 3 3 / 1 / 1 0 / 0 / 1 0 / 0 / 0 0 / 0 / 0 2.44 / 2.28 / 2.22 ± . / ± . / ± . wrinkles look more natural? 6 / 4 / 1 7 / 12 / 11 1 / 1 / 4 4 / 0 / 0 0 / 0 / 2 0 / 1 /0 0 / 0 / 0 2.17 / 2.06 / 2.50 ± . / ± . / ± . agent looks more like a real person? 4 / 5 / 5 6 / 6 / 3 4 / 3 / 6 3 / 4 / 4 0 / 0 / 0 1 / 0 / 0 0 / 0 / 0 2.56 / 2.33 / 2.50 ± . / ± . / ± . TABLE 3The response frequency of subjects in the study described in Sec. 4.8. Scores are normalized such that indicates a strong preference for ourmethod and indicates a strong preference for the LBS method. For each scenario (separated by “/”), participants prefer our method over the priormethod on several dimensions. Experimental Design:
The study was conducted basedon a within-subjects, paired-comparison design. Partici-pants explored two simulations with a fixed exposure timewearing a HTC Vive headset. The clothing deformations inthe two simulations were generated using our method andthe LBS method. The order of scenes was counterbalanced,as well as the order of the methods. After these two sim- ulations, participants answered a set of questions, and ourquestionnaire design is inspired by prior methods [73], [74],[75].
Comparison Methods:
Previous methods generallyadopt skinning methods to deform the clothing mesh of anagent in virtual environments. Therefore, we evaluated ourmethod against the Linear Blend Skinning (LBS) method.
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 12
Fig. 14. Convergence of synthesis error while using an increasing num-ber of data points.Fig. 15. The avatar in a VR scenario. We provide the user with animmersive VR experience from a first-person perspective with HTC Vive.The avatar looks around and observes an agent performing some tasks.
Environments:
We use three scenarios for the user study.The first scenario was comprised of a man sweeping in theliving room, while wearing a long-sleeved shirt and pants.The second scenario consisted of a man that has a differentshape and wiping the windows, while wearing the sameclothes as the first scenario. The last scenario correspondedto a woman wandering in a living room, while wearing a t-shirt and shorts. Participants can walk around and observethe agent doing some tasks. Please refer to Figure 15 and thesupplemental video for more details.
Metrics:
Participants were asked to indicate their pref-erence for a method using a 7-point Likert scale, with 1 in-dicating a strong preference for the method presented first,7 indicating a strong preference for the method presentedsecond, and 4 indicating no preference. In terms of reportingthe results, we normalized the participant responses so that1 indicates a strong preference for our method.
Results:
Our study was taken by 18 participants, 9 male,with a mean age of . ± . years. The participantresponses clearly demonstrate the benefits of our algorithm.For each question, we performed a one-sample t -test com-paring the mean of the question with a hypothetical meanof 4 (no preference or no impact). The question “Whichclothing animation looks more realistic?” was shown to besignificant for each scenario: t (17) = -6.336, p < . ; t (17)= -8.166, p < . ; t (17) = -7.114, p < . . The question“Which wrinkles look more natural?” was also significantfor each scenario: t (17) = -6.761, p < . ; t (17) = -7.432, p < . ; t (17) = -6.101, p < . , as was “Which agentlooks more like a real person?”: t (17) = -4.579, p < . ; t (17) = -6.216, p < . ; t (17) = -5.532, p < . ;. Figure 16 and Table 3 provide further details on participant responses. ISCUSSION
The rationale of our sampling method is that Taylor ex-pansion locally approximates surrounding values of ananchoring point while cluster centers are the best set ofpoints to approximate all the points, i.e. they are the bestset of anchoring points. Compared to the MCMC samplingprocess used in SOR [6], our sampling process is able tosample more typical poses in real life, while their samplingstep finds a pose that maximizes the error of the synthesizedclothing, which might produce weird poses. Experimentalresults show that SOR results can be significantly improvedwhen using our sampling method. Additionally, thanks tothe mutual independence of our data points, our databasecan be constructed in parallel, while the MCMC process inSOR must run serially since their sampling process dependson the previously constructed data. Please refer to Sec. 4 fordetails.
Limitations.
Our method assumes that the clothing co-ordinate differences introduced by body shape variation arelinear with body pose, i.e. the clothing coordinate differ-ences caused by body shape variation under the new posecan be predicted from the anchoring pose using LBS. Thishinders the application of our approach to garments that areloose on the avatar.Furthermore, the final results of our method are highlydependent on the clothing pose model used. If the clothingpose model is not accurate, then our result is also notaccurate. In addition, the efficiency of the clothing posemodel creates a bottle-neck in our method. To overcome thisdifficulty, we plan to develop our own clothing pose modelin the future.The wrinkles and folds generated in the animation sufferfrom some sudden changes when body poses change tooquickly. The reason for this is that our scheme is trained onclothing instances of static equilibrium, and the estimationsfrom different poses are inconsistent. Although we haveemployed a decaying factor to smooth the animation, itcannot solve this problem completely.Finally, our method cannot guarantee that all the pen-etrations are resolved, especially when the clothing is tootight on the body. In this case, the LBS result for eachanchoring pose may result in deep penetrations, and theblending result will make penetrations even worse, whichis beyond the ability of our penetration resolving method toaddress. When this is the case, we believe that the clothingis not suitable for the body, suggesting that the clothing is abad fit. In addition, when the clothing has too many folds,self-penetration might occur in our synthesis results.
Future works.
We want to extend our method to othercontrol parameters like clothing material and clothing pat-terns. For clothing material, for example, we first need todevise a clothing material model. Given a set of clothingmaterial parameters, this model can predict the correspond-ing clothing deformation. Then, we sample some valuableposes as the Taylor expansion anchoring points. Finally, weblend local approximation results at run-time.We currently use the average sensitivity of 17 trainingshapes to calculate the distances of anchoring points. How-
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 13
OursLBS
Scenario one Scenario two Scenario three
Fig. 16. Participant preferences in user evaluation. The questions were “Which clothing animation looks more realistic?”, “Which wrinkles look morenatural?”, and “Which agent looks more like a real person?”. Scores are normalized such that 1 indicates a strong preference for our method and7 indicates a strong preference for LBS method. For each scenario, participants rated our method preferably in terms of generating more realisticor plausible clothing animation ( . ± . , . ± . , . ± . ), generating more natural-looking wrinkles ( . ± . , . ± . , . ± . ), and enhancing the presence of the agent ( . ± . , . ± . , . ± . ). ever, as the body shape changes, the sensitivity changes ac-cordingly. In the future, we plan to train a sensitivity model,i.e. a model that can predict the corresponding sensitivity ofthe clothing given a new set of body shape parameters. Webelieve this will help us to find better anchoring points atrun-time. ONCLUSION
In this paper, we have presented a clothing synthesisscheme that uses Taylor expansion to combine two indepen-dent components: the pose-dependent deformation and theshape-dependent deformation. As a result, our method canadd the shape dimension to various clothing pose models.Our method does not need to modify or retrain the clothingpose model. The core innovation here is that we regardclothing deformation as a function of body shape param-eters and body pose parameters and use Taylor expan-sion to locally approximate clothing deformations aroundanchoring points. Due to the high computational cost ofhigher-order derivatives, we use linear Taylor expansion inboth offline and online processes. Our clothing shape modelcan efficiently predict realistic clothing deformations undervarious body shapes and has the potential for use as a stand-alone application.Using only a CPU, our method can generate realisticclothing deformations under various body poses and shapesin real time without resizing the cloth model. Furthermore,we believe that our method can be extended to add otherparameters such as clothing material and clothing pattern toclothing pose models. The performance can improve basedon data-driven parameter estimation methods [76]. A CKNOWLEDGMENTS
Xiaogang Jin was supported by the National Key R&DProgram of China (Grant No. 2017YFB1002600), the NingboMajor Special Projects of the “Science and Technology In-novation 2025” (Grant No. 2020Z007), the National Nat-ural Science Foundation of China (Grant Nos. 61732015,61972344), and the Key Research and Development Programof Zhejiang Province (Grant No. 2020C03096). Qianwen Chao was supported by the National Natural Science Foun-dation of China (Grant No. 61702393). We thank ZhejiangLinctex Digital Technology Co. Ltd for the help on designingthe female T-shirt in Figure 12. R EFERENCES [1] J. N. Bailenson, K. Swinth, C. Hoyt, S. Persky, A. Dimov,and J. Blascovich, “The independent and interactive effects ofembodied-agent appearance and behavior on self-report, cogni-tive, and behavioral markers of copresence in immersive vir-tual environments,”
Presence: Teleoperators & Virtual Environments ,vol. 14, no. 4, pp. 379–393, 2005.[2] D. Baraff and A. Witkin, “Large steps in cloth simulation,” in
Proceedings of the 25th annual conference on Computer graphics andinteractive techniques . ACM, 1998, pp. 43–54.[3] R. Narain, A. Samii, and J. F. O’Brien, “Adaptive anisotropicremeshing for cloth simulation,”
ACM Trans. Graph. , vol. 31, no. 6,p. 152, 2012.[4] G. Cirio, J. Lopez-Moreno, D. Miraut, and M. A. Otaduy, “Yarn-level simulation of woven cloth,”
ACM Trans. Graph. , vol. 33, no. 6,p. 207, 2014.[5] D. Kim, W. Koh, R. Narain, K. Fatahalian, A. Treuille, and J. F.O’Brien, “Near-exhaustive precomputation of secondary cloth ef-fects,”
ACM Trans. Graph. , vol. 32, no. 4, p. 87, 2013.[6] W. Xu, N. Umentani, Q. Chao, J. Mao, X. Jin, and X. Tong,“Sensitivity-optimized rigging for example-based real-time cloth-ing synthesis,”
ACM Trans. Graph. , vol. 33, no. 4, p. 107, 2014.[7] P. Guan, L. Reiss, D. A. Hirshberg, A. Weiss, and M. J. Black,“Drape: Dressing any person.”
ACM Trans. Graph. , vol. 31, no. 4,pp. 35–1, 2012.[8] I. Santesteban, M. A. Otaduy, and D. Casas, “Learning-based animation of clothing for virtual try-on,” arXiv preprintarXiv:1903.07190 , 2019.[9] M. Hauth, O. Etzmuß, and W. Straßer, “Analysis of numericalmethods for the simulation of deformable models,”
The VisualComputer , vol. 19, no. 7-8, pp. 581–600, 2003.[10] P. Volino and N. Magnenat-Thalmann, “Implicit midpoint inte-gration and adaptive damping for efficient cloth simulation,”
Computer Animation and Virtual Worlds , vol. 16, no. 3-4, pp. 163–175, 2005.[11] B. Fierz, J. Spillmann, and M. Harders, “Element-wise mixedimplicit-explicit integration for stable dynamic simulation ofdeformable objects,” in
Proceedings of the 2011 ACM SIG-GRAPH/Eurographics Symposium on Computer Animation . ACM,2011, pp. 257–266.[12] X. Provot et al. , “Deformation constraints in a mass-spring modelto describe rigid cloth behaviour,” in
Graphics interface . CanadianInformation Processing Society, 1995, pp. 147–147.
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 14 [13] R. Goldenthal, D. Harmon, R. Fattal, M. Bercovier, and E. Grin-spun, “Efficient simulation of inextensible cloth,”
ACM Trans.Graph. , vol. 26, no. 3, p. 49, 2007.[14] B. Thomaszewski, S. Pabst, and W. Strasser, “Continuum-basedstrain limiting,” in
Computer Graphics Forum , vol. 28, no. 2. WileyOnline Library, 2009, pp. 569–576.[15] H. Wang, J. O’Brien, and R. Ramamoorthi, “Multi-resolutionisotropic strain limiting,” in
ACM Trans. Graph. , vol. 29, no. 6.ACM, 2010, p. 156.[16] G. Ma, J. Ye, J. Li, and X. Zhang, “Anisotropic strain limiting forquadrilateral and triangular cloth meshes,” in
Computer GraphicsForum , vol. 35, no. 1. Wiley Online Library, 2016, pp. 89–99.[17] E. Grinspun, A. N. Hirani, M. Desbrun, and P. Schr¨oder, “Discreteshells,” in
Proceedings of the 2003 ACM SIGGRAPH/Eurographicssymposium on Computer animation . Eurographics Association,2003, pp. 62–67.[18] E. English and R. Bridson, “Animating developable surfaces usingnonconforming elements,” in
ACM Trans. Graph. , vol. 27, no. 3.ACM, 2008, p. 66.[19] K.-J. Choi and H.-S. Ko, “Stable but responsive cloth,” in
ACMSIGGRAPH 2005 Courses . ACM, 2005, p. 1.[20] P. Volino, N. Magnenat-Thalmann, and F. Faure, “A simple ap-proach to nonlinear tensile stiffness for accurate cloth simulation,”
ACM Trans. Graph. , vol. 28, no. 4, pp. Article–No, 2009.[21] R. Bridson, R. Fedkiw, and J. Anderson, “Robust treatment ofcollisions, contact and friction for cloth animation,” in
ACM Trans.Graph. , vol. 21, no. 3. ACM, 2002, pp. 594–603.[22] D. Harmon, E. Vouga, R. Tamstorf, and E. Grinspun, “Robusttreatment of simultaneous collisions,”
ACM Trans. Graph. , vol. 27,no. 3, p. 23, 2008.[23] C. Zheng and D. L. James, “Energy-based self-collision culling forarbitrary mesh deformations,”
ACM Trans. Graph. , vol. 31, no. 4,p. 98, 2012.[24] Q. Guo, X. Han, C. Fu, T. Gast, R. Tamstorf, and J. Teran, “Amaterial point method for thin shells with frictional contact,”
ACMTrans. Graph. , vol. 37, no. 4, p. 147, 2018.[25] T. Liu, A. W. Bargteil, J. F. O’Brien, and L. Kavan, “Fast simulationof mass-spring systems,”
ACM Trans. Graph. , vol. 32, no. 6, p. 214,2013.[26] S. Bouaziz, S. Martin, T. Liu, L. Kavan, and M. Pauly, “Projectivedynamics: fusing constraint projections for fast simulation,”
ACMTrans. Graph. , vol. 33, no. 4, p. 154, 2014.[27] H. Wang, “A chebyshev semi-iterative approach for accelerat-ing projective and position-based dynamics,”
ACM Trans. Graph. ,vol. 34, no. 6, p. 246, 2015.[28] M. Fratarcangeli, H. Wang, and Y. Yang, “Parallel iterative solversfor real-time elastic deformations,” in
SIGGRAPH Asia 2018Courses , 2018, pp. 1–45.[29] M. M ¨uller, B. Heidelberger, M. Hennix, and J. Ratcliff, “Positionbased dynamics,”
Journal of Visual Communication and Image Repre-sentation , vol. 18, no. 2, pp. 109–118, 2007.[30] M. Tang, H. Wang, L. Tang, R. Tong, and D. Manocha, “Cama:Contact-aware matrix assembly with unified collision handling forgpu-based cloth simulation,” in
Computer Graphics Forum , vol. 35,no. 2. Wiley Online Library, 2016, pp. 511–521.[31] M. Tang, T. Wang, Z. Liu, R. Tong, and D. Manocha, “I-cloth:incremental collision handling for gpu-based interactive clothsimulation,”
ACM Transactions on Graphics (TOG) , vol. 37, no. 6,pp. 1–10, 2018.[32] M. Tang, R. Tong, R. Narain, C. Meng, and D. Manocha, “A gpu-based streaming algorithm for high-resolution cloth simulation,”in
Computer Graphics Forum , vol. 32, no. 7. Wiley Online Library,2013, pp. 21–30.[33] N. K. Govindaraju, M. C. Lin, and D. Manocha, “Quick-cullide:Fast inter-and intra-object collision culling using graphics hard-ware,” in
IEEE Proceedings. VR 2005. Virtual Reality, 2005.
IEEE,2005, pp. 59–66.[34] C. Li, M. Tang, R. Tong, M. Cai, J. Zhao, and D. Manocha, “P-cloth:interactive complex cloth simulation on multi-gpu systems usingdynamic matrix assembly and pipelined implicit integrators,”
ACM Transactions on Graphics (TOG) , vol. 39, no. 6, pp. 1–15, 2020.[35] D. Manocha and J. Demmel, “Algorithms for intersecting paramet-ric and algebraic curves i: simple intersections,”
ACM Transactionson Graphics (TOG) , vol. 13, no. 1, pp. 73–100, 1994.[36] M. Tang, R. Tong, Z. Wang, and D. Manocha, “Fast and exactcontinuous collision detection with bernstein sign classification,”
ACM Transactions on Graphics (TOG) , vol. 33, no. 6, pp. 1–8, 2014. [37] D. Manocha, “Solving polynomial equations,”
Applications of Com-putational Algebraic Geometry: American Mathematical Society ShortCourse, January 6-7, 1997, San Diego, California , vol. 53, p. 41, 1998.[38] ——,
Algebraic and numeric techniques in modeling and robotics .University of California at Berkeley, 1992.[39] J. T. Klosowski, M. Held, J. S. B. Mitchell, H. Sowizral, andK. Zikan, “Efficient collision detection using bounding volumehierarchies of k-dops,”
IEEE Transactions on Visualization and Com-puter Graphics , vol. 4, no. 1, pp. 21–36, Jan. 1998.[40] A. Fuhrmann, G. Sobotka, and C. Groß, “Distance fields for rapidcollision detection in physically based modeling,” in
Proceedings ofGraphiCon 2003 . Citeseer, 2003, pp. 58–65.[41] J.-M. Thiery, ´E. Guy, and T. Boubekeur, “Sphere-meshes: Shapeapproximation using spherical quadric error metrics,”
ACM Trans.Graph. , vol. 32, no. 6, p. 178, 2013.[42] N. Wu, D. Zhang, Z. Deng, and X. Jin, “Variational mannequinapproximation using spheres and capsules,”
IEEE Access , vol. 6,pp. 25 921–25 929, 2018.[43] M. Teschner, S. Kimmerle, B. Heidelberger, G. Zachmann,L. Raghupathi, A. Fuhrmann, M.-P. Cani, F. Faure, N. Magnenat-Thalmann, W. Strasser, and P. Volino, “Collision Detection forDeformable Objects,” in
Eurographics 2004 - STARs . EurographicsAssociation, 2004.[44] E. De Aguiar, L. Sigal, A. Treuille, and J. K. Hodgins, “Stable spacesfor real-time clothing,” in
ACM Trans. Graph. , vol. 29, no. 4. ACM,2010, p. 106.[45] H. Wang, F. Hecht, R. Ramamoorthi, and J. F. O’Brien, “Example-based wrinkle synthesis for clothing animation,” in
ACM Trans.Graph. , vol. 29, no. 4. ACM, 2010, p. 107.[46] F. Hahn, B. Thomaszewski, S. Coros, R. W. Sumner, F. Cole,M. Meyer, T. DeRose, and M. Gross, “Subspace clothing simulationusing adaptive bases,”
ACM Trans. Graph. , vol. 33, no. 4, p. 105,2014.[47] N. Jin, Y. Zhu, Z. Geng, and R. Fedkiw, “A pixel-based frameworkfor data-driven clothing,” arXiv preprint arXiv:1812.01677 , 2018.[48] N. Chentanez, M. Macklin, M. M ¨uller, S. Jeschke, and T.-Y. Kim,“Cloth and skin deformation with a triangle mesh based convolu-tional neural network,” in
Computer Graphics Forum , vol. 39, no. 8.Wiley Online Library, 2020, pp. 123–134.[49] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, andJ. Davis, “Scape: shape completion and animation of people,” in
ACM Trans. Graph. , vol. 24, no. 3. ACM, 2005, pp. 408–416.[50] T. Y. Wang, D. Ceylan, J. Popovic, and N. J. Mitra, “Learninga shared shape space for multimodal garment design,” arXivpreprint arXiv:1806.11335 , 2018.[51] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black,“Smpl: A skinned multi-person linear model,”
ACM Trans. Graph.) ,vol. 34, no. 6, p. 248, 2015.[52] Z. Lahner, D. Cremers, and T. Tung, “Deepwrinkles: Accurateand realistic clothing modeling,” in
Proceedings of the EuropeanConference on Computer Vision (ECCV) , 2018, pp. 667–684.[53] C. Patel, Z. Liao, and G. Pons-Moll, “Tailornet: Predicting clothingin 3d as a function of human pose, shape and garment style,”in
Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition , 2020, pp. 7365–7375.[54] Q. Ma, J. Yang, A. Ranjan, S. Pujades, G. Pons-Moll, S. Tang, andM. J. Black, “Learning to dress 3d people in generative clothing,”in
Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition , 2020, pp. 6469–6478.[55] J. Yang, J.-S. Franco, F. H´etroy-Wheeler, and S. Wuhrer, “Analyzingclothing layer deformation statistics of 3d human motions,” in
Proceedings of the European Conference on Computer Vision (ECCV) ,2018, pp. 237–253.[56] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, andH. Li, “Pifu: Pixel-aligned implicit function for high-resolutionclothed human digitization,” in
Proceedings of the IEEE InternationalConference on Computer Vision , 2019, pp. 2304–2314.[57] R. Natsume, S. Saito, Z. Huang, W. Chen, C. Ma, H. Li, and S. Mor-ishima, “Siclope: Silhouette-based clothed people,” in
Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition ,2019, pp. 4480–4490.[58] G. Pons-Moll, S. Pujades, S. Hu, and M. J. Black, “Clothcap:Seamless 4d clothing capture and retargeting,”
ACM Transactionson Graphics (TOG) , vol. 36, no. 4, pp. 1–15, 2017.[59] S. Yang, J. Liang, and M. C. Lin, “Learning-based cloth materialrecovery from video,” in
Proceedings of the IEEE International Con-ference on Computer Vision , 2017, pp. 4383–4393.
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 15 [60] A. Saltelli, S. Tarantola, F. Campolongo, and M. Ratto, “Sensitivityanalysis in practice: a guide to assessing scientific models,”
Chich-ester, England , 2004.[61] J. Kiendl, R. Schmidt, R. W ¨uchner, and K.-U. Bletzinger, “Isoge-ometric shape optimization of shells using semi-analytical sen-sitivity analysis and sensitivity weighting,”
Computer Methods inApplied Mechanics and Engineering , vol. 274, pp. 148–167, 2014.[62] J. Zehnder, E. Knoop, M. B¨acher, and B. Thomaszewski, “Metasil-icone: design and fabrication of composite silicone with desiredmechanical properties,”
ACM Trans. Graph. , vol. 36, no. 6, p. 240,2017.[63] M. Geilinger, R. Poranne, R. Desai, B. Thomaszewski, and S. Coros,“Skaterbots: Optimization-based design and motion synthesis forrobotic creatures with legs and wheels,”
ACM Trans. Graph. ,vol. 37, no. 4, p. 160, 2018.[64] S. Zimmermann, R. Poranne, J. M. Bern, and S. Coros, “Pup-petmaster: robotic animation of marionettes,”
ACM Trans. Graph. ,vol. 38, no. 4, p. 103, 2019.[65] N. Umetani, D. M. Kaufman, T. Igarashi, and E. Grinspun, “Sensi-tive couture for interactive garment modeling and editing.”
ACMTrans. Graph. , vol. 30, no. 4, pp. 90–1, 2011.[66] V. Barrielle and N. Stoiber, “Realtime performance-driven physicalsimulation for facial animation,” in
Computer Graphics Forum ,vol. 38, no. 1. Wiley Online Library, 2019, pp. 151–166.[67] Z. Shen, J. Huang, W. Chen, and H. Bao, “Geometrically exactsimulation of inextensible ribbon,” in
Computer Graphics Forum ,vol. 34, no. 7. Wiley Online Library, 2015, pp. 145–154.[68] M. Chen and K. Tang, “A fully geometric approach for devel-opable cloth deformation simulation,”
The visual computer , vol. 26,no. 6-8, pp. 853–863, 2010.[69] M. Bergou, S. Mathur, M. Wardetzky, and E. Grinspun, “Tracks:toward directable thin shells,”
ACM Trans. Graph. , vol. 26, no. 3,pp. 50–es, 2007.[70] CMU, “CMU graphics lab motion capture database.” http://mocap.cs.cmu.edu. , 2003.[71] L. Kavan, S. Collins, J. ˇZ´ara, and C. O’Sullivan, “Geometric skin-ning with approximate dual quaternion blending,”
ACM Trans.Graph. , vol. 27, no. 4, p. 105, 2008.[72] I. Baran and J. Popovi´c, “Automatic rigging and animation of 3dcharacters,” in
ACM Trans. Graph. , vol. 26, no. 3. ACM, 2007,p. 72.[73] M. Garau, M. Slater, D.-P. Pertaub, and S. Razzaque, “The re-sponses of people to virtual humans in an immersive virtual en-vironment,”
Presence: Teleoperators & Virtual Environments , vol. 14,no. 1, pp. 104–116, 2005.[74] M. Slater, C. Guger, G. Edlinger, R. Leeb, G. Pfurtscheller,A. Antley, M. Garau, A. Brogni, and D. Friedman, “Analysis ofphysiological responses to a social situation in an immersive vir-tual environment,”
Presence: Teleoperators and Virtual Environments ,vol. 15, no. 5, pp. 553–569, 2006.[75] S. Narang, A. Best, T. Randhavane, A. Shapiro, and D. Manocha,“Pedvr: Simulating gaze-based interactions between a real userand virtual crowds,” in
Proceedings of the 22nd ACM conference onvirtual reality software and technology , 2016, pp. 91–100.[76] D. Wolinski, S. J. Guy, A.-H. Olivier, M. Lin, D. Manocha, andJ. Pettr´e, “Parameter estimation and comparative evaluation ofcrowd simulations,” in
Computer Graphics Forum , vol. 33, no. 2.Wiley Online Library, 2014, pp. 303–312.
Nannan Wu received his BSc degree in Com-puter Science from Nanjing University of Sci-ence and Technology, China, in 2013. He is cur-rently a PhD candidate in the State Key Lab ofCAD&CG, Zhejiang University, China. His mainresearch interests include cloth simulation, vir-tual try-on and computer animation.
Qianwen Chao is a Lecturer of Computer Sci-ence at Xidian University, China. She earnedher Ph.D. degree in Computer Science from theState Key Laboratory of CAD&CG, Zhejiang Uni-versity in 2016. Prior that, she received her B.S.degree in computer science in 2011 from XidianUniversity. Her main research interests includecrowd animation and swarm micro-robotics.
Yanzhen Chen received the BSc degree in digi-tal media technology from Zhejiang University, P.R. China, in 2020. He is currently working towardthe MSc degree at Zhejiang University. His re-search interests include computer graphics andcomputer animation.
Weiwei Xu is a researcher with the State KeyLab of CAD & CG, College of Computer Science,Zhejiang University, awardee of NSFC Excel-lent Young Scholars Program in 2013. His main-research interests include the digital geometryprocessing, physical simulation, computer visionand virtual reality. He has published around 70papers on international graphics journals andconferences, including 19 papers on ACM TOG.He is a member of the IEEE.
Chen Liu is the CEO of Zhejiang Linctex DigitalTechnology Co. Ltd., China. He received his BScdegree in 1993 from Zhejiang Institute of SilkTechnology, MSc degree in MBA in 2000 fromZhejiang University, and another MSc degree inapplied computer science in 2003 from Vrije Uni-versiteit Brussel (VUB). His current research in-terests mainly focus on computer-aided clothingdesign, cloth animation, digital face modeling,and new retail.
Dinesh Manocha
XAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS FOR VIRTUAL AGENTS 16
Wenxin Sun is currently an undergraduate stu-dent in Zhejiang University majoring in digitalmedia technology. Her main research interestsinclude image processing and facial animation.
Yi Han is a Ph.D. candidate in the State Key Labof CAD & CG, Zhejiang University. Prior to that,he received his bachelor’s degree in softwareengineering in 2017 from Sun Yat-sen University.Recently, his main researches focus on crowdanimation, traffic simulation, and computer ani-mation.
Xinran Yao is a master candidate in theState Key Lab of CAD&CG, Zhejiang University,China. She graduated from Zhejiang Universitywith a bachelor’s degree in 2018 majoring indigital media technology. In recent years, hermain researches focus on crowd animation andgame AI.