Fat Pad Cages for Facial Posing
Adèle Colas, Florent Guiotte, Fabien Danieau, François Le Clerc, Quentin Avril
FFat Pad Cages for Facial Posing
Adèle Colas
IMT Atlantique, [email protected]
Florent Guiotte
ESIR, [email protected]
Fabien Danieau
InterDigital, [email protected]
François Le Clerc
InterDigital, [email protected]
Quentin Avril
InterDigital, [email protected]
Figure 1: Given any facial mesh (a), fat pads are computed following a template fat map (b), attenuation matrices are computedusing geodesic distances (c top), pad’s handles are mapped onto the mesh (c bottom) and a cage is built using convex hull ofDelaunay triangulation (d). This automatically generated Fat Pad cage enables to easily and quickly sculpt facial poses (e).
ABSTRACT
We introduce Fat Pad cages for posing facial meshes. It combinescage representation and facial anatomical elements, and enablesusers with no artistic skill to quickly sketch realistic facial expres-sions. The model relies on one or several cage(s) that deform(s) themesh following the human fat pads map. We propose a new func-tion to filter Green Coordinates using geodesic distances preventingglobal deformation while ensuring smooth deformations at the bor-ders. Lips, nostrils and eyelids are processed slightly differently toallow folding up and opening. Cages are automatically created andfit any new unknown facial mesh. To validate our approach, wepresent a user study comparing our Fat Pad cages to regular GreenCoordinates. Results show that Fat Pad cages bring a significantimprovement in reproducing existing facial expressions.
Facial expressions are key elements in the realistic characters cre-ation process [18]. Recent movie characters have facial rigs withthousands of blend shapes. Blend shapes represent facial elementarydisplacements and are often related to muscles [11]. Modeling themrequires strong modeling skills and a lot of time (weeks for herocharacters). Posing faces can be achieved through linear piece-wisemodeling approaches [24] that require a lot of facial performancesmotion capture data. Physics-based approaches are also possible [8]but a high-end polyhedral facial anatomical representation is neces-sary to simulate bones, muscles and fat. Simpler solution would useany free-form deformation (FFD) techniques [22]. However, FFD aregeneric and do not take into account facial semantics. They can leadto severe uncanny artifacts for non-experts as a face is deformed as any other meshes. These non-expert users may be professionals,like stage directors who want to sketch facial expressions on a tabletto give guidelines, or beginners such as digital art school students.The final shape of a facial expression is a complex combination ofthe skull, the muscles and the fat [28]. As skin is soft and thin, itmay rather be considered as a protective and aesthetic layer thana dynamic and active one. The skull gives the macroscopic shapeof the face, and except for the jaws, it should not move. Actually,muscles are the cause of the action, and fat is the main visual ele-ment of the final shape (wrinkles, dimples, gaps and folds). Fat onthe face is divided in several pads. The name "fat pad" comes frommedical dictionaries and plastic surgery. Each pad moves almostindependently if it is activated by the related muscle, and the skinjoins all the pads together. We present the novel concept of fat padsfor facial posing, enabling an interactive mesh deformation thatrespects the constraints of a face and is intuitive and easy to usefor beginners. We propose a new automatic way for creating facialcages and a new weighted function based on geodesic distancesto localize and limit the influence of cage deformation. Finally, wepresent a user study with 17 naive users that validates the easinessand intuitiveness of the Fat Pad cages.
Cages.
Cages are polyhedra with a small number of polygons en-compassing a mesh. Mesh vertices are linear sums of the cage’svertices multiplied by weight functions (coordinates). Floater etal. [7] first proposed a way to use cages with a random geometrywith the Mean Values Coordinates (MVC). But undesired artifactsappear with negative coordinates. Joshi et al. then proposed the a r X i v : . [ c s . G R ] O c t rxiv.org, October, 2020 Colas et al. Harmonic Coordinates (HC) that resolve Floater’s issues but re-quire a long computation time [10]. Lipman et al. put forward theGreen Coordinates (GC) that do not have negativity issues [16].Computation time allows interactivity, surface’s details and meshvolume are correctly preserved. Jacobson et al. propose three rulesfor cage generation [9]: it must be low resolution to reduce manualinteractions, it must fully and tightly bind the enveloped model andit should respect the mesh topology for the manipulation to be in-tuitive. Yang et al. propose to use a template based cage generationtype [27]. The nested cage [19] uses the character mesh to buildprogressive decimated cage mesh around the model. Xian proposesto generate oriented bounding boxes for each mesh part and thenregistering them together [26] but it is hard to apply on faces dueto the lack of articulation. Chen et al. proposes a skeleton basedapproach [3] but face meshes do not have any skeleton. Lee et al.’scage relies on user inputs [13] that cuts the model and the slides setis used to construct an initial cage, but our target is an automaticapproach with no user interaction.
Facial Posing.
To model faces, there are several approachessuch as blend shapes [11, 14, 15, 20], region-based model [24] andphysics-based model [5, 8, 21]. Blend shapes remain a predominanttechnique and are in most digital content creation tools. Tena etal. [24] propose a region-based linear model, that uses PCA. It sep-arates the mesh in several models and the formulation restrictsthe solutions to have semi-consistent boundaries while enforcinguser-given constraints. It does not allow to model other expressionsthan the ones from the training set. It is also possible to rely oneigenbases of the Laplacian to generate low-frequency bases [2]to track human face using RGB-D camera. But it is not compatiblewith static facial mesh as our input. Waters proposed a musclemodel that uses muscle vectors and radial functions derived fromlinear and sphincter muscles to deform a skin mesh [25]. Lee et al.presented an algorithm that automatically constructs functionalmodels of heads, subjected to laser-scanned range and reflectancedata [13]. Contractile muscles within a dynamic skin model are theninserted and rooted in an estimated skull structure with a hingedjaw. Kähler et al. [12] proposed a model for muscle-based facialanimation composed of three layers: skin with fatty tissue, musclesattached to the skull, and the underlying bone structure, composedof immovable skull and rotating jaw. Muscle-based models are accu-rate but not interactive. Ichim et al. have presented a physics-basedapproach to model faces [8] that optimizes facial physical interac-tions and prevents undesired artifacts, but it requires a completefacial anatomical model.This highlights a lack between intuitive cage deformers, not usedto pose faces, and dedicated professional tools, requiring time andstrong artistic skills. Fat Pad cages enable to fill this gap proposingan intuitive, interactive and accurate modeling approach.
We propose to divide a face in smaller areas called "fat pads". Oneach pad, some vertices are manually labeled as "handles" and usedto build the cage. Their displacement induces smooth deformationsof the corresponding area. This method represents the way facialmuscles would deform the skin. The fat pads approach limits thedeformation, induced by the cage, to a local area. We use Green Coordinates (GC) [16] but the concept is applicable to any coor-dinates system [10, 17]. We use GC because they better preservemesh details, are computed at interactive time and the cage doesnot need to entirely contain the mesh. However, cage’s handlesonly affect the vertices of their own fat pad. To ensure a smoothand continuous deformation between pads, it must be attenuated atthe border. Hence, a weight between 0 and 1 smooths the GC. It isassociated to each vertex according to its position in a pad. Whilethe vertices close to the handle move normally, the ones close tothe border area remain still or almost still.
Template Map.
Fat pads are designed following a referencebook used by modelers in VFX studios [28]. As Cong et al. proposedto simulate flesh and muscles [4], we use a template map where wemanually painted fat pads. This operation has only to be performedonce. The template fat pad map is then used for all new meshes. Ifa character requires specific fat motion behaviors, the template iseasily re-paintable and handles freely movable. The face has smallerfat pads or tissues in-between main fat pads. To avoid a complexmodeling and fill the gaps between the pads, we propose to overlapthem as shown on Figure 1. It avoids sharp borders and ensuresmooth borders’ deformations. A plugin in Maya was developed topaint fat pads on meshes and place handles.
Attenuation Matrix.
A weight 𝑤 𝑣,ℎ is computed for each pad’svertex 𝑣 and handle ℎ , the closer the vertex to the border, the higherthe attenuation. To avoid any border aberration, the weight mustdecrease slowly to zero at the border. We propose to use the fol-lowing function that decreases according to the geodesic distancebetween the vertex position and the border, 𝑤 𝑣,ℎ = ( 𝑑 ( 𝑣, ℎ ) − 𝑑 ( 𝑖, ℎ )) 𝑑 ( 𝑖, ℎ ) , (1)with 𝑑 the geodesic distance between two vertices, and 𝑖 the vertexlocated at the intersection of the border of the pad and the line (cid:174) 𝑣ℎ (see Figure 2 and 3). The use of a quadratic radial basis functionkernel appears appropriate as it smoothly goes from 1 (at the handle)to 0 (at the borders), thus restricting the influence on the fat pads.The function is used to compute the weight matrix 𝑊 ℎ for eachvertex of each fat pad. This kernel has a zero derivative at theboundaries but it is not an issue in our case as we set the boundaries’vertices fixed and unmovable (weight is 0). Specific Borders’ Vertices.
Lips are a good example wherefat pads must be processed differently. Indeed, it has to follow thedeformation of the fat pads to allow folding up, opening, and closing.Such borders can be specified in our model as an exception not toset the attenuation at the border to zero as displayed in Figure 3.It is the left upper lip fat pad with its two handles (blue) and itsmovable border (red). This is applied to the nostrils and the eyelidsas well. In our posing process, we set the weight for this specificborders’ vertices to 1, thus it moves freely according to the handlemotion as if it was not linked to another adjacent pad.The final positions 𝑉 ′ of vertices after moving a handle ℎ is: 𝑉 ′ = 𝑉 + ( 𝑉 𝑔𝑐 − 𝑉 ) 𝑊 ℎ (2)with 𝑉 the current vertices’ positions, 𝑉 𝑔𝑐 the GC positions and 𝑊 ℎ the weight matrix associated to the handle ℎ . Pad Shape Processing.
As fat pads may have different shapes,the weight matrix 𝑊 ℎ must be computed for any new mesh. 𝑑 ( 𝑖, ℎ ) , at Pad Cages for Facial Posing arxiv.org, October, 2020 hv A V B v C i A i B i F i C d(i,h)d(v,h) A B C
Figure 2: A: vertex 𝑣 𝐴 with handle ℎ . 𝑑 ( 𝑖 𝐴 , ℎ ) , geodesic dist.between ℎ and 𝑖 𝐴 intersection between pad’s border and (cid:174) ℎ𝑣 𝑎 .B, C: non-convex fat pad, many intersections may exist. distance between the handle and the border passing through vertex 𝑣 , is a key value to compute 𝑊 ℎ . It is different for each vertex andeach handle. To compute 𝑑 ( 𝑖, ℎ ) , we set plane 𝑃 , defined by line 𝐿 (between handle and vertex) and handle’s normal 𝑁 (see Figure 3). 𝑃 Figure 3: Left: Fat pad movable border and handle. Middle:given a handle and a vertex, plane 𝑃 intersects the border.Right: nostril fat pad has a concave shape. may intersect the border at many points if the fat pad is non-convex(see Figure 3). Figure 2 shows cases A, B and C on a non-convex fatpad with vertices 𝑣 𝐴 , 𝑣 𝐵 and 𝑣 𝐶 , and handle ℎ . Plane 𝑃 remains thesame, and therefore the list of intersection points 𝑙 = [ 𝑖 𝐴 , 𝑖 𝐵 , 𝑖 𝐶 , 𝑖 𝐹 ] .A way to remove intersection candidates is to filter according todirection: ( (cid:154) −→ ℎ𝑖, −→ ℎ𝑣 ) = 𝑣 𝑛 must be between ℎ and 𝑖 𝑛 : 𝑑 ( ℎ, 𝑣 ) < 𝑑 ( ℎ, 𝑖 ) (B and C). The mindistance is kept 𝑖 𝑠𝑒𝑙𝑒𝑐𝑡 | 𝑑 ( 𝑣, 𝑖 𝑠𝑒𝑙𝑒𝑐𝑡 ) = 𝑚𝑖𝑛 𝑖 ∈ 𝑙 ( 𝑑 ( 𝑣, 𝑖 )) . The weightmatrix is computed offline using Surazhsky’s algorithm for thegeodesic distances [23]. Cage Construction.
We select cages for two reasons. First, theshape preservation property of the cage combined with GC reducesundesired artifacts while preserving face volume. Second, the in-tuitiveness, simplicity of usage and direct manipulation given bycages suit our needs for non-experts. The cage is built using theposition of the pad’s handles. To get the cage topology, we computea Delaunay triangulation on the handles, generating a tetrahedralmesh from which we extract the convex hull. This ensures a ho-mogeneous connection with almost all the nearest neighbors. Aswe want to model facial poses, we need the face to be dynamicand the rest of the head to remain static. Thus, we do not wantthe cage to entirely encapsulate the head mesh but only the face.To build local cages several conditions are required [16]: (1) theborders’ vertices need to be fixed, (2) the cage must be scaled toprevent edges to intersect the model, and (3) the cage needs to beclosed. To respect (1), cage borders’ vertices are duplicated, scaledand then fixed. As cage vertices are based on pads’ handles, fixingthe initial borders’ vertices do not make sense as it strongly limits the interaction with the pads. The duplication prevents this issue.To respect (2), cage’s vertices are moved away from the mesh alongthe normal to ensure no intersection between the mesh and thecage. First, we apply a uniform scaling to the upper part of theface and a double uniform one to the lower part due to smallerand tightener number of handles on the jaws. Then, based on theirposition on the face, a specific non-uniform adjustment scaling isapplied to some vertices. For example, nose handles are only scaledaccording to z-axis (towards the image plane) to keep them in themiddle of the face. To respect (3), two new vertices are positionedat the back of the head of the model. GC are meant to affect allthe vertices within the cage. For instance if the upper lip handle ismoved up to open the mouth, and the lower lip handle is moveddown, vertices of the lower lip are moved relatively to the newposition of those of the upper lip. The expected behavior is thatthey should be independent. To tackle this, two cages are built, onefor the upper part of the head and one for the lower one.
Neutral Cages outerBrowRaiser lipCornerPuller
Reference ABC
Generic mesh deformers [Sorkine04]
Figure 4: Results of modeling two primary facial actionunits. References are made by an artist.
Figure 4 presents posing results, obtained by a non-expert onthree different facial meshes, compared to a professional artist (toprow). Only parts with movable handles (not the ones added to closethe cage) are displayed. When a handle is grabbed, the associatedfat pad is colored in light blue and the deformation center with a redsphere (actual handle’s position on the pad). The artist used genericmesh deformers [22] whereas the non-expert has used our Fat Padcage. The three meshes were obtained through a digital doublecreation pipeline [6]. All meshes have the same mesh topologyallowing the transfer of the template fat pads map. We selected twofundamental action units: AU-2 (Outer Brow Raiser) and AU-12 (LipCorner Puller). They are involved in many facial human expressions rxiv.org, October, 2020 Colas et al. (e.g. Happiness, Sadness, Fear etc.). Heat maps show differencesbetween the initial mesh and the final modeling. Dark blue is static.We notice how close to the ground-truth the deformations are. Italso emphasizes that our new Fat Pad cage prevents undesiredmesh deformation due to global deformation (recomputation ofthe whole mesh when a cage vertex is moved) and ensures smoothdeformation at the borders of the pads. With our cage, most ofthe mesh remains static, deformations only occur in the specificlocations the user wanted to model. Sadness Disgust reference reference
Figure 5: Example of user results. From left to right: sadness(GC | ref | Fat Pad) and disgust (GC | ref | Fat Pad).
A user study was conducted with two cage systems: GC and Fat Pad.Two expressions were used as reference to reproduce: "sadness"and "disgust". With each cage participants had to realize the twoexpressions, leading to four tasks. We have chosen to compare ourmethod to a single GC cage as used in the literature, and because atwo-parts GC cage leads to strong artifacts around the mouth. Bydesign, a local GC cage moves the whole mesh. Objective measureswere time, number of handles’ manipulation, number of undos, andthe root mean square (RMS) of the Hausdorff distances betweeneach participant’s result and the reference model [1]. Subjectivemeasures were the following questionnaire evaluated on a 5-pointLikert scale, from 1 (totally disagree) to 5 (totally agree): [Q1] Theresult is similar to the model, [Q2] The task was easy, [Q3] Themanipulation was easy, [Q4] I am happy with my result. Participantshad two screens, a front one displaying the interactive UI and a sideone showing the reference meshes. It started with an explorationphase of 10 min to get used to the UI. The four tasks were thenrandomly presented. They stopped whenever they were satisfiedwith the result. Total duration was about 40 min. 17 participantstook part in the study (age 𝑥 = . 𝜎 = . 𝑥 = . 𝜎 = .
069 on a Likertscale 1 to 5) but no expertise in 3D modeling ( 𝑥 = . 𝜎 = . 𝑝 = . 𝑝 = . 𝑒 − for disgust). However,no significant differences were found for other objective metrics.Figure 6 shows answers to questionnaire. In the case of the sadnesstask, we did not observe statistical differences between the two SadnessDisgust
Figure 6: Left: mean of the RMS of the Hausdorff distancebetween meshes. The lower the better. Right: answers forsadness (up) and disgust (bottom). The higher the better. cages ( 𝑝 > . 𝑝 = . 𝑝 = . 𝑝 = .
03, and Q4: 𝑝 = . We presented Fat Pad cages, a first combination between cage-based deformation and facial anatomical model. The paper hasdescribed three main contributions: the Fat Pads concept enablingan interactive mesh deformation that respects facial anatomicalconstraints, a new automatic way of creating personalized cagesfor any facial mesh, and a user study validating the high interestand preference of Fat Pad cages. The new filter function appears tobe more suitable to pose faces than Green Coordinates. It preventsglobal deformation and ensures smooth deformation at the bordersof the pads. The generated cages closely fit the shape of the meshand can be considered as an adaptive extension of the head. Theuser study confirmed the interest of our approach and providedvaluable insight to improve our system. For instance, allowing thesymmetrical control of the handles would be necessary to speed upthe design of expressions. In a longer perspective, we want to focusour work to include machine learning to improve facial posing.Acquiring and analyzing facial fat motion would allow to betterdefine pad’s shape and regions of impact. We also want to extendthe concept of multiple cages to other facial parts while keepingmaking facial posing easier and more accessible to non experts.
REFERENCES [1] Nicolas Aspert, Diego Santa-cruz, and Touradj Ebrahimi. 2002. MESH: MeasuringErrors between Surfaces Using the Hausdorff Distance.
Proceedings of the IEEEInternational Conference in Multimedia and Expo (ICME)
ACM Trans. Graph
32, 4 (2013), 40:1–40:10.[3] Xue Chen and Jieqing Feng. 2014. Adaptive Skeleton-driven Cages for MeshSequences.
Comput. Animat. Virtual Worlds
25, 3-4 (May 2014), 447–455. https://doi.org/10.1002/cav.15774 at Pad Cages for Facial Posing arxiv.org, October, 2020 [4] Matthew Cong, Michael Bao, Jane L. E, Kiran S. Bhat, and Ronald Fedkiw. 2015.Fully automatic generation of anatomical face simulation models. In
ACM Sym-posium on Computer Animation . 175–183.[5] Matthew Cong, Kiran S. Bhat, and Ronald Fedkiw. 2016. Art-directed muscle sim-ulation for high-end facial animation. In
ACM Symposium on Computer Animation .119–127.[6] F. Danieau, I. Gubins, N. Olivier, O. Dumas, B. Denis, T. Lopez, N. Mollet, B. Frager,and Q. Avril. 2019. Automatic Generation and Stylization of 3D Facial Rigs. In . 784–792.https://doi.org/10.1109/VR.2019.8798208[7] Michael S. Floater. 2003. Mean Value Coordinates.
Comput. Aided Geom. Des.
ACM Trans. Graph
36, 4(2017), 153:1–153:14. https://doi.org/10.1145/3072959.3073664[9] Alec Jacobson, Zhigang Deng, Ladislav Kavan, and J. P. Lewis. 2014. Skinning:Real-time Shape Deformation. In
ACM SIGGRAPH 2014 Courses (Vancouver,Canada) (SIGGRAPH ’14) . ACM, Article 24, 24:1–24:1 pages. https://doi.org/10.1145/2614028.2615427[10] Pushkar Joshi, Mark Meyer, Tony DeRose, Brian Green, and Tom Sanocki. 2007.Harmonic Coordinates for Character Articulation.
ACM Trans. Graph.
26, 3,Article 71 (July 2007). https://doi.org/10.1145/1276377.1276466[11] Pushkar Joshi, Wen Tien, Mathieu Desbrun, and Frederic Pighin. 2003. LearningControls for Blend Shape Based Realistic Facial Animation. In
SIGGRAPH ’06ACM SIGGRAPH 2006 Courses . 187–192. https://doi.org/10.1145/1185657.1185857[12] Kolja Kähler, Jörg Haber, and Hans-Peter Seidel. 2001. Geometry-based MuscleModeling for Facial Animation. In
Proceedings of Graphics Interface 2001 (Ottawa,Ontario, Canada) (GI ’01) . Canadian Information Processing Society, 37–46. http://dl.acm.org/citation.cfm?id=780986.780992[13] Yuencheng Lee, Demetri Terzopoulos, and Keith Waters. 1995. Realistic Modelingfor Facial Animation. In
Proceedings of the 22nd Annual Conference on ComputerGraphics and Interactive Techniques (SIGGRAPH ’95) . ACM, New York, NY, USA,55–62. https://doi.org/10.1145/218380.218407[14] John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin,and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models.
Eurographics (State of the Art Reports)
1, 8 (2014), 2. https://doi.org/10.2312/egst.20141042[15] J. P. Lewis and Ken-ichi Anjyo. 2010. Direct Manipulation Blendshapes.
IEEEComput. Graph. Appl.
30, 4 (July 2010), 42–50. https://doi.org/10.1109/MCG.2010. 41[16] Yaron Lipman, David Levin, and Daniel Cohen-Or. 2008. Green Coordinates.
ACM Trans. Graph.
27, 3, Article 78 (Aug. 2008), 10 pages. https://doi.org/10.1145/1360612.1360677[17] Jesús R Nieto and Antonio Susín. 2013. Cage based deformations: a survey. In
Deformation models . Springer, 75–99. https://doi.org/10.1007/978-94-007-5446-1_3[18] Verónica Orvalho, Pedro Bastos, Frederic I Parke, Bruno Oliveira, and XenxoAlvarez. 2012. A Facial Rigging Survey. , 183–204 pages. https://doi.org/10.2312/conf/EG2012/stars/183-204[19] Leonardo Sacht, Etienne Vouga, and Alec Jacobson. 2015. Nested Cages.
ACMTrans. Graph.
34, 6, Article 170 (Oct. 2015), 14 pages. https://doi.org/10.1145/2816795.2818093[20] Jaewoo Seo, Geoffrey Irving, J. P. Lewis, and Junyong Noh. 2011. Compressionand Direct Manipulation of Complex Blendshape Models.
ACM Trans. Graph.
ACMTrans. Graph
24, 3 (2005), 417–425.[22] O. Sorkine, D. Cohen-Or, M. Alexa, C. Rössl, and H.-P. Seidel. 2004. Laplaciansurface editing. (2004), 179–188.[23] Vitaly Surazhsky, Tatiana Surazhsky, Danil Kirsanov, Steven J. Gortler, andHugues Hoppe. 2005. Fast Exact and Approximate Geodesics on Meshes.
ACMTrans. Graph.
24, 3 (July 2005), 553–560. https://doi.org/10.1145/1073204.1073228[24] J. Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive Region-based Linear 3D Face Models.
ACM Trans. Graph.
30, 4, Article 76 (July 2011),10 pages. https://doi.org/10.1145/2010324.1964971[25] Keith Waters. 1987. A Muscle Model for Animation Three-dimensional FacialExpression.
SIGGRAPH Comput. Graph.
21, 4 (Aug. 1987), 17–24. https://doi.org/10.1145/37402.37405[26] Chuhua Xian, Hongwei Lin, and Shuming Gao. 2012. Automatic Cage Generationby Improved OBBs for Mesh Deformation.
Vis. Comput.
28, 1 (Jan. 2012), 21–33.https://doi.org/10.1007/s00371-011-0595-6[27] Xiaosong Yang, Jian Chang, Richard Southern, and Jian J. Zhang. 2013. AutomaticCage Construction for Retargeted Muscle Fitting.
Vis. Comput.
29, 5 (May 2013),369–380. https://doi.org/10.1007/s00371-012-0739-3[28] Uldis Zarins. 2017.