Functionality-Driven Musculature Retargeting
Hoseok Ryu, Minseok Kim, Seunghwan Lee, Moon Seok Park, Kyoungmin Lee, Jehee Lee
FFunctionality-Driven Musculature Retargeting
Hoseok Ryu , Minseok Kim , Seungwhan Lee , Moon Seok Park , Kyoungmin Lee , Jehee Lee Seoul National University, Korea Seoul National University Bundang Hospital, KoreaFigure 1: Various musculoskeletal models of different body sizes and proportions. The full-body musculature of the reference model onthe left was retargeted to the others. (Left to right) Musculoskeletal model : reference model, Hulk, Dwarf, and Alien.
Abstract
We present a novel retargeting algorithm that transfers the mus-culature of a reference anatomical model to new bodies with dif-ferent sizes, body proportions, muscle capability, and joint rangeof motion while preserving the functionality of the original mus-culature as closely as possible. The geometric configuration andphysiological parameters of musculotendon units are estimatedand optimized to adapt to new bodies. The range of motion aroundjoints is estimated from a motion capture dataset and edited fur-ther for individual models. The retargeted model is simulation-ready, so we can physically simulate muscle-actuated motor skillswith the model. Our system is capable of generating a wide va-riety of anatomical bodies that can be simulated to walk, run,jump and dance while maintaining balance under gravity. We willalso demonstrate the construction of individualized musculoskele-tal models from bi-planar X-ray images and medical examinations.
The human body in computer graphics has evolved from a stickfigure with torque actuators to a muscle-driven skeleton. The mus-culoskeletal model for physics-based simulation includes manyparts and parameters that require careful orchestration to make itanatomically plausible. Designing such a musculoskeletal modelis labor-intensive and hand-crafted models are often inaccurate toactuate an anatomical body in physically simulated environments.We present a new method that retargets a reference musculoskele-tal model to new bodies of different sizes, body proportions, mus- cle capability, and joint Range of Motion (ROM).The musculature is a system of muscles and tendons actuatingthe skeleton. The motion of the skeleton is the result of the harmo-nious coordination of many muscles. Muscles are correlated witheach other in various ways. A group of muscles works together toactuate a single joint and the ROM around the joint is also deter-mined by multiple adjacent muscles. Some adjacent muscles sharea tendon and thus the contraction of one muscle may affect theactivation of the others. A single (multi-articular) muscle may ac-tuate two or more joints simultaneously.The functional role of each individual skeletal muscle is thor-oughly studied and well documented in anatomy textbooks [46].Each muscle is responsible for particular movement (e.g., flex-ion/extension, adduction/abduction, internal/external rotation) ofbody parts. For example, the active contraction (shortening) of therectus femoris (a muscle in the thigh) results in knee extension andhip flexion. Even if the muscle is inactive, its background elasticitygenerates passive force to prevent the adjacent joints from exces-sive movement, which influences the ROM around the joints.Our focus is on adapting the musculature of a referenceanatomical model to new bodies while preserving the function-ality of the musculature. The muscles are expected to perform thesame functional roles and have the same ROMs in the new bodyafter retargeting. The key parameters that affect muscle functionsand joint ROMs are the length of musculotendons and their ge-ometric routing paths. The core of our retargeting algorithm is anumerical solver that optimizes the key parameters throughout theentire body. The algorithm allows us to create a variety of muscu-loskeletal models including exotic creatures.We also implemented a human body editing system that al-1 a r X i v : . [ c s . G R ] A ug ows the user to interactively edit the height, width, bone length,bending and torsion angles, and joint ROMs of the skeleton. Themusculature of the anatomical model is automatically retargeted tofit the new skeletal body. Additionally, the user is allowed to editphysics parameters, such as the mass and inertia of individual bodyparts, and make the body stronger or weaker by adjusting Hill-typemuscle parameters, such as force-length curves and force-velocitycurves. The physics and muscle parameters are exploited in a phys-ically based simulation of muscle-actuated characters.The simulation and control of under-actuated biped locomotionunder gravity have been studied for decades in computer graphicsand robotics. The recent progress in deep reinforcement learningresearch has been very successful in continuous control problemsand made it possible to simulate dynamic human activities withoutcompromising fundamental physics laws [38, 22]. We will demon-strate that our muscle-actuated anatomical model is compatiblewith state-of-the-art simulation algorithms. Our retargeted char-acters with extreme body proportions can be physically simulatedand controlled to walk, run, kick, and jump while maintaining bal-ance under gravity. There has been a stream of studies for reproducing natural hu-man motion by incorporating the increasingly more accurate mod-els of human anatomy and the mechanics of anatomical struc-tures. Muscle-based anatomical modeling and simulation havebeen extensively explored and exploited in Computer Graphicsand Biomechanics.
Muscle Modeling
The anatomical model includes an articulated skeleton driven byits musculature. The Hill-type muscle model [12, 54, 34] has beenbroadly adopted to encode the nonlinear contraction dynamics ofmuscles. The geometry of a muscle is often simplified into a se-quence of line segments for the efficiency of computation. Therehave been continuous efforts in Computer Animation and Biome-chanics to demonstrate the computational plausibility of volu-metric FEM muscles [23, 43, 9, 3].
In vivo estimation of mus-cle parameters relies primarily on medical 3D imaging. Matias etal. [32] studied the estimation of muscle-bone attachments basedon bony landmarks. Levin et al. [28] studied the estimation ofmuscle fiber directions from diffusion tensor images. Arnold etal. [2] and Holzbaur et al. [14] provided a comprehensive refer-ence of muscle modeling parameters for lower and upper limbs.Anatomical modeling for specific body parts, such as face [44,15], feet [37], hands [48, 41], shoulders [50, 18, 33], tongue [47],and jaw [55], has been explored for last two decades. Comprehen-sive full-body musculoskeletal modeling and simulation systemsare also available freely [7] and commercially [5]. Muscle routingplays an important role in muscle-actuated full-body simulation.Many modeling approaches, such as conditional waypoints [6],obstacle sets [10], surface SSD [35], and wrapping surfaces [39],have been exploited to improve the estimate of muscle lengths dur-ing joint motion.
Joint Modeling
The joints of a skeletal model are often simplified as either revolute(1 degree of freedom) or ball-and-socket (3 degrees of freedom)joints, though anatomical joints are structurally more complicatedthan simple mechanical joints [24]. Maurel and Thalman [33] de-signed a skeletal model of human shoulders and represented theflexibility of the ball-and-socket joint by a combination of the si-nus cone and an interval of the arm twist angle. Lee [19] stud-ied a general approach of describing the range of motion arounda joint based on quaternion half-spaces and their boolean combi-nation. Akhter and Black [1] collected a motion capture datasetthat explores a wide range of human poses, and learned the pose-dependent range of motion around each joint. Jiang and Liu [16]represented the boundary of valid human joint configurations byusing a fully-connected neural network. Measuring the joint rangeof motion and muscle lengths are of great clinical interest. A vari-ety of clinical examination techniques are performed frequently inclinical practice [40].
Muscle-Driven Simulation and Control
Recent accomplishments in Computer Animation made it possi-ble to reproduce realistic human motion in physics-based simula-tion [31, 4, 25, 45]. The emergence of deep reinforcement learn-ing accelerated the progress and successfully simulated skillfulactions, such as jump, flip, cartwheel, basketball dribbling, andeven aerobatic flapping flight [38, 30, 53, 52, 22]. Muscle-drivenanatomical simulation poses further challenges of dealing withthe complexity of anatomical modeling and scalability in physics-based simulation. Wang et al. [51] demonstrated a muscle-actuatedbiped with eight musculotendon units on each leg. The biped wasessentially two-dimensional because all muscles are aligned inthe sagittal plane. Geijtenbeek et al. [11] improved the flexibil-ity in biped design by allowing off-sagittal muscles and optimiz-ing muscle-attachments sites for locomotion control. Comprehen-sive 3D musculoskeletal models with up to 120 musculotendonunits were successfully simulated and controlled using a two-levelcontrol architecture that consists of low-level muscle coordinationbased on quadratic programming and higher-level gait modulationbased on stochastic optimization [26]. Lee et al. [21] demonstratedthat volumetric FEM muscles are viable for dextrous manipula-tion, such as juggling, that requires a high level of control pre-cision. Nakada et al. [36] built a comprehensive full-body neu-romuscular system to learn biomimetic sensorimotor control ofhuman animation. The muscle-actuated control system proposedby Lee et al. [22] employed a hierarchical deep reinforcementlearning algorithm to cope with the full details (46 joints and 346muscles) of the musculoskeletal model and successfully simulateddynamic motor skills. They also demonstrated pre-operative andpost-operative simulation of pathologic gaits. We built our mus-culoskeletal model on top of their simulation system, which pro-vides favorable features such as analytic differentiation of muscle-actuated dynamic systems, muscle routing based on LBS (linearblend skinning), and efficient handling of joint ROMs based onLCP (linear complementary problem).2igure 2: Our reference musculoskeletal model.
Anatomy Transfer and Retargeting
Constructing a high-fidelity anatomical model is time-consumingand labor-intensive. Therefore, there has been a series of stud-ies that retarget a reference anatomical model to a range of tar-get bodies. Dicko et al. [8] transferred the geometry of internalanatomy from an input template to a target body while maintaininganatomic constraints. Saito et al. [42] generated a spectrum of hu-man body shapes with various degrees of muscle growth. Kadleˇceket al. [17] generated a personalized anatomic model by retargetinga template model to fit full-body 3D scans while accounting forbody shape variations and pose variations. Our work is on the lineof these approaches with emphasis that our retargeted model is op-timized to function as closely as possible to its reference model inthe realm of muscle-driven simulation and control.
Our musculoskeletal model has an articulated skeleton with 5revolute joints, 13 ball-and-socket joints, and 6 DoFs (Degrees ofFreedom) at the skeletal root yielding 50 DoFs in total (see Fig-ure 2). Our work is focusing on simulating human motion drivenby four limbs and spinal joints. The model includes 282 musculo-tendon units corresponding to skeletal muscles, which are attachedto bones on each end by tendons. The attachment site on the prox-imal end is called the origin of the muscle, while the attachmentsite on the opposite end is called its insertion . Muscle contractionpulls the attached bones and moves the joint between them. Ourmodel comprehensively includes skeletal muscles that contributeto the motion in any limb joints. Since our model has no fingers ortoes, the muscles that originate and insert within hands or feet areomitted.
The Hill-type muscle is a three-element scheme that modelsthe muscle contraction-force relation (see Figure 3). A musculo-tendon unit is defined by an active contractile element (CE), a pas-sive parallel element (PE), and a passive serial element (SE). The Figure 3: Hill-type model of musculotendon units. The force-length curve plots maximum isometric force of the active fiber andpassive fiber force as a function of normalized muscle fiber length.active force of the contractile element comes from muscular con-traction. All skeletal muscles have a rest length. When the muscleis stretched to its ideal length, it can maximize muscular contrac-tion. The serial element represents the tendons on either side of thecontractile segment of the muscle. The tendon is a tough band offibrous connective tissue that connects muscle to bone. The paral-lel element models the background elasticity of the muscle. Theparallel element is responsible for muscle passive behavior whenit is stretched.Let l m , l t and l mt be muscle fiber length, tendon length, and thetotal muscle-tendon length, respectively. Optimal fiber length l m is the length of muscle fibers when it develops maximum isomet-ric force. Let l t be tendon slack length. Pennation θ is the anglebetween the muscle fibers and the tendon axis. Assuming a quasi-static setting, the tension in muscle fibers is a function of musclelength and its activation a , while the tension in the tendon is afunction of only muscle length. The tension in the three elementsholds F = (cid:0) f CE (˜ l m , a ) + f PE (˜ l m ) (cid:1) · cos θ = f SE (˜ l t ) , (1)where ˜ l m = l m /l m and ˜ l t = l t /l t are normalized muscle and ten-don lengths, respectively. Here, we assume that the pennation an-gle is constant regardless of muscle contraction.Actual muscles wrap around bones and soft tissues during jointmotion and they are also constrained by surrounding ligaments andskin tissue layers. Therefore, the straight line segment between theorigin and insertion sites of a muscle is a lousy approximator of itsactual length. Better approximators can be achieved by exploitinggeometric proxies, such as waypoints [6], obstacle sets [10], andwrapping surfaces [39], that approximate muscle deformation dur-ing joint motion. We use waypoints as geometric proxies and ex-pressed the location of a waypoint relative to nearby bones usingthe idea of linear blend skinning (LBS) to improve the accuracyand flexibility over bone-attached waypoints (see Figure 4). p = (cid:88) j w j T j x j (2)where T j ∈ R × is the transformation matrix of j -th bone, w j is skinning weight, and x j ∈ R is the coordinates relative to j -th bone. Since the derivatives of LBS-based waypoints can be3igure 4: Polyline approximation of (Top) Vastus Intermedius and(Bottom) Gluteus Maximus. Since the waypoints are representedin LBS coordinates, they move during knee/hip flexion to better fitmuscle deformation.derived in an analytic form, they are well-suited for efficient dy-namics simulation with inequality constraints [22]. The skeletal body is a parametric model that can generate awide variety of human body shapes (see Figure 5). The model hasa reference shape and a set of shape parameters. The trunk and fourlimbs are parameterized to have variations in their length, size, andalignment relative to the reference shape. The limb bones includ-ing Femur, Tibia, Humerus, Ulna exhibit an elongated shape withits proximal head, long shaft, and distal head. The limb bone hasfour parameters: proximal head scale, distal head scale, shaft elon-gation, and torsion. Even though the trunk consists of many bones,the trunk is considered as a lumped body and parameterized bythree parameters: elongation, expand, and bend. The hands andfeet have one parameter for scaling their size. We use a geometricdeformation method [17] to produce parametrically-varied shapesfrom the reference model.We scaled physics and muscle parameters relative to geomet-ric parameters inspired by the work of Hodgins and Pollard [13].Assuming uniform scaling by factor L in all dimensions, they sug-gested to scale time, force, mass, moment of inertia, velocity, stiff-ness, and damping by factors L / , L , L , L , L / , L , and L / , respectively. This scaling rule provides a rough guideline asto how physics and muscle parameters should be adjusted relativeto body scaling. The user can edit the mass and inertia of each in-dividual body part and scale the force-length curve of each musclefollowing the guideline. The guideline is not strict since stable dy-namics simulation can be often accomplished for a wide range ofparameters. Figure 5: Parametric modeling of the trunk and limbs. Many articulated characters in computer graphics have theROM around joints described by an interval of min/max angles foreach DoF. This approach is inherently limited because the pose-dependence of joint ROMs cannot be captured in per-DoF inter-vals. An alternative approach is to estimate a high-dimensionalfunction from a motion capture dataset, which fits the boundaryof valid poses. The isValid function V ( q ) : R N → { , } takesa full-body pose as input and outputs a boolean value to answerwhether the pose is valid or not. The function estimation requiresnonlinear regression over a large collection of full-body poses [1,16]. The musculoskeletal system readily provides an isValid func-tion without requiring nonlinear regression because the muscu-loskeleton represents the fundamental mechanism of joint move-ments and their limits. The joint ROM is affected by many anatomic factors, such asbony structures, surrounding ligaments, joint capsules, and mus-cular tension. We are particularly interested in the factors inducedby muscles. The elasticity of muscle fibers represented by a par-allel element determines the distance a joint can move to its fullpotential in the direction of the muscle. This extension of musclestretching is described by an implicit function C i ( q ) : R N → R ,where q is the generalized coordinates of the full-body skeletalpose, and N is its dimension. The constraint induced by passivefiber tension is C i ( q ) = k m l − l m ( q ) , (3)where muscle length l m ( q ) is a function of skeletal pose. Notethat there exists a direct mapping between skeletal poses and mus-cle lengths. Given a pose, the lengths of all muscles are uniquelydetermined and vice versa. We assume that passive elongationreaches its limit when muscle tension is larger than maximumisometric tension in muscle fibers by a certain margin (see Fig-ure 3(right)). Deriving from muscle contraction dynamics [49], weset the maximal muscle-tendon length by l maxmt = k m l + k t l , (4)where k m = 1 . and k t = 1 . . We can also express other fac-tors induced by bones and ligaments as implicit functions (e.g., the4igure 6: Full-body poses that maximize individual musclelengths. (Left) Right Bicep Brachii Short Head. (Right) Left PsoasMajor.constraint by the kneecap can be denoted by a function of skeletalpose). The collection of all constraints defines an isValid function. V ( q ) = (cid:26) , if C i ( q ) ≥ ∀ i , otherwise (5)It is often the case that multiple muscles cross a joint and allthose muscles affect the ROM in the joint. Among those muscles,some are multi-articular meaning that it crosses multiple joints.The presence of multi-articular muscles makes joint ROM pose-dependent. For example, we have a wider range of motion in thehip when the knee is flexed than it is fully extended. Our isValid function based on the musculoskeletal model inherently capturesthe pose-dependence of ROM. Even though carefully specifying muscle parameters wouldyield appropriate joint ROMs, parameter tuning is laborious andtime-consuming. We estimate muscle parameters (in particular,muscle fiber and tendon lengths) automatically such that the is-Valid function fits the boundary of full-body poses captured in amotion dataset. Specifically, we used a data set collected by Akhterand Black [1], which includes an extensive variety of stretchingand yoga poses performed by trained athletes and gymnasts. Inthe dataset, we can find a full-body pose that maximizes the length l mt for each muscle (see Figure 6). Assuming that muscle fiber andtendon length ratios are invariant, solving Equation (1) and Equa-tion (4) gives us optimal fiber length l and tendon slack length l at the maximal length l maxmt . This estimation of muscle lengthsguarantees that all poses in the dataset are valid.The dataset actually provides us with the lower bound for themaximal length of individual muscles and therefore it tends tounder-estimate the maximal lengths. For example, the range ofknee extension is bounded by the bone and ligament structuresaround the knee rather than muscle passive force. It is highlylikely that knee flexor muscles (e.g., Hamstring and Gastrocne-mius) do not extend to their maximal length when the knee isstraight. Hence, the dataset does not provide a precise estima-tion of the maximal length of the knee flexors. We address thisunder-estimation issue by measuring joint torques at several key-poses, for which muscles should not develop passive force. The Figure 7: Rotation Decomposition.key-poses include standing, zero-gravity, and T-poses. We gradu-ally increase l and l of muscles if they develop passive forceuntil the force magnitude is reduced below a certain threshold. Muscle lengths thus estimated yield an isValid function V ( q ) ,which serves as a reference ROM model. The reference modelrepresents the level of flexibility of athletes, which is far higherthan the flexibility level of average, non-athletic individuals. Wewould like to be able to edit or modify the reference model to cre-ate various individualized ROM functions. ROM editing involvesimproving or reducing the ROM in a particular joint and shiftingthe range. Note that ROM editing is a supplementary step that pro-vides flexible and convenient user control over the modeling andretargeting procedures. The ROM editing step can be skipped if itis not necessary.Since V ( q ) is a high-dimensional function defined implicitlydepending on many factors and complex anatomical structures, itis not practical to edit the function directly. Alternatively, we de-fine a new, modified function ˜ V ( q ) = V ( T ( q )) indirectly withtransformation T over the input pose. The action of T affects theROM inversely. For example, if the transformation exaggeratesposes, the ROM with the modified function ˜ V becomes narrowerthan the reference model because the test pose will be exaggeratedfirst and then passed to the reference isValid test V .We can define the transformation for each joint or each DoFindependently. For a revolute joint, the ROM is simply an interval [ φ − ψ, φ + ψ ] , where φ is the average of the joint angles in thedataset. The transformation T ( θ ) = s ( θ − φ ) + φ + t of jointangle θ in pose q results in s -scaling of the interval followed by t -shifting. Note that we do not need to know the value of ψ todefine the transformation. Similarly, we can define ROM-editingtransformations for ball-and-socket joints. The configuration of aball-and-socket joint is a three-dimensional rotation, which can bedecomposed into a twist about the principle (bone shaft) axis inthe reference pose followed by another rotation about its orthog-onal axis (see Figure 7). The decomposed configuration is ( ω, ˆ v ) ,where ω ∈ R is a twist angle and ˆ v ∈ S is a unit vector indicatingthe direction of the bone shaft. The ROM of ω is a simple interval,while the ROM of ˆ v is a cone with its center direction estimatedfrom the dataset. The scaling and shifting transformation on thecone is also defined in a straightforward manner and can be ex-ploited for ROM editing. Through simple user interfaces, the usercan edit ROMs interactively by modulating scale and shift factors.5able 1: Examples of muscle activities during joint motion. Thefull version with details can be found in anatomy textbooks [46].*Biarticular (two-joint) muscles. Joint Motion Active Muscles
Hip Flexor Psoas major, Iliacus, Rectus femoris*,Tensor fasciae latae*, Sartorius*Hip Extensor Gluteus maximus, Biceps femoris*,Semitendinosus*, Semimembranosus*Hip Gluteus medius, Pectineus, Tensor fasciae latae*Medial RotatorHip Quadratus femoris, Obtrator externus,Lateral Rotator Biceps femoris*Hip Abductor Gluteus minimus & medius,Tensor fasciae latae*Hip Adductor Adductor longus & brevis & magnusKnee Extensor Vastus medialis & intermedius & laterlias,Rectus femoris*Knee Flexor Sartorius*, Biceps femoris*, Semitendinosus*,Semimembranosus*, Gastrocnemius*, Plantaris*Foot Gastrocnemius*, Plantaris*, Soleus,Plantarflexor Tibialis posteriorFoot Tibialis anterior, Extensor hallucis longus,Dorsiflexor Extensor digitorum longus, Fibularis tertius
In this section, we describe a retargeting algorithm that adaptsthe full-body musculature to the change of the skeleton (see Fig-ure 8). Our parametric skeletal model can generate a rich variety ofhuman body shapes. The goal of retargeting is to make the variedmodels viable for physics-based simulation and control.The functional roles of skeletal muscles are classified by theplot of their length changes and the direction of muscle forces act-ing on the attached bones. For example, a muscle is called a flexor(or extensor) of the joint if the muscle shortens as the joint flexes(or extends). Table 1 shows examples of muscle actions duringvarious joint motions. If the muscle is functionally significant forcausing a particular joint motion (either as an agonist or an antag-onist), its length-angle curve is monotonically increasing, mono-tonically decreasing, or close to unimodal (see Figure 9). There-fore, the length-angle curves characterize the functional roles ofmuscles. Na¨ıve scaling of the musculoskeletal model could alterthe monotonicity and modality of the length-angle curves, mus-cle force directions at attachment sites and joint ROMs, eventuallyaffecting the functionality of muscles in the new body.Musculature retargeting involves three categories of parame-ters. The parameters in each category have the desired impacts onmuscle functionality. Our algorithm solves for three categories ofparameters sequentially. • Optimize muscle routing to modulate length-angle curvesand force directions, where the muscle routing is expressedby
LBS coordinates of muscle waypoints . • Determine maximal muscle lengths to achieve desired ROMs(as described in Section 4.2 and Section 4.3). • Optimize muscle fiber and tendon length ratios to modulatethe angle of peak torque at joints.Since we already discussed the second step in the previous section, this section focuses on the first and third steps which are formu-lated as nonlinear optimization.
Waypoints play an important role in estimating muscle lengthsduring joint motion. While the use of LBS coordinates improvesthe estimation accuracy substantially, manual tuning of LBS co-ordinates is laborious. Our algorithm decides LBS coordinates ofwaypoints such that the energy function is minimized: E waypoint = E direction + w l E length . (6)The first term encourages muscle force directions through thepolyline approximations of muscles to be preserved E direction ( m ) = (cid:88) k (cid:88) j (cid:90) θ =0 (cid:107) f m ( k ; j, θ ) × f (cid:48) m ( k ; j, θ ) (cid:107) dθ, (7)where m is the index of muscles, k is the index of waypoints of themuscle m (including the origin and insertion), j is the type of jointmotion in which the muscle m participates, and θ ∈ [0 , is thenormalized joint angle. f and f (cid:48) are normalized force directionsin the reference and retargeted bodies, respectively, at point k to k + 1 .The second term encourages the qualitative characteristics oflength-angle curves to be preserved in the retargeted body. Assum-ing that the length-angle curve L m ( j, θ ) is monotonic or unimodalon interval [0 , , the curve can be characterized by three param-eters: θ max , θ min , ∆ . The curve has its maximum value at θ max ,minimum value at θ min , and the difference between the maximumand minimum value is ∆ (see Figure 10). The second term is de-fined using the characteristics parameters. E length ( m ) = (cid:88) j ( θ max − θ (cid:48) max ) +( θ min − θ (cid:48) min ) + w ∆ (∆ − ∆ (cid:48) ) . (8)We use a gradient descent algorithm to minimize the energy func-tion. The gradient of the energy function is estimated by finite dif-ference and line search along gradient direction accelerates the al-gorithm. In our example, w l = 10 and w ∆ = 50 .The performance and quality of gradient-based optimizationdepend on the initial guess of the optimized parameters. Duringthe scaling of the skeleton, each waypoint moves together with theskeleton anchored to the nearest point on the bone. The anchorposition relative to the skeleton serves as an initial guess of theoptimization. Joint torque is the sum of muscle torques crossing the joint.Peak torque usually occurs in the joint ROM when the musclefibers are close to their optimal length. For example, knee and el-bow joints generate their maximum bending torque when they aremoderately flexed. The angle of peak torque is related to musclefiber and tendon length ratios.Peak torque angle adjustment is a constrained optimizationprocess based on gradient descent. The optimization objective isthat all joints in the retargeted model have their peak torque at the6igure 8: The overview of musculature retargeting.Figure 9: Types of length-angle curves. (Top) The monotonicallydecreasing (or increasing) curve indicate that the muscle is an ag-onist (or antagonist) during the joint motion represented by angle θ . The Non-monotonic curve indicates that the muscle is maybefunctionally insignificant for the particular joint motion. (Bottom)Examples of hip flexors and hip extensors.same angles as in the reference model. The optimization param-eters are fiber-tendon length ratios at all muscles. The ratios areconstrained to vary within from their original values. E apt = (cid:88) j || θ ref ( j ) − argmax θ τ ( j, θ ) || , (9)where θ ref is the angle of peak torque in the reference model and τ ( j, θ ) is the net torque in joint motion j when its normalizedangle is θ . We can generate a wide spectrum of body proportions throughinteractive user interfaces. The user interface provides a set of slide Figure 10: Muscle retargeting. (Red) The length-angle curve ofBiceps Brachii Short Head during elbow flexion. (Blue) shortening of Humerus and Ulna/Radius yields a length-anglecurve that is non-monotonic. (Green) Muscle retargeting makesthe curve closer to its original shape.bars to manipulate body parameters and allows the user to editjoint ROMs. In this section, we show experiments conducted withfour representative characters:
Human , Hulk , Dwarf , and
Alien (see Figure 1 and Table 2). The Human is a reference model wecreated based on a human skeleton geometry. We annotated ori-gins, insertions, and waypoints of all muscles and tuned their pa-rameters to create a model that can be used for physics-based sim-ulation. The Hulk has a large muscular body with long arms andshort legs and the trunk leans slightly forward. The Alien has ashort torso and extremely long limbs, while the Dwarf is the oppo-site with a long torso and short limbs. The musculature of the Hu-man is retargeted to the others to build musculoskeletal dynamicsystems.
We constructed the ROMs of our reference model from a mo-tion capture data set [1]. We excluded corrupted outliers from thedataset, doubled the data by mirror reflection, and subsampled at a1/10 ratio to take approximately 70,000 full-body poses. The mir-ror reflection makes the ROMs symmetric. The maximum lengthsof all muscles are estimated from the pose set and further relaxed7able 2: Skeleton scaling parameters. Positive angle denotesclock-wise torsion. Left and Right sides are symmetrically ad-justed.
Reference Alien Hulk DwarfTrunk Elongate 1.0 1.0 1.0 2.0Expand 1.0 1.0 2.0 0.6Mass 1.0 1.0 1.0 2.0Femur Elongate 1.0 2.5 0.8 0.6Torsion . ◦ . ◦ . ◦ − . ◦ Mass 1.0 2.5 0.8 0.6Tibia Elongate 1.0 2.5 0.8 0.6Torsion . ◦ . ◦ . ◦ . ◦ Mass 1.0 2.5 0.8 0.6Humerus Elongate 1.0 2.5 1.8 0.6Torsion . ◦ − . ◦ . ◦ . ◦ Mass 1.0 2.5 1.8 0.6Ulna Elongate 1.0 2.5 1.4 0.6Torsion . ◦ . ◦ . ◦ . ◦ Mass 1.0 2.5 1.4 0.6Neck Elongate 1.0 2.0 1.0 1.0Mass 1.0 2.0 1.0 1.0 to address the under-estimation issue as explained in Section 4.2.Muscle lengths thus estimated represent pose-dependent joint lim-its. Figure 11 shows that the hip ROM is wider when the knee isflexed than it is straight. This observation is due to the presenceof biarticular muscles. The hamstring muscles (Biceps femoris,Semitendinosus, and Semimembranosus) are all biarticular, origi-nating at the bottom of the Pelvis, crossing both the hip and kneejoints, and inserting at the head of the Tibia. The hamstring mus-cles therefore involved in both hip extension and knee flexion. Thehamstring muscles are extended when the knee is straight and thusthe potential for the hip flexion is limited.The ROM around a joint is strongly influenced by the change ofthe body shape. Figure 12 shows how the longitudinal elongation(in the range of 70% to 130% of the original length) and torsion(in the range of -30 ◦ to 30 ◦ ) of the femur affect the hip ROM.Na¨ıve linear scaling of the muscles attached to the femur resultsin undesirable changes in the hip ROM, which is discretized into × × cells. Each cell has a boolean value. We measured theerror rate by the percentage of false positives and true negatives inthe cells. Without retargeting, the error rates are in the range of5.8% to 75.4%. The maximum error drops down to 1.7% afterretargeting.Our system provides a simple user interface that allows the userto edit the ROM of any joint interactively. The user can make theROM wider, narrower, or shifted on the space of conic and twistrotations. The editing operations are defined by transformationsas explained in Section 4.3. Our system recalculates the musclesat relevant cites such that the muscle-induced ROM matches theuser’s intention approximately. In Figure 13, the hip ROM tiltedforward by 30 ◦ and shrank in its width on the cone by a factorof 0.63 by user-specified transformations. Recalculating muscleslengths approximates the user editing in the error rate of 7.3%. We found that LBS-based waypoints work noticeably betterthan fixed-anchor type waypoints in the previous studies [11, 26].Nonetheless, transplanting the musculature of the reference model Figure 11: The pose-dependent ROM of the hip joint. (Top) Validhip configurations captured in the reference dataset. The range onthe left figures (the knee angles smaller than 30-degree) is nar-rower than the range on the right figures (the knee angles largerthan 90-degree). (Bottom) The muscle-induced hip ROM of ourmusculoskeletal model. The hip configuration is decomposed intoa twist angle and the direction of the bone shaft. The bone shaft di-rection is depicted as a yellow-red range on the unit sphere at thehip. The color spectrum indicates the range of the twist angle. Theyellow color represents narrow ranges, while the red color repre-sents wider ranges.to varied skeletons without retargeting often results in undesirabledistortion in length-angle curves and disturbance in muscle forcedirections. Figure 14 shows the length-angle curves of the refer-ence and varied models with and without retargeting. The quali-tative and quantitative changes of muscle curves can be observedin many major muscles (e.g., Iliacus for the Hulk and the Alien,and Gluteus Medius of the Dwarf in the lower limbs). The influ-ence on force direction is broader. The force direction at almost allmuscle origins and insertions are disturbed by the change of bodyproportions(see Figure 16). Both curve distortions and force direc-tion disturbances affect the maneuverability and ROM of joints.Our muscle routing optimization addresses both problems simul-taneously to build a retargeted model that functions as closely aspossible to the reference model.Figure 15 shows that many muscles change their functional roleas the limbs lengthen or shorten in the range of 60% to 250%of their original length. Each value indicates the percentage offunctional disorders (increasing-to-decreasing and vice versa) inthe length-angle curve during joint motion with and without re-targeting. Some muscles (e.g., Tibialis Anterior, Tibialis Poste-rior, Pectineus, and Semimembranosus) change their function withshorter limbs, while functional disorders occurs for some muscles(e.g., Rectus Femoris and Vastus muscles) with longer limbs. Forexample, the functional disorders for Pectineus muscles occur withshorter (60% of its original length) legs in the half of the range ofhip extension and flexion. Our retargeting algorithm alleviate thisissue substantially with mild body shape variations, though theproblem is not completely fixed with extreme body proportions8igure 12: The hip ROM affected by the elongation and torsion ofthe femur. (TOP) Na¨ıve linear scaling of the muscles results in un-desirable changes in the ROM. (Bottom) Our retargeting algorithmprecisely recalculates the muscle lengths to preserve the ROM ofthe reference model regardless of body shape variations.Figure 13: Interactive ROM editing. (Top) The estimated ROMof the hip joint. (Middle) The user edits the ROM interactively bydefining transformations that make the ROM shift forward and nar-rower in its width. (Bottom) Recalculating muscle lengths matchthe muscle-induced ROM to the user’s intention. (e.g., longer limbs by 250%).
We tested the usability of our retargeted models in our muscle-actuated simulation system. We briefly summarize the dynam-ics formulation and its control methods based on Quadratic Pro-gramming (QP). The Lagrangian dynamics of the musculoskeletalmodel is defined by M ( q )¨ q + c ( q , ˙ q ) = (cid:88) m J (cid:62) m f m ( a m ) + J (cid:62) c f c + τ e xt (10)where q , M ( q ) , and c ( q , ˙ q ) are generalized coordinates, the massmatrix, and Coriolis/centrifugal force, respectively. f m and f c aremuscle and constraint forces, respectively. Jacobian matrices J m and J c map the forces to generalized coordinates. τ e xt is externalforces. We compute the constraint forces by solving linear com-plementary conditions: v c ≥ , f c ≥ , and v (cid:62) c f c = 0 , (11)where v c = J c ˙ q is the constraint velocity representing the rate ofchange of the constraint.Exercising a particular joint is an under-specified control prob-lem, since there are more muscles crossing the joint than mini-mally required to actuate the joint. Solving the control problem re-quires muscle coordination to decide activation levels at the mus-cles that achieve desired joint accelerations ¨ q d . A typical solutionmethod based on QP formulates the problem such thatm in a (cid:107) ¨ q d − ¨ q ( a ) (cid:107) + w r eg (cid:107) a (cid:107) subject to M ¨ q + c = (cid:88) m J (cid:62) m f m ( a m ) + J (cid:62) c f c + τ e xt ≤ a i ≤ f or ∀ i, (12)where a is a vector concatenating muscle activations. This for-mulation solves for muscle activations while satisfying the La-grangian equation of motion and regularizing large activations.The formulation allows us to move the joint in an arbitrarydirection by specifying desired joint accelerations if the muscu-loskeletal model is carefully calibrated. The QP with a variedmodel without retargeting often fails to solve for muscle activa-tions because force directions are degenerate and unable to spanthe solution space. Our retargeting algorithm recovers the forcedirections to better condition the problem (see Figure 16). The re-sults may be best viewed in the supplementary video. Our characters also learned to perform challenging full-bodymotor skills. Deep Reinforcement Learning has recently shown itspromise in the control of physically-simulated bipeds and muscle-actuated skeletal figures. The goal of trajectory tracking controlis to learn a control policy that imitates reference motion data inphysics simulation. It has been shown in previous studies that thecontrol policy represented by deep networks can reproduce highly-detailed human movements [38, 22].9igure 14: Musculature retargeting experiments with Hulk, Alien, and Dwarf characters. Length-angle curves (top three rows) of thereference model and the varied models with/without retargeting are plotted for comparison. (Top) Iliacus during hip flexion. (Second row)Gastrocnemius during knee flexion. (Third row) Gluteus Medius during hip medial rotation. (Bottom) The torque-angle curve of the kneeshows the angle of peak torque.Let s t be the dynamic state of all musculotendon units at time t . The state includes muscle lengths and the rate of their lengthchanges. Let a t be the level of muscle activations and r t be a re-ward of taking action a t at state s t . The agent interacts with theenvironment by taking actions according to its control policy π θ .The objective of policy learning is to find a control policy thatmaximizes the expected cumulative rewards θ ∗ = arg max θ E s , a , s , ··· (cid:20)(cid:88) t =0 γ t r t (cid:21) , (13)where γ is a discount factor.The reward evaluates how well joint angles and end-effectorpositions match the reference trajectory. The reward r = r q r e isdefined by a multiplication of a joint angle match reward and an end-effector match reward, where r q = exp (cid:0) − σ q (cid:88) j (cid:107) ˆ q j ( t ) (cid:9) q j ( t ) (cid:107) (cid:1) r e = exp (cid:0) − σ e (cid:88) e (cid:107) ˆ p e ( t ) − p e ( t ) (cid:107) (cid:1) , (14)where q j are the joint configurations in generalized coordinatesand p e are end-effectors positions. j and e are the indices of jointsand end-effectors, respectively. Our implementation of the end-effector reward includes the positions of both hands, both feet, andthe head relative to the moving coordinate frame of the skeletalroot (pelvis). The hat symbol indicates the positions and joint con-figurations taken from the reference trajectory. The joint configu-rations are represented by unit quaternions. The geodesic distance q (cid:9) q = ln( q − q ) measures the shortest rotation angle be-10igure 15: The degree of functional disorders for the change oflimb lengths with (red) and without (blue) retargetingFigure 16: Vastus lateralis is supposed to be a knee extensor. Itsfunction may be disoriented if skeleton scaling disturbs the mus-cle force direction at its insertion. (Left) The vastus lateralis actsas expected in the reference model. (Middle) Na¨ıve scaling of themusculoskeleton results in its opposite action. (Right) Our retar-geting algorithm recovers the force direction and its action. tween two joint configurations [20]. We employed a hierarchicalRL algorithm proposed by Lee et al. [22] to learn muscle-actuatedcontrol policies mimicking reference motion data.We collected motion capture data available on the web and re-targeted each motion clip to our kinematic models using AutodeskMotionBuilder TM . The DRL algorithm is able to learn muscle-actuated control policies successfully with mild body shape varia-tions. The DRL algorithm can also cope with larger variations to acertain extent. The highly-varied characters were able to success-fully imitate motor skills, including walking, running, jumping,kicking, and dancing while maintaining their balance (see Fig-ure 17 and the results may be best viewed in the accompaniedvideo). The moment arm of a muscle m to a joint j is a quantity ofinterest often referred to by biomechanics literature. The momentarm is the perpendicular distance from the joint to the line of mus-cle tension force and serves as a measure of effectiveness that themuscle has on joint torque. The moment arm about a particularaxis ˆ n of action at joint angle θ can be calculated by [29] r θ = τ θ f = J (cid:62) m f m ( a m ) · ˆ nf ( a m ) (15)where τ θ is the effective torque applied to joint j by muscle m and f ( a m ) is the tension force at the attachment site generated bymuscle activation.Moment arm is not an invariant factor we want to preserveacross different body sizes and proportions. We rather wish tohave moment arms to be adjusted appropriately during the retar-geting process. In Figure 18, the angle-moment arm curves in theretargeted models differ from thes curves in the reference model.The Alien has larger moment arms for hip flexors and extensors tocompensate for longer Femurs. As the femurs gets longer, largermoment arms are preferred to use flexor and extensor musclesmore efficiently. The retargeting algorithm reduces the momentarms for the Hulk and the Dwarf on the contrary to compensatefor their short Femurs. We would like to create individualized anatomic models frommedical images and physical examination data. There are stan-dard physical examination procedures of measuring joint ROMsin physical practice, such as popliteal angle, Staheli, and Silfver-ski¨old tests. In this example, we used biplanar X-ray images ofan anonymous person to create a skeletal model and demonstratedhow ROM editing can be performed according to physical exami-nation data.The biplanar image data include lateral and anteroposterior X-ray images acquired simultaneous at precisely calibrated view an-gles (see Figure 19). The calibrated orthographic images allow usto reconstruct 3D points from a pair of 2D feature points on theimages. We selected joint points on the X-ray images and scaledour parametric skeleton model to match the reconstructed 3D jointpoints. Musculature retargeting to the scaled skeleton creates anindividualized musculoskeletal model.11igure 17: Our characters learned a variety of motor skills using DRL. All results are physically simulated and controlled by modulatingthe level of muscle activations and solving muscle contraction dynamics.The ROM measurements in physical examination data can beincorporated into our anatomic model. For example, the unilateralpopliteal angle is a test for measuring hamstring tightness, whichis related to the ROM in the knee. The test measures the range ofa knee angle when the subject is in the supine position with itsleg flexed to 90 ◦ at the hip and the knee is extended passively. Wesimulated this procedure to have our musculoskeletal model takethe specified pose (see Figure 20). Interactive ROM editing allowsus to change the knee ROM to match any target range by decidingthe scaling and shifting factors as explained in Section 4.3. Our attempt to build a musculoskeletal model for a varietyof body shapes is an important step toward the goal of expand-ing the applicability of anatomic human modeling and simulation.Nonetheless, our current framework still has several limitationsin its scope of applicability and biomechanical accuracy. Humananatomy is an extremely complicated system. There has been atrail of research works that incrementally added details to the com-puter model of human anatomy. Currently, our model ignores a lotof important anatomic features and phenomena, such as ligamentsand contact coupling. Joint ROMs are largely influenced by thecollision between bony features and tensions generated by liga-12igure 18: The relation between moment arm and joint angle. (Top) Gluteus Maximus during hip extension. (Middle) Iliacus during hipflexion. (Bottom) Rectus Femoris during knee extension.Figure 19: The 3D skeleton reconstructed from biplanar X-rayimages.ments around the joints. Muscle paths wrapping around bones andother muscles will best be approximated by contact coupling be-tween anatomic features if their high-resolution geometry and vol-umetric FEM simulation are provided [21]. The use of waypointsis a crude, yet computationally-feasible approximation.Our demonstration of reconstructing a musculoskeletal modelfrom biplanar X-rays and clinical examination shows the poten-tial applicability of our study in clinical practice. Ideally, we wishto be able to construct individualized musculoskeletal models for [!ht]Figure 20: The unilateral popliteal angle of our reference modelis 30 ◦ assuming that the threshold for the net passive torque is 50 N · m . ROM editing can enhance or reduce the ROM in the knee.The reduced ROM shown in the green curve results in a largerunilateral popliteal angle.13ny human subjects with minimal effort. Although the potential isapparent, we have not tried to validate or verify its effectivenessin medical applications, which may require rigorous experimentswith human subjects [27]. This ambitious goal remains the subjectof future research beyond the scope of this paper.Our algorithm assumes the presence of a reference model,which can be retargeted only to structurally equivalent bod-ies. If we want to build a musculoskeletal model of non-anthropomorphic characters, there is no high-quality referencemodel to begin with. An interesting direction for future researchis to build anatomical models from scratch, potentially with helpof various types of measurements including 3D scans, medical im-ages, EMG, motion capture, and videos [17]. References [1] Ijaz Akhter and Michael J. Black. “Pose-conditioned jointangle limits for 3D human pose reconstruction”. In:
Pro-ceedings of the IEEE Conference on Computer Vision andPattern Recognition . 2015, pp. 1446–1455.[2] Edith M. Arnold et al. “A model of the lower limb for anal-ysis of human movement”. In:
Annals of biomedical engi-neering
International Conference on Medical Image Comput-ing and Computer-Assisted Intervention . Springer. 2014,pp. 65–72.[4] Stelian Coros, Philippe Beaudoin, and Michiel Van dePanne. “Generalized biped walking control”. In:
ACMTransactions on Graphics (TOG) . Vol. 29. 4. ACM. 2010,p. 130.[5] Michael Damsgaard et al. “Analysis of musculoskeletalsystems in the AnyBody Modeling System”. In:
Simula-tion Modelling Practice and Theory
IEEE Transactions on Biomedical engineering
IEEE Transactions on Biomedical Engineering
ACMTransactions on Graphics (TOG)
ACM Transactions onGraphics
Computer methods in biomechanics andbiomedical engineering
ACM Transactions on Graphics(TOG)
Proc. R. Soc. Lond. B
Proceedings of the24th annual conference on Computer graphics and interac-tive techniques . ACM. 1997, pp. 153–162.[14] Katherine RS Holzbaur, Wendy M Murray, and Scott LDelp. “A model of the upper extremity for simulating mus-culoskeletal surgery and analyzing neuromuscular control”.In:
Annals of biomedical engineering
ACM Transactions on Graph-ics (TOG) . IEEE. 2018, pp. 1098–1103.[17] Petr Kadleˇcek et al. “Reconstructing personalized anatom-ical models for physics-based body animation”. In:
ACMTransactions on Graphics
Journal of biomechanics
Korean AdvancedInstitute of Science and Technology, Daejeon, PhD thesis (2000).[20] Jehee Lee. “Representing Rotations and Orientations in Ge-ometric Computing”. In:
IEEE Computer Graphics and Ap-plications
ACM Transactions on Graph-ics (TOG)
ACM Transactions on Graph-ics
ACM Transactions on Graphics
ACM Transactions on Graphics(TOG)
ACM Transactions on Graphics (TOG)
ACM Transactions on Graphics
ACM Transactions on Graphics
Medical ImageAnalysis
Physicaltherapy
ACM Transactions on Graphics(TOG)
ACM Trans. Graph.
Journal ofbiomechanics
Computers & Graphics
Journal of biomechanical engineering
IEEERobotics and Automation Letters
ACM Transactions on Graphics
Proceedings of the11th Annual International Conference on Motion, Interac-tion, and Games . ACM. 2018, p. 16.[38] Xue Bin Peng et al. “DeepMimic: Example-guidedDeep Reinforcement Learning of Physics-based CharacterSkills”. In:
ACM Trans. Graph.
IEEE Trans. Biomed. Engineering
Joint Range of Mo-tion and Muscle Length Testing . Saunders, 2017.[41] Prashant Sachdeva et al. “Biomechanical simulation andcontrol of hands and tendinous systems”. In: 34.4 (2015),p. 42. [42] Shunsuke Saito, Zi-Ye Zhou, and Ladislav Kavan. “Com-putational bodybuilding: Anatomically-based modeling ofhuman bodies”. In:
ACM Transactions on Graphics
ACM Transactions onGraphics
ACM Transactionson Graphics
ACMTransactions on Graphics (TOG) . Vol. 26. 3. ACM. 2007,p. 107.[46] Susan Standring, Harold Ellis, and Caroline Wigley.
Gray’sanatomy : the anatomical basis of clinical practice . ElsevierChurchill Livingstone, 2005.[47] Ian Stavness, John E. Lloyd, and Sidney Fels. “Automaticprediction of tongue muscle activations using a finite el-ement model”. In:
Journal of biomechanics
ACM Transactions on Graphics
Journal of biomechanical engineering
Journal of biome-chanics
ACMtransactions on graphics arXiv preprintarXiv:1801.08093 (2018).[54] Felix E. Zajac. “Muscle and tendon Properties models scal-ing and application to biomechanics and motor”. In:
Criti-cal reviews in biomedical engineering