Efficient Generation of Motion Transitions using Spacetime ConstraintsCharles RosePrinceton University(currently at Microsoft Research)Brian GuenterMicrosoft ResearchBobby BodenheimerMicrosoft ResearchMichael F. CohenMicrosoft ResearchAbstractThis paper describes the application of space time constraints to creatingtransitions between segments of human body motion. Themotion transition generation uses a combination of spacetime constraintsand inverse kinematic constraints to generate seamless anddynamically plausible transitions between motion segments. Weuse a fast recursive dynamics formulation which makes it possibleto use spacetime constraints on systems with many degrees of freedom,such as human figures. The system uses an interpreter of amotion expression language to allow the user to manipulate motiondata, break it into pieces, and reassemble it into new, more complex,motions. We have successfully used the system to create basis motions,cyclic data, and seamless motion transitions on a human bodymodel with 44 degrees of freedom.Additional Keywords and Phrases: computer animation, inversekinematics, motion capture, motion control, human figure animation,cyclification.CR Categories and SubjectDescriptions: I.3.7 [Computer Graphics]:Three Dimensional Graphics and Realism: Animation; I.6.3[Simulation and Modeling]: Applications; G.1.6 [Constrained Optimization];I.3.5 [Physically-Based Modeling].1 IntroductionExisting 3D animation tools primarily provide support for creating asingle linear stream of animation where the entire motion is plannedin advance and computed off-line. Interactive 3D character animation,however, is characterized by a degree of uncertainty that isnot present in animation for film or television. Characters are underthe control of the user and must be able to change the way theymove at any time. Crafting animation for an interactive applicationpresents a new set of problems and requires a different set of specializedtools.One solution for these problems is to generate a set of high qualitymotions, called basis motions in the remainder of the paper, andthen create transitions between these motions so they can be strungtogether into animations of unlimited length and great variety. BasisEmail: ft-chuckr, briangu, bobbyb, email@example.com are typically short and can be combined into motions representativeof the type of actions that the body has to perform. Forexample, basis motions might be walk cycles, arm waves, karatekicks, and so forth. Basis motions need not specify all the degreesof freedom in the body; they can specify just those of one limb oreven a part of a limb. Using motion capture techniques, it is relativelyeasy to make high quality basis motions, but generating highqualitytransitions among those basis motions is still difficult andinvolves significant manual labor. The techniques presented workwell with motion capture data, but would work equally well withhand-animated basis motions.We have developed an algorithm for generating these transitionssemi-automatically, greatly reducing the time spent and the numberof parameters an animator must specify. The system provides twosemi-automatic mechanisms for generating motion: motion transitiongeneration and cyclification. Motion transition generation usesa combination of spacetime constraints  and inverse kinematicconstraints  to generate transitions between basis motions. Afast dynamics formulation makes it practical to use spacetime transitiongeneration on high degree of freedom systems. With this dynamicsformulation, the algorithm achieves the lower bound timecomplexity for spacetime algorithms that use gradient based optimizationtechniques.The motion transitions satisfy both dynamic and kinematic constraints.This differs from the work described in [2, 11, 9]. Thesepapers described various mechanisms, such as dynamic time warping,Fourier interpolation, and multi-resolution signal processing,for transforming existing motion data. Transitions between motionclips were achieved using linear combinations of the two motionsbeing transitioned. This can result in motion which does not haverealistic dynamic qualities and which may not necessarily satisfykinematic or anthropomorphic constraints. The transition mechanismdescribed here generates motion which minimizes the torque requiredto transition from one motion to another while maintainingjoint angle constraints. Inverse kinematics are used to ensure thatkinematic constraints are satisfied.Additionally, we have defined a motion expression language toallow the user to manipulate motion data, break it into pieces, andreassemble it into new, more complex motions. We have successfullyused the system to create basis motions, cyclic data, and motiontransitions on our human body model which has 44 degrees offreedom.Section 2 of the paper describes the human body model and howmotion capture data is processed before it gets into the system.Section 3 describes the semi-automatic spacetime and inverse kinematictransition mechanism. This section also explains the fastdynamic formulation which allows spacetime constraint optimizationto run quickly on systems with many degrees of freedom. Section3 also describes our method of motion cyclification. Section 4explains how body motion is internally represented and describesthe motion expression language. Section 5 presents results of transitionsgenerated with the system. Section 6 concludes the paper.Appendix A contains an explanation of the notation used in the dynamicsformulation, the dynamics equations, and their partial derivativesas they are used in the spacetime optimization.
R is a function of the joint angles in the bodyR = f (q 1(t);:::;q n(t))which are themselves functions of the B-spline control points definingeach joint angle functionq i(t) =g(b i;1;:::;b i;m)where the b i;j are the control points of the B-spline curve for jointangle function q i(t). We minimize R using the BFGS optimizationalgorithm described in more detail in Section 3.2.3. For the purposesof the present discussion the only relevant part of the BFGSalgorithm is that it requires the gradient of R at each iteration in theoptimizationrR =@R@b i;1; ;@R@b i;j=@r k@b i;jZ t2t 1K Xk=1@R@b n;m@r k@b i;jdt=2(p k(t),^p k(t)) (u i d ki) @qi@b i;jwhere u i is the axis of rotation of joint i and d ki is the vector fromjoint i to the constrainedframe k. Figure 2 showsthe effect using theinverse kinematic constraint to fix the feet of the body on the groundduring a motion transition period. The left leg is constrained to beon the ground during the entire transition interval while the right legis constrained to be on the ground only at the end of the interval. Inthe image on the left the inverse kinematic constraints were turnedoff; the left leg drifts from its desired position and the right leg failsto meet its desired position. In the image on the right the inversekinematic constraints are satisfied. The left leg remains fixed duringthe transition period and the right leg touches down at the end of theinterval.Figure 2: Effect of inverse kinematics constraint on placement offeet.3.2.3 Spacetime Dynamics FormulationThe energy required for a human to move along a path is actually acomplex non-linear function of the body motion since energy can bestored in muscle and tendon in one part of the motion and releasedlater on. As shown in , joint torques are a reasonable predictor ofmetabolic energy, so minimizing torque over time should be a reasonableapproximation to minimizing metabolic energy. Experiencehas shown that motion that minimizes energy looks natural. Thisleads to the minimization problem:minimize e =Z t2t 1Xi 2 i (t) dt:We use the BFGS optimization algorithm  to find a minimumof this integral equation. BFGS belongs to the class of quasi-Newton algorithms which progress toward a solution by using thegradient of the objective functiong = re:The gradient is used to incrementally update a matrix decompositionof a pseudo-Hessianmatrix, H, and to compute a new step directiond = ,H ,1 g:The relative amount of computation for each subtask required atevery iteration of the algorithm is common to several quasi-Newtonalgorithms: gradient computation, pseudo-Hessian update, andcomputation of the step direction.Since each of the i is potentially a function of all the q i, _q i, q i thegradient requires the evaluation of O(n 2 ) partial derivatives wheren is the number of degrees of freedom of the body. This is in facta lower bound for the asymptotic time complexity of space time algorithmswhich use gradient-based optimization techniques.If m is the number of B-spline coefficients used to define the timefunction of each degree of freedom then the pseudo-Hessian is ofsize nm by nm. The update of the pseudo-Hessianand computationof the step direction are both O((nm) 2 ).Form small, less than 20,and n large, more than 30, the time required to computeg dominatesall other computation thus an efficient formulation for g will pay thegreatest dividends in reducing computation.Computing g requires finding the joint torques and a varietyof subsidiary quantities, such as angular velocity and acceleration.This is the inverse dynamics problem which has been extensivelystudied in the robotics literature. See  for a good overview ofmany of these algorithms. Many inverse dynamics formulationshave been proposed in the robotics literature ranging from O(n 4 )non-recursive to O(n) recursive algorithms. The inverse dynamicsformulation we use is due to Balafoutis . This is an O(n)recursive formulation which requires 96n , 77 multiplications and84n,77 additions to solve the inverse dynamicsproblem for a robotmanipulator with n joints. This is faster than the O(n) Lagrangianrecursive formulation developed by Hollerbach  and used in ,which requires 412n , 277 multiplications and 320n , 201 additions.The efficiency of the Balafoutis algorithm derives from the computationalefficiency of Cartesian tensors and from the recursivenature of the computations. These efficiencies carry over to thecomputation of the gradient terms.The Balafoutis algorithm proceeds in two steps. In the first stepvelocities, accelerations, net torques, and forces at each joint arecomputed starting from the root node and working out to the tips ofall the chainsin the tree. In the secondstep the joint torques are computedstarting from the tips of the chains back to the root node. Seethe appendix for details. These recursive equations can be differentiateddirectly to compute g or one can use Cartesian tensor identitiesto compute the derivatives. Since the differentiation is tediousand somewhat involved we have included some of the partial derivativesof the recursive equations in the appendix as an aid to thoseattempting to reproduce our results.3.3 Motion CyclificationIf cyclic motions, such as walking and running, come from motioncapture data they will not be precisely cyclic due to measurementerrors and normal human variation in movement. The discontinuitiesin motion will likely be small enough that the full power of aspacetime transition will not be necessary in order to splice a motionback onto itself smoothly. In this case we use a much simpler andfaster algorithm for generating seamless cycles.
The cyclification algorithm proceeds in two steps. First the usermarks the approximate beginning and end of a cycle. We createtwo time regions I s and I f centered about the markers the user haschosen. The time regions are set to be one-fifth the length of the timebetween markers. The system then finds one time point in each intervalthat minimizes the difference between position, velocity, andacceleration of the body:mint 1 2I s;t 2 2I fka,bk 2where a =[q(t 1);_q(t 1);q(t 1)] T and b =[q(t 2);_q(t 2);q(t 2)] T .For most motions there will still be a discontinuity at the timewhere the motion cycles. We distribute this discontinuity error overthe entire time interval so that the end points of the cycle match exactlyby adding a linear offset to the entire time interval. We thenconstructa C 2 motion curve by fitting a least squares cyclic B-splineapproximation to the modified motion.I 1^I 2 =I 1 , I 2 =((I 1 + I 2 = I 1 , I 2;I 1 ^I 2 t 1 t 2(f 1 ;:::;f n;g 1 ;:::;g n;t 3 ;t 2 ) t 3
above. Figure 6 shows an example of a spacetime transition from awalk arm motion to a wave arm motion and then back to a walk armmotion. The arm wave MU defines only the DOFs of the arm andthe walk MU defines all the DOFs of the body. First we perform anaffine transformation on the arm wave MU and undefine this fromthe walk MU. This will undefine the arm degrees of freedom of thewalk MU during a time that is slightly longer than the arm wave MU.When we add a time shifted version of the arm wave MU to the resultingMU there will be two undefined regions surrounding it whichthe spacetime operator will fill in with spacetime transition motion.The result will be a smooth transition from a walking arm motion toan arm wave motion and back to a walking arm motion. This operationis shown in both a time line view and an operator tree view inFigure 6. Letting SP denote the spacetime optimization, the algebraicrepresentation of this motion isWithout a biomechanical model to guide a large motion, our minimalenergy model will often prove insufficient.The beginning of the transition is colored blue and the end iscolored red with intermediate times a linear blend of the two colors.This motion is one transition from a longer animation which has 5transitions between 6 motions.SP (ane 2 (wave) + (walk , ane 1 (wave))) :SpacetimetransitionregionsWalkWalkaffine2(wave)affine2(wave)WalkWalkSp+Figure 8: Multiple time exposure of transition generated from themotions in Figure 7Walkaffine2(wave)affine1 (wave)Walk-aff1aff2Figure 9 shows an example of a motion transition which affectsonly the arm degrees of freedom of the motion. This sequenceactuallyconsists of two space time transitions: one from a walking armmotion to the salute motion and another back to the walking armmotion. Each transition motion is .3 seconds long.WaveWaveWalkWalkTime line viewFunction composition viewFigure 6: Spacetime composition of motions.5 ResultsFigure 9: Arm walk motion transitioning to salute motion and backto walk motion. Arm degrees of freedom affected by the transitionare colored green.Figure 7: End position of motion 1 and beginning position of motion2 for a motion transitionWe have successfully applied the motion transition algorithm onmany motions. For this example the transition time was set to .6,and the number of B-spline coefficients to 5. The resulting transitionis shown in Figure 8. Our experience has been that successfultransitions are quite short, usually in the range of .3 to .6 seconds.Computation times for transitions are strongly dependent on thenumber of degrees of freedom involved since the spacetime formulationwe use is O(n 2 ) in the number of degrees of freedom. Forthe transition of Figure 8 generating the spacetime transition motiontook 72 seconds. This transition involved 44 degrees of freedom.For the transition of Figure 9 generating the spacetime transitiontook 20 seconds. All timings were performed on a 100 MHzPentium processor.Spacetime transitions are more costly to generate than joint angleinterpolation techniques, but they often produce more realistic motion.One type of motion that demonstrates this superiority is motionthat has identical joint space beginning and ending conditionson some of the degrees of freedom of the figure. An example of thistype of motion is shown in Figure 10. This motion begins with theforearm nearly vertical, held close to the shoulder with zero initial
AAppendix: Dynamics DetailsA.1 List of Symbolso ic iorigin of the i-th link coordinate frame.center of mass of the i-th link.! i i angular velocity of the i-th link.Figure 10: Joint angle interpolation versus spacetime optimization.z i is i i;jjoint axis of the i-thlink expressed in the i-thcoordinate frame.vector from o i to o j expressed in the i-th coordinate frame.velocity. The motion ends with the forearm held horizontal also withzero velocity. Because the upper arm and the wrist have identicaljoint space starting and ending conditions any simple interpolationtechnique, which would include linear interpolation, polynomial interpolation,and most other types of interpolation which simply takea weighted sum of the two endpoint conditions will yield a motionsuch as that shown on the left in Figure 10. This is an unnatural motionsince there is no joint space motion at the shoulder or the wrist.The spacetime motion, however, has motion at every joint and looksmuch more like the kind of motion a person might make.6 ConclusionThis paper has presented a powerful animation system for manipulatingmotion data using a motion expression interpreter. Data canbe positioned in space and time, and complete control of the degreesof freedom in the system allows motions to be spliced and mixed inarbitrary manners.The system is capable of generating seamless transitions betweensegments of animations using spacetime constraints and inverse kinematics.Using a new, fast, dynamics formulation we can applyspacetime constraints to systems having a large number of degreesof freedom and still have reasonable computation time. An additionalcapability of the system is the generation of arbitrary lengthperiodic motions, such as walking, by cyclifying segments of motiondata which are approximately periodic.The results of using our system to generate animations startingfrom a base library of soccer motions are quite good. Cyclificationof a segment of such motions as a walk produces a quite realisticwalk of arbitrary length. The spacetime constraint and inverse kinematicoptimization produce transitions between diverse motionswhich are seamless and invisible to the viewer. While the optimizationcannot be done in real time, it is relatively fast, and quite usableby an animator designing motions.We plan to extend our motion model to more accurately modelthe dynamics of the human body model. The current approximationused for computing root motion works reasonably well for the classof transitions we have worked with but is not accurate for free bodymotion.AcknowledgementsThe authors thank Jessica Hodgins, College of Computing, GeorgiaInstitute of Technology,for providing inertia matrices for the humanbody model. The Microsoft Consumer Division generously sharedmotion capture data with us for use in this project.r i i;jA iI k c iJ k c i i iF i c iM i c ivector from o i to c j expressed in the i-th coordinate frame.3x3 coordinate (or 4x4 homogeneous) transformation relatingthe i-th coordinate frame to the (i , 1)-th frame.inertia tensor of the i-th link about c i expressed in the k-th coordinateframe.Euler’s inertia tensor of the i-th frame about c i expressed inthe k-th coordinate frame.angular acceleration tensor of the i-th link expressed in the i-thcoordinate frame.force vector acting on c i expressed in the i-th coordinateframe.moment vectoraboutc i expressedin the i-th coordinate frame.f i i force vector exerted on link i by link (i , 1). i i moment vector exerted on link i by link (i , 1). i torque at joint i.m ig gravity.mass of the i-th link.In the above, the subscript indicates the coordinate frame beingrepresented and superscript the coordinate frame in which it is represented.We use + and , on index variables to denote relative placementin the joint hierarchy. Thus, i, is the predecessor of i which is thepredecessor of i+. For example, in the equation ! i+z i+i+ = AT i+! i i +_qi+, the variable i+ !i i is the angular velocity in the coordinateframe which precedes the coordinate frame of ! i+ .Inotherwords,i+coordinate frame i is closer to the root coordinate frame than isframe i+. Note that there is no guarantee of a uniquely defined successor.g =[0:0;,9:80655; 0:0] TJ i c i = 1 trace, I i c2 i 1 , Iici" 0 ,v3 v 2dual(v) =~v= v 3 0 ,v 1,v 2 v 1 0dual(~v) =v#
A.2 Forward Dynamics EquationsBase conditions at the root of the creature:! 0 0 = z 0 0 _q 0_! 0 0 = z 0 0 q 0s 0 0;0 = A T 0 gRecursive forward dynamics equations:! i+i+ = A T i+! i i + z i+i+ _q i+_! i+i+ = AT i+ _! i i + ~! i+i z i+i+ i+i+ = ~! _ i+i+ + ~! i+i+ ~!i+ i+s i+0;i+ = A T i+ si0;i + i is i i;i+r i+0;i+ = i+ i+r i+i+;i+ + si+ 0;i+F i+c i+= m i+ r i+0;i+_qi+ + zi+i+ qi+~M i+c i+= , i+J i+c i+,, i+i+ J i+c i+ TA.3 Backward Recursive Equations (Torque Equations)At a joint controlling an end-effector:At an internal joint:f i i = F i c i+f i i = F i c i i i = ~r i i;iF i c i+ M i c i i = i i z i i i i = ~r i i;iF i c i+ M i c i+ i = i i z i iX Ai+f i+i+i+X Ai+ i+i+ + ~si i;i+f i i+A.4 The Energy Function and its PartialZe =tXi 2 i dti+R t 2@e @t= 1Pi 2 i dt=2@q j @q jZ t2t 1X i@ i@q jdtA.5 Forward Partials with Initial Conditions! i+i+ =i+>j A T ! i ii+q j q j! i+i+ = A T ji+=j ! j,j,q j q j _! i+i+ =i+>j A T _! i ii+ +^! i+iq j q j _! i+i+ = A T ji+=j _! j,q j q j i+i+_!=^ i+i+q js i+0;i+q j=i+>j A T i++^! i+i+q jz i+i+ _q i+q j^ A Tj, + j_! j,j,q jq j~! i+i+ + ^! i+i+ si0;i+ i is i i;i+q j q js i+0;i+ = A T ji+=j sj,0;j,q j q + j,jj,s j,j,;jM i+c i+q jF i+c i+= i+ i+q jJ i+c i+,0;i+q j= m i+r i+q jz j j _qj~! i+i+q j i+ Ti+J i+cq i+ j!TA.6 Reverse Partials with Initial Conditionsf i iq j=9i+F i c iq j+f i iq j= 69i+F i c iq jii =q j9i+]F iri c ii;i+q jXi+ Ai+f i+i+q + f iAi+ ij q j+^M i c iq j+A i+ i+i+q j+ g sii;i+A i+q jf i+ii =q j69i+]F iri c ii;i+ +^M i c iq j q j iq j= i iq j z i iReferencesXi+Ai+q j i+i+ +i+ + g sii;i+ A i+f i+i+q j BALAFOUTIS,C.A.,AND PATEL,R.V.Dynamic Analysis ofRobot Manipulators: A Cartesian Tensor Approach. KluwerAcademic Publishers, 1991. BRUDERLIN, A.,AND WILLIAMS, L. Motion signal processing.In Computer Graphics (Aug. 1995), pp. 97–104. Proceedingsof SIGGRAPH 95. BURDETT,R.G.,SKRINAR,G.S.,AND SIMON,S.R.Comparisonof mechanical work and metabolic energy consumptionduring normal gait. Journal of Orhopaedic Research 1,1(1983), 63–72. COHEN, M. F. Interactive spacetime control for animation. InComputer Graphics (July 1992), pp. 293–302. Proceedings ofSIGGRAPH 92. GILL, P.E.,MURRAY,W.,AND WRIGHT,M.H. PracticalOptimization. Academic Press, 1981.
 HOLLERBACH, J. M. A recursive lagrangian formulation ofmanipulator dynamics and a comparative study of dynamicsformulation complexity. IEEE Transactions on Systems, Man,and Cybernetics SMC-10, 11 (Nov. 1980). LIU, Z.,AND COHEN, M. F. An efficient symbolic interfaceto constraint based animation systems. In Proceedings ofthe 5th EuroGraphicsWorkshop on Animation and Simulation(Sept. 1994). LIU, Z.,GORTLER, S.J.,AND COHEN, M. F. Hierarchicalspacetime control. In Computer Graphics (July 1994), pp. 35–42. Proceedings of SIGGRAPH 94. UNUMA,M.,ANJYO,K.,AND TEKEUCHI, R. Fourier principlesfor emotion-based human figure animation. In ComputerGraphics (Aug. 1995), pp. 91–96. Proceedings of SIG-GRAPH 95. WITKIN,A.,AND KASS, M. Spacetime constraints. In ComputerGraphics (Aug. 1988), pp. 159–168. Proceedings ofSIGGRAPH 88. WITKIN, A.,AND POPOVÍC, Z. Motion warping. In ComputerGraphics (Aug. 1995), pp. 105–108. Proceedings ofSIGGRAPH 95. ZHAO,J.,AND BADLER, N. I. Inverse kinematics positioningusing non-linear programming for highly articulated figures.ACM Transactions on Graphics 13, 4 (Oct. 1994), 313–336.