CVFX Lecture 21: Inverse kinematics and motion editing

okay so today’s lecture is on what I would call inverse kinematics and then we’re going to talk what’s called motion ending and so where we left off last time was talking about forward kinematics of a human body that was basically like saying okay so if I told you where all the joint angles were I would be able to produce basically the positions of any any position any point on the body that I wanted to right so basically forward kinematics is basically the following where I have a formula like this that tells me if I tell you the you know all the joint angles and so on can you tell me then where are the 3d points on the surface of the body okay and so we kind of talked about that last time and dysfunction which I’m kind of leaving deliberately uncomplicated as F is basically made up of a whole bunch of trigonometry right so basically we talked about last time if I want to know where is the position of my wrist well you as first I tell you what is the angle my shoulder how long is my for or how long of the upper arm what’s the angle of my elbow how long is my lower arm that tells you where my wrist is right so to get any given point on the left hand side requires me to go through a whole bunch of cosines and sines and you know then lengths and so on on the right hand side on their hand you know a lot of times what we care about in real nursing capture scenarios is going the other way right so again think about what we saw at the motion capture studio right when you actually see in the motion capture studio our blood counts line right you see some points on the surface of the body and from that you want to infer what are these joint angles right so we’re kind of doing the inverse problem that’s why it’s called inverse kinematics the problem is that we can’t just take this function and undo it right you know it’s not easy to do because there’s all these strain of metric functions inside so it’s not like a nice easy inverse of that function instead we have to do there are a couple of different ways we can we can solve this problem so inverse kinematics is basically estimating theta from our okay and there are kind of two different situations that we might be in okay one situation is you keep and the future is depend on what is dimension of this and was the dimension of that so if we say for example and we get my notation straight so if we say that if R is in you know n dimensional space and theta is in P dimensional space then we have basically two situations one situation is that if n is less than P right that means that I have more joint angles that I have observations right and that means that the observations don’t totally constrain the droid angle position that’s right this is an underdetermined system and that means that we’re kind of doing is trying to find the best of the many solutions for theta that satisfy the observed positions of r so this is like the under constraint situation and this is like finding the best solution that satisfies you know the constraints the other situation is when n is bigger than P that means that I have more observations on the body than I do joint ankles and that’s an over constraint situation right that means that basically I have too much information and I can’t find some Thetas that are going to be exactly consistent with all of those observed surface positions and so in that case I have to find some sort of optimization algorithm that tells me what is the closest I can get to match the observed body positions to the ones that I would predict with theta so this is like over constrained and in this case we need to do something that’s along the lines of like a least square solution and so both of these situations come up in different ways in motion capture problems right so the problem where we were looking at in the motion capture lab where you observe all these markers and you want to infer what the underlying skeleton is that’s probably in this lower case right because we have a lot of markers on the body if you calculate if you count up the number of markers XYZ positions and a number of degrees of freedom inside the body I think that you’ll find that

for many motion capture scenarios there are more markers than you need okay so that’s usually we’re going to be here and that’s the second thing I’m going to talk about the position that the situation here is more like saying okay you know if I want to get the position on the body to a certain position in the world then what are the joint angles that accomplish that so for example I’ll show you a video in just a few minutes where we can say okay you know I’ve got a human kinematic model and now I want to reach out and I want to grab this position right I don’t tell you anything else about what the other joints are doing in that case I’m really just pinning a couple of positions in 3d space and I’m asking the rest of the joints to kind of follow along in a realistic way right so in that case maybe I’m only specifying a few nailed down positions in the 3d world and I have to figure out all the joints to kind of match up with that right and so I’ll show you a video in a moment that will make that hopefully a lot more concrete okay so let’s talk about kind of the the most mathematical way of doing it then we’ll talk about a little bit more practical ways of doing it right so if frozen air approaches problem mathematically one way to start is what’s called inverse differential kinematics in verse because we want to undo the relationship between the joint angles and the surface points and differential because we’re gonna be taking some derivatives right so if I take my formula from here I take the derivative of both sides with respect to T right so basically the idea is what I have here is actually a series of great locations over taught and right because my body is evolving over time and I want to use that information that there’s some sort of temporal coherence and consequently I observe some points on the surface of the body over time right and so if I take the derivative of both sides of the previous equation what I get is basically dr DT is equal to DF D theta D theta at DT right this is just the chain rule okay and so this thing here is called the Jacobian and kind of what it means is that you know again F is a function that takes the joint angles in R P and puts them on two 3d positions in RM and so what we’re kind of doing is we’re saying okay so that means that the result of this function is some n by 1 vector and then this DF D theta is basically saying how does every entry of this n by 1 vector depend on each of the P parameters of there right so this is going to be an N by P matrix okay and so even though I didn’t go into it too much last time the jacobian the constitute was recovery and also comes up when we’re talking about bundle adjustment and instruction promotion right because we had a similar situation where we had basically every observation of a image point on the image plane right x IJ depended on the unknown parameters before the camera parameters for camera eye and the scene parameters for point J all right so there’s again this Jacobian we talked about how the structure of that Jacobian the sparsity structure helps us solve all the adjustment problems more easily so if you go back you can see there there’s another place we’re talking about Jacobian is already in class okay and so kind of what I want to do is I want to get the evolution of these things over time so kind of what I’d like to do is let me just rewrite this with J so if I knew R and I wanted to infer theta what I could do is kind of just take the inverse right when I’d like to be able to just move the J over to the left hand side the problem is that I don’t have an actual inverse of J because J is not a square matrix right J is actually going to be let’s assume that I guess I should say up here assuming that and is less than P we’re going to start in this case where I have fewer markers than joints and so in this case and being less than P means that my Jacobian looks like this it’s kind of like a wider matrix and it’s tall okay and so I can take the inverse of that kind of matrix right but what I can’t do is take what’s called the pseudo-inverse right so as the pseudo inverse is basically kind of like the inverse except you know it isn’t well it’s a slight some properties of the universe but it doesn’t work because it’s not square so I could say okay my d theta/dt is j and i’m going to put like a little cross here like this that’s usually what you mean when you say pseudo inverse and this is going to be a P by n matrix

and the suniverse is defined by basically J transpose not seer I guess it’s going to be J I have the right J transpose J J transpose inverse okay and so in this sense what I get is okay if I if I take the derivatives of the observed R is I can get the derivatives of the observer of Thetas and so again one way I think about this is this is this is kind of like only one solution that will satisfy all the observed arms because I’m under constrained and so this solution minimizes this norm which is kind of like the norm between dr/dt and j dt there are other solutions there are many solutions at all satisfied the same observed position so gotta remember kinda the way to think about this right is that you know if I think about pinning one point in 3d space you know there are some other solutions for the Thetas that all get my end effector to the same point 3d space right so you can so for example also if I’m only pinning one point in space then my other arm could be flailing around right all those are also solutions right so this is kind of like saying what is the one that has a certain kind of minimum norm you know characterization and so then how would I get back to an actual path for theta so then what I would do is to recover theta of T I would basically start at a known initial position theta 0 and basically evolve that’ll be out saying that theta of T is like where I started plus I integrate from 0 to T the derivative right I did this for my inverse differential kinematics this is my starting point and say it is if I know where the person started right so for example if I put them in some sort of a known position where I say ok stand like this so you can see that when we were in the motion capture lab there were a couple of instances where we’re basically saying okay standing this known position right and then if I know the derivative of things I can evolve from that initial position okay and so you know there are some twists on this idea and so I’m not going to go into the full details of it but basically this is the kind of thing that you could do when you had just a few pinned constraints okay the more often or the more common thing that we want to do for the purposes of actually you know if I want to estimate that real time moving skeleton from the live 0.73 data is that I would have you know when n is greater than P right that means that I have more marker points than joints kind of speaking colloquially then what I’m trying to do is I’m trying to find the Thetas that match up with the observed ours as best as possible and that may be that’s going to be an over constrained situation assuming that I have more information that I really need okay and so in that case what I could do is I could just minimize a cost function that looks something like okay I’m gonna look at the sum over all of the you know I’m gonna say okay I’m say this is the j f– observed 3d point i’m gonna try and bring it in line with the trajectory that i observed as close as possible so select saying this is the j f– observed barker and this is like the ford kinematic model for the jake marker right that is I want to try and make those things as close together as possible right with the understanding that I may not be able to you know exactly nail down all the observed markers in the world right you know there may be some errors now on top of that what I could do is say okay you know in addition to just trying to match the markers really well what I could do is I could put some additional constraints on them right so for example I could say okay you know I want this here’s my kinematic model of the sky

right so what I could do is I could say okay I also want the vector for example on these two points of the kinematic model to be in the same direction as those guys step sorry all right well okay then you take this the grain of salt so basically what I could say is okay I want this vector between here and here of course this doesn’t satisfy it to mark to match up with the vector on two markers of a human body right so I could kind of say okay in addition to wanting the points to match up I also want the you know I can say okay I put a marker on this part of the surface of the body and this pry the surface of the body and I want that vector to be parallel to the underlying you know what’s this for are going this is the radians other yeah okay okay so so you can kind of say okay in addition to just kind of making sure the points are right I also want to make sure that these vectors match up right and then you could also put some further constraints on saying okay so let me just make a list of possible constraints so one constraint is you know observed marker vectors let’s say this is pretty casual match underlying limb vectors that’s kind of what I try to draw here you know another one is basically saying you know human motion constraints right so what I mean by that is that right now when I solve this problem there are no constraints at all on the Thetas that I estimate right which are the joints but I know that for a human there are certain reasonable joint limits right so for example you know I can’t bend my shoulder behind my back very far right where I can bend it this way pretty far and so there have been kind of these biomechanical tables that have measured for the you know average human population what is the range of motion of each of these joints right so for example my elbow can bend you know you can get pretty close to straight but I certainly can’t bend the other way you’re right unless I’m double-jointed or something like that okay and so what you can say is okay for every joint I also have a natural range of motion and top of that there are kind of coupled ranges of motion right so for example I can bend my knee you know however I want but I can’t kind of like simultaneously then my knee however I want and then my other knee however I want there’s probably some sort of a coupling between all my joints at once right that’s a little bit harder to measure in a table but there are things that exist right so for example I know that if I a good example like since my shoulders of my arms are relatively you know unconstrained but I think that you know if you think about yeah yeah I think about what you can do with your lower body and then how it restricts the range of motion that you might otherwise be able to do right so if I bend my body in one way it’s a lot less comfortable to me that for me to bend my other leg that’s not supporting the weight that it is if I’m if I’m supporting the weight in a more natural way right so you can kind of talk about these you know both individually and coupled let’s say you could also put things like not necessarily just the position but also the acceleration right because really what I’m doing is I’m solving for dysfunction of joint angles over time right so I know that I can’t like instantaneously move my arm much faster than a certain rate right and so you want to make sure that your skeleton doesn’t like jump around because you know that you can’t move from one place with its godlike derivative constraint that says you know velocity and acceleration and these again have been measured by studies and there also might be some sort of like physics based constraints right so for example it may be suppose that you were in well I guess camera talk about being in zero-g but suppose that you were well so when you’re walking right there’s a natural sequence there’s natural you know kind of lowest energy motion that comes from the fact that you are a body with a certain mass and you could bow your bones and limbs as a bunch of Springs and weights and stuff like that and see if you imagine that when you’re walking you know you’re trying to find the lowest energy position at every possible time and so you could kind of add these physics space constraints into reconstructing the trajectories now that’s like it’s probably not going to be in the typical

realm of a system like we saw last week I mean I think that most of the constraints that are probably being imposed on the vikon system are probably along the lines of you know some you know some human motion constraints some velocity constraints even so like when you when you saw the the legs get mixed up right so they’re something that obviously if you were imposing some sort of reasonable human constraints certainly like physics based constraints you know that that should never even have been considered as a possible you know configuration but you know part the issue was that what you were seeing before was kind of like instantaneous estimates of the joint positions right or the joint angles so that the user of the mocap system could kind of watch the skeleton moving around in real time and practice what you’d want to do is you save all those you know marker positions and then you approach process them afterwards to get as good of a set of joint you know angles in trajectories as possible right so you know so what you saw on the screen is really not necessarily the stopping point of inverse kinematics really what you would do is you would feed all that data into a more advanced inverse kinematics solver and watch it go you know and then you got after the fact okay so the other thing that you can do so another thing you can do basically is what I would say you know learned from observed data and so what I mean by this is something where you have a model of how humans are naturally likely to move right all these things so far are a little bit more I would say their biomechanical but they’re not based on kind of natural human motion and so what I could do instead is I could say okay I’m going to observe lots of people of my mocap studio and I’m going to figure out you know what are natural ranges of what are natural types of motions for humans and then I can use that model to basically only generate poses that match up with kind of things I’ve seen before right and so here’s kind of a video that explains that idea this was from Groucho at all and so what you’re seeing here is this is the effect of if I were to basically pin the white dots which are the two arms and this leg and move the red dot around if I didn’t have any sort of model for inverse kinematics what I’m gonna see is these really weird human poses right so if you watch the body here you know these are things that are kind of like rag doll level motion right you would never see a person doing that stuff right whereas if you add some learned information from watching people move you can see if these poses look a lot more like what a person would do if they were stuck in these bizarre situations right and so I got the sound off basically here’s an example of basically what they did they recorded lots of mocap data and then they kind of record in this kind of lower dimensional space where was each pose and then when they generate new poses they only allow yourself a lot of them to generate poses that are close to these observed positions so we see here right so here what they’re doing kind of is they’re dragging this point around 2d space and you can see that every point that they drag corresponds to a different 3d pose most of which looked like this you know ball throwing position right so all these you know poses are pretty reasonable and so the idea is that when I generate a new possible set of Thetas I don’t allow myself to stray very far from this manifold right and so kind of here you see is like a heat map that tells me how likely is every pose given the training gear like I play this whole thing but you can basically train on different kinds of motions so you can say okay I have a different manifold depending on whether I’m looking at a guy pitching a ball or a guy doing a jump shot right and so the idea is that what you could do is learn this is natural you know baseball behavior this is natural basketball behavior and then if you know that that’s the kind of motion capture data that you’ve collected then you can kind of restrict yourself to say okay I’m going to apply the basketball model to this basketball mocap and only generate you know good you know guys shooting hoops the nice thing about this is that then you can deal with things like if you were to lose markers

this gives you another way to predict what’s happening so we see here so here what they’re showing basically is you know what happens if I simulate the effect of marker dropouts right so now I’ve got this model for how all the markers are coupled then I can say okay well what if I were to lose some markers on the body right so here they say okay I took away the all the markers on the entire right hand arm and the synthesized motion still looks pretty good because it learns that people walk like this right from observing data and now I’m a particularly on the entire torso and now I took away all the legs and actually what happens is that still if all I knew were the markers on the feet if I learn this model of how people move you know the rest of the body follows along in a pretty convincing way right and so you know this is kind of an example say okay I know that in theory there’s nothing mechanically wrong with me walking you know like this learning how people really move right there’s nothing in the you know I mean if you go to a physics-based model you might discover that because of the way your body kind of moves like a pendulum like we were talking about motion capture lab then your arms naturally swing in this you know counterbalanced way right but there’s nothing that makes that any more uncomfortable for a human than going like this right I think that’s all I wanted to show out of this then what they kind of show is that you can kind of interpolate between you know if you wanted to interpolate between catching motion and the best motion you can kind of do that that’s kind of relate to what I’m going to talk about next this is just kind of showing that you can now more realistically drag the positions of 3d points in the world and make the rest of the body follow along if you’ve learned you know what’s going on so here they pin the feet and they’ve just specified the motion of the arm and then all the rest of the joints follow along with what has been learned you know typical human motion and so you can imagine this would be really helpful for a animator and here this is kind of an example of a 2d to 3d example where they kind of try and pin the joints corresponding to one view of that person in a picture and they say okay for a walker motion you know you get a pretty good looking reconstruction and then this is like the Train basketball model applying it to a picture of a basketball guys so here they’re moving the joints around to kind of match up as well as possible with the 3d picture right so one thing to be aware of in these kinds of situations is that you know if you apply the wrong model to the data that you have it’s gonna look pretty strange right so if you have a ballerina and you apply the basketball model or a boxer model then your reconstructions are going to look really weird right because the ballerina is doing very non boxer like behavior but the boxer model is the only thing that the model has ever seen right so one thing that’s really careful is that you have to be able to match up you know what you learned on with what you’re seeing right now so like I said I think this is probably you know this is probably more advanced than what the off the shelf motion capture system is going to do for you right this is more like post-processing at the motion capture data to you know get better and better you know better a better constraints okay one of the thing I want to mention I guess well in here is that even after you do all that stuff you know there may be still some stuff to take care of and here’s one problem that’s called foot scape and so the idea is that if you watch the person’s foot here you’ll see that it doesn’t really look right it kind of slides along the ground you’re gonna see it on the slow down motion here in just a second so watch the feet and see how they kind of slide along the ground in a way that is not realistic right once you put your foot down it can’t you know move around that freely and so what you really want is something where the reconstructed motion has the person really clearly planting their feet on the ground right no skitter around right you can’t just you know so you can’t have a motion-capture guy who looks like walking across the floor right you need to have something where they really come down on the ground and so I think there’s another video here these are from Lukas Kovar I think there was one where’s the other one yes here’s another foot skate example where you can see as this person comes towards the camera or this is the original motion they’re gonna do something we’re going to talk about next which is basically trying to move that onto there but here you can really see the foot skate right you can see this person is kind of like you know it’s kind of skating along the floor with their feet right and instead you want to apply some additional constraints to say okay that person needs to plant their feet on the floor right their feet can’t penetrate the floor and they can’t move once they’re planted on the floor so you kind of pay attention to when does the foot hit the

ground right and so that’s kind of an additional you know additional consideration style I guess I’ll also say here you know foot skate okay all right so any questions or comments about this junk okay so the next thing I talk about is what I call motion editing I don’t call it what is called Bushman right and so what I mean by that is that you know motion capture sessions are inevitably kind of short right like it’s not like you have somebody who acts out a ten minute scene in a motion capture studio and you just go back and you apply that to a character in a movie or in a video game right more likely what you’re asking someone to do let’s say go on video games for a second so what probably someone do is okay give me a good punch right okay now give me ten of those right now I want you to kind of you know throw this guy over your shoulder right so what you’re doing is you’re collecting a whole bunch of shorter mocap sequences and then you want to kind of string those together as one example into a fluid sequence right so if you’re playing a fighting game on your PC you’re seeing a lot of motion captured video that is strung together in a way that you don’t notice the transitions between different captain sequences right and the same thing is probably true for you know four characters in movies a different motion captures right so one example that that someone’s talking about when I visited was along the lines of say okay you know suppose you’ve got two characters that are kind of like walking across the room to shake hands with each other right and so in that case now suppose I’m gonna make one of them a giant like Gandalf and one of them smaller like a hobbit right and so now you know if I think about what I’ve actually captured right if I scale all that data down the hobbit is going to you know say say the pilot really took four steps to meet Gandalf now if the Hobbit is you know point eight is big he’s not even gonna get to Gandalf by the time he has to shake the hand right so somewhere we have to add some additional walk cycles in there right and so what you may say is okay you capture a bunch of examples of okay just walk across the for a bunch of times and you splice in walk cycles from other mocap sessions to make it look like you’ve got the full thing right and so kind of what I would say is there are these things I would call motion blending or interpolation where we have basically two motions and we want to put them together in a realistic way and then another option is what would be called motion path editing and so what I mean here is something where you know again at the time that something is filmed you don’t necessarily know exactly how you’re gonna use that mocap data in your finished product right and so say that you capture somebody who is like you know walking they say actually I don’t I don’t want them to walk in a straight line in my finished product anything to kind of walk off to the left right and so now I have to kind of transform that that video to or that book app data to conform with the path anything to take in the real world right and so what we’re gonna basically do then is assuming that we already have these theta eyes right this is like my captured trajectory kind of what I’m doing in some sense immediate senses I’m transforming these over into some new you know positions over time right and so the obvious thing the easiest thing I could do would be is say okay for example I can you know I can make each of these guys be a linear function of what I had before right so for example you know what I could do is I could you know someone example this would be kind of like a simple filter on the you know if I wanted to amplify or decrease the motion of a joint what I could do is I could change this a of T to attenuate or amplify what’s going on right so if I wanted to kind of observe someone walking writing like this and how about I want to make them appear more cartoony what I might do is I might kind of just turn all the joints up by one point two and then that person’s gonna look kind of like over-exaggerated and I haven’t really recorded them doing that all I’m doing is I’m changing the application of these of these joint angles right so you could do some kind of fun things like that the other thing you could do is kind of you can imagine filtering these joint angles to do like slowing down or speeding up offense right so if I know that I’ve gotten from yeah that’s how I’ve done this kind of punched right and I’ve observed thirty frames of that I could interpolate those fails to get sixty frames of a

slow-motion punch right just by kind of interpolating between adjacent potatoes so you know you could basically do anything that you want to these captured joint angles and you know you can get to come some kind of fun looking emotions like that you’re starting to kind of stray from what actually was captured but for just kind of simple motion tuned up that’s a pretty straightforward thing you do so kind of a more natural or a more common thing to want to do is something like like this where you have a couple of motion capture sequences at the top like a walking motion and a running motion and you want to stitch those together but you don’t want them to immediately kind of stick together like a hard edge right and sense of what you want is that there is this nice smooth transition between you know the person walking and before they start to run they begin to kind of jog into the run right so you kind of want something where instead of going from you know motion a demotion be just bugged up against each other you have this kind of nice transition where I am interpolating from one to the other right and so how do I get the person to kind of jog naturally you know between the walk and the run and so kind of the way I think about this is kind of like a dynamic programming problem where you say okay I’ve got some I got I’ve got to sync up the frames as well as possible between motion one and motion two and then over the overlap I can kind of smoothly interpolate between the joint angles right and so for example here’s what I could do I could say okay if I have a cost function that says how close is frame I for a motion 1 to frame J emotion – how can I find kind of like the best path to match up frame to frame this chunk of motion right and so really it is what I would call a dynamic programming problem something secure that’s called dynamic time warping the idea being that there may be more phrase in image 1 then there are an image – so for example if this bottom motion was a walking motion and the and the vertical motion was a running motion then basically you can imagine that over one you know cycle of you know left foot to left foot right it takes the walker maybe 20 frames to do that and it takes the runner 10 frames to them because they’re moving faster right so what I’m doing is I’m kind of trying to match up as best as possible which of these ten frames match up as well as possible these 20 frames right and so one big consideration that might be for example looking at where the feet came down so here what I’m showing is that you know when the foot comes down there’s this black dot right and so for certain I want to you know make sure that the black dots match up between the two sequences because you might say okay what I’m going to do is I’m going to say I’m going to match the interval between these two foot plants to this longer interval between foot plants and then I’m going to generate you know I’m just kind of interpolate between the short amount of time it takes the walker between foot plants and the long time well not time but spaces like physical space covered in some sense so I kinda want to make the person kind of moved their feet gradually faster as I as I interpret between the motions and so really that’s nothing more than kind of a problem where you say ok I have a I have a cost function that says how similar so I have a cost function on poses that says ok you know so here this is kind of like this is the cost of like a matching path between pose sequence 1 and post sequence 2 and so what I might say is I have a cost between you know the vector of joint angles at this time and the vector of joint angles from the other motion at this other time and while I’m trying to do is I’m trying to minimize the overall cost between the poses to a given point in time right so it is that I kind of say okay if I have a if I have a cost function that tells me you know how similar are two poses like to instantaneous buzzers instant and I kind of sum that up over all the matching along the graph right this is exactly the kind of thing you can do with dynamic programming fairly easily okay one thing to keep in mind is that when I try and match these things up so this could be as simple as don’t ever say this could be as simple as like you know sum of squared differences maybe not the best idea but it could be done

like this so one thing to keep in mind is that again think about how we talked about parameterizing the forward kinematics sorry it’s just like saying okay you know this joint angle is always measured with respect to the root of the body right the root is this kind of 3d point inside your pelvis probably that kind of says okay you know this is the orientation and the position of the person so all these droid angles are kind of relative to that root so that’s like saying okay I don’t really care about the I don’t really care about the 3d angle in some coordinate system in space all I care about is what’s like relative to the body right so if I’m positioned like this and then I compare it to somebody who’s kind of like this the fact that we’re looking in different directions in their bodies point a different way shouldn’t make a big difference in terms of saying our my elbow and my shoulder joints lined up the right way alright so this is all kind of like these are all relative angles from the forward kinematics with respect to the roof and so once I’ve gotten the correspondence how do I actually do the interpolation right so now kind of say okay well I’ve got the you know angle from this person and the angle from that person that I want to kind of smoothly work between them so to actually interpolate interpolate between you know two poses well then what we have here is kind of an issue of trying to interpolate between angles or rotation matrices right and so the best way I do that is what’s called spherical linear interpolation or and so per that is that you know put simply if I’m thinking about you know how do I interpret between two poses right there are all these rotation and translation rotation matrices and translation factors tied up in that question right and so if I have a rotation matrix that describes you know one joint angle and I have another rotation matrix over here I don’t want to just like average since they want a halfway position I don’t want to average two rotation matrices right that doesn’t work because the reason is it doesn’t give me your rotation matrix so what do I think about this is that suppose I have a circle and I want to average this point in the circle this way in a circle you know that gives me this point and that’s not on the circle anymore right what I want to do is I want to triple in such a way that I’m always staying on the circle and so that means I kind of want to go half way from this point this point over here along the circle right and so this is a kind of a sketchy way of saying that I don’t want to just average rotation matrices I want to find the way where if I go half way from one to the other I’m always continuing to be your rotation matrix and I’ve already plot that path it would kind of look like taking a great circle along the surface of my globe instead of cutting through the center of the globe right as that’s one of the homework problems as I give you partition matrix one rotation matrix – what you do is you turn them into quaternions and then you apply this spherical linear interpolation formula to the quaternions you get a Nuke return Ian and you turn that back into a rotation matrix right that’s the best way to do it and so I’m not going to give away you know the rest of it but that’s basically one of the homework problems is to figure out how to do that it’s just a kind of a coordinate you know transformation problem and so let me show you a couple videos just to show kind of like how this process works so here’s a video that kind of again from Lucas Khobar that shows time warping and so the idea is that you know if I didn’t time warp between two motions and I just were to directly interpolate between corresponding you know positions in time you would see that the interpolation will look really weird right so like that was like bad interpolation let’s go back and see that again so if interpolating from walk to run you see that over that transition the person kind of does this weird like feet together hop that is physically implausible right right so that kind of weirdo skip is not something that you would ever see whereas if your time warp that’s kind of like showing that there’s just nice you know so here you’re seeing basically effective time warping between different kinds of motions running

speaking you know I believe these are like different interpolations you know between different positions I think the main thing I want to show let’s see that again is the time warping so basically you know if you do it right there’s this nice gradual speed-up and there are some other pictures let me show you this one so basically here you’re gonna see is basically transitions between different motions so you say here’s motion number one guy walking around motion number to dial walking like a crazy crab for some reason and now what you’re doing here is you’re saying okay this is where I want the crazy crab motion to start and then I move this guy and say this is where I want the transition to take place and then this algorithm basically puts these two together does the dynamic time warping and says okay now what I’ve got is you know neural walking motion and then smoothly transitioning into crazy crab motion right and I believe this is another example of interpolations so it’s like saying you’ve got two different kicks you know different ways of interpolating between those two different kicks that all look like reasonable looking kicks in different ways right so it’s kind of saying you know these are all kind of newly generated kicks that are built from just those two originals like you know who’s breakdancing guy one you know handspring guy – I know what you get are these kinds of interpolations between you know the the motions that all look like reasonable looking intermediate motions and so again you can imagine that this would be handy if you were doing something along the lines of you know authoring a video game where you didn’t want to capture like a hundred kicks you basically said okay instead what I want to do is I want to capture ten kicks and then the kick guide you on date you know on the moment is some combination of these ten kicks that doesn’t look quite like what the person has seen before so you kind of can mix it up by interpolating and this is kind of similar row shows just second but here’s an idea where they’re pushing the sky around depending on kind of real-time motion and what you’re seeing is interpolation between differents files and also a long different paths and so we’re going to talk about that in just a second and let’s see what else this is just a kind of a picture showing that what you need to do is align the coordinate frame so if I just were to take the input motions that if I would just kind of average those coordinate frames what I see is this guy who kind of like suddenly walks in place in this weird way and then flips around right we want to do is you want to kind of get something where the average of the blue guys is the red guy right and you do that with properly align the coordinate frames before you do the interpolation okay so let me talk a little bit about this notion of path editing okay and so one thing is that okay before I talk about path lighting let me talk about motion graphs so motion graphs again this is from Kovar at all the idea here is that okay you know I’ve got all this motion capture data that I’ve collected now I want to kind of find the best ways to string them together in as realistic a way as possible right and so this is like realistically strained together motions from a library and so kind of one thing that I wanna think about is well where is the best transition point to go from one motion to the other right so I’m so I’m going to do a little bit of blending between the motions probably but to make my life easier what I want to do is I want to find the points where its most imperceptible to go from motion one to motion to right if I’ve got a lot of motions in my library I should be able to do that and so here’s kind of a picture of I guess actually this is just kind of pictured somewhere I was reading before so here’s kind of an idea where I say okay you know what I do is I build this set of possible transitions right that’s like saying okay well certainly here the white dots are like frames from the vocab okay and certainly between any pair of adjacent frames from the same captured sequence I know that those

naturally go together right because they were captured from the same person so there are kind of arrows between every pair of adjacent frames but then there also may be some points where actually this sum this frame transitions nicely into this one and so I can draw an arrow that says hey you know this actually would be a reasonable place to jump from this motion over to that motion and what I can do is I can kind of build so here the light gray lines are like all sorts of possible transitions and the dark gray lines are places that I can kind of naturally jump from one to the other without you noticing right and so if I want to take a path through this graph what I might say is okay I take a little bit from motion one I jump over and Triple H at this point motion two I move here I jump over this point motion three I go along on that for a little while I go back to motion two I loop around back to motion one and go through this again so basically you can kind of find your way through the different actual captured mocap points you know if you do that right if you kind of assess how similar are any two given poses you can do that in a kind of realistic looking way so here’s kind of a sketch of say okay you know what i do as i look for between frames in one sequence and frames near sequence i have a cost function that says how similar those two things and the places where this cost function is really low right so think about this as white is like a bump and black is like a hollow right and so places where this thing is really low means the cost function is really pretty good to transition between this point in frame 1 and this point in frame 2 and so that could also be kind of periodic in a sense that you know if you imagine that one of these motions is like walking and while the motions is like running that’s natural I think there are lots of points where I could drop from one motion of walking into the other motion running basically every time that I’ve got like a foot that goes down is a good point for me to say ok that pose actually it’s pretty similar to the other guy I can swap over to there so it is that you know when I’m transitioning between frames from the same video sequence I just kind of play them back in order when I’m transitioning between frames from different video sequences I do this interpolation to make sure that I don’t have this weird you know skittering step that we fixed with our motion or pollution from before and so right so kind of what I would say is that you know that is to build a graph between all frames from all motions and then what you could say it’s kind of walk along the edges of the ground corresponding to good transitions and then kind of the final thing that you can do is you can kind of combine these things to you know or you can extend these things I can kind of like walk along predefined paths while doing certain motions so let me show you a couple videos about that but here’s the idea is that you know you have one captured version like this and now you want to make that person walk in a different direction right so you can do that with some of these techniques we talked about here I think that the yellow is the direction this is the you know the path that the user dragged and the black is what you can actually do with it with the data that you have and then here you see that you’re kind of making that person to walk a path that you never actually collected in real life but you’re stringing together you know a bunches of cycles from this observed motion to make it look pretty natural right and you can see this person can kind of do these abrupt loop-de-loops and turns and stuff like that or you can make them you know here the guy is fighting his way along the hello right so you’ve got lots of kickboxing data that you’re stringing together to make the person kickbox along a certain path and then here’s an example where you know not only are you going along a certain path but you are saying okay when you’re yellow walk normally and then when you’re green you know sneak around right and then when you’re blue I’m not sure what happens

when you’re blue we’ll find out crazy crowd now kickbox right and so again you can kind of basically draw on the floor what you want to happen and then as you pass the right point you kind of smoothly transition into doing the next thing and so these are all basically strung together shorter motion clips that you know are estimated to match up well with each other and then also kind of transformed so that you never see kind of this really bad transition so you do all sorts of goofy stuff like illustrate is it a turn it could be it could be a teenager be a ninja turtle yes and so you know there are lots of ways to kind of extend this idea I think that probably you know as you can read about in the in the kind of interview part of the chapter there are lots of times when you have to do something to the mocap data that you capture it’s it’s not really that typical to just immediately take what came from mocap studio and map it onto this character because you know you may be planning what their character does six months after you captured the vocab and all of a sudden you know instead of having you know you know three sword fights you’re gonna have two instance you have to kind of change the caption mocap data to fit what the narrative of the story is what the vision of the story is right and so you know you’re definitely in the in the process you’re living in a position of having to do a lot of vocab editing you know all the time of your productions and certainly I think that stuff I’ve talked about here is really relevant to stuff like video game design at mocap where you are clearly seeing these motions of these video game characters that look pretty realistic but obviously you can’t predict what all these motions that you need to do on the fly as a player is moving around to be right so if I am playing you know something like Skyrim with lots of people moving around you know if I shove somebody they have to be able to kind of like move realistically and then keep on moving along with their day or you never necessarily recorded you know show right you have to kind of figure out okay I’m gonna push them over this way a little bit and see what happens right yeah so it’s interesting stuff okay so any questions can’t be done in real time yeah I don’t think the interpolation is that bad so basically what you do is you pre compute yes actually it can be done in real time for sure in video games right so basically you’ve got a whole list of pre computed motions and then you’re kind of on the fly toggling between them and interpolating between them to make sure they look good right so i’m not i sir i don’t know anything about exactly how it happens under the hood but video games were proof that you can basically do mow caps you know editing in real time yeah the boovie stuff obviously you’re gonna spend a lot more time refining all the little subtle motion nuances because you know you can rewind a tree watch a movie but you can’t do that with a video game right so if something looks a little bit weird on a video game you’re willing to let it go because you’re in the middle of something right you’re you’re actively participating as opposed to just kind of passively watching and picking away at how bad it was yeah so why don’t I talk about next time is a couple things one is facial motion capture right so all the stuff we’re talking about so far is pretty much body motion captured right like if you remember from the vocab visit we had last week there were no markers on the face so I was only these headband markers that gave me coarse positions ahead so what do you do when you’ve got someone’s face if you want to capture and so we’ll talk a little about how that works with markers all it turns out that in the real world a lot of facial stuff is not done with markers in quite the same way the other thing I want to talk about is can you do this kind of joint estimation without having someone go into the skin type bodysuit with the goofy ping pong balls I mean is there a way to do it with purely video based normal street clothes what I would call marvelous motion capture and so certainly that’s been a big area of interest in the Puritan community for time is you know look at a video find people and find out what they’re doing right so I mean there are some things from if your vision you can transfer over to do this kind of markerless motion capture it sounds good by any means as market motion capture but still got its uses so we’ll talk a little bit about how far you can go without the Mart okay so with that I will shut myself down here