Discussion: Bone orientation including bone roll

[aaronfranke]

In sufficiently complex implementations, it is completely reasonable to expect a skeleton to be retargeted on import. Doing this operation has a variety of benefits, and is recommended for implementations to do.

However, that said, not all implementations are able to do this. Very simple metaverse implementations may lack the capability to retarget. To support these cases, we should, as part of Reference Canonical Skeleton Framework, define not only the names and positions of bones, but also the rotations of those bones. This allows simple implementations to only worry about setting bone rotations without performing complex retargeting. However, aside from supporting such applications, it is still a generally useful concept to define a single recommended set of bone orientations.

In every application I have seen, bones "point" in their local +Y direction. Runtimes often don't really have this concept cemented inside them, because they just deal with transforms, but any time a tool does care about bone direction, it's +Y. This is even consistent across coordinate systems, in Y-up applications unrotated bones point up, but in Z-up applications, bones still point in the +Y direction, so they're horizontal.

For the hips/spine/chest/neck/head/eyes, the bone orientation is straightforward: these already point up, so they match up perfectly with an unrotated glTF node. For the rotation matrix of these bones, +X can point in the +X direction, +Y can point in the +Y direction, and +Z can point in the +Z direction, with minor perturbations to make the bones point in slightly different directions.

What I have seen as more conflicting is the roll orientation of the arm and leg bones. Mixamo and Ready Player Me skeletons have their arms rolled such that their local +Z direction points down towards the hips, and have their legs rolled such that their local +Z direction points forward. Thus, for the upper arms, upper legs, and lower legs, the rotation around the local X axis serves as the primary rotation.

However, I have also seen avatars which do the polar opposite, with for example the legs having their local +Z direction pointing back, and I believe I've seen other rolls as well. One interesting fact is that, relative to a glTF-unrotated bone (pointing up), the Mixamo/RPM leg bones are effectively rolled 180 degrees around the Z axis, thus flipping the X and Y axes, while if the Z was pointing back as I've seen on some other avatars, those bones are effectively rolled 180 degrees around the X axis, thus flipping the Y and Z axes.

Which system is better, or the most common, or worthy of becoming the standard? I don't know, thus I am opening this as a discussion rather than a proposal. Is there any existing standard which these avatars follow, or are we just witnessing a wild west of character/avatar bone rolls because nobody has defined a standard yet? I'm not sure, but it would be worth doing research into this. I have not yet found evidence of a commonly accepted industry standard for bone roll.

Regardless, there is just one statement I want to declare, which is that we should define something as the recommended roll orientation for all the bones. If we do not standardize bone roll now, it becomes harder later, because it invalidates previously-compliant existing models which don't match up with the new specification.

[meshula]

I agree, however I'd like to have the specification be built on layers. Like the joint hierarchy defines topology a topology of bones in terms of joints, and the bone specification defines the geometry of the bones. I think your skeleton.md document that describes geometry would be a good starting point.

[aaronfranke]

I chatted about this with @fire and he pointed me to this article which talks about the work that @TokageItLab has done in this area.

The article links to the Godot retargeting documentation https://docs.godotengine.org/en/stable/tutorials/assets_pipeline/retargeting_3d_skeletons.html#rest-fixer which explicitly states a preferred bone roll, written in terms of rotation:

+X rotation bends the joint like a muscle contracting

This is a simple rule, which makes a lot of sense. Out of the arbitrary ways to do it, prefer the one that results in a positive-numbered rotation. This rule matches the bone roll found in the Mixamo / Ready Player Me screenshot above.

I have not dug too deep into the details, but @fire told me over voice chat that him and @TokageItLab looked into the existing ecosystem, and I remember he mentioned that they referenced something from Microsoft, and that this approach matches what Microsoft was doing, and I think he said one other thing they referenced that I forgot what it is.

It's probably good to do more research just in case, but in general, I think we've found a winner here... everyone is doing it this way. Godot, VRM, Mixamo, and Microsoft all have this as the preferred bone roll.

[meshula]

"prefer the rotation that results in a positive-numbered rotation."

That does make a lot of sense.

[fire]

I cannot find the original samples when we looked.

However you can see how https://learn.microsoft.com/en-us/previous-versions/azure/kinect-dk/body-joints defines bone roll orientation (the default rotation).

[fire]

Please see the strongly argued positions for defining a joint's bone roll (vrm-c/vrm-specification#337) in Japanese.

The argument resulted in VRM0 changing to return setting a bone roll of Quaternion identity to be any bone roll (we weren't able to argue for a specific bone roll in vrm1).