Discussion: Definition of blend shape / morph target names

[aaronfranke]

The skeleton bone hierarchy is the most important part of a character/avatar standard. The Reference Canonical Skeleton Framework proposal already covers this - great! However, there is more than just skeleton bones that could be defined.

I noticed that the CSV document defines a bunch of face bones, such as nose, chin, cheek, mouth, lip, brow, etc: https://github.com/meshula/LabRCSF/blob/b99b7742d2802679b88f4081ce5eda44a9cef57a/joints.csv

I'm not against having such bones defined, especially if it allows OpenUSD bones to be consistently transcribed to/from RCSF and glTF. However, from what I have seen, facial animation is far more common to do with blend shapes, also known as shape keys in Blender terminology, also known as morph targets in glTF terminology.

In my specification, G4MF, I have attempted to define standard names for these. Character/avatar blend shapes can be divided into 2 groups:

• Visemes, short for "visual phonemes", are blend shapes that represent the visual aspect of phonemes during human speech. They are used for lip syncing, the process of matching a character's mouth movements to spoken words.
  • I have a proposed set of names here, as a superset of what platforms provide: G4MF Viseme Blend Shape Naming.
    • These names are basically the same as other platforms, except with a consistent "Viseme" prefix on all of them. Some use "E" instead of "EE", VRM uses single letters only, but I went with always 2 letters for consistency and clarity.
    • As far as I can tell, my definition is the only one to actually include IPA symbols. Other places use English letters only, plus pictures sometimes, but are ambiguous on the sounds, especially for a non-English speaker.
• Face tracking is the process of using a camera to track a person's facial movements, and applying those movements to a character/avatar's face. While visemes are used for lip syncing to audio, face tracking is used to mimic a much broader range of facial expressions.
  • I have a proposed set of names here, as a superset of what platforms provide: G4MF Face Tracking Blend Shape Naming.
    • These are basically the same as "Unified Expressions" from VRCFaceTracking, which already did the work of unifying several face tracking standards together, except that I prefixed all the names with "Face".
    • I am far from an expert on face tracking, so I'm basically just blindly trusting that VRCFT did a good job. I'd say that more research is required before declaring a standard. Anyway, see their docs for their much more detailed comprehensive documentation.
• Aside from those 2 groups, there are plenty of models out there with blend shapes like "Happy", "Sad", "Angry", etc. But for these types of things, any level of standardization feels inappropriate, so I propose we decide to ignore these.

With these in place, it is possible that we don't have a need for the facial bones, but again I'm not necessarily proposing that they be removed if others have a genuine use for them. Regardless, defining blend shapes is vital for enabling interoperability for characters/avatars that have them. It is possible that these are out-of-scope for the RCSF repo, but it should still be considered, and it still has to be defined somewhere.

[meshula]

I agree that the "soft" features like lips don't belong in a joint hierarchy. We might declare where they are attached to a joint hierarchy though, as a separate table.

Another twist with visemes is that to be useful we have to specify the composition of joint and blend deformations of a skin to compute the final vertex positions.

So another layer to this would be to write out some math...

[fire]

In Godot Engine we made an effort to synchronize with VRCFT face tracking defintions which blends FACS, ARKIT, SRanipal and others https://docs.godotengine.org/en/stable/classes/class_xrfacetracker.html

There is some emerging technologies for audio to IPA vismes. openai/whisper#318 (comment)

See also audio to FACS facial blend shapes. https://github.com/NVIDIA/Audio2Face-3D

Also some research on ARKIT to FACS https://melindaozel.com/arkit-to-facs-cheat-sheet/

[fire]

If we use the current VRM / KHR_avatar definition of 0.0-1.0 blend shape values as a synonym for 0.0s to 1.0s facial bones, technologies like https://www.ea.com/seed/news/open-source-dem-bones (Blender addon https://superhivemarket.com/products/blender-dembones) allow us to switch from one type to the other.

[Kjakubzak]

I'd recommend keeping facial bones in the translation framework; they're used all over the industry and not having them be present would be an odd omission. There's still a wide use of them in pipelines, games, etc.. Even in the case of characters where you have expression morphs; they more likely than not had a joint rig for the face (and the original joints and joint-form of the expressions were thrown out after the morphtargets were generated).

For a current example of a product using joint-based facial animation, Roblox uses them for animating character faces:
https://create.roblox.com/docs/art/characters/facial-animation

Also, I think there's a bit of a conflation here with expression and actuated geometry/joints. Functionally this is like saying we can throw away the leg joints because there are representations of leg animations in mesh animation form. It's a separate topic entirely.

IMO any explorations on expression set translation frameworks should be a related, but different repository/project altogether to ensure RCSF stays clean and tightly-focused. FWIW, we're proposing a follow-up exploration specifically on this topic in the MSF 3D Asset Interop meetings. As part of that separate effort, it definitely makes sense to reference this effort and note what surveyed face bones would (likely) actuate for specific expressions.