MeshHandler: Fix weightsData dequantization error#338
Merged
K0lb3 merged 2 commits intoK0lb3:masterfrom Aug 8, 2025
Merged
Conversation
In the Editor's serialization code, normalized weights are quantized as 5-bit UNORM values then packed as ints. This implies flooring - while the sum of the quantized values are guaranteed to be `31` this can lead to a loss of precision where the _de_quantized sum (div by 31 then sum) of weights is less than 1.0. Such cases can be trivally reproduced by search, for example a tuple of (1,25,5) summed as (1/31+25/31+5/31) in _that exact order_ will yield 0.9999999999999999 instead of 1.0, and ((1/31+25/31+5/31) >= 1.0) will be falsey. This PR implements the sum of weight with ints instead. Alternatively an epsilon value could be chosen, but this is what the player code does.
K0lb3
reviewed
Aug 1, 2025
Owner
|
Thanks for the write-up and the example. |
K0lb3
requested changes
Aug 1, 2025
Owner
K0lb3
left a comment
K0lb3
left a comment
There was a problem hiding this comment.
Just the removal of the mentioned line, and then it can be merged.
K0lb3
approved these changes
Aug 8, 2025
K0lb3
pushed a commit
that referenced
this pull request
Aug 9, 2025
fix(MeshHandler): fix `weightsData` dequantization error In the Editor's serialization code, normalized weights are quantized as 5-bit UNORM values then packed as ints. This implies flooring - while the sum of the quantized values are guaranteed to be `31` this can lead to a loss of precision where the _de_quantized sum (div by 31 then sum) of weights is less than 1.0. Such cases can be trivally reproduced by search, for example a tuple of (1,25,5) summed as (1/31+25/31+5/31) in _that exact order_ will yield 0.9999999999999999 instead of 1.0, and ((1/31+25/31+5/31) >= 1.0) will be falsey. This PR implements the sum of weight with ints instead. Alternatively an epsilon value could be chosen, but this is what the player code does.
K0lb3
pushed a commit
that referenced
this pull request
Aug 11, 2025
fix(MeshHandler): fix `weightsData` dequantization error In the Editor's serialization code, normalized weights are quantized as 5-bit UNORM values then packed as ints. This implies flooring - while the sum of the quantized values are guaranteed to be `31` this can lead to a loss of precision where the _de_quantized sum (div by 31 then sum) of weights is less than 1.0. Such cases can be trivally reproduced by search, for example a tuple of (1,25,5) summed as (1/31+25/31+5/31) in _that exact order_ will yield 0.9999999999999999 instead of 1.0, and ((1/31+25/31+5/31) >= 1.0) will be falsey. This PR implements the sum of weight with ints instead. Alternatively an epsilon value could be chosen, but this is what the player code does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the Editor's serialization code, normalized weights are quantized as 5-bit fixed point UNORM values then packed as ints.
This implies flooring - while the sum of the quantized values are guaranteed to be
31this can lead to a loss of precision where the dequantized sum (div by 31 then sum) of weights is not correct with IEEE float.Such cases can be trivally reproduced by search, for example a tuple of$$(1,25,5)$$ summed as $$(\frac{1}{31}+\frac{25}{31}+\frac{5}{31}$$ in that exact order will yield $$0.9999999999999999$$ instead of $$1.0$$ , and
((1/31+25/31+5/31) >= 1.0)in Python interperter will evalutate asFalse.This PR implements the sum of weight with ints instead. Alternatively an epsilon value could be chosen, but this is what the player code does.
Attachments in #333 can reproduce this issue. See screenshots below.