Empty RPU

[Anuskuss]

I've extracted a RPU from a profile 5 MP4 and I'd like to convert it to 8.1.

Frames: 202264
Profile: 5
DM version: 1 (CM v2.9)
Scene/shot count: 1163
RPU mastering display: 0.0001/1000 nits
RPU content light level (L1): MaxCLL: 996.13 nits, MaxFALL: 109.50 nits
L6 metadata: Mastering display: 0.0001/38528 nits. MaxCLL: 38528 nits, MaxFALL: 38528 nits
L5 offsets: top=N/A, bottom=N/A, left=N/A, right=N/A
L2 trims: 100 nits, 600 nits, 1000 nits

JSON

{
  "dovi_profile": 5,
  "header": {
    "rpu_nal_prefix": 25,
    "rpu_type": 2,
    "rpu_format": 18,
    "vdr_rpu_profile": 0,
    "vdr_rpu_level": 0,
    "vdr_seq_info_present_flag": true,
    "chroma_resampling_explicit_filter_flag": false,
    "coefficient_data_type": 0,
    "coefficient_log2_denom": 23,
    "coefficient_log2_denom_length": 23,
    "vdr_rpu_normalized_idc": 1,
    "bl_video_full_range_flag": true,
    "bl_bit_depth_minus8": 2,
    "el_bit_depth_minus8": 2,
    "ext_mapping_idc_0_4": 0,
    "ext_mapping_idc_5_7": 0,
    "vdr_bit_depth_minus8": 4,
    "spatial_resampling_filter_flag": false,
    "reserved_zero_3bits": 0,
    "el_spatial_resampling_filter_flag": false,
    "disable_residual_flag": true,
    "vdr_dm_metadata_present_flag": true,
    "use_prev_vdr_rpu_flag": false,
    "prev_vdr_rpu_id": 0
  },
  "rpu_data_mapping": {
    "vdr_rpu_id": 0,
    "mapping_color_space": 0,
    "mapping_chroma_format_idc": 0,
    "num_x_partitions_minus1": 0,
    "num_y_partitions_minus1": 0,
    "curves": [
      {
        "num_pivots_minus2": 7,
        "pivots": [
          0,
          173,
          110,
          109,
          110,
          109,
          110,
          109,
          193
        ],
        "mapping_idc": "Polynomial",
        "poly_order_minus1": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
        ],
        "linear_interp_flag": [
          false,
          false,
          false,
          false,
          false,
          false,
          false,
          false
        ],
        "poly_coef_int": [
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ]
        ],
        "poly_coef": [
          [
            7775741,
            1414859,
            0
          ],
          [
            7775741,
            1414859,
            0
          ],
          [
            7775741,
            1414859,
            0
          ],
          [
            7775741,
            1414859,
            0
          ],
          [
            7775741,
            1414859,
            0
          ],
          [
            7775741,
            1414859,
            0
          ],
          [
            7775741,
            1414859,
            0
          ],
          [
            7775741,
            1414859,
            0
          ]
        ]
      },
      {
        "num_pivots_minus2": 3,
        "pivots": [
          0,
          288,
          224,
          224,
          287
        ],
        "mapping_idc": "Polynomial",
        "poly_order_minus1": [
          1,
          1,
          1,
          1
        ],
        "linear_interp_flag": [
          false,
          false,
          false,
          false
        ],
        "poly_coef_int": [
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ]
        ],
        "poly_coef": [
          [
            7789421,
            1196032,
            0
          ],
          [
            7789421,
            1196032,
            0
          ],
          [
            7789421,
            1196032,
            0
          ],
          [
            7789421,
            1196032,
            0
          ]
        ]
      },
      {
        "num_pivots_minus2": 3,
        "pivots": [
          0,
          288,
          224,
          224,
          287
        ],
        "mapping_idc": "Polynomial",
        "poly_order_minus1": [
          1,
          1,
          1,
          1
        ],
        "linear_interp_flag": [
          false,
          false,
          false,
          false
        ],
        "poly_coef_int": [
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ],
          [
            -1,
            1,
            0
          ]
        ],
        "poly_coef": [
          [
            7789421,
            1196032,
            0
          ],
          [
            7789421,
            1196032,
            0
          ],
          [
            7789421,
            1196032,
            0
          ],
          [
            7789421,
            1196032,
            0
          ]
        ]
      }
    ]
  },
  "vdr_dm_data": {
    "compressed": false,
    "affected_dm_metadata_id": 0,
    "current_dm_metadata_id": 0,
    "scene_refresh_flag": 1,
    "ycc_to_rgb_coef0": 8192,
    "ycc_to_rgb_coef1": 799,
    "ycc_to_rgb_coef2": 1681,
    "ycc_to_rgb_coef3": 8192,
    "ycc_to_rgb_coef4": -933,
    "ycc_to_rgb_coef5": 1091,
    "ycc_to_rgb_coef6": 8192,
    "ycc_to_rgb_coef7": 267,
    "ycc_to_rgb_coef8": -5545,
    "ycc_to_rgb_offset0": 0,
    "ycc_to_rgb_offset1": 134217728,
    "ycc_to_rgb_offset2": 134217728,
    "rgb_to_lms_coef0": 17081,
    "rgb_to_lms_coef1": -349,
    "rgb_to_lms_coef2": -349,
    "rgb_to_lms_coef3": -349,
    "rgb_to_lms_coef4": 17081,
    "rgb_to_lms_coef5": -349,
    "rgb_to_lms_coef6": -349,
    "rgb_to_lms_coef7": -349,
    "rgb_to_lms_coef8": 17081,
    "signal_eotf": 65535,
    "signal_eotf_param0": 0,
    "signal_eotf_param1": 0,
    "signal_eotf_param2": 0,
    "signal_bit_depth": 12,
    "signal_color_space": 2,
    "signal_chroma_format": 0,
    "signal_full_range_flag": 1,
    "source_min_pq": 7,
    "source_max_pq": 3079,
    "source_diagonal": 42,
    "cmv29_metadata": {
      "num_ext_blocks": 5,
      "ext_metadata_blocks": [
        {
          "Level1": {
            "min_pq": 2,
            "max_pq": 2081,
            "avg_pq": 1229
          }
        },
        {
          "Level2": {
            "target_max_pq": 2081,
            "trim_slope": 1771,
            "trim_offset": 2019,
            "trim_power": 1235,
            "trim_chroma_weight": 2048,
            "trim_saturation_gain": 2048,
            "ms_weight": 2048
          }
        },
        {
          "Level2": {
            "target_max_pq": 2851,
            "trim_slope": 2059,
            "trim_offset": 2048,
            "trim_power": 1955,
            "trim_chroma_weight": 2048,
            "trim_saturation_gain": 2048,
            "ms_weight": 2048
          }
        },
        {
          "Level2": {
            "target_max_pq": 3079,
            "trim_slope": 2049,
            "trim_offset": 2048,
            "trim_power": 2047,
            "trim_chroma_weight": 2048,
            "trim_saturation_gain": 2048,
            "ms_weight": 2048
          }
        },
        {
          "Level6": {
            "max_display_mastering_luminance": 38528,
            "min_display_mastering_luminance": 1,
            "max_content_light_level": 38528,
            "max_frame_average_light_level": 38528
          }
        }
      ]
    }
  },
  "rpu_data_crc32": 135159509
}

But it just results in an empty RPU (Final metadata length: 0). Is this a bug or is the RPU broken? I even tried Mode 0 ("rewrites it untouched") but that also didn't work:

EditConfig {
  "mode": 0,
  "remove_cmv4": false,
  "remove_mapping": false
}
Parsing RPU file...
Final metadata length: 0
Writing RPU file...

[quietvoid]

Is this a bug or is the RPU broken?

I don't know, share the RPU.

[quietvoid]

It's not displayed but the L6 is invalid.

Condition failed: self.max_display_mastering_luminance <= MAX_PQ_LUMINANCE (38528 vs 10000)

If you had extracted with mode 0 you would have hit an error.
Clearly we need a way to validate an already extracted RPU.

[Anuskuss]

So what shall I do now? Is the extraction process broken or the actual RPU data in the MP4? Do you want the first X MBs of the MP4 to investigate?
Also why did you close the issue? Is it on your internal todo list or did you close it accidentally?

[quietvoid]

Nothing is broken, your RPU has invalid metadata so it can't be rewritten.
Use the editor to fix the L6.

[Anuskuss]

You mean like this?

{
  "mode": 2,
  "level6": {
    "max_display_mastering_luminance": int,
    "min_display_mastering_luminance": int,
    "max_content_light_level": int,
    "max_frame_average_light_level": int
  }
}

I think I'm just gonna copy those values from the non-DV version then (the L1 MaxCLL is 54% and the MaxFALL is 14% of the HDR10 values but I'm assuming that as long as the real values are smaller it's gonna be fine). Since it's all ints I guess it doesn't want the MediaInfo values (e.g. min: 0.0050 cd/m2, max: 1000 cd/m2) but rather the ffprobe values ("min_luminance": "50/10000", "max_luminance": "10000000/10000"), right? Unfortunately that doesn't seem to work:

Error: invalid value: integer 10000000, expected u16 at line 1 column 63

Giving it seemingly "incorrect" values (1000 and 50) gives me a fixed RPU though:

L6 metadata: Mastering display: 0.0050/1000 nits. MaxCLL: 1839 nits, MaxFALL: 766 nits

P.S. Not to be annoying but doesn't that trigger any alarm bells in your head? Like is the data in the RPU really invalid or does dovi_tool write invalid data? The file plays fine on my LG so the original data is either correct, or the TV silently ignores it. 38528 is a pretty unremarkable number though.

[quietvoid]

L6 follows SMPTE ST2086 and is for ST2084 which means the limit is 10k nits.
I don't care if there are files with higher values, they shouldn't be valid.

Either way, the level is practically useless and isn't used by devices.

Since it's all ints I guess it doesn't want the MediaInfo values (e.g. min: 0.0050 cd/m2, max: 1000 cd/m2) but rather the ffprobe values ("min_luminance": "50/10000", "max_luminance": "10000000/10000"), right? Unfortunately that doesn't seem to work:

No, it expects nits values except for min_display_mastering_luminance which is nits * 10 000.

[quietvoid]

With changes from #377 we get this kind of output to the user:

Failed writing invalid RPU: Index 0
  CM v2.9: Level 6: Condition failed: `self.max_display_mastering_luminance <= MAX_PQ_LUMINANCE` (38528 vs 10000)
Final metadata length: 0

It only displays the first error but should be plenty to know what's going on.