Skip to content

Use new (misc) keys in MTG2 actions#251

Open
tweska wants to merge 5 commits intodevelopfrom
fix/misc-prefix
Open

Use new (misc) keys in MTG2 actions#251
tweska wants to merge 5 commits intodevelopfrom
fix/misc-prefix

Conversation

@tweska
Copy link
Copy Markdown
Member

@tweska tweska commented Mar 24, 2026

New prefixed (with misc-) keys should be used in scale, statistics-mtg2 and encode-mtg2 actions.

If you review this PR, I suggest going commit by commit.

The main changes in this PR:

  • All non-mars keys are prefixed with misc- in their definition
  • The MiscRecord can be dumped without this misc- prefix, this is needed in the encode-mtg2 action
  • Removed unused code in the statistics-mtg2 action
  • Created records for the metadata in the statistics-mtg2 action to enable the proper parsing/defaulting upon reading

Some improvements that we should discuss + my opinion:

  • There are still direct accesses to the metadata in the statistics-mtg2 action, this should be refactored further to access everything through the record
  • We parse the record (and create the config) several times in statistics-mtg2, it should be parsed once and then passed around
  • FullMarsRecord and MiscRecord are defined outside the encode-mtg2 action, but to my knowledge only used there, we should define the records in that action header instead
  • The FlushKind enum is not really a statistics-mtg2 specific thing, it should be moved
  • May need to discuss serverRank and restartDateTime keys
  • MarsMiscGeo.h now contains some random keys which are not mars, nor grib, perhaps we should create a new file MultioKeys.h?
  • We should check misc-bitmapPresent before reading misc-missingValue Implemented for scale and average-rate actions
  • misc-bitmapPresent and misc-missingValue don't have default values, this is especially confusing for the bitmapPresent. It can now be optional, and it is unclear if that means the bitmap is present or not. I should either be a required key, or defaulted to true/false. Defaulted bitmapPresent to false

🌈🌦️📖🚧 Documentation 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/dev-section/multio/pull-requests/PR-251

@tweska tweska force-pushed the fix/misc-prefix branch 2 times, most recently from e7ed8dc to 4249573 Compare March 25, 2026 11:49
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 25, 2026

Codecov Report

❌ Patch coverage is 53.70370% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.32%. Comparing base (210c956) to head (4b03c9a).
⚠️ Report is 5 commits behind head on develop.

Files with missing lines Patch % Lines
src/multio/action/statistics-mtg2/MetadataKeys.cc 27.02% 27 Missing ⚠️
src/multio/datamod/core/EntryDumper.h 0.00% 13 Missing ⚠️
...c/multio/action/statistics-mtg2/OperationWindow.cc 40.00% 12 Missing ⚠️
src/multio/action/statistics-mtg2/Statistics.cc 63.63% 8 Missing ⚠️
...ion/statistics-mtg2/cfg/StatisticsConfiguration.cc 87.80% 5 Missing ⚠️
src/multio/action/average-rate/AverageRate.h 57.14% 3 Missing ⚠️
src/multio/action/encode-mtg2/EncodeMtg2.cc 0.00% 3 Missing ⚠️
src/multio/action/scale/Scale.h 62.50% 3 Missing ⚠️
...statistics-mtg2/mappings/StatisticsParamMapping.cc 83.33% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #251      +/-   ##
===========================================
+ Coverage    56.30%   56.32%   +0.01%     
===========================================
  Files          321      323       +2     
  Lines        21177    21118      -59     
  Branches      1684     1686       +2     
===========================================
- Hits         11924    11894      -30     
+ Misses        9253     9224      -29     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tweska tweska marked this pull request as ready for review March 30, 2026 08:50
tweska added 5 commits April 9, 2026 07:43
Prefix all misc keys and add `dumpUnscopedRecord` method
Access new keys through records to apply checks and defaults
Rename missingValue -> misc-missingValue
Also default bitmapPresent to false instead of allowing it to be unset
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants