Skip to content

Add pickle backed load cache to fetch_fcst#1905

Draft
jfrost-mo wants to merge 1 commit intomainfrom
loading_cache
Draft

Add pickle backed load cache to fetch_fcst#1905
jfrost-mo wants to merge 1 commit intomainfrom
loading_cache

Conversation

@jfrost-mo
Copy link
Member

This does the loading of the data into cubes (with all associated callbacks) once in the fetch_fcst step, vastly speeding up the loading during recipe baking.

The read operator is updated to use this cache when it is provided a directory, and that directory contains a loadcache.pickle file. Otherwise loading will continue as normal.

TODO

  • Unit testing
  • Validation that it is actually faster (currently running)
  • The callbacks that realise data need fixing to not do that.

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

  • Documentation has been updated to reflect change.
  • New code has tests, and affected old tests have been updated.
  • All tests and CI checks pass.
  • Ensured the pull request title is descriptive.
  • Conda lock files have been updated if dependencies have changed.
  • Attributed any Generative AI, such as GitHub Copilot, used in this PR.
  • Marked the PR as ready to review.

This does the loading of the data into cubes (with all associated
callbacks) once in the fetch_fcst step, vastly speeding up the loading
during recipe baking.

The read operator is updated to use this cache when it is provided a
directory, and that directory contains a `loadcache.pickle` file.
Otherwise loading will continue as normal.
@jfrost-mo jfrost-mo self-assigned this Feb 10, 2026
@jfrost-mo jfrost-mo added the enhancement New feature or request label Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant