Add scripts/aimip_postprocessing for CMIP6-compliant AIMIP result processing#1023
Add scripts/aimip_postprocessing for CMIP6-compliant AIMIP result processing#1023
Conversation
…cessing Converts the 2025-11-26 notebook workflow into a reproducible script that reads raw ACE inference netCDFs from GCS, applies CF-convention transforms, and uploads to GCS and DKRZ. Includes typed config dataclasses with YAML-backed defaults, a Click CLI, and unit tests for key helpers.
- Replace click CLI with argparse (_get_parser + main pattern) - Remove DKRZ upload (s3fs, dotenv, upload_to_dkrz, DKRZ_REMOTE_PREFIX) - Remove create_environment Makefile target (fme env sufficient as-is) - Add cftime to pyproject.toml dependencies - Simplify README
| return ds | ||
|
|
||
|
|
||
| def stack_vertical_dimension( |
There was a problem hiding this comment.
Feels like we have some version of the same function in many places (e.g., this is used for some of the coupled analysis work), would it make sense to consolidate them? I think it's kind of the idea for that ace-tools repo though we didn't want to rigorously maintain it.
In fact, this applies to many functions in this file. Just wondering if want to consider unify them.
There was a problem hiding this comment.
Agree that there's a lot of duplication overall, and if we make some of these postprocessing steps part of the the real ACE workflow we'd consolidate some of the code.
At least here, I'll look into whether using the existing Stacker might work.
There was a problem hiding this comment.
It looks like the current Stacker hardcodes that variable names are <name>_<number> which doesn't work here because we lack the underscore. But this is good motivation to maybe try to refactor this code into the data writer and use a common set of tools in a future PR.
| } | ||
| ) | ||
| result = stack_vertical_dimension(ds, "plev", "ta", r"[0-9]+$") | ||
| assert list(result["plev"].values) == [100.0, 500.0, 850.0] |
There was a problem hiding this comment.
Does it sort correctly if you have 100., 500., 850., 1000.?
There was a problem hiding this comment.
I think it does, but can test, or use the stacker.
| dynamic = ["version"] | ||
| dependencies = [ | ||
| "cartopy>=0.22.0", | ||
| "cftime", |
There was a problem hiding this comment.
Should this be added here or analysis-deps.txt?
There was a problem hiding this comment.
While it's unlikely that xarray would ever drop cftime as a transitive dependency in our repo, we directly import cftime in a some places already in ace (eg here) because we like to write CF calendar time coordinates, so I figured it was probably worth adding explicitly. But probably not critical either way.
|
Don't need to address making the common functions widely available in this PR, but I think it makes sense to have some sort of shared common ace analysis tools. Still not sure what's the best way. |
2 participants
Notebook from explore2 2025-11-26-process-aimip-inference-results.ipynb contains a one-off workflow for post-processing ACE2.1-ERA5 AIMIP inference results into CMIP6-compliant netCDF files. This PR converts it to a script alongside the related
scripts/aimip_forcing/pipeline.To some extent, this PR is coupled to the configurations used to run inference on ACE2.1-ERA5 and specify the names of the data writer files, which is being merged in a separate PR. Maybe there's useful added functionality to handle these "CMIP-style" runs automatically in ACE without the need for a postprocessing script, but leaving that for later.
Changes:
scripts/aimip_postprocessing/postprocess.py— script with typedSimulationConfig/FileConfigdataclasses, YAML-backed defaults, and all CF-convention transformation helpers from the notebookscripts/aimip_postprocessing/simulations.yaml/files.yaml— default configuration lists (15 simulations, 48 variable/table/grid entries)scripts/aimip_postprocessing/test_postprocess.py— unit tests for key helpersscripts/aimip_postprocessing/README.md/Makefile— usage docs and convenience targetsTests added
If dependencies changed, "deps only" image rebuilt and "latest_deps_only_image.txt" file updated