Update vertical coordinate with deptho during inference

[elynnwu]

We previously did not include sea floor depth (deptho) to account for partial bottom cells which caused vertical integral such as ocean heat content to be inaccurate. We have since updated the dataset to incorporate deptho in the vertical coordinate, however existing checkpoints do not have this information and cause inference/evaluator to have the wrong ocean heat content. This PR automatically updates the vertical coordinate in an existing checkpoint with deptho from the data, all other fields stay the same (e.g., idepth, mask).

• Tests added

[elynnwu]

Verification here.

Prediction loader uses a checkpoint with deptho in the vertical coordinate, whereas *-update_deptho uses SamudrACE-CM4 GRL checkpoint and a coupled ocean dataset with deptho and gets updated automatically from using the feature in this PR. They are not identical because they start at slightly different IC (0310-12-29T12:00:00 vs. 0311-01-01T00:00:00).

[oliverwm1]

Is it possible to do this in an offline script (i.e. save a new version of the checkpoint with the updated vertical coordinate) instead of complicating the generic ACE/coupled inference entrypoints?

[elynnwu]

Is it possible to do this in an offline script (i.e. save a new version of the checkpoint with the updated vertical coordinate) instead of complicating the generic ACE/coupled inference entrypoints?

Yes, we can do this offline instead. The original intention was to not merge this, since we just need it for baseline comparison in wandb. James and I thought it could be helpful to do this automatically and not have to think about it, but I agree that it makes the inference/evaluator code messy. @jpdunc23 thoughts?

[jpdunc23]

Is it possible to do this in an offline script (i.e. save a new version of the checkpoint with the updated vertical coordinate) instead of complicating the generic ACE/coupled inference entrypoints?

Yes, we can do this offline instead. The original intention was to not merge this, since we just need it for baseline comparison in wandb. James and I thought it could be helpful to do this automatically and not have to think about it, but I agree that it makes the inference/evaluator code messy.

Was hoping this could be done a bit more unobtrusively. Possibly at the load_stepper() level but not sure that is going to be any better. Agreed this is too specific, so maybe best to just use this branch as needed to create updated baselines while leaving the existing checkpoints as is.