Improvement suggestion: use a yml card as script input to manage processing of all the simulations

[pmathiot]

To organise myself during the processing of all the simulation, I add option to specify a yml card as input. The yml contains, the regions, sections and moorings you want to process, the simulation you want to process and the metadata for the output file like this (NEMO4.0_OceanA-hind.yml):

processing:
  #'Amundsen' and, or 'Weddell'
  Amundsen:
    sections: [1,2]             # section number (1,2 for Amundsen, 1,2,3 for Weddel). Need to be the same lenght as regions.
    moorings: [1,2,3,4,5,6,7,8] # mooring number ([1:8] for Amundsen and [1:6] for Weddell). Need to be the same lenght as regions.

  Weddell:
    sections: [1,2,3]
    moorings: [1,2,3,4,5,6]

simulation:
# General settings for ISOMIP2 data processing
  test_case: "eORCA025_prod"  # Choose between NEMO_test, MITGCM_test, ROMS_test, or eORCA025_test

# Directory where data is stored (supports placeholders like {test_case})
  data_dir: "test_cases/oce/eORCA025.L121-OPM026"

# Model-specific configurations
  model: "NEMO4.0"        # Model name and version
  original_sim_name: "eORCA025.L121-OPM026"  # Description of the simulation

# Metadata for the experiment
  institute: "IGE-CNRS-UGA"  # Institute name (use "-" for multiple entities)
  abc: "a"                   # Single letter to distinguish setups
  exp: "OceanA-hind"         # MISOMIP2 experiment name (e.g., Ocean-A1, Ocean-W1)

metadata:
  project: "MISOMIP2"
  contact: "...."  # Name(s) of the person(s) who produced the simulation <email>
  computing_facility: "France/GENCI/TGCC/Irene/Rome"                 # Computing center where the simulation was run
  interpolation_method: "linear triangular barycentric"              # e.g., 'linear triangular barycentric', 'bi-linear', 'nearest-neighbor', 'conservative'
  reference: "Mathiot and Jourdain (2023): Mathiot, P. and Jourdain, N. C.: Southern Ocean warming and Antarctic ice shelf melting in conditions plausible by late 23rd century in a high-end scenario, Ocean Sci., 19, 1595–1615, https://doi.org/10.5194/os-19-1595-2023, 2023."                   # Main publication or website describing the simulation
  bathymetry: "BedMachine-2"                  # Bathymetry dataset (e.g., 'BedMachine-1.33', 'Bedmap2')
  ice_shelf_draft: "BedMachine-2"             # Dataset for the depth of ice-shelf base
  atmosphere: "JRA55do (Tsujino et al. 2018)" # Atmospheric forcing, with a reference (e.g., 'ERA5 (Hersbach et al. 2020)')
                                              #    `ERAint (Dee et al. 2011)', `JRA55do (Tsujino et al. 2018)',
                                              #    `MARv3.9.3 (Donat-Magnin et al. 2020)'
  iceberg: "Lagrangian model (March et al., 2015)"  # Method used to account for melting icebergs, with a reference,
                                                    #     e.g. `None', `Lagrangian model (Martin and Adcroft 2010)',
                                                    #          `Prescribed freshwater (Merino et al. 2016)',
                                                    #          `Prescribed Freshwater and Heat (Merino et al. 2016)' 
  sea_ice: "Dynamics-Thermodynamics Model (SI3) [NEMO Sea Ice Working Group: Sea Ice modelling Integrated Initiative (SI3) – The NEMO sea ice engine, Zenodo, https://doi.org/10.5281/zenodo.3878122, 2019. a] "                  # Method used to simulate or prescribe the ocean--sea-ice interaction,
                                     # with a reference, e.g. 'Dynamics-Thermodynamics Model (Hibler 1979)',
                                     #                        'Thermodynamics Model (Bitz and Lipscomb 1999)',
                                     #                        'Prescribed Freshwater and Heat'
  ocean_lateral_bdy: "None"          # Type of lateral boundary conditions, e.g. 'Simulation (Merino et al. 2018)',
                                     #     'Reanalysis (Mazloff et al. 2016)', 'Observations (Locarnini et al. 2018)',
                                     #     'Corrected simulation (explain method)', 'None' 
  tides: "Parameterized through non-uniform tidal velocity (CATS2008)"   # Method used to account for the effect of tides on ice-shelf melt, and dataset
                                                                         #     e.g. `Barotropic tidal harmonics prescribed at lateral boundaries (CATS)',
                                                                         #          `Forced by a tidal potential', `None',
                                                                         #          `Parameterized through uniform tidal velocity (utide=0.1 m s-1)',
                                                                         #          'Parameterized through non-uniform tidal velocity (FES2012)'
  vertical_coordinate: "Stretched Geopotential (Zstar)" # e.g. `Geopotential (Z)', `Stretched Geopotential (Zstar)', `Pressure (P)',
                                                        #      `Stretched Pressure (P*)', `Isopycnal', `Terrain-Following (Sigma)',
                                                        #      `Arbitrary Lagrangian-Eulerian (ALE)'
  is_melt_param: "3-equation (velocity-dependent gamma)"   # Parameterization used to calculate ice-shelf basal melt rates,
                                                           #      e.g. `3-equation (constant gamma)', `3-equation (velocity-dependent gamma)',
                                                           #           `3-equation (stability and velocity-dependent gamma)'
  eos: "TEOS10"                      # Equation of state in original simulation (e.g., 'TEOS10', 'EOS80')
  advection: "tracer: flux corrected scheme 2nd order ; momentum: vector form - 2nd centered scheme"     # Brief description of the momentum and tracer advection schemes 
                                                                           #      (centered, third-order with limiter, etc)
  horizontal_mixing: "biharmonic"    # Brief description of how ``horizontal'' mixing was performed 
                                     #      (harmonic, biharmonic, etc.; within model levels, along geopotentials, 
                                     #       along isopycnals, etc.; using the Gent-McWilliams parameterization; etc)
  vertical_mixing: "Modified version of the Gaspar et al. (1990) turbulent kinetic energy (TKE) scheme (Madec, 2016)" 
                                     # Brief description of how ``vertical'' mixing was performed (constant diffusivity,
                                     #       k-profile parameterization, etc.; harmonic, biharmonic, etc)
  convection: "Parameterized using enhanced vertical mixing"  # Brief description of the procedure for handling convection, 
                                                              #       e.g. `Explicitly modeled', `Parameterized using enhanced vertical mixing'

If there is some interest, I can build a dummy yaml card for the default proposed by the interp script in example.

Let me know if there is some interest as it will disrupt a bit the usage.

[dgwyther]

I love this idea @pmathiot !

[dgwyther]

@pmathiot Just a note that you didn't add a *.yaml file in your latest commit. An example file would be good so we can adapt our own?

[pmathiot]

From my pull request, you should find it in examples/test_cases/oce. If you simply run the interpolation script as before, only the NEMO_example is run (default path: test_cases/oce/NEMO_test.yml

[dgwyther]

My apologies - I thought i pulled your branch but it didn't add those yaml files - I can see them in the GitHub repo though. Looking forward to testing it out with ROMS.

…

On Fri, 29 May 2026, 7:57 pm Pierre Mathiot, ***@***.***> wrote: *pmathiot* left a comment (misomip/misomip2#3) <#3 (comment)> You should find it in examples/test_cases/oce. If you simply run the interpolation script as before, only the NEMO_example is run (default path: test_cases/oce/NEMO_test.yml — Reply to this email directly, view it on GitHub <#3?email_source=notifications&email_token=ABIZWM5DLYHY3SCRVIKCIS345FNIZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJXGM2TIMRQG42KM4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4573542074>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIZWM5R6V2Z5GSC4QGECML45FNIZAVCNFSM6AAAAACZJJSHBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKNZTGU2DEMBXGQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ABIZWM3PBDSVYGYMNCQCG3T45FNIZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJXGM2TIMRQG42KM4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/ABIZWM7KGSBZSARNCGKBHCL45FNIZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJXGM2TIMRQG42KM4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you commented.Message ID: ***@***.***>

[pmathiot]

@dgwyther : as you may have looked at it and tested it, do you mind doing the review of the pull request ?

[dgwyther]

@pmathiot I'll have a good look at it - I'm just waiting to see if I can run the script with my full dataset - there seems to be a process using huge amounts of RAM.

[pmathiot]

I found too that the memory consumption is huge. Once my HPC is back, I will create a branch with my version (triangulation and memory). The idea for the memory were to:

• remove as much as possible the temporary array.

For example:

tmp = interp(....)
out['var1'] = tmp

Become:

out['var1'] = interp(....)

• Do vertical and then strait away the horizontal array and then del (or re-use) the remaining temporary variables.

But anyway, I need use a full HPC node for about 1.5h to run 10x1 years.

[dgwyther]

Yep, I'm also getting huge memory use and long wall-times.
It takes ~12 hrs to run the load script, and at least as long again to run the interpolate script. Memory usage seems to be over 300G, at which point I ran out of memory on the machine.

I've split the processing at the load script - now it will save the output from the load script into a netcdf. I then need to wipe the memory and load the dataset (100G in size for the 1992-2016 dataset). Unfortunately, python isn't great with memory management, so it's not easy to free up memory, other than killing the process and starting from clean. With a 100G dataset, it's easy to see how a few temporary arrays can rapidly increase the usage.

The process may be significantly sped up by using dask and running in parallel, but I haven't experimented with this yet.

[dgwyther]

Some comments @pmathiot

1. if run from the base directory, the yaml is not found.

(devel3.11) gadi-cpu-clx-1110: /g/data/jk72/deg581/amundsen-isom/ana/misomip2$ python examples/interpolate_to_common_grid_oce.py
card test_cases/oce/NEMO_test.yml not found

Maybe a variable can be set for the base directory path location, then update the load_arguments() accordingly:

analysis_path = '/Users/jourdain/MY_SCRIPTS'
sys.path.append(analysis_path)
...
misomip_path = analysis_path+"/misomip2/"
...
parser.add_argument("--config"  , type=str, default=misomip_path+'test_cases/oce/NEMO_test.yml', help="Path to YAML configuration file (optional).")

Alternatively, we could just direct the user to cd misomip2/examples first...

2. It would be nicer to have a more upfront specification of the config file:

parser.add_argument("--config"  , type=str, default='test_cases/oce/NEMO_test.yml', help="Path to YAML configuration file (optional).")

Setting the default argument earlier on above the function definitions would make it clearer.

3. The specification of the files looks like its a bit bespoke between models. e.g. NEMO has files as NEMO_m012_psi.nc (for example) while ROMS has files like ROMS_avg_1992.nc (for example). This could be hardcoded into the loading section:

f_ALL  = [data_dir+'/'+test_case+'_m0'+month.astype('str')+'.nc' for month in np.arange(1,4)]

        oce = mp.load_oce_mod_roms( files_M=f_grid, files_T=f_ALL, rho0=1026.0, teos10=False, region=reg )

But it might be nice to expand the yaml settings to specify more of the file name string. I know you have it so that data_dir and test_case are specified in the yaml, but it would be more general if the general format of the filename could be specified in the yaml.

What do you think?

I think that this update is a great addition - nice work!