FROM: ACCESS-NRI Intake Catalog --> TO: ESMValTool .yml Recipe 'Datasets'

[almek]

Hi all,

I'm sorry for not posting this sooner. I had to complete a few edits before posting this.
In case anyone is interested, during the CMIP7 Hackathon in Aspendale I've developed a simple Python notebook that takes the "catalog" output of the ACCESS-NRI Intake, and returns the 'datasets' input for the .yml recipe of ESMValTool. You can find the code at the end of this post.

I hope this will help anyone who doesn't want to manually write the list of datasets in the .yml file each time.
Some further edits are needed on the code, such as getting the 'exp' section of the .yml from the ACCESS-NRI intake catalog. I am not sure the intake catalog has this coded. Also, 'start year' and 'end year' are different between CMIP5 and CMIP6 phases. It would be nice if we could automatise all these entries.

To access the ACCESS-NRI Intake catalog I have opened an ARE session on Gadi following the instructions at this link:
https://access-nri-intake-catalog.readthedocs.io/en/latest/usage/how.html

I hope this helps.
Cheers,
Alberto

====
CODE:

import warnings
warnings.filterwarnings("ignore") # Suppress warnings for these docs

import intake
import yaml

catalog = intake.cat.access_nri
freq = '3hr'
var = 'uas'

df = catalog.search(frequency=freq, variable=var).df

datasets = []
for index, row in df.iterrows():
entry = {
'dataset': row['model'][0],
'project': row['name'],
'exp': 'historical',
'ensemble': 'r1i1p1f1',
'start_year': '1985',
'end_year': '2014'
}
datasets.append(entry)

Write the updated YAML to a new file

with open('datasets.yml', 'w') as output_file:
output_file.write("datasets:\n")
for entry in datasets:
output_file.write(f"- {{dataset: {entry['dataset']}, project: {entry['project']}, exp: {entry['exp']}, ensemble: {entry['ensemble']}, start_year: {entry['start_year']}, end_year: {entry['end_year']}}}\n")

[hot007]

@paolap this is what we were discussing could be done with Clef or Intake, FYI if you're interested in looking at how hard it'd be to plug into Clef as an optional output type?

[rbeucher]

Thanks for sharing @almek! That's useful.

[paolap]

@almek what is "dataset" in the final yaml? Is that the official dataset_id ? Is it needed by ESMValTool or is just a way to distinguish between records?
I have something nearly ready to go but it would help to have an example output file needed by the tool. Also is this different for CMIP5/6 or Cordex?

[almek]

Hi @paolap ,

I'm sorry for not getting back to you sooner. I will check this thread more often from now on.
At this link is an example of an ESMValTool evaluation .yml.

The idea behind these few lines of code was to avoid the annoying part of manually populating the datasets section.
I thought that linking the ACCESS-NRI Intake with the ESMValTool .yml input would help speed up this process. It would be nice if we could extend this also to Clef tool that you made.

I think that it shouldn't vary much between CMIP5, CMIP6 or CORDEX datasets, as long as these products have the same variables nomenclature.

Let me know if you need any additional info.

Best,
Alberto