ECMWF's public open-data S3 bucket starts around April 2024. To complete the IFS ENS record on dynamical.org, we need to retrieve the historical period from ECMWF's MARS tape archive and stage it on S3 as GRIBs for downstream reformatting.
Scope
- Period: 2016-03-08 to 2024-04-01 (cycle 41r2+, when ENS resolution increased to 18km)
- Parameters: sp, 2t, 10u, 10v, 100u, 100v, tp, strd, ssrd, msl, 2d, ptype (surface) + z at 500 hPa
- Members: 0 (control) + 1–50 (perturbed)
- Steps: 3-hourly 0–144h, 6-hourly 150–360h (85 steps)
- Resolution: 0.25°
- ~2,946 dates, ~117 GB/date, ~350 TB total
Approach
A staging script submits requests to the MARS API, polls for completion, downloads the GRIBs, generates byte-range index files (.grib.idx), and uploads both to S3. State is tracked in SQLite for full resumability. MARS is tape-backed, so requests are ordered to minimize tape contention — estimated 3–8 weeks wall time with 3 API keys.
ECMWF's public open-data S3 bucket starts around April 2024. To complete the IFS ENS record on dynamical.org, we need to retrieve the historical period from ECMWF's MARS tape archive and stage it on S3 as GRIBs for downstream reformatting.
Scope
Approach
A staging script submits requests to the MARS API, polls for completion, downloads the GRIBs, generates byte-range index files (.grib.idx), and uploads both to S3. State is tracked in SQLite for full resumability. MARS is tape-backed, so requests are ordered to minimize tape contention — estimated 3–8 weeks wall time with 3 API keys.