ECMWF IFS GRIBs in S3: kerchunk template and reuse for multi-year reference creation

[nishadhka]

A follow-up to the GFS / GEFS template-and-.idx-merge approach discussed in #530 and #572.
We've applied the same Grib-Index-Kerchunk (GIK) pattern to ECMWF IFS 0.25° ensemble forecasts (https://registry.opendata.aws/ecmwf-forecasts/ s3://ecmwf-forecasts/, anonymous, 51 members × 85 timesteps per run hour, ~3–4 GB per timestep file packing all members together) and want to share the result + ask for feedback.

The ECMWF-specific difference vs GEFS

• ECMWF publishes one GRIB file per (date, run-hour, timestep) packing all 51 members together (vs GEFS's one-file-per-member). So a single GRIB file is huge but its sidecar is small.
• The sidecar is .index (JSON-lines, one object per message with _offset, _length, param, number, step, ...) rather than GEFS's plain-text .idx. This made the Stage-2 byte-position lookup a direct JSON-lines parse, no kerchunk._grib_idx.parse_grib_idx / map_from_index needed at runtime.

The 4 steps as applied to ECMWF

1. Template (one-time, frozen at reference date 20240529) ecmwf/ecmwf_index_preprocessing.py runs kerchunk._grib_idx.build_idx_grib_mapping over every (member, timestep) = 51 × 85 ≈ 4,335 GRIB files on the reference date. The per-(member, timestep) mapping parquets are tarred into gik-fmrc-v2ecmwf_fmrc.tar.gz (~30 MB) and published to HuggingFace + baked into the production Docker image. After this, kerchunk's GRIB-scanning machinery is never invoked again.
2. Index (every new date): fetch each daily .index sidecar (a few KB), parse the JSON-lines, extract [_offset, _length, param, number, step] per message. No GRIB bytes downloaded.
3. Build & save parquet: for each of 51 members, walk the deflated zarr store recovered from the rt000 slice of the bundled tar.gz, inject the freshly-discovered [s3_url, byte_offset, byte_length] triplets keyed by zarr chunk path, and write one ~140 KB parquet per member: {date}{run}z-{member}.parquet. Implemented as a self-contained Lithops worker on Cloud Run (ecmwf/lithops-cr-gik-ecmwf/run_lithops_ecmwf.py), fexec.map(...) fans out one container per date, so a multi-year backfill runs in parallel without a long-running driver.
4. Stream → xarray: read the parquet, fetch only the needed byte ranges from S3, decode chunks with gribberish (Rust-based, ~25 ms/chunk vs cfgrib's ~2 s), wrap in an xr.Dataset. Single-member helper: ecmwf/run_single_ecmwf_to_zarr_gribberish.py. Distributed Coiled-Dask version: ecmwf/stream_cgan_variables_coiled_simple.py.

Published parquet references (HuggingFace)

E4DRR/gik-ecmwf-par - per-member parquets covering 2024-03 → 2026-02 across all four run hours, plus monthly aggregates with row-group-level filter pushdown (one row group per date, row_group_size=102, zstd) so a single-date filter against an 80 MB monthly file pulls only ~few MB via HF range requests.

Validation against Herbie — bit-equivalence on the source bytes

Because GIK and Herbie ultimately both read the same source bytes out of s3://ecmwf-forecasts/, the only thing that can differ is the decode path. We validated this directly:

• Script: ecmwf/validate_gik_vs_herbie_2025.py — for a sweep of dates, downloads ECMWF's TP via Herbie + cfgrib, streams the same TP via the GIK parquet refs + gribberish, computes per-pixel and aggregate diffs, writes side-by-side maps + JSON metrics.

The validation artefacts (NetCDFs of GIK-streamed vs Herbie-fetched TP, side-by-side comparison PNGs, JSON metrics) are published alongside the parquet refs at gik-ecmwf-par/validation_gik_vs_herbie_2024.

With the template precomputed and shipped, kerchunk's scan_grib / grib_tree / build_idx_grib_mapping machinery becomes a one time step which can be duplicated for different dates. The production daily run never imports kerchunk.grib2 at all; it just opens a baked-in parquet and parses JSON-lines .index sidecars. The cost per date ends up around $0.092 blended on Cloud Run without expensive scan_grib of 4.5GB grib files. The dataset at https://registry.opendata.aws/ecmwf-forecasts/ from 20240131 only has 0p25 dataset available, the period from 202301 onward it is in the form 0p4-beta/ and have to make specific template for these periods.

[martindurant]

Super cool, thanks for posting. Are there any next steps you foresee?
I think this is the first kerchunk dataset I've seen published at huggingface.

[nishadhka]

Thanks for the Kerchunk library. The next steps involve addressing the issue of encoding the pressure levels and exploring the Icechunk virtual form of the Parquet files.

[TomNicholas]

@nishadhka you can also do this with VirtualiZarr - see https://www.earthmover.io/blog/virtual-grib-nbm

[nishadhka]

Thanks @TomNicholas for the VirtualiZarr tooling and the GIRB history blog post — the GTS/undersea-cable framing is a great read. I'll dig into the methodology, especially the DataTree structuring in VirtualiZarr.

On the grib-index-kerchunk(GIK) side: we've converted our Parquet references(made using Lithops cloudruns https://github.com/icpac-igad/grib-index-kerchunk/blob/main/ecmwf/lithops-cr-gik-ecmwf/run_lithops_ecmwf.py) into Icechunk virtual stores for the full ECMWF ENS corpus (2023-01-18 → 2026-05, 00z) and the GEFS corpus (2020 → 2026, 00z), both published on source.coop:

• ECMWF: https://gist.github.com/nishadhka/3917c3d1b5391bb97c65fd98b06f6ca7
• GEFS: https://gist.github.com/nishadhka/8f2c3191b51aa7e7670c8979081484bc

What makes the GIK Lithops path cost-effective is that Kerchunk scan_grib runs once to build a reusable template per product (a single cycle — ~85 ECMWF step-files at ~5–6 GB each); every subsequent date is then resolved from its index files alone. Across the corpus that's roughly 106,000 ECMWF .index files (one per date × step, all 51 members bundled, ~2 MB each) and
~4.9 million GEFS .idx files (one per date × member × step, ~1.5 KB each), about 5 million index reads totalling ~214 GB, versus the ~712 TB of GRIB they reference (~1/3000th the bytes). That's why an earlier full Parquet build of the ECMWF 00z corpus came in around US$25–40.

The index sidecars (.idx/.index) are the real enabler here, and I see GribberishParser(use_index=...) already uses them to locate messages, is cross-file template reuse (deriving the structure once per stream, then resolving every other cycle from the index alone) something VirtualiZarr's parser is considering, or is it per-file?

[TomNicholas]

is cross-file template reuse (deriving the structure once per stream, then resolving every other cycle from the index alone) something VirtualiZarr's parser is considering

We could add a feature like this to the GribberishParser. Please raise an issue on the Gribberish repo describing your problem and suggesting how you would like the solution to work.