Skip to content
WindgramDocs

Provider transports

The repository uses two broad transport strategies. They share download accounting, site sampling guards, builder invariants, and the profile contract while preserving each provider’s storage semantics.

Two transports, one job

NOAA exposes indexed byte ranges while ECCC publishes whole files, so each builder follows its provider's transport contract.

A comparison of NOAA indexed byte ranges and ECCC whole-domain streaming. Four HRRR records are located at byte offsets read from the repository fixture for 2026-08-06 12Z; ECCC builders stream one file at a time, sample configured sites, and discard the file.

Byte ranges versus whole-domain streamsTransport A, indexed byte ranges, NOAA HRRR and GFS: a plain-text .idx sidecar names each record's byte offset inside the GRIB2 object — for example record 449,HGT:850 mb, at byte 216,059,484 of the 2026-08-06 12Z run — so the builder issues one HTTP range request per needed record and the rest of the408-plus-megabyte file never leaves the bucket. Transport B, whole-domain stream, ECCC HRDPS, RDPS, GDPS, REPS, and GEPS: Datamart has no index and one message per file, so the builder fetches each whole-domain file, samples the catalogued launches in memory, and drops it before the next fetch — 4 to 8 gibibytes moved per deterministic run, about 9 for REPS, and about 14 for GEPS, while only kilobytes of profile JSON per site are kept and published.AINDEXED BYTE RANGESNOAA · HRRR, GFS.idx SIDECAR — PLAIN TEXT, FREE1HGT:850 mbbyte 216,059,4842UGRD:850 mbbyte 221,543,7463TMP:2 mbyte 329,529,8794DPT:2 mbyte 333,678,240hrrr.t12z.wrfprsf24.grib2 · run 2026-08-06 12Z · one record per field and level1234unshaded bytes never leave NOAA's bucketthe index alone places recordsbeyond byte 407,874,688 — ≥ 408 MBONE RANGE REQUEST PER NEEDED RECORDmegabytes cross the network; the file stays on the shelfBWHOLE-DOMAIN STREAMECCC · HRDPS, RDPS, GDPS, REPS, GEPS — Datamart has no index1 · FETCHthe whole domain,one message per file2 · SAMPLE IN MEMORYread once; keep only thecatalogued launch cells3 · DROPreleased before thenext fetch begins↺ repeat, file after file, through the run — memory never holds more than a handful of files4–8 GiBmoved per deterministic run;≈ 9 GiB REPS · ≈ 14 GiB GEPSkilobytes keptper site and run — the profile JSONis what actually gets published

Fixture: tests/fixtures/hrrr.t12z.wrfprsf24.excerpt.idx

Record numbers and offsets come from the committed HRRR index fixture; run-volume context comes from the project measurements recorded with the pipeline research.Units byte offsets and transferred GiB where labelled
Strategy Current home Data movement Important boundary
ECCC Datamart GRIB datamart.py, grib.py, ECCC builders Fetch one whole-domain field file, sample all sites, release bytes Paths, field tokens, accumulations, sentinels, and schedules are model declarations
NOAA Open Data indexed GRIB noaa.py, NOAA builders Read .idx, fetch only byte ranges for needed records, sample sites Record names, level strings, grid rotation, and accumulation windows are model-specific

Preserve provider semantics

Section titled “Preserve provider semantics”

Two fields with the same output unit can describe different windows. For example, a precipitation rate can be an instantaneous diagnostic or a mean over the publishing step. A gust can be an instantaneous diagnostic or an hour maximum. Transport code recovers the provider quantity; the builder’s verified semantics declaration keeps its meaning attached to the profile.

Sampling and domains

Section titled “Sampling and domains”

Grid readers sample the nearest model point for every configured site and check distance. A distant result usually means a library clamped an out-of-domain coordinate to the grid edge. The builder must reject that sample. Model terrain elevation is sampled separately and remains distinct from surveyed launch elevation.

Projected grids may publish winds relative to grid north. Where provider metadata requires it, builders rotate components to true north before computing speed and meteorological FROM-direction. Regular latitude/longitude earth-relative grids require no such correction. These are verified feed facts, not global transport assumptions.

Failure and accounting

Section titled “Failure and accounting”

The shared manifest core records downloads, bytes, retries, and duration. Transport-specific numeric counters may be added, but consumers cannot build logic on those unstable extension keys. Missing required records fail a build; optional capability records may remain absent only when catalogue and builder behaviour agree.

Live-provider evidence belongs in the dated forecast feed reference.