Pipeline architecture

The Python publication pipeline exposes the windgram CLI and the versioned documents it writes as its supported interface.

Publication flow from model cycle to browser

A run is published only after completeness and derivation checks, then read as static files.

A five-stage sequence from upstream forecast publication through completeness probing, model-specific building, one repository commit, and a consumer consistency check.

1. 1
   Upstream model productsECCC or NOAA publishes a forecast cycle.
2. 2
   Completeness probeRequest the final required hour before fetching the run.
3. 3
   Per-model builderFetch required records; verify; derive; serialize.
4. 4
   Workflow commitCommit every completed model directory and append one history record per site and run.
5. 5
   Consumer readFetch manifest and profile; compare referenceTime; expose a torn pair as stale.

Git repository publicationmanifest.json · site profiles · append-only history

Builders complete independently. Each polling cycle commits every finished model update once; consumers compare independently cached static files and choose their own stale-pair policy.Units Ordered stages; no numeric scale

Inside the builder stage, provider bytes move through module-owned steps:

models.json + sites.json
              │
provider transport ──→ gridpoint sampling ──→ source-shaped hours
                                                   │
                                                   ▼
                                      derive_windgram_profile
                                                   │
                                                   ▼
                                   contract rounding + validation tests
                                      │            │             │
                                      ▼            ▼             ▼
                                  profile      manifest      history

Responsibility by module

Area	Repository home	Responsibility
CLI and scoped paths	pipeline/src/windgram/cli.py, pipeline/src/windgram/config.py	Select model(s), validate site/output paths, cap smoke steps, dispatch safely
Provider clients	datamart.py, noaa.py, grib.py, geotiff.py	Fetch, range-read, decode, sample, and account for transport work
Builders	builders/*.py	Verify run completeness, request declared fields, preserve absence, assemble source hours
Derivation	derive.py	Produce published profile blocks and model-dependent derived values
Ensemble aggregation	ensemble.py	Aggregate member profiles, including circular wind and censored counts
Publication	publish.py	Round contract fields, write JSON, append gzip history, build run index
Teaching scenarios	scenarios.py	Generate fixed inputs through the same derivation authority

Authority boundary

Which quantities the pipeline owns and which belong to the npm package — including the one parameterized exception — is defined once in the project overview. The windgram derivations define the pipeline equations, constants, fallbacks, and renderer-only transformations.

Generated scenarios

Scenario generation starts from committed inputs with fixed identity and time, then calls the same profile derivation used by builders. Teaching figures consume the generated scenario registry.

The builder contract defines the required inputs, validation, and publication behavior for model modules.