Environment Parity for Spatial Pipelines

Most “works on my machine” spatial bugs trace back to environment parity for spatial pipelines — a GDAL or PROJ version that differs by a point release between a developer’s laptop and a production worker, quietly changing resampling defaults or which datum-shift grid a transform selects. This page shows how to close that gap: assert the exact native library versions and PROJ grid availability the moment a worker boots, install the same lock across dev, CI, and prod, and detect the datum-grid differences that shift coordinates before they corrupt output. A startup assertion is the single highest-leverage guard you can add.

When to Use This Pattern#

• A reprojection produces subtly different coordinates in production than in development.
• Developers run macOS, CI runs Linux, and prod runs a different Linux image — three chances for skew.
• A pipeline upgraded GDAL and mosaic seams or nodata handling changed without any code change.
• You have a lockfile from Pinning Geospatial Dependencies with conda-lock but no runtime proof that every worker actually honors it.

Why Version Skew Moves Coordinates#

The reason spatial skew is so dangerous is that it rarely raises an exception. A generic web service that drifts between library versions usually throws or returns an obvious error; a geospatial pipeline instead returns a plausible but wrong coordinate. Understanding the two mechanisms makes the guard below make sense.

The first mechanism is algorithmic. GDAL’s default resampling, overview generation, and nodata propagation can change subtly across minor releases. A mosaic built with GDAL 3.6 and one built with 3.8 may differ at seams by a fraction of a pixel — invisible in a spot check, real in a change-detection analysis downstream. Nothing errors; the numbers are simply not identical, which breaks any workflow that assumes byte-stable output, including the content-checksum comparison a canary rollout relies on.

The second mechanism, and the more insidious one, is datum-grid selection. When you transform between EPSG:4326 and a national grid like EPSG:2056, PROJ looks for a high-accuracy datum-shift grid file. If that grid is present, you get sub-centimetre accuracy; if it is absent, PROJ does not fail — it falls back to a “ballpark” transform that ignores the datum shift and can be off by a metre or more. A developer laptop with grids installed and a production image without them will therefore produce coordinates that differ by exactly the datum offset, with no error anywhere in the logs. This is the single most common cause of “the same code gives different coordinates in prod,” and it is why the guard below checks both the PROJ version and whether a real transform silently degraded to ballpark accuracy.

Complete Working Example#

Parity is enforced at two moments: install time (everyone installs the same lock) and boot time (the worker refuses to start if reality disagrees with the pin). The startup guard below runs before the worker accepts any task. It asserts GDAL, PROJ, and GEOS versions, confirms the PROJ data directory resolves, checks that a named datum-shift grid is actually present, and performs a canonical transform whose result is bounded — catching the grid-selection differences that silently move features.

"""Startup parity guard — import and call assert_spatial_parity() before
the Prefect/Dagster worker begins pulling tasks."""
from __future__ import annotations

import sys
import rasterio
import pyproj
import shapely
from pyproj import Transformer, CRS
from pyproj.datadir import get_data_dir

# The single source of truth — must equal the locked conda-lock.yml versions.
PINNED: dict[str, str] = {
    "gdal": "3.8.4",
    "proj": "9.3.1",
    "geos": "3.12.1",
}

# A datum-shift grid this pipeline requires. Missing grids are the classic
# cause of coordinates that differ by a metre or more between machines.
REQUIRED_GRID: str = "ch_swisstopo_CHENyx06a.tif"

def assert_spatial_parity() -> None:
    # 1. Native library versions must match the lock exactly.
    actual: dict[str, str] = {
        "gdal": rasterio.__gdal_version__,
        "proj": pyproj.proj_version_str,
        "geos": ".".join(str(n) for n in shapely.geos_version),
    }
    mismatched = {k: (actual[k], PINNED[k]) for k in PINNED if actual[k] != PINNED[k]}
    if mismatched:
        raise RuntimeError(f"native version skew vs lock: {mismatched}")

    # 2. PROJ data directory must resolve, else every transform degrades.
    data_dir: str = get_data_dir()
    if not data_dir:
        raise RuntimeError("PROJ_LIB unset; datum grids unavailable")

    # 3. The specific datum-shift grid must be on disk, not just referenced.
    transformer = Transformer.from_crs(
        CRS.from_epsg(4326), CRS.from_epsg(2056), always_xy=True
    )
    op = transformer.description or ""
    # pyproj marks a ballpark (grid-free, low-accuracy) transform explicitly.
    if "ballpark" in op.lower():
        raise RuntimeError(
            f"transform fell back to ballpark accuracy; grid {REQUIRED_GRID} missing"
        )

    # 4. A canonical point must land inside its expected bounds.
    easting, northing = transformer.transform(7.4474, 46.9480)  # Bern
    if not (2_590_000 < easting < 2_610_000):
        raise RuntimeError(f"reprojection drift: easting {easting:.2f} out of range")

    print(f"spatial parity OK — {actual}, grids from {data_dir}")

if __name__ == "__main__":
    try:
        assert_spatial_parity()
    except RuntimeError as exc:
        print(f"PARITY CHECK FAILED: {exc}", file=sys.stderr)
        sys.exit(1)

Wire this into the worker’s entrypoint so a skewed image never processes a single feature. In Prefect, call it from a flow-run or worker startup hook; in Dagster, invoke it from a resource setup before ops run.

from prefect import flow

@flow(name="reproject-tiles")
def reproject_tiles(tile_index: str) -> None:
    assert_spatial_parity()  # abort before any spatial work if the env drifted
    # ... open rasters with rasterio, reproject, write COGs ...

Where to Run the Guard#

A parity assertion only helps if it runs before the worker does real work, and in the right places. Put it at three checkpoints. The first is the container HEALTHCHECK or entrypoint, so an image whose native stack does not match the lock never reports healthy — a fast, coarse gate that catches gross skew before the orchestrator routes any task to the worker. The second is the worker or flow startup hook, shown above, which runs inside the Python process the tasks will actually use and therefore catches a mismatch that a shell-level check could miss, such as a PROJ_LIB that resolves for the OS but not for the interpreter’s pyproj. The third is CI, where the golden-coordinate test runs against the freshly built image so drift is caught before publish rather than after deploy.

Keep the assertion cheap and loud. It should add milliseconds, not seconds, to startup, and on failure it must exit non-zero with a message naming the exact mismatch — GDAL 3.6.4 != 3.8.4, not a generic stack trace. That specificity is what turns a 3am page into a two-minute fix: the on-call engineer sees precisely which library drifted and on which pool. Feed the failure event into your alerting so a parity break is treated as a deploy-blocking incident, not a warning that scrolls past in the logs.

Parameter & Option Reference#

Verification & Testing#

Beyond the boot guard, compare the two environments directly. A quick CLI diff of gdalinfo --version and the PROJ grid list across dev and a prod container catches skew before deploy:

# Run identically in each environment and diff the output.
gdalinfo --version
projinfo --list-crs | wc -l           # rough CRS catalog size
ls "$(python -c 'from pyproj.datadir import get_data_dir; print(get_data_dir())')" \
    | grep -c '\.tif$'                 # count installed datum-shift grids

Capture the golden values once, on the pinned environment you consider authoritative, and commit them next to the test. From then on the test is a tripwire: any change to the resolved PROJ version or grid set that would move a coordinate fails CI with a concrete before/after, rather than surfacing as a mysterious analytical discrepancy weeks later. Run it in the same job that builds the image so the environment under test is exactly the one that will ship.

For a regression test that fails CI when a transform result moves, freeze a golden coordinate and assert against it:

from pyproj import Transformer, CRS

def test_bern_reprojection_is_stable() -> None:
    t = Transformer.from_crs(CRS.from_epsg(4326), CRS.from_epsg(2056), always_xy=True)
    easting, northing = t.transform(7.4474, 46.9480)
    # Golden values captured against the pinned PROJ 9.3.1 grid set.
    assert round(easting, 2) == 2_600_669.53, easting
    assert round(northing, 2) == 1_199_647.75, northing

Common Pitfalls#

• Asserting Python package versions but not native ones. pip freeze shows rasterio==1.3.9 on both machines while the linked libgdal differs. Assert __gdal_version__, proj_version_str, and shapely.geos_version — the native libraries, not the wrappers.
• Trusting that a referenced grid exists. A transform can name a datum grid it cannot find and silently downgrade to ballpark accuracy. Detect the ballpark marker explicitly, as the guard above does, and bake grids at build time per Containerizing GDAL Workers with Docker.
• Different lock per environment. Regenerating the lock on each machine reintroduces skew; install one committed lock everywhere, exactly as Pinning Geospatial Dependencies with conda-lock prescribes.
• Comparing only a single test point. One golden coordinate near the centre of a projection can pass while edges drift, because datum-grid coverage is regional. Pick test points near the corners of your working extent, not just its middle, so a partially-installed grid set is caught.
• No boot-time gate. Without a startup assertion, a skewed worker runs for hours before anyone notices drifted coordinates. Surface the failure loudly and route it into Observability & Monitoring for Geospatial Pipelines so a parity failure pages someone, and let State Management in Geospatial Flows decide whether partial output must be rolled back.

Related: Parity depends on the lock from Pinning Geospatial Dependencies with conda-lock, the grids baked by Containerizing GDAL Workers with Docker, and the staged rollout in Deployment & CI/CD for Spatial Workers that catches drift in a canary before it spreads. The PROJ datum-transformation guide explains grid selection and ballpark fallbacks in depth.

← Back to Deployment & CI/CD for Spatial Workers