Environment Parity for Spatial Pipelines

“Works on my machine” is a minor irritation for a web service and a correctness problem for a spatial pipeline, because the thing that differs between machines is the library that decides where things are on the ground. Parity means one image, one set of environment variables and one lock file everywhere the code runs — and, because parity always erodes, a report that names the differences rather than asserting there are none.

When to Use This Pattern#

• Development happens outside the worker image, which it usually does at first.
• A bug reproduces in one environment and not another, which is the symptom that starts this work.
• CI runs tests against its own libraries rather than inside the artefact being shipped.
• Environment variables differ between local and production, which is almost always true and rarely written down.

Complete Working Example#

One image, three entry points, and a report that compares them.

# Local development: the fleet's image, with the source mounted.
docker run --rm -it \
  -v "$PWD/src:/app/src:ro" \
  -v "$PWD/tests:/app/tests:ro" \
  --env-file .env.local \
  ghcr.io/example/spatial-worker@${WORKER_DIGEST} \
  bash

# CI: the same digest, no mount, tests run inside the artefact.
docker run --rm --network none \
  ghcr.io/example/spatial-worker@${WORKER_DIGEST} \
  python -m pytest tests/ -q

The parity report is a module that dumps everything which could differ, so two environments can be diffed rather than argued about:

# src/parity.py
from __future__ import annotations

import json
import os
import platform
import sys

import fiona, pyproj, rasterio, shapely
from osgeo import gdal

# Variables that change results or resource use. Absence is as significant as value.
TRACKED_ENV = (
    "PROJ_DATA", "PROJ_NETWORK", "GDAL_DATA", "GDAL_CACHEMAX", "GDAL_NUM_THREADS",
    "GDAL_DISABLE_READDIR_ON_OPEN", "CPL_VSIL_CURL_ALLOWED_EXTENSIONS",
    "VSI_CACHE", "VSI_CACHE_SIZE", "OGR_DRIVER_LIST", "GDAL_DRIVER_LIST",
)

PROBES = {
    # transformation, input, expected — one per grid the pipeline depends on
    "EPSG:4326->EPSG:27700": ((55.9533, -3.1883), (325863.108, 673795.836)),
    "EPSG:4326->EPSG:2154": ((48.8566, 2.3522), (652351.245, 6862142.263)),
}


def report() -> dict:
    transforms = {}
    for name, (src, _expected) in PROBES.items():
        a, b = name.split("->")
        e, n = pyproj.Transformer.from_crs(a, b).transform(*src)
        transforms[name] = [round(e, 3), round(n, 3)]

    return {
        "platform": f"{platform.system()}-{platform.machine()}",
        "python": sys.version.split()[0],
        "versions": {
            "gdal": gdal.__version__.split()[0],
            "gdal_via_rasterio": rasterio.__gdal_version__,
            "gdal_via_fiona": fiona.__gdal_version__,
            "proj": pyproj.proj_version_str,
            "geos": shapely.geos_version_string,
        },
        "env": {k: os.environ.get(k) for k in TRACKED_ENV},
        "drivers": sorted(gdal.GetDriver(i).ShortName
                          for i in range(gdal.GetDriverCount()))[:400],
        "transforms": transforms,
    }


if __name__ == "__main__":
    print(json.dumps(report(), indent=2, sort_keys=True))

Comparing two environments is then a diff, and CI can enforce it:

def test_ci_matches_the_fleet(fleet_report_path) -> None:
    here, fleet = report(), json.loads(Path(fleet_report_path).read_text())
    for section in ("versions", "env", "transforms"):
        assert here[section] == fleet[section], (
            f"{section} differs from the fleet:\n"
            + "\n".join(f"  {k}: local={here[section].get(k)!r} "
                        f"fleet={fleet[section].get(k)!r}"
                        for k in set(here[section]) | set(fleet[section])
                        if here[section].get(k) != fleet[section].get(k))
        )

The environment section of the report treats an unset variable as a value, and that is deliberate. Most parity failures in spatial pipelines are not a variable set differently but a variable set in one place and absent in another — GDAL_CACHEMAX unset on a laptop means a cache sized from physical memory, PROJ_NETWORK unset means whatever the build defaulted to, GDAL_DISABLE_READDIR_ON_OPEN unset means a prefix listing per open. Reporting null rather than omitting the key makes those visible in a diff instead of invisible by absence.

Parameter & Option Reference#

Verification & Testing#

def test_report_is_stable_within_an_environment() -> None:
    assert report() == report(), "the parity report is not deterministic"


def test_every_tracked_variable_is_set_on_the_fleet(fleet_report) -> None:
    unset = [k for k, v in fleet_report["env"].items() if v is None]
    assert not unset, f"relying on defaults for: {', '.join(unset)}"


def test_probes_match_their_expected_values() -> None:
    for name, (src, expected) in PROBES.items():
        a, b = name.split("->")
        e, n = pyproj.Transformer.from_crs(a, b).transform(*src)
        assert abs(e - expected[0]) < 0.01 and abs(n - expected[1]) < 0.01, name


def test_the_two_gdal_bindings_agree() -> None:
    r = report()["versions"]
    assert r["gdal_via_rasterio"] == r["gdal_via_fiona"] == r["gdal"]


def test_driver_list_matches_the_fleet(fleet_report) -> None:
    local, fleet = set(report()["drivers"]), set(fleet_report["drivers"])
    assert local == fleet, f"only local: {local - fleet}; only fleet: {fleet - local}"

The second test is the one that changes behaviour most. “Relying on defaults” sounds harmless and is the mechanism by which a laptop, a CI runner and a worker end up with three different driver caches, three different threading behaviours and three different read patterns while running identical code. Asserting that every tracked variable is explicitly set — even to the value the default would have given — converts a set of implicit environmental assumptions into a list somebody has looked at.

There is a limit to how far parity can be pushed, and knowing where it sits saves a lot of wasted effort. Some differences are irreducible: a laptop has different cores, different memory, no cloud identity and a network with different characteristics. Chasing those produces elaborate local infrastructure that is itself a source of divergence. The useful boundary is between differences that change results and differences that change performance — the first must be eliminated, the second must merely be known. A local run that is four times slower is fine; a local run that transforms a coordinate to a different place is not, and the report is arranged to make the second kind obvious while not pretending the first does not exist.

That boundary also decides what belongs in the report. Versions, environment variables, driver availability and probe results all change what the pipeline computes, so they are compared strictly. Core count, memory size and network latency change how long it takes, so they are worth recording for context and not worth asserting on. Mixing the two produces a check that fails constantly for reasons nobody can act on, which is the fastest way to have it disabled.

Common Pitfalls#

• A second way to install the stack for development. The moment a laptop has its own GDAL, parity is a matter of coincidence.
• Testing in CI against the runner’s libraries. It proves the tests pass somewhere that is not the artefact being shipped.
• Omitting unset variables from the report. Absence is the commonest difference and the easiest to miss.
• One transformation probe for a multi-jurisdiction pipeline. It proves one grid is present and says nothing about the others.
• Treating parity as achieved. It erodes with every base-image bump and every new deployment target; measure it.
• Ignoring the driver list. A driver present locally and missing on the fleet turns a working local test into a production failure on an unusual delivery.

Frequently Asked Questions#

Related#

• Deployment & CI/CD for spatial workers — the promotion chain parity supports
• Containerizing GDAL workers with Docker — the image everything shares
• Pinning geospatial dependencies with conda-lock — per-platform locks for real platform differences
• Testing spatial flows in CI with synthetic fixtures — what runs inside the image
• Rolling out GDAL upgrades without breaking flows — parity across a version change

← Deployment & CI/CD for Spatial Workers