Deployment & CI/CD for Spatial Workers

In short: a spatial worker is a Python process wrapped around a large native stack, and the native stack is what makes deployment different. Build one immutable image, prove it can actually reproject before publishing it, assert the library and grid versions at boot, and promote the same digest through a canary. The failure this prevents is not a crash — it is a silently different transformation result on a fraction of the fleet.

An ordinary Python service is a set of wheels; a spatial worker is GDAL, PROJ, GEOS, their transitive C dependencies and a directory of transformation grids, with Python bindings on top that are compiled against specific versions of all of it. That has two consequences. The first is ordinary: the image is large and slow to build. The second is not: a difference in the native stack between two workers does not usually produce an error. It produces different numbers — a coordinate transformed through a different grid, a resampling kernel with a changed default, a driver that reads an extra tag — and those numbers flow into published products with nothing anywhere suggesting a problem.

So the discipline is stricter than for a service. The unit that gets promoted is a digest rather than a tag, the version assertions run at worker boot rather than in CI alone, and the canary’s job is to prove a real reprojection matches a known answer rather than to prove the process stays up.

It is worth naming the shape of the risk before the mechanics, because it decides which controls are worth their cost. Spatial deployment failures divide cleanly into two kinds. The loud kind — a missing driver, an import error, a binary that will not link — is annoying and self-announcing, and ordinary deployment practice handles it. The quiet kind changes a number: a transformation grid that is absent, a resampling default that moved between releases, a driver that now reads an overview it previously ignored. Nothing errors, the run is green, and the difference reaches a published product. Everything unusual in this section exists for the second kind, and it is why the checks compare values rather than exit codes.

Prerequisites & Architecture Baseline#

• A base image with the native stack pinned by version, not by latest
• A lock file covering every Python dependency, resolved on the same platform the worker runs
• PROJ transformation grids present in the image, or a deliberate decision to fetch them
• A registry that supports pulling by digest, and deployments that reference digests
• Fixtures with known-correct transformation answers, small enough to run in CI
• Separate work pools for canary and full fleet, so a promotion can be partial

Core Principles#

1. Promote a digest, never a tag. A tag is a mutable pointer. Two workers pulling worker:2026.8 a week apart can run different code, and nothing in the deployment records that they did.

2. The native stack is pinned or it is undefined. pip install rasterio resolves a wheel that bundles some GDAL. Which one depends on the day.

3. Assert at boot, not only in CI. CI proves the image was right when it was built. A boot-time assertion proves this worker, now, has the versions the pipeline expects.

4. Correctness is a reprojection, not a startup. The canary must transform a known point and compare it to a known answer. A worker that starts is not a worker that computes the same numbers.

5. Grids are part of the version. PROJ’s transformation grids change results by metres. An image without them silently falls back to a less accurate transformation.

6. Build time is worth spending once. A large image built well and cached is cheaper than a small image that installs the native stack at start-up, and enormously cheaper than an inconsistent fleet.

Production Implementation#

The image pins the native stack, installs from a lock file and carries the grids.

# syntax=docker/dockerfile:1.7
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.9.2 AS base
# The tag above is itself resolved to a digest by the build; see the CI step below.

ENV PROJ_NETWORK=OFF \
    PROJ_DATA=/usr/share/proj \
    GDAL_CACHEMAX=512 \
    GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR \
    CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff,.vrt,.gpkg \
    PYTHONDONTWRITEBYTECODE=1

FROM base AS grids
# Grids change transformation results by metres. Bake them in rather than fetching
# at run time, so an offline worker is not a quietly less accurate worker.
RUN --mount=type=cache,target=/root/.cache \
    projsync --system-directory --bbox -12,49,3,61 --quiet

FROM base AS deps
COPY requirements.lock /tmp/
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --require-hashes --no-deps -r /tmp/requirements.lock

FROM base AS runtime
COPY --from=grids /usr/share/proj /usr/share/proj
COPY --from=deps /usr/lib/python3/dist-packages /usr/lib/python3/dist-packages
COPY src/ /app/src/
WORKDIR /app
# Fail fast at boot rather than at the first reprojection.
ENTRYPOINT ["python", "-m", "src.worker_boot"]

The boot check is short and it is the difference between a fleet you can reason about and one you cannot:

# src/worker_boot.py
from __future__ import annotations

import sys

from osgeo import gdal, osr
import pyproj

EXPECTED = {"gdal": "3.9.2", "proj": "9.4.1", "geos": "3.12.1"}
# A point whose transformed value is known to sub-millimetre precision.
PROBE = (55.9533, -3.1883)                     # Edinburgh, EPSG:4326
PROBE_BNG = (325863.108, 673795.836)           # EPSG:27700, via the OSTN15 grid


def main() -> int:
    actual = {
        "gdal": gdal.__version__.split()[0],
        "proj": pyproj.proj_version_str,
        "geos": gdal.__version__ and geos_version(),
    }
    for name, want in EXPECTED.items():
        if actual[name] != want:
            print(f"native stack mismatch: {name} {actual[name]} != {want}", file=sys.stderr)
            return 1

    t = pyproj.Transformer.from_crs("EPSG:4326", "EPSG:27700", always_xy=False)
    e, n = t.transform(*PROBE)
    if abs(e - PROBE_BNG[0]) > 0.01 or abs(n - PROBE_BNG[1]) > 0.01:
        # Almost always a missing grid: the fallback transformation is metres out.
        print(f"transformation probe failed: got ({e:.3f}, {n:.3f})", file=sys.stderr)
        return 1

    print(f"worker ready: gdal {actual['gdal']}, proj {actual['proj']}, grid probe ok")
    return start_worker()


if __name__ == "__main__":
    raise SystemExit(main())

The grid probe is the check that earns its place most often. A PROJ installation without the relevant transformation grid does not fail; it falls back to a coarser method — for Britain, a seven-parameter Helmert instead of OSTN15 — and returns coordinates that are a metre or two out. Every downstream check passes, the tiles line up with each other, and the layer is subtly misregistered against everything produced by anyone else. One probe point with a known answer catches it at boot on every worker, which is the only place the fleet’s variation can be observed.

The --no-deps flag in the dependency stage looks reckless and is deliberate. A lock file that has already resolved the full transitive set does not need the installer to resolve anything, and letting it try is precisely how a wheel with its own bundled native library gets pulled in beside the system one. Two GDALs in one process is not a hypothetical: it produces a worker where gdalinfo on the command line and rasterio in Python report different versions, support different drivers, and disagree about transformations. Installing exactly what the lock file names, and nothing else, is what keeps the image’s native stack singular.

The multi-stage split serves the same purpose from a different angle. Grids, Python dependencies and application code change at completely different rates — the grids perhaps twice a year, the dependencies monthly, the code several times a day — so keeping them in separate stages means an ordinary code change rebuilds one small layer. That is a build-time convenience, and it is also a correctness property: a layer that is not rebuilt cannot change, so the native stack stays byte-identical across every deployment between dependency bumps.

Step-by-Step Walkthrough#

1. Pin the base image by digest in CI, resolving the tag once and recording what it resolved to.
2. Resolve the Python lock file on the target platform, with hashes, so a wheel cannot be substituted.
3. Bake the grids for the geographies the pipeline covers, and record which bounding box was synced.
4. Run the correctness suite inside the built image, not against a CI runner’s own libraries.
5. Publish by digest and record it in the deployment, not a tag that will move.
6. Record the digest in every output’s metadata, so the outputs of a build can be identified after the fact.
7. Promote through the canary pool, comparing its outputs against the previous digest’s on the same inputs before the fleet follows.

Edge Cases & Failure Recovery#

A wheel bundles its own GDAL. Several do, and the bundled version can differ from the system one, so the process ends up with two GDALs and which one answers depends on link order. Installing with --no-deps from a lock file and preferring distribution packages for the bindings avoids the situation entirely.

The canary passes and the fleet fails. Almost always a difference the canary did not have: a different instance type, a missing environment variable, a mounted volume. Making the canary a pool within the same cluster rather than a separate environment removes most of these.

A grid is missing for one region only. The probe covers the region it probes. Pipelines spanning several jurisdictions need a probe per transformation they rely on, which is a list of three or four points rather than one.

A rollback is needed after tiles have been published. Rolling back the image is fast; the tiles built by the bad digest are not automatically wrong-flagged. Recording the image digest in each tile’s metadata makes “which outputs came from the bad build” a query.

The base image is withdrawn. Public base images do disappear. Mirroring the resolved digest into your own registry at build time costs a few gigabytes and removes a dependency on somebody else’s retention policy.

Two pipelines share a worker image. Common, reasonable, and it means a GDAL upgrade needed by one is imposed on the other. Where the two have different tolerance for change — a research pipeline and a published basemap, say — separate images with separate promotion schedules cost a little duplication and remove a recurring argument.

The probe itself is wrong. Worth stating because it happens: a reference value copied from a different transformation path, or computed once by the same code it is meant to check. Take probe values from an authoritative published source rather than from your own pipeline, or the assertion only proves the pipeline is consistent with itself.

Configuration Reference#

One last observation about the canary, which is the control people trim first when a release feels urgent. Its value is not that it catches crashes — CI catches those — but that it produces two sets of outputs from two digests over the same inputs, which can be diffed. A byte-identical diff is strong evidence that an upgrade changed nothing; a diff confined to one layer localises the change immediately; a diff everywhere stops the promotion. None of that is available from a canary that only checks whether the process stayed up, so the comparison step is the part worth protecting when time is short.

Frequently Asked Questions#

Related#

• Containerizing GDAL workers with Docker — the image, in detail
• Pinning geospatial dependencies with conda-lock — resolving the native stack reproducibly
• Environment parity for spatial pipelines — keeping laptop, CI and production the same
• Testing spatial flows in CI with synthetic fixtures — fixtures small enough to run every commit
• Rolling out GDAL upgrades without breaking flows — the canary and the diff
• Cost optimization for spatial compute — where the environment variables in the image come from

← Geospatial Orchestration Architecture Fundamentals