Deployment & CI/CD for Spatial Workers

Shipping a Prefect or Dagster worker that carries GDAL, PROJ, and GEOS is fundamentally different from shipping a plain Python service: the deployment and CI/CD pipeline for spatial workers must guarantee that the exact native library versions, PROJ transformation grids, and GDAL format drivers baked into your image at build time are the same ones that execute a reprojection in production. This guide walks through building reproducible container images, pinning the ABI-sensitive native stack, testing against a real GDAL in CI, and rolling worker pools out in stages so a bad image never touches every tile job at once.

TL;DR#

Build one immutable image that contains your pinned geospatial stack, prove it works by running gdalinfo --formats and a reprojection smoke test inside CI before you publish, and promote that identical digest through dev, staging, and production worker pools with a canary step. Never install GDAL at container start, never let pip install resolve native versions at deploy time, and always assert library versions and PROJ grid availability when the worker boots. The rest of this page is the concrete mechanics.

This section sits under Geospatial Orchestration Architecture & Fundamentals and pairs naturally with the delivery concerns covered in Cost Optimization for Spatial Compute and Observability & Monitoring for Geospatial Pipelines, since an image you cannot observe is an image you cannot safely promote.

Prerequisites & Architecture Baseline#

Before you wire a delivery pipeline, confirm the foundational pieces are in place. The versions below are the ones this guide assumes; treat them as a floor, not a ceiling.

• A container runtime with BuildKit enabled (docker 24+ or buildx) for multi-stage builds and layer caching.
• GDAL 3.8+ with its bundled PROJ 9.3+ and GEOS 3.12+ — the trio whose ABIs must agree.
• rasterio 1.3+ and geopandas 0.14+ built against that same GDAL, not a wheel that ships its own copy.
• prefect 2.14+ or dagster 1.6+ for the worker runtime you deploy.
• conda-lock 2.5+ available in CI to produce the pinned lockfile described in the companion how-to.
• A container registry that supports immutable digests (GHCR, ECR, or Artifact Registry).
• A CI runner with enough disk for a full GDAL image (2–3 GB) and network access to fetch PROJ grids.

If your workers touch a spatial database, review Security Boundaries for Spatial Data before you bake credentials or connection logic into an image — secrets belong in the runtime environment, never a layer.

Core Principles#

Five ideas separate a spatial worker image that behaves in production from one that quietly reprojects coordinates to the wrong place.

1. Immutability by digest. The artifact you test must be the artifact you run. Reference images by sha256 digest, not a mutable latest tag, so a rebuild can never swap the native stack underneath a running pool.
2. Native ABI cohesion. GDAL, PROJ, and GEOS are C/C++ libraries linked at build time. rasterio and shapely bind to specific ABI versions. Mixing a rasterio wheel that carries GDAL 3.9 with a system libgdal at 3.6 produces import-time segfaults or, worse, silent driver mismatches. Pin all of them together.
3. Data files are dependencies too. PROJ ships proj.db and datum-shift grids; GDAL ships GDAL_DATA support files. If PROJ_LIB or GDAL_DATA point at nothing, a EPSG:4326 to EPSG:2056 transform can fall back to a coarse approximation and shift features by meters. Treat these paths as first-class build outputs.
4. Verification is part of the build. A build that compiles but cannot open a GeoTIFF is a failed build. CI must execute real GDAL calls against the assembled image and fail the pipeline on any missing driver or grid.
5. Progressive exposure. Native library upgrades are the riskiest change you can ship because they alter numerical output. Roll a new image to a small canary worker pool, compare its spatial output to the incumbent, then promote.
6. Reproducibility over convenience. The convenient path — pip install rasterio in a RUN line, :latest base tags, grids fetched on first use — is precisely the path that produces two workers that disagree. Every convenience that defers a decision to runtime is a place where two environments can diverge. Push all resolution to build time and freeze it.

The Three Companion Guides#

This delivery discipline decomposes into three focused how-tos, each of which this page assembles into the end-to-end flow above. Read them in order when you are building a pipeline from scratch, or jump to the one that matches the failure you are chasing.

The first, containerizing GDAL workers with Docker, is where the image itself is defined. It settles the base-image question — the tested OSGeo GDAL image versus a hand-compiled python-slim — and walks a multi-stage Dockerfile that exports PROJ_LIB and GDAL_DATA, bakes datum grids, and drops the compiler from the runtime layer. If your worker segfaults on import rasterio or cannot find a GTiff driver, that page is the place to start.

The second, pinning geospatial dependencies with conda-lock, supplies the lockfile the Dockerfile installs from. It explains why the ABI-sensitive native trio must be resolved once into a multi-platform lock rather than re-resolved on every build, and how to regenerate and verify that lock in CI. Reach for it when two builds a week apart produced different GDAL versions.

The third, environment parity for spatial pipelines, closes the loop at runtime. It provides the boot-time assertion that refuses to start a worker whose native versions or PROJ grids disagree with the lock, and the golden-coordinate regression test that catches reprojection drift. Turn to it when output differs between a laptop and production despite an identical-looking install.

Production Implementation#

The reference below is a GitHub Actions workflow that builds a spatial worker image from a fully pinned lockfile, runs real GDAL verification, publishes an immutable digest, and hands off to a staged deploy. It assumes the Dockerfile from Containerizing GDAL Workers with Docker and the lockfile from Pinning Geospatial Dependencies with conda-lock.

name: spatial-worker-delivery

on:
  push:
    branches: [main]

env:
  IMAGE: ghcr.io/${{ github.repository }}/spatial-worker

jobs:
  build-test-publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Set up Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # Build the worker image from the pinned conda-lock file.
      - name: Build image
        uses: docker/build-push-action@v5
        with:
          context: .
          load: true
          tags: ${{ env.IMAGE }}:ci
          cache-from: type=gha
          cache-to: type=gha,mode=max

      # Prove GDAL works BEFORE anything is published.
      - name: Verify native stack in the built image
        run: |
          docker run --rm ${{ env.IMAGE }}:ci gdalinfo --version
          docker run --rm ${{ env.IMAGE }}:ci python /app/ci/verify_stack.py

      - name: Push immutable digest
        id: push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}

  canary:
    needs: build-test-publish
    runs-on: ubuntu-latest
    steps:
      - name: Deploy image digest to canary worker pool
        run: |
          ./deploy/promote.sh \
            --pool canary \
            --image "${{ env.IMAGE }}@${{ needs.build-test-publish.outputs.digest }}" \
            --traffic 0.05

  full-rollout:
    needs: [build-test-publish, canary]
    runs-on: ubuntu-latest
    environment: production   # requires manual approval gate
    steps:
      - name: Promote same digest to all pools
        run: |
          ./deploy/promote.sh \
            --pool production \
            --image "${{ env.IMAGE }}@${{ needs.build-test-publish.outputs.digest }}" \
            --traffic 1.0

The verification script the workflow runs inside the image is deliberately small but exercises the parts that break silently:

import sys
import rasterio
import geopandas as gpd
from pyproj import CRS, Transformer
from pyproj.datadir import get_data_dir
from osgeo import gdal

def verify() -> None:
    # 1. GDAL must expose the drivers our pipeline actually uses.
    required_drivers: list[str] = ["GTiff", "COG", "GPKG", "PostGISRaster"]
    available = {gdal.GetDriver(i).ShortName for i in range(gdal.GetDriverCount())}
    missing = [d for d in required_drivers if d not in available]
    assert not missing, f"GDAL missing drivers: {missing}"

    # 2. PROJ data directory must resolve, or datum shifts silently degrade.
    assert get_data_dir(), "PROJ_LIB is empty; grids unavailable"

    # 3. A real transform must round-trip within tolerance.
    transformer = Transformer.from_crs(
        CRS.from_epsg(4326), CRS.from_epsg(2056), always_xy=True
    )
    x, y = transformer.transform(7.4474, 46.9480)  # Bern, CH
    assert 2_590_000 < x < 2_610_000, f"unexpected easting {x}"

    # 4. rasterio and geopandas must import against THIS GDAL, not a wheel copy.
    assert rasterio.__gdal_version__.startswith("3."), rasterio.__gdal_version__
    assert gpd.options.use_pygeos is not True  # shapely 2.x path expected

    print("spatial stack verified:", rasterio.__gdal_version__)

if __name__ == "__main__":
    try:
        verify()
    except AssertionError as exc:
        print(f"VERIFICATION FAILED: {exc}", file=sys.stderr)
        sys.exit(1)

Step-by-Step Walkthrough#

1. Checkout and Buildx. The workflow checks out the repo and enables BuildKit so the multi-stage Dockerfile can cache the expensive native-library layer across runs. Cache keys are scoped to the lockfile, so an unchanged stack reuses layers.
2. Build, don’t push yet. The first build-push-action uses load: true to leave the image in the local Docker daemon without publishing. Nothing reaches the registry until it passes verification.
3. Run real GDAL. The verify step runs gdalinfo --version and then verify_stack.py inside the freshly built image. This is the load-bearing gate: it opens drivers, checks PROJ_LIB, and performs an actual reprojection whose numeric result is asserted.
4. Publish by digest. Only after verification passes does the second step push, tagging with the commit SHA. The step exposes the resulting sha256 digest as a job output so downstream jobs reference the exact bytes that were tested.
5. Canary first. The canary job promotes that digest to a small worker pool handling roughly 5% of tile jobs. Metrics from this pool — feed them into your Grafana dashboards for GIS workflows — reveal reprojection drift or driver errors before broad exposure.
6. Gated full rollout. The full-rollout job targets a production environment protected by a manual approval gate, then promotes the identical digest to every pool. Because it is the same digest, there is zero chance of a rebuild introducing a different GDAL.

Edge Cases & Failure Recovery#

PROJ grid missing only in production. A build that fetches datum-shift grids at build time works in CI but fails in an air-gapped production network. Detection: the startup assertion in Environment Parity for Spatial Pipelines checks grid availability. Fix: bake all required grids into the image with projsync --all during the build, never at runtime.

Rasterio wheel shadows system GDAL. Installing rasterio from a binary wheel via pip after a conda-based GDAL leaves two GDAL copies in the image; imports may bind to either. Detection: python -c "import rasterio; print(rasterio.__gdal_version__)" disagrees with gdalinfo --version. Fix: install rasterio from the same conda channel as GDAL, or build it --no-binary rasterio against the system library.

Canary output diverges numerically. A GDAL minor upgrade changes resampling defaults, so mosaic seams differ. Detection: compare a checksum of canary-produced tiles against incumbent output for a fixed test extent. Fix: hold the rollout, pin the previous GDAL, and open an upstream issue before promoting.

Registry tag reused. A teammate re-pushes :main, and a worker pool restart pulls different bytes than were tested. Detection: workers log the running image digest at startup. Fix: reference @sha256:... everywhere and enable tag immutability on the registry.

CI green, worker OOMs on first mosaic. The image is correct but the pool’s memory limit is below what a raster mosaic needs. Detection: exit code 137 on the first heavy job. Fix: size pools per the guidance in right-sizing workers for raster mosaic jobs.

Configuration Reference#

The tunables below govern build reproducibility and rollout safety. Defaults are conservative starting points for a mid-sized raster pipeline.

[image]
base = "ghcr.io/osgeo/gdal:ubuntu-small-3.8.4"  # pinned by tag AND verified by digest
proj_lib = "/usr/share/proj"                    # exported as PROJ_LIB
gdal_data = "/usr/share/gdal"                   # exported as GDAL_DATA
projsync_grids = "all"                          # bake datum grids at build time

[ci]
verify_drivers = ["GTiff", "COG", "GPKG", "PostGISRaster"]
fail_on_missing_grid = true
lockfile = "conda-lock.yml"

[rollout]
canary_traffic = 0.05      # fraction of tile jobs to canary pool
canary_bake_minutes = 30   # observation window before full rollout
require_manual_approval = true
reference_by = "digest"    # never "tag"

Promotion & Rollback Strategy#

Because a spatial worker upgrade can change numbers rather than throw errors, the promotion path deserves as much rigor as the build. Treat every image digest as a candidate that must earn its way to the full fleet.

Promotion is a ratchet, not a switch. The canary pool runs the new digest against a fixed, representative slice of work — ideally the same tile indices every time, so results are comparable across releases. During the bake window, compare three signals: task success rate, per-task duration (a GDAL change can quietly double warp time), and a content checksum of output tiles for a frozen extent. Only when all three match the incumbent within tolerance does the manual approval gate open and the identical digest fan out to every pool. Nothing is rebuilt at promotion; you are re-pointing pools at bytes that already passed.

Rollback must be equally boring. Keep the previous known-good digest recorded alongside the pool definition so reverting is a single re-point to that sha256, not a rebuild that might resolve a different stack. Since images are immutable and referenced by digest, a rollback restores the exact prior native trio and grid set — there is no ambiguity about which GDAL you fell back to. Retain at least the last three digests in the registry so you can step back more than one release if a regression is discovered late. This is also why the startup guard from the parity guide matters: if a rolled-back pool somehow pulls the wrong bytes, the worker refuses to start rather than silently reprojecting against a mismatched grid set.

Coordinate the ratchet with your monitoring. Wire canary metrics and the checksum comparison into the same panels you use for steady-state health so the same on-call view answers “is this release safe?” and “is the fleet healthy?” — the dashboards in Observability & Monitoring for Geospatial Pipelines are the natural home for both.

Frequently Asked Questions#