Containerizing GDAL Workers With Docker

The image is where a spatial worker’s behaviour is actually decided. GDAL, PROJ, GEOS and the transformation grids determine what a reprojection returns, and Python’s role is to call them — so the Dockerfile is closer to a specification of correctness than to packaging. Build it in stages ordered by how often each part changes, bake the grids, install exactly what the lock file names, and finish with a smoke test that runs inside the image rather than beside it.

When to Use This Pattern#

• Workers run reprojections, warps or format conversions, which is every spatial pipeline.
• More than one worker exists, so consistency between them stops being automatic.
• The pipeline publishes something whose registration against other data matters.
• Builds happen often and a two-minute rebuild for a code change is worth engineering for.

Complete Working Example#

Four stages, ordered so that the fastest-changing thing is last.

# syntax=docker/dockerfile:1.7
ARG GDAL_DIGEST=sha256:8c1f...            # resolved in CI, recorded in the build record
FROM ghcr.io/osgeo/gdal@${GDAL_DIGEST} AS base

ENV PROJ_NETWORK=OFF \
    PROJ_DATA=/usr/share/proj \
    GDAL_CACHEMAX=512 \
    GDAL_NUM_THREADS=2 \
    GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR \
    CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif,.tiff,.vrt,.gpkg,.fgb \
    VSI_CACHE=TRUE \
    VSI_CACHE_SIZE=26214400 \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

# ---- grids: change perhaps twice a year -----------------------------------
FROM base AS grids
RUN projsync --system-directory --bbox -12,49,3,61 --quiet

# ---- python deps: change monthly ------------------------------------------
FROM base AS deps
COPY requirements.lock /tmp/requirements.lock
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --require-hashes --no-deps -r /tmp/requirements.lock

# ---- runtime: application code changes several times a day ----------------
FROM base AS runtime
COPY --from=grids /usr/share/proj /usr/share/proj
COPY --from=deps  /usr/lib/python3/dist-packages /usr/lib/python3/dist-packages
RUN useradd --uid 10001 --create-home worker
COPY --chown=worker:worker src/ /app/src/
USER worker
WORKDIR /app
HEALTHCHECK --interval=60s --timeout=10s CMD python -m src.health
ENTRYPOINT ["python", "-m", "src.worker_boot"]

The smoke test is a build stage, so a bad image cannot be published:

FROM runtime AS smoke
USER root
COPY tests/fixtures/ /fixtures/
RUN set -eu; \
    gdalinfo --formats | grep -q " GTiff "; \
    gdalinfo --formats | grep -q " GPKG "; \
    python - <<'PY'
import pyproj, rasterio
from rasterio.warp import calculate_default_transform

# 1. The bindings and the system library must agree.
assert rasterio.__gdal_version__ == "3.9.2", rasterio.__gdal_version__

# 2. A transformation with a known answer, to prove the grid is present.
t = pyproj.Transformer.from_crs("EPSG:4326", "EPSG:27700")
e, n = t.transform(55.9533, -3.1883)
assert abs(e - 325863.108) < 0.01 and abs(n - 673795.836) < 0.01, (e, n)

# 3. A real warp, because reading a header is not the same as resampling.
with rasterio.open("/fixtures/scene.tif") as src:
    calculate_default_transform(src.crs, "EPSG:3857", src.width, src.height, *src.bounds)
print("smoke ok")
PY

GDAL_NUM_THREADS=2 is the setting most often left at its default and most often wrong on a worker. GDAL will happily use every core it can see for a single warp, which is excellent on a workstation and counterproductive on a worker running four concurrent tiles — four warps each claiming sixteen cores produces contention, not throughput, and inflates memory as each thread takes its own buffers. Setting it explicitly to a small number, and letting the orchestrator’s concurrency provide the parallelism, gives predictable per-task behaviour, which is what the memory arithmetic in right-sizing workers depends on.

Parameter & Option Reference#

Verification & Testing#

The smoke stage above is the primary gate. These run in CI against the built image and cover what it cannot.

import subprocess


def run_in_image(digest: str, script: str) -> str:
    return subprocess.run(
        ["docker", "run", "--rm", "--network", "none", digest, "python", "-c", script],
        capture_output=True, text=True, check=True,
    ).stdout


def test_no_network_needed_for_a_transformation(image_digest) -> None:
    # --network none: proves the grids are baked rather than fetched.
    out = run_in_image(image_digest, TRANSFORM_PROBE)
    assert "325863.108" in out


def test_bindings_and_system_library_agree(image_digest) -> None:
    out = run_in_image(image_digest,
                       "import rasterio;print(rasterio.__gdal_version__)")
    cli = subprocess.run(["docker", "run", "--rm", image_digest, "gdalinfo", "--version"],
                         capture_output=True, text=True).stdout
    assert out.strip() in cli


def test_image_runs_as_non_root(image_digest) -> None:
    out = subprocess.run(["docker", "run", "--rm", image_digest, "id", "-u"],
                         capture_output=True, text=True).stdout
    assert out.strip() == "10001"


def test_no_credentials_baked_in(image_digest) -> None:
    layers = subprocess.run(["docker", "history", "--no-trunc", image_digest],
                            capture_output=True, text=True).stdout
    assert "AWS_SECRET" not in layers and "PGPASSWORD" not in layers

The --network none test is the sharpest of the four. A transformation that works with networking enabled proves nothing about the image, because PROJ may have fetched the grid it needed from a CDN at run time — which will also happen in production, until the day the worker has no egress or the CDN is slow, at which point the transformation silently becomes less accurate. Running the probe with networking disabled is the only way to prove the grid is in the image.

A last word on the environment block, because its placement is not incidental. Every one of those variables changes what the library does — how much it caches, how many threads it uses, whether it lists a prefix before opening a file, which extensions it will fetch over HTTP — and every one of them has a default that is reasonable for an interactive workstation and wrong for a worker running four tasks at once. Putting them in the image means they are part of the artefact that gets promoted, tested and rolled back as a unit. Putting them in the flow means a new code path can omit one, and the omission shows up as a cost or a memory problem weeks later with nothing connecting it to a change.

The corollary is that the image, not the code, is where the pipeline’s operational tuning lives, and it deserves to be reviewed as such. A change from GDAL_CACHEMAX=512 to 2048 is a change to every worker’s memory arithmetic and belongs in a pull request that says so, rather than in a hurried edit to fix one slow layer. Treating the Dockerfile as configuration that happens to be code — reviewed, versioned, promoted by digest — is the practice that keeps a fleet’s behaviour explicable.

Common Pitfalls#

• Installing the native stack at container start. Every worker then resolves independently, and the fleet drifts within a day.
• Copying application code before installing dependencies. Every code change rebuilds everything, including the grid sync, so two builds of one commit can differ.
• Leaving PROJ_NETWORK=ON with no baked grids. Accuracy then depends on egress, and the degradation is silent.
• Letting pip resolve dependencies. A wheel with a bundled GDAL beside the system one produces a worker with two native stacks.
• Default GDAL_NUM_THREADS. One warp claims every core, and four concurrent warps contend rather than scale.
• Running as root. Nothing needs it, and scratch directories written as root are awkward for everything that follows.

Frequently Asked Questions#

Related#

• Deployment & CI/CD for spatial workers — where this image sits in the promotion chain
• Pinning geospatial dependencies with conda-lock — producing the lock file this installs
• Environment parity for spatial pipelines — using the same image everywhere
• Cutting egress costs with COG range reads — why those environment variables are in the image
• Right-sizing workers for raster mosaic jobs — the memory settings this pins

← Deployment & CI/CD for Spatial Workers