Rolling Out GDAL Upgrades Without Breaking Flows

A GDAL upgrade rarely breaks a pipeline; it changes its output. A resampling default moves, a driver starts honouring a tag it used to ignore, a transformation picks a different pipeline from an updated database — and the run stays green while the tiles differ. So the rollout procedure is not “deploy and watch for errors” but “build both, run both on the same real inputs, and diff the results against a tolerance you decided in advance”.

When to Use This Pattern#

• The native stack version is changing — GDAL, PROJ, GEOS, or the grid data.
• Outputs are published to consumers who would notice a shift, or who would not notice and should.
• A previous upgrade caused a surprise, which is the experience that makes this obviously worth an afternoon.
• The pipeline has a canary pool or can have one, since the comparison needs somewhere to run.

Complete Working Example#

A shadow run builds the same tiles under both digests and compares them numerically.

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

import numpy as np
import rasterio
from prefect import flow, task, get_run_logger


@dataclass(frozen=True)
class TileDiff:
    tile: str
    identical: bool
    max_abs: float
    changed_fraction: float
    shifted: bool                      # geotransform moved: never acceptable


@task(timeout_seconds=300)
def compare(old: Path, new: Path, tile: str) -> TileDiff:
    with rasterio.open(old) as a, rasterio.open(new) as b:
        if a.transform != b.transform or a.crs != b.crs:
            return TileDiff(tile, False, float("inf"), 1.0, shifted=True)
        x, y = a.read(masked=True), b.read(masked=True)
        delta = np.abs(x.astype("float64") - y.astype("float64"))
        return TileDiff(
            tile=tile,
            identical=bool(np.all(delta == 0)),
            max_abs=float(delta.max()),
            changed_fraction=float((delta > 0).sum() / delta.size),
            shifted=False,
        )


@flow(name="stack-upgrade-shadow")
def shadow(tiles: list[Tile], old_digest: str, new_digest: str,
           max_abs_allowed: float = 1.0, max_changed_fraction: float = 0.02):
    log = get_run_logger()
    old_out = run_in_image(old_digest, tiles, out=Path("/scratch/old"))
    new_out = run_in_image(new_digest, tiles, out=Path("/scratch/new"))

    diffs = compare.map(old=old_out, new=new_out, tile=[t.path for t in tiles])
    results = [d.result() for d in diffs]

    shifted = [d for d in results if d.shifted]
    if shifted:
        raise RuntimeError(f"{len(shifted)} tile(s) moved on the ground: {shifted[:5]}")

    identical = sum(d.identical for d in results)
    worst = max(results, key=lambda d: d.max_abs)
    log.info("shadow: %d/%d byte-identical, worst delta %.3f on %s",
             identical, len(results), worst.max_abs, worst.tile)

    over = [d for d in results
            if d.max_abs > max_abs_allowed or d.changed_fraction > max_changed_fraction]
    if over:
        raise RuntimeError(
            f"{len(over)} tile(s) exceed tolerance (worst {worst.max_abs:.3f} "
            f"on {worst.tile}) — review before promoting"
        )
    return {"identical": identical, "total": len(results), "worst": worst.max_abs}

The rollout itself is then a promotion with a recorded digest per output:

@task
def publish(tile: Tile, path: Path, image_digest: str) -> str:
    # The digest travels with the product, so "which tiles came from the new stack"
    # is a query rather than an inference from timestamps.
    return upload_with_metadata(path, key=tile.path,
                                metadata={"stack-digest": image_digest})

The geotransform check is separate from the pixel tolerance, and it is not negotiable in the way the tolerance is. A changed pixel value within a fraction of a digital number is a resampling difference and usually acceptable; a changed transform means the tile covers different ground, which makes every value in it wrong regardless of how close the numbers look. Treating them as one comparison, with one threshold, loses that distinction — and a shifted tile can easily produce a smaller numerical delta than a legitimate resampling change, because adjacent ground looks similar.

Parameter & Option Reference#

Verification & Testing#

def test_shadow_blocks_a_moved_transform(monkeypatch, tmp_path) -> None:
    old = write_tile(tmp_path / "a.tif", transform=T)
    new = write_tile(tmp_path / "b.tif", transform=T_shifted_half_pixel)
    d = compare.fn(old, new, "12/2147/1398")
    assert d.shifted and not d.identical


def test_small_scattered_delta_passes(tmp_path) -> None:
    old = write_tile(tmp_path / "a.tif", data=BASE)
    new = write_tile(tmp_path / "b.tif", data=BASE + jitter(max_abs=1, fraction=0.01))
    d = compare.fn(old, new, "t")
    assert d.max_abs <= 1.0 and d.changed_fraction <= 0.02


def test_widespread_small_delta_is_caught(tmp_path) -> None:
    # Every pixel off by one is not "within tolerance": something systematic moved.
    old = write_tile(tmp_path / "a.tif", data=BASE)
    new = write_tile(tmp_path / "b.tif", data=BASE + 1)
    d = compare.fn(old, new, "t")
    assert d.changed_fraction > 0.02


def test_categorical_layer_uses_zero_tolerance(monkeypatch) -> None:
    with pytest.raises(RuntimeError, match="exceed tolerance"):
        shadow(LANDCOVER_TILES, OLD, NEW, max_abs_allowed=0.0)

The third test is the one that justifies having two thresholds rather than one. An upgrade that shifts every pixel by a single digital number produces a maximum delta of one, which a max-only rule waves through — and yet a systematic, universal shift is exactly the signature of a changed scaling or a changed nodata interpretation, which is much more significant than a few scattered rounding differences of the same magnitude. Checking how much of the image changed alongside how much it changed by separates the two.

One organisational point makes the difference between this procedure being followed and being skipped: the shadow run has to be cheap enough to run on every upgrade, including the boring ones. A patch release that nobody expects to change anything is exactly when a change slips through, because that is the release nobody bothered to shadow. Keeping the sample to a few hundred tiles, running both builds on the canary pool overnight and producing a one-line summary — “381 of 412 byte-identical, worst delta 0.4” — makes it a routine step rather than a project, and a routine step survives a busy month.

The summary line is worth designing as carefully as the comparison. It goes into the promotion record, and six months later it is the only evidence of what an upgrade did. “Promoted GDAL 3.10.0; 381/412 identical, 28 within ±1 DN, 3 investigated and traced to a nodata convention in the coastal source” is a sentence that answers a question somebody will eventually ask. “Upgraded GDAL, all green” is not, and the two cost the same to write.

Common Pitfalls#

• Deploying and watching for errors. The failure mode of a stack upgrade is a changed number, not an exception.
• One tolerance for every layer. Imagery tolerates a digital number of rounding; a categorical raster tolerates nothing.
• Comparing only the maximum delta. A universal one-unit shift is systematic and passes a max-only rule.
• Shadowing on synthetic fixtures. They are for CI. The shadow needs real sources, because the surprises come from real sources’ quirks.
• No digest recorded on outputs. After a bad promotion, identifying affected products becomes an inference from timestamps.
• Skipping the backfill during the soak. Backfills exercise code paths a nightly run does not, and they are where a driver change tends to surface.

Frequently Asked Questions#

Related#

• Deployment & CI/CD for spatial workers — the promotion chain this fits into
• Pinning geospatial dependencies with conda-lock — where the version being changed is declared
• Testing spatial flows in CI with synthetic fixtures — the layer above, with known answers
• Environment parity for spatial pipelines — proving the shadow ran under the versions you think
• Caching reprojected rasters with content hashing — why the recipe version must move with the stack

← Deployment & CI/CD for Spatial Workers