Resilience & Failure Handling for GIS Pipelines

Resilience in a geospatial pipeline is not a bolt-on. It is a set of decisions about identity, retry timing, shared knowledge of upstream health, durable capture of what failed, and bounded time — made early, because each of them changes the shape of the code around it. A pipeline designed without them can be made to retry, but it cannot be made safe to retry, and the difference shows up as duplicated features, half-written mosaics and a nightly run whose worst case nobody can state.

What makes this a distinct discipline rather than an application of general data-engineering practice is the cost and shape of spatial work. A retried HTTP call to a JSON API costs milliseconds; a retried GetMap costs the far side a full render. A duplicated row in a tabular warehouse is a deduplication query; a duplicated polygon in PostGIS breaks topology for every downstream overlay. A tabular job that is killed halfway leaves a partial load; a raster job that is killed halfway leaves a file that gdalinfo opens without complaint and that quietly corrupts every product derived from it. Every pattern on this page exists because one of those asymmetries made the generic version of the pattern insufficient.

Core Architectural Layers#

A resilient spatial pipeline is built from five layers, each of which owns one question. They compose, and they are most useful when each is allowed to answer only its own question.

1. Identity — what work is this? An idempotency key derived from the inputs that determine the output: the source digest, the CRS, the resampling method, the nodata value. Identity is the foundation, because without it “have we already done this?” is unanswerable and every other layer is guessing.

2. Timing — when should we try again? A backoff policy with exponential growth, full jitter, a cap and a wall-clock budget. Timing decides how a transient failure is absorbed, and it is the only layer that can make things worse by being too eager.

3. Shared health — is this endpoint worth trying at all? A circuit breaker whose state is shared across the fleet, so one worker’s discovery of an outage becomes everyone’s. Retries handle a blip; the breaker handles the hour.

4. Capture — where does failure go? A dead-letter queue holding the geometry, the CRS, the parameters and the cause, so a batch can complete while its failures remain re-drivable rather than lost in a log.

5. Bounds — when do we stop? A timeout budget derived from one deadline, and a cancellation path that actually terminates the GDAL subprocess it claims to have stopped.

Key Design Constraints Imposed by Geospatial Data#

Effects are not rows. Much resilience advice assumes the unit of work is a record, so idempotency reduces to an upsert. Spatial pipelines write GeoTIFFs, publish tile pyramids, refresh materialised views and invalidate caches — effects an upsert cannot express. That is why the identity layer here uses a ledger rather than relying on the database’s own conflict handling, and why the unit of work has to be chosen deliberately rather than inherited from the schema.

The coordinate reference system is part of the input. Two runs with identical bytes and different CRS declarations produce different data. Any digest that omits the CRS will treat them as the same work and skip the second — a silent, geographically-shaped corruption that no amount of testing on a single-projection fixture will find. The same applies to the nodata value, the resampling method and the target grid: they are inputs, not settings.

Retries are expensive on the far side. A rendering endpoint spends real CPU on every request, including the ones it fails. This inverts the usual advice about retry aggressiveness and is the reason full jitter, caps and shared breakers matter more here than in a microservice mesh. It is also why concurrency limits and retry policy have to be designed as one thing.

Memory is non-linear in the input. Doubling a raster’s dimensions quadruples the pixels and can more than quadruple peak memory once warp buffers and overviews are counted. Failures therefore cluster on the largest inputs, which are also the most expensive to retry — the exact opposite of the comfortable case where failures are random and cheap.

Partial output is indistinguishable from complete output. A truncated GeoTIFF opens. A partially loaded feature table queries. Neither announces itself, which is why the write-to-temporary-and-rename discipline appears in every pattern here: it is the only cheap way to make “incomplete” and “absent” the same state.

Failures are geographically correlated. This is the constraint with the fewest analogues outside spatial work, and the most useful one for triage. Errors do not scatter randomly across a tile grid: they concentrate at projection edges, at the antimeridian, at datum boundaries, in the one municipality whose export tool writes a malformed .prj, and in the mountainous tiles whose geometry is densest. That correlation is a diagnostic gift — a map of your failures usually names the cause — but it also means sampling-based testing systematically misses them, because a random sample of tiles is overwhelmingly ocean and farmland.

Topic Deep-Dives#

Idempotency keys in spatial ETL#

Idempotency keys give a unit of work a deterministic name derived from its inputs, so a task can ask “have I already done this?” before doing it again. The spatial specifics are what goes into the digest — normalise the CRS through pyproj so EPSG:3857 and its WKT equivalent collapse to one value, include the nodata and resampling parameters, and exclude anything that varies per attempt.

The unit matters as much as the hash. Feature-level keys are precise and cost a lookup per geometry; tile-level keys cost one lookup per tile and re-do at most one tile’s work; extent-level keys are cheapest and re-do a region. Most raster pipelines sit at the tile level for the same reason their DAGs do. Two recipes cover the ends of the range: hashing a whole shapefile bundle for delivery-level safety, and idempotent PostGIS upserts for row-level safety without a ledger at all.

Exponential backoff for API rate limits#

Backoff decides when a failed call is tried again. The defaults from general practice — three attempts, doubling delay — are wrong for spatial services in two ways: they do not account for a single request costing the server seconds of rendering, and they count attempts where the meaningful bound is wall-clock time against a deadline.

The two recipes here cover the endpoints that behave least like a JSON API. A slow WFS needs paging so that a retry costs one page rather than a twenty-minute stream, and a read timeout set above the healthy p99 rather than at a library default. A geocoder that sends Retry-After knows more about its own quota window than any schedule you can invent, and its pause has to be shared across every worker holding the same API key.

Circuit breakers for external WMS services#

A circuit breaker is the layer that converts one worker’s knowledge into fleet-wide knowledge. Without it, sixty workers each walk a full retry ladder against a service that is comprehensively down, which is both futile and a meaningful fraction of that service’s capacity.

Three decisions carry most of the value. Thresholds should be derived from the endpoint’s measured request rate and healthy failure ratio rather than copied, and must count slow renders as failures, because map servers degrade before they error. Half-open recovery needs exactly one probe fleet-wide and several consecutive successes before closing. And the pipeline must decide in advance what an open breaker returns — failing fast, or serving a cached tile with honest provenance.

Dead-letter queues for failed geotasks#

A dead-letter queue is where work goes when its retries are spent. Its value is that a batch can complete while its failures stay re-drivable, which decouples “the pipeline is healthy” from “the data is complete” — two things that a run-success dashboard conflates by default.

The spatial part is the payload. A PostGIS queue table must accept the geometries the rest of the schema rejects, which means no validity check, no typed geometry column and no SRID constraint — the queue is the one table whose job is to keep what everything else throws away. Draining it is a flow rather than a script, because re-driving safely requires claiming rows, re-entering at the original key and capping attempts. And because a queue nobody watches is a delete with extra steps, alerting on its growth — the trend, never the depth — is part of the pattern rather than an optional extra.

Timeout budgets and cancellation for geotasks#

Timeout budgets replace a scattering of independently chosen constants with one deadline and a set of derivations. Every layer’s limit is min(its own maximum, time remaining), and every layer’s limit is strictly shorter than the layer enclosing it, so the innermost failure is the one that gets to explain itself.

The cancellation half is where the spatial specifics bite hardest. GDAL work is out of process, so sizing the subprocess limit has to follow the shape of the operation — output megapixels, resampling cost, whether the source is on object storage — and a single constant will always be wrong for a job whose tiles differ in cost by two orders of magnitude. And cancelling cleanly means signalling the process group, discarding the partial output, bounding the database session and releasing the ledger claim — four obligations, each of which leaves distinctive debris when skipped.

How the five layers are usually adopted#

Nobody builds all five at once, and the order in which they arrive is fairly consistent across teams. Retries come first, because the orchestrator offers them as a checkbox and a flaky endpoint makes the case for them immediately. That is also where the first real incident comes from: retries applied to a non-idempotent write, duplicating features until someone notices the feature count drifting upward on every re-run. Identity therefore usually arrives second, as a repair rather than as a design.

Bounds tend to arrive third, prompted by a run that did not finish before the morning deadline and could not be explained, because nothing in the pipeline recorded which step was waiting on what. Capture arrives fourth, when the accumulated small failures become large enough that “we re-run it and hope” stops being an acceptable answer. Shared health arrives last and often only after an upstream provider gets in touch about traffic — which is a slightly embarrassing way to discover that sixty workers were retrying a dead endpoint for four hours.

There is no strong reason to follow that order deliberately. Identity first and bounds second is a better sequence, because those two make everything else safe and legible, and both are cheap to add to a pipeline that does not yet have them. The rest can follow the pain.

Implementation Patterns & Code Scaffold#

The scaffold below is the composition of all five layers around one tile. It is deliberately unexciting: identity first, breaker outside the retry, the retry inside a budget, dead-letter capture outermost, and cleanup in a finally.

from __future__ import annotations

from pathlib import Path

from prefect import flow, task


@task(retries=0)                       # retries live in the client, not the task
def build_tile(tile: TileWork, deps: Deps, budget: Budget) -> None:
    key = work_key(tile)                                  # 1 · identity

    if not claim(deps.conn, key, tile):                   #     already done or in flight
        return

    final = Path(f"/data/tiles/{tile.tile_z}/{tile.tile_x}/{tile.tile_y}.tif")
    try:
        with atomic_output(final) as tmp:                 # 5 · bounds: no partial promote
            source = deps.breaker.call(                   # 3 · shared health
                lambda: fetch_with_backoff(               # 2 · timing
                    deps.http, tile.source_uri, params={}, policy=deps.backoff,
                )
            )
            shape = shape_for_tile(tile.source_uri, tile.dst_crs, tile.resampling)
            run_cancellable(
                ["gdalwarp", "-t_srs", tile.dst_crs, "-r", tile.resampling,
                 "-dstnodata", str(tile.nodata), "-co", "COMPRESS=DEFLATE",
                 source, str(tmp)],
                timeout=budget.slice(timeout_for(shape)),
            )
        complete(deps.conn, key, feature_count=0)
    except BreakerOpen:
        # An open breaker is not a data failure: leave it for the next run.
        release_claim(deps.conn, key)
        raise
    except Exception as exc:                              # 4 · capture
        dead_letter_geometry(
            deps.conn, work_key=key, source_uri=tile.source_uri, feature_id=None,
            geom=None, declared_crs=tile.src_crs, failure_class=classify(exc),
            error_text=str(exc), params=tile.params,
        )
        release_claim(deps.conn, key)
        raise


@flow(timeout_seconds=7200)
def build_mosaic(tiles: list[TileWork], deps: Deps) -> None:
    budget = Budget.of(7200 - 120)                        # reserve for finalisation
    for tile in tiles:
        if budget.remaining() < 60:
            break                                         # stop cleanly, not by force
        build_tile(tile, deps=deps, budget=budget)

Failure Modes & Operational Guardrails#

Projection mismatch that survives a retry. A source that changes its declared CRS produces geometry in the wrong place, and retrying reproduces it faithfully. Detection: coordinate ranges outside the target CRS’s valid area; a sudden change in the extent of a layer. Mitigation: include the CRS in the idempotency key so the change re-keys the work, and validate ranges against the declared authority before writing.

OOM on a raster mosaic. Warp buffers scale with the output, and a worker running eight tiles concurrently sizes them from what is available. Detection: the kernel OOM killer in worker logs; failures correlated with tile size rather than with time. Mitigation: bound GDAL with -wm, size the concurrency from memory rather than cores, and split the tiles that cannot fit rather than raising the machine size indefinitely.

Geometry corruption on retry. A retried feature load that is not idempotent inserts a second copy, and the duplicate breaks every subsequent overlay and topology check. Detection: feature counts that grow on re-runs; ST_IsValid failures appearing in derived layers rather than in source data. Mitigation: upsert on a natural key, or claim before writing — never both partially.

Stale tile cache after an outage. Tiles served from cache while a breaker was open persist indefinitely if nothing re-renders them. Detection: object metadata showing tile-source: cache outside an incident window. Mitigation: treat the outage as an invalidation event and re-queue the affected coordinates once the breaker closes.

Orphaned GDAL processes. A cancellation that signalled only the direct child leaves helpers holding memory. Detection: ps -eo etimes,comm finding GDAL processes older than the maximum timeout. Mitigation: start_new_session=True plus killpg, with SIGKILL escalation, and a start-up check on every worker.

A dead-letter queue that only grows. Failures are captured and never triaged, so the pipeline looks healthy while its output gets steadily less complete. Detection: the oldest unresolved entry ageing past a week. Mitigation: alert on the trend and on the age, schedule a drain for transient classes, and read the unknown bucket weekly.

Toolchain & Dependency Matrix#

A note on how these guardrails are usually discovered, since it affects how you should read the list. Almost none of them are found by design review; they are found the night they happen, and the useful artefact afterwards is not the fix but the detection signal. A team that responds to an OOM by raising the instance size has bought time; a team that responds by recording which tile sizes correlate with the failure has bought a threshold they can act on before the next one. Each row above is written detection-first for that reason — the mitigation is usually obvious once you can see the thing, and invisible until then.

The other pattern worth naming is that these failures compose badly. A projection mismatch produces geometries in the wrong place, which land in tiles that were not expected to have data, which makes those tiles far more expensive to render, which trips timeouts, which fills the dead-letter queue with upstream_timeout entries whose real cause is a CRS change three steps upstream. Triage that starts from the failure class alone will chase the timeout. Triage that starts from the geography — where are these tiles? — finds the projection change in minutes. That is the practical argument for capturing the spatial context on every dead letter, and it is worth more than any individual pattern here.

Frequently Asked Questions#

Related#

• Idempotency keys in spatial ETL — naming work so retries are safe
• Exponential backoff for API rate limits — retry timing for expensive endpoints
• Circuit breakers for external WMS services — fleet-wide knowledge of an outage
• Dead-letter queues for failed geotasks — durable capture of what failed
• Timeout budgets and cancellation for geotasks — one deadline, derived limits, real cancellation