DAG Design Principles for Spatial ETL

In short: a spatial DAG’s shape should come from the data’s natural partitions — tiles, scenes, municipalities — not from the steps a script happens to perform. Get the granularity right and retries, caching, resumption and partial failure all follow; get it wrong and every one of them has to be worked around individually for the life of the pipeline.

The default shape of a geoprocessing pipeline is one node per verb: extract, reproject, mosaic, publish. It draws well and it hides the property that matters, which is that “reproject” is four hundred independent operations and “mosaic” is one operation that depends on all of them. Those two nodes have completely different failure characteristics, memory profiles and retry semantics, and a graph that renders them as equal boxes is a graph that will surprise its operators.

It is worth being specific about why this matters more here than in a tabular pipeline. A warehouse job’s parallelism is the engine’s business: you write a query and the planner decides how many workers touch how many partitions. A spatial pipeline has no planner. The orchestrator does exactly what the graph says, so the graph is the execution plan — every decision about parallelism, memory, retry scope and failure isolation is encoded in its shape, and none of it is inferred. That places the whole burden on a design that is easy to draw badly and expensive to redraw.

Prerequisites & Architecture Baseline#

• Prefect 2.14+ or Dagster 1.5+, both of which can build graph structure at runtime
• A stated unit of work per stage: tile, scene, municipality or delivery
• Concurrency limits configured per resource, not only per work pool
• An idempotency key per unit, so a partial re-run is cheap
• Measured task durations from at least one full run, since granularity decisions depend on them
• Scratch storage sized from concurrency × the largest intermediate, not from the total

Core Principles#

1. Partition on what the data already partitions on. Sources arrive per scene, per municipality, per map sheet. A DAG whose units match those boundaries can retry one of them, cache one of them and skip one of them. A DAG that concatenates everything and re-splits later has thrown away a structure it will spend the rest of its life trying to recover.

2. A task should be minutes, not hours or milliseconds. Hours means a failure is expensive and progress is invisible. Milliseconds means the orchestrator’s bookkeeping — state writes, logging, result persistence — costs more than the work. Between about one and fifteen minutes, almost everything works: retries are cheap, progress is legible, and overhead is negligible.

3. Dependency edges must be real. In Prefect the edge exists because one task consumed another’s return value; in Dagster because an op declared an input. Writing tasks in order and having them agree on a hard-coded path is not a dependency, and it produces a graph that runs everything concurrently while looking sequential.

4. Fan-out is bounded twice. Once by the orchestrator, which decides how many task runs may execute, and once by the resource, which decides how many may touch one upstream service. Those numbers are rarely the same, and setting only the first is how a fan-out that is polite to your workers overwhelms someone else’s endpoint.

5. Reductions are their own stage. A mosaic, a merge, a publish is one node depending on many, and it has a different memory profile from every node feeding it. Making it explicit lets it be retried alone, and it forces the question of what should happen when only 398 of 412 inputs arrived.

6. The graph should be inspectable before it runs. A manifest written to storage, a logged item count, a guard that refuses an implausible fan-out. A runtime-shaped graph is powerful and it is also the easiest way to launch four hundred thousand tasks by accident.

The reason the partition-shaped view matters operationally, rather than merely aesthetically, is that it is the only one that supports a conversation about failure. Ask of the verb-shaped graph “what happens if reproject fails” and the honest answer is “it depends which of the four hundred reprojections you mean”. Ask it of the partition-shaped graph and the answer is a sentence: that tile is retried twice, then dead-lettered, and the mosaic publishes if coverage stays above the threshold. Every operational property people want from a pipeline — resumability, partial success, per-unit caching, progress — is a property of the second shape and unavailable in the first.

Production Implementation#

The flow below shows the shape: a partition per scene, a bounded fan-out per tile, and an explicit reduction that accounts for what arrived.

from __future__ import annotations

from pathlib import Path
from typing import Optional

from prefect import flow, task, get_run_logger
from prefect.concurrency.sync import concurrency


@task(retries=1, timeout_seconds=600)
def build_manifest(scene_uri: str, zoom: int, out: Path) -> list[TileItem]:
    """The graph's width is decided here, from the data, and recorded."""
    items = [
        TileItem(zoom, t.x, t.y, scene_uri, work_key=tile_key(scene_uri, zoom, t.x, t.y))
        for t in tiles_intersecting(footprint_of(scene_uri), zoom)
    ]
    out.write_text(json.dumps([asdict(i) for i in items]))
    get_run_logger().info("manifest: %d tiles at z%d", len(items), zoom)
    if len(items) > 20_000:
        raise ValueError(f"fan-out of {len(items)} is implausible — check the zoom")
    return items


@task(retries=2, retry_delay_seconds=[15, 60], timeout_seconds=900, tags=["tile"])
def render_tile(item: TileItem, scratch: Path) -> Optional[str]:
    """Minutes, not hours: one tile, independently retryable, idempotent."""
    if ledger_has(item.work_key):
        return str(scratch / f"{item.z}_{item.x}_{item.y}.tif")
    # The resource limit, distinct from the work pool's: it bounds how many
    # tasks read the SOURCE, which is what the upstream service experiences.
    with concurrency("scene-reads", occupy=1):
        dest = warp_window(item, scratch)
    ledger_mark(item.work_key)
    return str(dest)


@task(retries=1, timeout_seconds=3600)
def reduce_to_mosaic(paths: list[Optional[str]], expected: int, dest: Path) -> Path:
    """One node depending on many, with a different memory profile from all of them."""
    present = [p for p in paths if p]
    if len(present) / expected < 0.995:
        raise RuntimeError(f"coverage {len(present)}/{expected} — not publishing")
    build_vrt_and_translate(present, dest)      # bounded memory, one sequential pass
    return dest


@flow(name="scene-to-mosaic", timeout_seconds=7200)
def scene_to_mosaic(scene_uri: str, zoom: int, scratch: Path) -> Path:
    items = build_manifest(scene_uri, zoom, scratch / "manifest.json")
    # The edge is the ARGUMENT. Calling these in order without passing `items`
    # would let the reduction start before any tile existed.
    states = render_tile.map(items, scratch=scratch, return_state=True)
    paths = [s.result() if s.is_completed() else None for s in states]
    return reduce_to_mosaic(paths, expected=len(items), dest=scratch / "mosaic.tif")

Step-by-Step Walkthrough#

1. Decide the partition before writing any tasks. Scene, tile, municipality — the choice determines retry cost, cache granularity and memory profile, and it is the hardest thing to change later.
2. Derive the fan-out from the data. The manifest comes from the scene’s footprint, so a coastal scene produces fewer tiles than its bounding box would suggest. Enumerating the envelope instead is the most common way a fan-out doubles for nothing.
3. Guard the width. One comparison catches a wrong zoom before the orchestrator schedules a hundred thousand runs — which is far easier than cancelling them afterwards.
4. Bound concurrency at the resource. concurrency("scene-reads") is about the upstream service; the work pool’s limit is about your workers. Both are needed and they are rarely the same number.
5. Make every mapped task idempotent. The ledger check turns a re-run after a partial failure from four hundred renders into fourteen, which is the difference between a re-drive being routine and being avoided.
6. Pass values to create edges. render_tile.map(items, …) depends on build_manifest because it consumes its return. A hard-coded path would produce a graph with no edges and a race.
7. Make the reduction explicit and accountable. It takes the expected count as well as the paths, so “398 of 412” is a decision rather than a silent publication.

Edge Cases & Failure Recovery#

A partition that is far larger than its siblings. One municipality with ten times the features, or one scene at a much finer resolution, will dominate the run’s wall-clock and may not fit in a worker at all. Split on a second axis where this happens — by tile within the municipality — rather than sizing every worker for the outlier.

A reduction that runs before its inputs exist. The classic symptom of a graph whose edges are implied by call order rather than by data flow. It passes on small test data, where the first task finishes before the second starts, and fails intermittently at scale.

Tasks too small to be worth scheduling. Four hundred thousand mapped runs at fifty milliseconds each means the orchestrator is the workload. The signal is the ratio of summed task duration to run duration; below about half, batch the items and map over batches instead.

A fan-out that starves its own tail. Tile cost varies by an order of magnitude, so the last few tasks can run long after the rest have finished and the workers have gone idle. Ordering the manifest most-expensive-first is a scheduling hint that costs nothing and routinely recovers a quarter of the wall-clock.

Scratch sized for one task. Sixteen concurrent tiles each writing a 300 MB intermediate need five gigabytes, and the failure arrives as a disk-full error in whichever task was writing at the time — never the one that caused it. Size from concurrency, and clean up per item.

A dependency on a side effect rather than on a value. Two tasks that both write to the same PostGIS table have a real ordering requirement that the graph knows nothing about, because neither consumes the other’s return value. The orchestrator will happily run them concurrently and the result is a deadlock or a lost update. Where the dependency is a side effect, return something — a table name, a row count, a token — purely so the edge exists.

A graph that cannot be resumed. If the manifest lives only in memory, a re-run recomputes it and may produce a different list against a republished source. Persisting it is what makes a re-drive comparable to the original run rather than merely similar.

Configuration Reference#

The two concurrency numbers deserve to be written down next to each other with their derivations, because they are the settings most likely to be changed by someone who only knows about one of them. A work-pool limit raised to speed up a backlog will, without a resource limit, raise the load on every upstream service the pipeline touches — and the first symptom will be a partner’s rate-limiting rather than anything visible in your own telemetry. Keeping both in one configuration file, each with a comment naming what it protects, is a small piece of documentation that prevents a recurring incident.

Frequently Asked Questions#

Related#

• How to structure a DAG for raster processing — the raster shape in full
• Parametrizing spatial DAGs by tile index — the partition key in practice
• Mapping tasks over a tile grid in Dagster — the same shape with dynamic outputs
• Limiting DAG fan-out with concurrency groups — the two limits, configured
• Spatial task design & dependency mapping — the wider set of decisions this sits inside

← Geospatial Orchestration Architecture Fundamentals