Geospatial Orchestration Architecture & Fundamentals

A geospatial pipeline is an ordinary data pipeline right up to the point where it isn’t, and every difference comes from the same source: the data has an extent, a projection and a wildly uneven distribution over the ground. Those three properties decide the shape of the graph, the size of the workers, where the money goes, what a credential must reach and how a deployment can fail. This section is about the decisions that follow from them — the ones taken early, encoded everywhere, and expensive to revisit.

There are seven of them. What is the unit of work. Where does state live. Which orchestrator models that unit best. What does a run cost and why. Who can reach the data, and how precisely. How does the native stack ship. And what does the pipeline divide the world into. None is difficult in isolation; together they determine whether a pipeline that works in June is still operable in December.

The order in which they bind matters more than the order in which they are implemented. The unit of work constrains everything else — a ledger cannot be keyed on something that has no canonical name, a partition cannot be smaller than the thing being retried, a credential cannot be scoped to a prefix that does not exist. So the sequence below is a dependency order rather than a schedule; in practice all seven are built together, and the value of naming them separately is knowing which one to revisit when something further along turns out to be awkward.

Core Architectural Layers#

The unit layer. A tile index, a scene identifier, a municipality code — the thing a run is made of. It fixes the retry granularity, the cache key, the failure blast radius and the fan-out width simultaneously, which is why DAG design principles for spatial ETL treats it as the first decision rather than an implementation detail.

The state layer. What has been built, how far a long job got, and how current the output is. Three different questions with three different homes, covered in state management in geospatial flows.

The orchestration layer. Whether the tool models runs or the things runs produce, which decides how much bookkeeping you write yourself. Prefect vs Dagster for GIS workloads sets out the trade.

The resource layer. Worker memory, upstream concurrency, bytes read, instance market. Cost optimization for spatial compute shows where the money actually goes, which is rarely where people look first.

The trust layer. Credentials scoped to a job, sources treated as untrusted input, coordinates treated as personal data. Security boundaries for spatial data covers all three.

The delivery layer. One immutable image, a boot-time assertion, a canary that reprojects. Deployment & CI/CD for spatial workers explains why the discipline is stricter here than for a service.

The partition layer. Grids, administrative units or hierarchical cells, and what to do about the skew all three have. Partitioning strategies for spatial workloads profiles them against real data.

None of these layers is optional, and several are frequently implicit. A pipeline with no explicit state layer still has one — it is the orchestrator’s task history, which happens to be the wrong granularity. A pipeline with no partition decision still has a partition, which is whatever the first fan-out happened to iterate over. The difference between an implicit and an explicit choice is not usually correctness on day one; it is whether the choice can be found and changed by somebody who was not there when it was made.

Key Design Constraints Imposed by Geospatial Data#

Extent means the graph’s width is data-dependent. A footprint decides how many tiles a run produces, so a wrong zoom is a hundred thousand scheduled tasks rather than a raised exception. Every fan-out needs a guard and a manifest written before it happens.

Projection means correctness is invisible. A wrong CRS produces valid files, complete coverage and a layer in the wrong place. Only an explicit comparison against a known answer catches it, which is why validating coordinate systems before ETL exists and why a boot-time transformation probe is worth having on every worker.

Density means uniform partitions carry non-uniform work. The largest cell in a national grid is routinely hundreds of times the median, so a run’s duration is set by one city while the fleet idles.

Size means memory is the binding constraint. Spatial work is memory-hungry in short spikes, so per-task peak times concurrency is the number that sizes a fleet — and the number nobody has measured.

Sources are other people’s. Publishers change projections, truncate deliveries, republish unchanged files nightly and occasionally stop. A pipeline over public geodata spends much of its design budget on that fact.

The native stack decides the answers. GDAL, PROJ and their grids determine what a reprojection returns, so the deployment artefact is closer to a specification of correctness than to packaging.

Topic Deep-Dives#

DAG design principles for spatial ETL#

Shape the graph from the data’s natural partitions rather than from the verbs a script performs. The topic covers granularity, real dependency edges and the two limits every fan-out needs; its guides cover structuring a raster DAG, parametrizing by tile index, mapping over a grid in Dagster and bounding the fan-out.

State management in geospatial flows#

The ledger records what exists, the checkpoint records how far a job got, the watermark records how current the source is. The topic keeps them apart; its guides cover checkpointing mosaics, incremental vector deliveries, choosing a home for each and resuming from a checkpoint.

Prefect vs Dagster for GIS workloads#

One models runs, the other models the things runs produce, and the difference decides how you answer “which tiles are missing”. The topic compares them properly; its guides cover what Prefect’s states mean, migrating between them, where Airflow fits and what a task run costs.

Cost optimization for spatial compute#

Three levers — measured memory, bytes read, and work skipped — dwarf the instance-price question everyone starts with. The topic covers all three; its guides cover sizing workers, interruptible capacity and range reads.

Security boundaries for spatial data#

A coordinate can identify a person, a source file is input to a large C parser, and a fan-out multiplies every credential decision by its width. The topic covers the three boundaries; its guides cover database connections, logs and object-storage credentials.

Deployment & CI/CD for spatial workers#

The native stack decides the numbers, so the deployment artefact has to be pinned, asserted and compared rather than merely started. The topic covers the chain; its guides cover the image, the lock file, parity, fixtures and upgrades.

Partitioning strategies for spatial workloads#

Grids are simple and skewed, administrative units match the data and vary hugely, hierarchical cells sit between. The topic profiles all three on real data; its guides cover raster grain, H3 cells and the skew every scheme has.

Where to start if you are starting now#

Decide the unit of work, give it a canonical name, and derive a content-based key from it. Almost everything else in this section becomes easier once that exists: the ledger has a primary key, the cache has something to hash, the partition has a name, the credential has a prefix to scope to, and the coverage question has a denominator. Pipelines that skip it spend the following year adding each of those separately.

How the layers show up in one run#

It is worth walking a single nightly run through all seven, because they are much easier to hold together as a narrative than as a list. The run begins by asking the source what has changed, which is the watermark and the digest — the state layer. It computes a manifest of affected units from a footprint, which is the unit and partition layers deciding the graph’s width. It acquires a session scoped to one source prefix and one output prefix, which is the trust layer. It fans out under two concurrency limits sized from measured memory and a publisher’s tolerance, which is the resource layer. Each task consults the ledger, does nothing if the key matches, and writes a checkpoint if the work is long. The reduction refuses to assemble below a coverage floor, and the publish step stamps the image digest onto every object, which is the delivery layer making the run auditable afterwards.

Nothing in that narrative is exotic, and every step of it is a place where a pipeline that skipped the corresponding decision has to improvise. The improvisations are individually reasonable and collectively expensive: a manifest computed inline so the width is never logged, a credential shared because scoping it was fiddly, a ledger that is really a cache, a coverage check that was going to be added later. Each is a small shortcut whose cost arrives during an incident, when the question is which tiles are wrong and the pipeline cannot say.

Implementation Patterns & Code Scaffold#

The scaffold below is the smallest thing that has all seven decisions in it. Everything else in this section is an expansion of one of these lines.

from __future__ import annotations

import hashlib
from dataclasses import dataclass
from pathlib import Path

from prefect import flow, task, get_run_logger
from prefect.concurrency.sync import concurrency


@dataclass(frozen=True, slots=True)
class Tile:
    """1. The unit of work, canonical and validated at construction."""

    z: int
    x: int
    y: int

    def __post_init__(self) -> None:
        if not (0 <= self.z <= 22 and 0 <= self.x < 2 ** self.z and 0 <= self.y < 2 ** self.z):
            raise ValueError(f"non-canonical tile index z{self.z}/{self.x}/{self.y}")

    @property
    def path(self) -> str:
        return f"{self.z}/{self.x}/{self.y}"


def work_key(tile: Tile, source_digest: str, recipe: str) -> str:
    """2. Content-derived, so an unchanged source does no work."""
    return hashlib.sha256(f"{tile.path}|{source_digest}|{recipe}".encode()).hexdigest()[:16]


@task(timeout_seconds=300)
def plan(footprint_wkt: str, zoom: int) -> list[Tile]:
    """3. The graph's width, decided and guarded before anything fans out."""
    tiles = [Tile(zoom, x, y) for x, y in tiles_intersecting(footprint_wkt, zoom)]
    if not tiles:
        raise ValueError("empty manifest — footprint and zoom disagree")
    if len(tiles) > 20_000:
        raise ValueError(f"{len(tiles)} tiles at z{zoom} is implausible")
    get_run_logger().info("manifest: %d tiles at z%d", len(tiles), zoom)
    return tiles


@task(retries=2, retry_delay_seconds=[15, 60], timeout_seconds=900, tags=["tile"])
def build(tile: Tile, source: SourceSpec, recipe: str, session) -> str | None:
    """4. Bounded twice, idempotent, and recorded only once the write is durable."""
    key = work_key(tile, source.digest, recipe)
    if ledger_has(key):
        return tile.path
    with concurrency(f"source:{source.host}", occupy=1, timeout_seconds=1800):
        window = fetch_window(source, tile, session)     # 5. scoped session
    write_tile(warp(window, tile), tile, session)
    ledger_mark(key, tile)                                # 6. after the object exists
    return tile.path


@flow(name="tile-build", timeout_seconds=7200)
def tile_build(footprint_wkt: str, zoom: int, source: SourceSpec,
               recipe: str = "v4", coverage_floor: float = 0.995):
    tiles = plan(footprint_wkt, zoom)
    with job_session(source.prefix, output_prefix=f"tiles/{recipe}") as session:
        states = build.map(tiles, source=source, recipe=recipe,
                           session=session, return_state=True)

    done = [s for s in states if s.is_completed()]
    coverage = len(done) / len(tiles)
    get_run_logger().info("tiles: %d/%d, coverage %.4f", len(done), len(tiles), coverage)
    if coverage < coverage_floor:
        # 7. The run's verdict reflects the product, not merely the execution.
        raise RuntimeError(f"coverage {coverage:.4f} < {coverage_floor} — not publishing")
    return publish(done, recipe=recipe)

Four properties of that scaffold are worth naming, because they are what the rest of this section defends. The manifest is written before the fan-out, so the width is inspectable rather than discovered. The work key is content-derived, so a re-run over an unchanged source is nearly free. The ledger is written after the object exists, so a crash leaves an orphaned object rather than a phantom record. And the coverage floor makes the run’s final state a statement about the product rather than about whether the code returned.

A fifth property is easy to miss and worth pointing at: the domain functions the tasks call — tiles_intersecting, fetch_window, warp, write_tile — contain no orchestrator concepts at all. That separation is what makes the pipeline testable without a control plane, portable between orchestrators, and reviewable by somebody who knows geospatial processing but not this particular framework. It costs nothing to maintain if it is established at the start and is genuinely difficult to recover once decorators have accumulated business logic.

Failure Modes & Operational Guardrails#

The graph is wider than anyone expected. A footprint that grew, a zoom typo, a source whose extent became global. The guard in plan costs two lines and converts four hundred thousand scheduled tasks into a raised exception.

A retry storm reaches somebody else’s server. Task retries multiply by flow retries, and a fan-out multiplies both. Retry at one level, bound the concurrency at the resource, and see limiting DAG fan-out.

Everything rebuilds every night. A timestamp entered the work key, or the source republishes unchanged files. The skip rate is the metric, and a sudden drop to zero deserves an alert.

A worker runs out of memory intermittently. Concurrency times per-task peak exceeded the limit, usually because the driver cache was left at its default. Right-sizing workers has the arithmetic.

The layer is in the wrong place and nothing failed. A missing transformation grid, a flipped tile axis, a CRS assumed rather than checked. Only a comparison against a known answer catches any of them.

One partition sets the duration of the run. Adding workers does not help; the critical path is one unit. Weigh before scheduling, as in handling skewed partitions.

A dependency upgrade changed the outputs. Nothing errored. Diff a shadow run’s tiles against the previous digest’s before promoting; see rolling out GDAL upgrades.

Reading the guardrails as a checklist#

Each of those failure modes maps to a control that costs very little to add and a great deal to add late. A manifest guard is two lines in the planning task. A retry budget is a decision about which level retries rather than a mechanism. A skip-rate log line is one call. A memory histogram is three. A transformation probe is a known coordinate and an assertion. A weight query is one grouped SELECT. A shadow diff is a flow that already exists, run twice. The whole list is perhaps a day of work spread over the life of a pipeline, and it is the difference between an incident that is diagnosed in ten minutes and one that is diagnosed by reading code.

The reason to treat them as a checklist rather than as good practice is that none of them is prompted by anything. A pipeline without a manifest guard behaves perfectly until the day it does not; a pipeline without a skip-rate metric never complains about rebuilding everything. These are all controls whose absence is silent, so they have to be added deliberately, and the natural moment to do that is when the architecture is being decided rather than after the first surprise.

Toolchain & Dependency Matrix#

Frequently Asked Questions#

Related#

• DAG design principles for spatial ETL — the unit of work and the graph’s shape
• State management in geospatial flows — ledger, checkpoint and watermark
• Prefect vs Dagster for GIS workloads — choosing the orchestration model
• Cost optimization for spatial compute — where the money goes
• Security boundaries for spatial data — credentials, sources and coordinates
• Deployment & CI/CD for spatial workers — shipping the native stack
• Partitioning strategies for spatial workloads — dividing the world, and the skew that follows

← Geospatial Orchestration