Spatial Task Design & Dependency Mapping

Task design is the decision layer of a geospatial pipeline: what counts as one unit of work, what each unit promises about its output, how units depend on one another, and how many of them exist. Everything downstream — retries, caching, cost, the ability to resume a failed run — is determined by those four answers, and none of them can be changed cheaply once a pipeline is in production. They are worth getting right early.

What makes this a distinct discipline rather than a restatement of general data engineering is that spatial work resists every convenient default. The unit of work is a tile, an extent or a delivery rather than a row, so it is not implied by the schema. The dependency between steps is a data contract about projection and validity, not a foreign key. The cost of one unit varies by two orders of magnitude across a single job. And the number of units is not known until something inspects a footprint. A pipeline designed as though those four things were as tame as they are in a tabular warehouse does not fail loudly — it works, slowly and expensively, until the day it is asked to handle a continent.

Core Architectural Layers#

A spatial pipeline’s task design has five layers, each answering one question. They are ordered here roughly by how expensive they are to change later.

1. Unit of work — what is one task? A tile, a municipality, a scene, a delivery. This choice determines retry granularity, cache granularity, memory profile and how a partial failure presents. It is the hardest thing to change afterwards, because every other decision is expressed in terms of it.

2. Contract — what does each step promise? Format, CRS, geometry type, validity, deduplication. Building ETL chains for vector data treats the contract as the real design and the tasks as functions that satisfy it, which is what makes a step independently re-runnable.

3. Execution model — where does the work happen? Coroutines for waiting, processes for computing, subprocesses for anything that must be cancellable. Async execution for heavy GIS tasks covers the boundary between the two worlds and the failure that happens when it is crossed carelessly.

4. Reuse — what does not need doing again? A content-addressed cache for expensive deterministic work, and a branch that skips work whose inputs have not changed. Both are the same digest asking different questions.

5. Shape — how many tasks are there? Fixed for a chain, runtime-determined for a fan-out. Dynamic task mapping for tile fan-out is where a pipeline stops being a graph you drew and becomes a graph the data drew.

Key Design Constraints Imposed by Geospatial Data#

The unit of work is not implied by the data model. A tabular pipeline has rows, and a row is an obvious unit. A raster has no rows: the unit is whatever you decide a tile is, and that decision is a trade between retry granularity and per-task overhead that nothing in the data will make for you. Getting it wrong in the small direction means an orchestrator spending more time on bookkeeping than on work; in the large direction it means a failure costing an hour instead of a minute.

Dependencies are contracts, not keys. Step B depends on step A not because of a join key but because A promises geometry in EPSG:25832, made valid, deduplicated. That promise cannot be checked by the type system and is checked by nothing at all unless you make it explicit — which is why a contract object that each step asserts on entry earns its place so quickly.

Cost per unit varies by orders of magnitude. A tile over open sea and a tile over a mountain range are the same shape and differ hundredfold in runtime and memory. Every scheduling assumption that works for uniform units — fixed timeouts, fixed batch sizes, equal-sized chunks — misbehaves here, and the fix is always to derive the parameter from an inspection rather than to set it.

Memory scales non-linearly with the input. Doubling a raster’s edge quadruples its pixels and can more than quadruple peak memory once warp buffers and overviews are counted. That relationship is why memory, not CPU, is the binding constraint on almost every spatial worker, and why pool sizes and concurrency limits are memory calculations.

The graph’s width is data-dependent. How many tiles a scene covers is a property of its footprint. A pipeline that hard-codes the fan-out either over-provisions for the largest case or breaks on it, and the only correct answer is to inspect first and build the graph at runtime.

Topic Deep-Dives#

Building ETL chains for vector data#

Vector chains are the most common shape in this section: extract, validate, reproject, join, load. The design insight is that the chain’s value comes from its contracts rather than its steps — a small structure carrying format, CRS, geometry type and validity that each step asserts on entry and returns on exit. With it, a failure names its own cause; without it, an error surfaces three steps downstream as a database constraint violation.

The practical consequences are that intermediates are files rather than in-memory frames, that reprojection happens at exactly one point, and that validation happens at the boundary rather than defensively everywhere. Two recipes cover the ends: chaining GDAL tasks in Prefect for the orchestrator wiring, and streaming large GeoPackage loads into PostGIS for the load step once the file stops fitting in memory.

Async execution for heavy GIS tasks#

Async execution is worth exactly as much as the share of runtime a pipeline spends waiting, which is high for tile fetching and low for national reprojection. The design question is where the boundary sits between the waiting and the computing, and the failure mode is a synchronous call inside a coroutine — which produces a pipeline that looks concurrent and runs serially, with no error anywhere.

Running async GeoPandas tasks safely covers the boundary discipline and the debug-mode check that catches violations in CI. Offloading GDAL work to a process pool covers the other side: sizing the pool by memory rather than by cores, configuring workers correctly, and passing paths so the process boundary stays cheap.

Caching strategies for spatial tasks#

Caching is the highest-leverage optimisation available here and the one with the worst failure mode. A cache key must cover every input that determines the output — source digest, both CRSs, resampling, nodata, extent — because a key that omits one serves the wrong pixels under the right name, silently and indefinitely.

Caching reprojected rasters with content hashing works the raster case end to end, including how to identify a four-gigabyte source without reading it. Invalidating tile caches after a source update covers the other direction — turning a changed footprint into exactly the set of tiles that must be rebuilt, buffered by the resampling kernel’s reach.

Conditional branching in geospatial DAGs#

Branching is how a pipeline stays honest about a workload whose shape is genuinely bimodal. The discipline is to compute the predicate in its own task, from cheap header reads, and to record the decision — so that “why did it take the fast path?” is answerable months later rather than being an if nobody can see.

The two recipes are the two predicates that matter most. Branching on spatial extent chooses between single-pass and tiled processing from a memory estimate derived from the worker, not from a hard-coded constant. Skipping tiles with no new source data is a spatial join between the tile grid and the source footprints, and it is the difference between rebuilding a continent nightly and rebuilding what changed.

Spatial validation & sync tasks#

Validation belongs at the boundary, once, with the result recorded as a guarantee. Four checks cover nearly everything: is a CRS declared, do the coordinates fall inside it, is the geometry topologically valid, and is the attribute encoding right. Three of the four are nearly free and are the ones most often skipped.

Validating coordinate systems before ETL covers the declaration and range checks, which catch the most damaging class of spatial error for the price of a header read. Repairing invalid geometries before load covers the geometry half, and argues for classifying the defect and applying the narrowest fix rather than calling ST_MakeValid on everything and hoping.

Dynamic task mapping for tile fan-out#

Fan-out is where the graph’s width becomes a property of the data. Four things make it safe: a durable manifest, a concurrency limit bound to the resource rather than to the worker pool, tolerated partial failure, and a fan-in that moves paths rather than pixels.

Fanning out Prefect tasks over a tile manifest covers the map step, including the three separate limits that bound a fan-out and why their right values differ. Collecting mapped results into a single mosaic covers the reduce step, where a VRT keeps memory flat and an explicit coverage figure keeps an incomplete mosaic from passing as a complete one.

How the layers constrain each other#

The five layers are not independent, and the dependencies run downward. A unit of work of “one tile” makes tile-level caching possible and municipality-level caching awkward. A contract that carries a CRS makes the reprojection step cacheable, because the cache key can name what the step promised. An execution model of “subprocess per unit” makes cancellation possible, which makes a timeout budget meaningful, which makes a fan-out’s worst case computable. Each choice opens or closes the options above it.

That is also why retrofitting is painful in a specific way. Adding a cache to a pipeline whose unit of work is “the nightly run” is not hard — it is impossible, because there is nothing at a useful granularity to key on. Adding a fan-out to a pipeline whose steps pass in-memory frames means rewriting every step’s signature. The work of adopting any one of these patterns is usually the work of fixing a lower layer first, which is worth knowing before estimating it.

Implementation Patterns & Code Scaffold#

The scaffold below is one partition of a pipeline with all five layers visible: a contract, a decision, a bounded fan-out, and a reduce that accounts for what arrived.

from __future__ import annotations

from dataclasses import dataclass, replace
from pathlib import Path
from typing import Optional

from prefect import flow, task


@dataclass(frozen=True)
class Contract:
    """Layer 2 — what a step promises about its output."""

    path: Path
    crs: str
    geometry_type: str
    validity_guaranteed: bool
    feature_count: int

    def require(self, *, crs: Optional[str] = None, valid: bool = False) -> None:
        if crs and self.crs != crs:
            raise ValueError(f"expected {crs}, upstream produced {self.crs}")
        if valid and not self.validity_guaranteed:
            raise ValueError("this step requires repaired geometry upstream")


@flow(name="process-partition")
def process_partition(source_uri: str, scratch: Path, target_crs: str = "EPSG:25832") -> Path:
    # Layer 4 — reuse: has this exact work already been done?
    decision = inspect_source(source_uri, previous_digest=last_digest(source_uri))
    record_decision(source_uri, decision)
    if decision.path is Path_.SKIP:
        return existing_output(source_uri)

    # Layer 2 — the chain, each step asserting the contract it needs.
    raw = extract(source_uri, scratch / "01_raw.gpkg")
    clean = repair(raw, scratch / "02_valid.gpkg")
    clean.require(valid=True)
    projected = reproject(clean, target_crs, scratch / "03_projected.gpkg")
    projected.require(crs=target_crs, valid=True)

    # Layer 5 — shape: how wide the graph gets is decided here, from the data.
    manifest = build_manifest(projected, zoom=12, path=scratch / "manifest.json")
    states = render_tile.map(manifest, out_dir=scratch, return_state=True)
    paths = [s.result() if s.is_completed() else None for s in states]

    # Reduce by reference, and refuse to publish an incomplete product.
    result = collect_mosaic(manifest, paths, scratch / "mosaic.tif", min_coverage=0.995)
    if result.path is None:
        raise RuntimeError(f"coverage {result.coverage:.3f} below threshold — not published")
    return result.path

Failure Modes & Operational Guardrails#

A unit of work too large to retry. One task covering a whole country means a failure at ninety per cent costs the whole run. Detection: task durations in the tens of minutes with a bimodal outcome — either complete or nothing. Mitigation: split along the data’s natural partitions, which almost always exist and are almost always the right boundary.

A unit of work too small to schedule. Four hundred thousand tasks each taking fifty milliseconds means the orchestrator is the workload. Detection: summed task duration far below total run duration. Mitigation: batch items and map over the batches; the parallelism is unchanged and the bookkeeping falls by the batch factor.

A broken contract discovered downstream. A step receives data in the wrong CRS and fails at the load with a constraint violation. Detection: errors that name a database object rather than a pipeline step. Mitigation: assert the contract on entry to each step, so the failure names the handoff.

An event loop blocked by a library call. Concurrency is configured at sixty-four and throughput matches concurrency one. Detection: raising concurrency changes nothing; asyncio debug mode reports slow callbacks. Mitigation: move the call to a thread or a process pool, and keep the naming convention that makes the boundary visible.

A cache serving the wrong pixels. A parameter that affects the output is missing from the key. Detection: hit rate that rises after a parameter change — the signature of a key that ignores the parameter. Mitigation: enumerate every input, classify each as key material or configuration, and test that changing any one changes the key.

A fan-out that cancels itself. One failing tile takes four hundred siblings with it. Detection: runs that fail with far fewer completed tasks than the manifest. Mitigation: collect states rather than results, count what arrived against the manifest, and dead-letter the difference.

Each of those six has the same shape: a detection signal that is visible in ordinary telemetry, and a mitigation that is a design change rather than a parameter. That is characteristic of task-design failures generally — none of them are fixed by increasing a timeout or adding a retry, and several of them are made worse by it. A pipeline that responds to the second failure mode by raising worker counts, or to the fifth by shortening the cache TTL, will spend real money confirming that the problem is somewhere else.

The detection signals are worth building into a dashboard once rather than rediscovering per incident. Four numbers cover most of it: the ratio of summed task duration to run duration (which catches units that are too small), the distribution of task durations (which catches units that are too large and cost distributions that have drifted), the cache hit rate against the source change rate (which catches unstable or over-permissive keys), and completed tasks against manifest size (which catches a fan-out that cancelled itself). None of the four require instrumentation beyond what the orchestrator already records.

Toolchain & Dependency Matrix#

Two entries in that table are worth a comment because they constrain more than they appear to. pyproj is pinned not for API stability but because its authority database participates in cache keys and in area-of-use checks: an upgrade can change what EPSG:4326 resolves to for a deprecated code, which re-keys a cache and re-validates a dataset that was previously accepted. And GDAL is pinned per image rather than per environment, because a spatial pipeline’s reproducibility is a property of the container it runs in — a pip install that satisfies a version range while linking against a different libgdal produces results that differ from the ones the tests saw.

Frequently Asked Questions#

Related#

• Building ETL chains for vector data — contracts between steps
• Async execution for heavy GIS tasks — choosing the execution model
• Caching strategies for spatial tasks — reuse without stale results
• Conditional branching in geospatial DAGs — decisions the run history can explain
• Dynamic task mapping for tile fan-out — a graph the data draws