Airflow vs Prefect for Spatial Pipelines

Airflow was designed for a graph that is known before the run starts, and a spatial fan-out is not. Dynamic task mapping closed most of that gap, so the question is no longer “can Airflow do this” — it can — but “what does it cost”. The answer is a scheduler that has to materialise every mapped instance, a UI that becomes unusable past a few thousand of them, and a worker model that makes per-task resource limits awkward. Against that sits the strongest argument in the comparison: the platform is already running, and the team already knows it.

When to Use This Pattern#

• An Airflow deployment already exists and adding a spatial pipeline to it is being considered.
• The fan-out is a few hundred to a few thousand units, which is the range where the choice actually matters.
• Operations are shared with a wider data team that has Airflow conventions and on-call rotas.
• A move is being contemplated and somebody needs a concrete account of what changes.

Complete Working Example#

The same tile build in both, written the way each tool wants it. Airflow, with dynamic task mapping:

from __future__ import annotations

import pendulum
from airflow.decorators import dag, task
from airflow.models import Variable


@dag(
    schedule="0 2 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    max_active_tasks=64,                     # worker-side bound for this DAG
    default_args={"retries": 2, "retry_exponential_backoff": True},
)
def tile_build():
    @task
    def plan() -> list[dict]:
        tiles = tiles_intersecting(Variable.get("service_area_wkt"), zoom=12)
        if len(tiles) > 20_000:
            raise ValueError(f"{len(tiles)} tiles is implausible")
        return [{"z": 12, "x": x, "y": y} for x, y in tiles]

    @task(pool="scene-reads", pool_slots=1, max_active_tis_per_dag=64)
    def build(tile: dict) -> str | None:
        key = work_key(tile, source_digest(), RECIPE)
        if ledger_has(key):
            return tile_path(tile)
        path = render(tile)
        ledger_mark(key, tile)
        return path

    @task
    def reduce(paths: list[str | None], expected: int) -> str:
        present = [p for p in paths if p]
        if len(present) / expected < 0.995:
            raise ValueError(f"coverage {len(present) / expected:.4f} — not assembling")
        return assemble(present)

    items = plan()
    reduce(build.expand(tile=items), expected=412)


tile_build()

Prefect, with the same guards:

@task(retries=2, retry_delay_seconds=[15, 60], timeout_seconds=900, tags=["tile"])
def build(tile: Tile) -> str | None:
    key = work_key(tile, source_digest(), RECIPE)
    if ledger_has(key):
        return tile.path
    with concurrency("scene-reads", occupy=1):
        path = render(tile)
    ledger_mark(key, tile)
    return path


@flow(name="tile-build", timeout_seconds=7200)
def tile_build(footprint: str, zoom: int) -> str:
    tiles = plan(footprint, zoom)
    states = build.map(tiles, return_state=True)
    paths = [s.result() if s.is_completed() else None for s in states]
    return reduce(paths, expected=len(tiles))

The pool mechanism is the closest Airflow analogue to a named concurrency slot, and it is genuinely good: pools are global, defined in the UI or via configuration, and shared across DAGs, which is exactly the property a shared upstream source needs. What Airflow does not give you easily is the ability to hold the slot for only part of a task — the pool is acquired for the task’s whole duration — so the “acquire late, release early” discipline that keeps a source limit from becoming a throughput cap has to be expressed by splitting the fetch into its own task, with the intermediate written somewhere both can see.

Parameter & Option Reference#

Verification & Testing#

Whichever tool is chosen, the same properties need pinning, and the tests look different enough to be worth seeing.

# Airflow: the DAG parses, and the fan-out guard is real.
from airflow.models import DagBag


def test_dag_parses_without_errors() -> None:
    bag = DagBag(include_examples=False)
    assert bag.import_errors == {}
    assert bag.get_dag("tile_build") is not None


def test_plan_refuses_an_implausible_width(monkeypatch) -> None:
    monkeypatch.setattr("mymodule.tiles_intersecting", lambda *a, **k: [(0, 0)] * 25_000)
    with pytest.raises(ValueError, match="implausible"):
        get_task("tile_build", "plan").python_callable()


def test_pool_is_declared() -> None:
    task = get_task("tile_build", "build")
    assert task.pool == "scene-reads", "the source limit is not applied"

The DAG-parses test has no Prefect equivalent and it is not a trivial check. Airflow imports every DAG file on a loop in the scheduler, so an import error in one pipeline degrades the whole deployment — a missing GDAL binding in the scheduler’s environment, for instance, will break DAG parsing even though the actual work runs on a worker that has it. Keeping heavy spatial imports inside task functions rather than at module scope is the standard defence, and the parse test is what stops somebody undoing it.

The scheduling model is the other place where a spatial pipeline pushes against Airflow’s design. Airflow’s central concept is the logical date: a run is a date, backfilling means re-running dates, and the whole catchup mechanism assumes that a date identifies a unit of work. That is exactly right for a warehouse job over yesterday’s partition and only approximately right for a tile pyramid, where the unit of work is a tile and the thing that decides whether it needs rebuilding is a source digest, not a calendar. The two can coexist — schedule by date, key by content — but the pipeline has to hold the distinction deliberately, because every default in the platform will quietly encourage conflating them.

The clearest symptom of that conflation is a team that recovers from a bad delivery by clearing and re-running a date range. It works, in the sense that tiles get rebuilt; it also rebuilds every tile whose source never changed, and it will not rebuild a tile whose correction arrived under a different date than the one being cleared. Both failures come from treating the schedule as the identity of the work. Prefect avoids the trap less by design than by omission: it has no strong opinion about dates, so nothing invites the mistake.

Common Pitfalls#

• Importing rasterio, fiona or GDAL at module scope in a DAG file. The scheduler pays for it on every parse and a missing binding breaks the whole deployment.
• Expanding over tens of thousands of items. The metadata database and the grid view both degrade; batch into coarser units instead.
• Using the logical date as the work key. Spatial work keys should be content-derived; the logical date is a scheduling artefact and re-running a date is not the same as rebuilding what changed.
• Relying on max_active_tasks to protect an upstream source. It bounds your workers, not somebody else’s server; a pool is the mechanism for the latter.
• Holding a pool slot through an expensive warp. The slot becomes a throughput cap; split the fetch into its own task if it matters.
• Assuming migration means rewriting the spatial code. It does not, provided the domain functions were extracted from the decorators.

Frequently Asked Questions#

Related#

• Prefect vs Dagster for GIS workloads — the other axis of the same decision
• Migrating from Prefect to Dagster for spatial pipelines — the mechanics of moving, whichever direction
• Limiting DAG fan-out with concurrency groups — pools and slots compared
• Benchmarking orchestrator overhead for small geotasks — the per-task cost in numbers
• Containerizing GDAL workers with Docker — why the scheduler image should not carry GDAL

← Prefect vs Dagster for GIS Workloads