Prefect vs Dagster for GIS Workloads

In short: Prefect models runs and Dagster models the things runs produce. For spatial work that difference decides how you answer “which tiles are missing”, how a partial failure is recovered, and how much bookkeeping you write yourself. Prefect is lighter and better at runtime-shaped fan-outs; Dagster is heavier and better when the grid outlives the run. Neither is a mistake, but choosing the wrong one shows up as code you did not expect to write.

Both tools run Python, both retry, both have concurrency limits, and a demo built in either looks much the same. The divergence appears the first time a run fails half way. In Prefect you have four hundred task runs, some completed and some not, and a flow that finished with a failure; recovering means knowing which tiles those task runs corresponded to, which means you have written a ledger. In Dagster you have three hundred and one materialised partitions and a hundred and eleven that are not, and recovering means selecting the gap in the UI.

That is the whole comparison in miniature: Dagster ships the bookkeeping that spatial pipelines need, and charges for it in concepts and operational weight. Prefect ships a smaller, faster thing and leaves the bookkeeping to you, which is the right trade when the fan-out is transient and the wrong one when it is a national tile pyramid rebuilt every night for five years.

One clarification before the details, because the framing gets muddled in most comparisons. This is not a question about which tool has better retries, nicer decorators or a more pleasant interface; on those axes they are close enough that preference decides, and preference is a legitimate tiebreaker. It is a question about what the tool considers the noun of your pipeline. Everything downstream — recovery, backfills, coverage reporting, what an operator sees at three in the morning — follows from that one modelling decision, and it is the only part of the comparison that is expensive to change later.

Prerequisites & Architecture Baseline#

• A stated unit of work — tile, scene, municipality — and an idea of how many there are
• Knowledge of whether that unit's set is fixed or discovered at runtime
• A view on who operates the pipeline: the team that wrote it, or a wider group
• Somewhere to run a control plane, or a decision to pay for a hosted one
• Existing conventions worth respecting — a team already fluent in one tool has a real head start
• An honest estimate of how often partial recovery will be needed

Core Principles#

1. Prefect’s unit is the run; Dagster’s is the asset. Everything else follows. A run is ephemeral and a tile is not, so a Prefect pipeline over tiles needs a ledger of its own and a Dagster one gets the equivalent for free.

2. Runtime-shaped graphs are Prefect’s strength. task.map over a manifest computed seconds earlier is natural, cheap and unbounded. Dagster’s dynamic outputs do the same thing with more ceremony.

3. Fixed grids are Dagster’s strength. A partition set is declarative, backfillable by selection, and queryable. Expressing the same thing in Prefect means writing and maintaining the query yourself.

4. Operational weight differs by roughly a factor of two. Dagster wants a code location, a daemon and a database; Prefect wants a server and a work pool. Both are manageable, and neither is free.

5. Both need the same spatial discipline. Content-keyed work, bounded fan-out, idempotent writes and a coverage check before publishing are tool-independent. Choosing well saves bookkeeping, not thinking.

6. Migration is possible and rarely worth doing for its own sake. The task bodies port almost unchanged; the graph structure does not. Migrate when the model is wrong for the work, not when the syntax is unfamiliar.

Production Implementation#

The same tile build, written idiomatically in each, makes the difference concrete. In Prefect the manifest is a value and the fan-out is a call:

from prefect import flow, task
from prefect.concurrency.sync import concurrency


@task(retries=2, timeout_seconds=900, tags=["tile"])
def build_tile(tile: Tile, digest: str) -> str:
    if ledger_has(work_key(tile, digest)):        # bookkeeping you write
        return tile.path
    with concurrency("source:geodata.example.gov", occupy=1):
        render(tile)
    ledger_mark(work_key(tile, digest), tile)     # and maintain
    return tile.path


@flow(name="tile-build")
def tile_build(footprint: str, zoom: int, digest: str) -> int:
    tiles = plan(footprint, zoom)                 # the graph's width, decided now
    states = build_tile.map(tiles, digest=digest, return_state=True)
    return sum(s.is_completed() for s in states)

In Dagster the manifest is a partition set and the fan-out is the framework’s job:

import dagster as dg

TILES = dg.StaticPartitionsDefinition(
    [f"{z}_{x}_{y}" for z, x, y in tiles_intersecting(SERVICE_AREA_WKT, 12)]
)


@dg.asset(partitions_def=TILES, pool="scene-reads",
          retry_policy=dg.RetryPolicy(max_retries=2, delay=15))
def warped_tile(context: dg.AssetExecutionContext) -> dg.MaterializeResult:
    z, x, y = (int(v) for v in context.partition_key.split("_"))
    stats = render(Tile(z, x, y))                 # no ledger: the record is the point
    return dg.MaterializeResult(
        metadata={"valid_px_pct": dg.MetadataValue.float(stats.valid_fraction * 100)}
    )

Neither snippet is longer than the other, which is exactly the trap. The Prefect version’s cost is not in the lines shown but in ledger_has, ledger_mark, the table behind them, the migration that creates it, the query that reports coverage and the discipline that keeps all of it consistent when somebody changes the work key. That is perhaps two hundred lines and an ongoing obligation. It is entirely reasonable to take it on — plenty of teams have, and it gives you a ledger shaped exactly like your problem rather than like a framework’s idea of it.

The reverse also holds, and it is the argument people forget when Dagster wins the comparison on paper. A partition set is a commitment to a fixed vocabulary of units, declared in code and versioned with it. That is exactly right for a service area that changes twice a year and exactly wrong for a pipeline whose width comes from whatever a publisher delivered this morning. Expressing “the tiles intersecting today’s scene footprint” as partitions means dynamic partitions, a sensor that adds keys, and a growing set nobody prunes — which is more machinery than the Prefect version’s ledger, arrived at from the opposite direction. The tools are not better and worse; they are shaped for different questions, and a pipeline that asks the other one pays in the same currency either way.

There is a second, subtler difference in how the two treat failure. Prefect’s flow returns a value and its states are a property of execution: a task run is Completed or it is not, and the flow’s own state summarises its children. Dagster’s materialisation is a property of the world: an asset partition is materialised or it is not, and a run that failed after materialising three hundred partitions leaves three hundred true statements behind. When a pipeline is asked “what do we have”, the second framing answers directly and the first requires a translation step — which is, once more, the ledger.

Step-by-Step Walkthrough#

1. Count the units and ask whether the set is fixed. A few thousand fixed cells points at partitions; a width computed per run from a footprint points at mapping.
2. Ask who recovers a partial failure. If it is somebody who will not read code, the selectable gap in a partition view is worth a lot.
3. Ask what questions get asked of the past. “Which tiles are stale in this district” needs a queryable record, whether the framework supplies it or you do.
4. Estimate the operational budget. A code location, a daemon and a database is a real commitment for a two-person team.
5. Check the team’s existing fluency. A team that already runs one tool well should need a specific reason to run the other.
6. Prototype the awkward part, not the easy part. Both tools do the happy path well; build the partial-recovery story in each before deciding.

Step six deserves expanding, because it is the step teams skip and the one that decides correctly. Both tools make a happy-path tile build look identical in a demo — a manifest, a fan-out, a mosaic, a green run — and no useful information comes out of building it twice. What separates them is the awkward part: kill the run half way, then find out what you have and rebuild only the gap. In Dagster that is a selection in a partition view and a materialise; in Prefect it is a table you designed, a query you wrote and a filtered manifest you passed back in. Both work. Spending an afternoon building exactly that in each is the only prototype worth doing, and it answers the question the demo cannot.

A related test is worth running on the observability side. Ask each prototype “which tiles in this district are more than seven days old” and see how far the answer is from a query you can hand to somebody else. That single question exercises the record, the spatial index and the operator interface at once, and its answer tends to predict how the pipeline will feel in a year better than any feature comparison.

The other reason to keep that boundary clean is testing. Domain functions that take a footprint and return a manifest, or take a tile and return a raster, can be tested without any orchestrator at all — no test harness, no ephemeral database, no fixtures for a control plane. What remains for the orchestrator’s own tests is the graph’s shape and its guards, which is a much smaller surface. Pipelines that blur the line end up with a test suite that needs a running control plane to assert a reprojection, and that suite is slow enough that people stop running it.

Edge Cases & Failure Recovery#

The manifest is huge. Dagster’s static partitions become unwieldy past a few thousand keys, so partition at a coarser grain and iterate inside. Prefect maps happily over tens of thousands, bounded by a tag limit — this is the case that most favours Prefect.

The unit set changes. A footprint gains a municipality. In Prefect nothing happens, because the manifest is recomputed each run. In Dagster the partition set changes shape, historical coverage figures shift meaning, and the change is visible in a diff — which is either useful or annoying depending on how often it happens.

A single tile must be re-run by hand. Dagster: select the partition and materialise it. Prefect: call the task with the index, which works, and requires knowing the index and having a safe path to do it.

The pipeline spans several products. Dagster’s asset graph across layers is genuinely good at this, showing which downstream products a re-materialised source invalidates. Prefect models it as flows calling flows, which is workable and less visible.

Everything must run in one process for a test. Both support it. Prefect’s test harness is lighter; Dagster’s materialize is explicit about partitions, which makes the test more honest about what it covers.

A team of one. Worth calling out because the comparison usually assumes a team. A single maintainer running a spatial pipeline part-time gets more from Prefect’s smaller surface than from Dagster’s record-keeping, right up until they are on holiday and somebody else has to recover a run. That is the moment the asset view earns its keep, and it is not a moment anyone plans for.

The pipeline outlives the person who wrote it. This is the strongest argument for the heavier tool and the hardest to make in a design review, because it is a claim about a future nobody can evidence. A partition grid is legible to somebody encountering it for the first time; a hand-rolled ledger is legible only after reading the code that maintains it. Where a pipeline is expected to run for years and change hands, that difference compounds.

Configuration Reference#

Frequently Asked Questions#

Related#

• Prefect flow state transitions explained — what a Prefect run’s states actually mean
• Migrating from Prefect to Dagster for spatial pipelines — doing it without a flag day
• Airflow vs Prefect for spatial pipelines — the third option, and when it wins
• Benchmarking orchestrator overhead for small geotasks — the numbers behind the granularity rule
• Mapping tasks over a tile grid in Dagster — the partitioned form in full
• State management in geospatial flows — the ledger the Prefect column asks you to own

← Geospatial Orchestration Architecture Fundamentals