Mapping Tasks Over a Tile Grid in Dagster

Dagster’s answer to a tile fan-out is not a loop but a partition set: declare the grid as partitions, write one asset that materialises a single tile, and let the framework keep the record of which tiles exist. That inversion is the whole point — a Prefect-style map produces N task runs and forgets them, whereas a partitioned asset produces N materialisations Dagster can query, so “which tiles are missing for last Tuesday” becomes a question the system answers rather than one you write a script for.

When to Use This Pattern#

• Output is a grid whose cells are addressable — a tile pyramid, a map-sheet series, a fixed set of municipalities.
• Backfills are routine, and re-running only the gaps matters more than raw throughput.
• Coverage is a question people ask, so the record of what exists needs to outlive the run that produced it.
• Tiles and dates are independent axes, which is exactly what a multi-dimensional partition set models.

Complete Working Example#

The grid is a static partition dimension, the schedule is a daily one, and the asset materialises the intersection of the two.

from __future__ import annotations

import dagster as dg

# The spatial axis. Enumerated once, from the footprint, so the partition set is
# the manifest — there is no second list to keep in step with it.
TILES = dg.StaticPartitionsDefinition(
    [f"{z}_{x}_{y}" for z, x, y in tiles_intersecting(SERVICE_AREA_WKT, zoom=12)]
)

GRID = dg.MultiPartitionsDefinition(
    {"date": dg.DailyPartitionsDefinition(start_date="2026-01-01"), "tile": TILES}
)


@dg.asset(
    partitions_def=GRID,
    # One tile at a time per source; the pool is what protects the upstream server.
    pool="scene-reads",
    retry_policy=dg.RetryPolicy(max_retries=2, delay=15, backoff=dg.Backoff.EXPONENTIAL),
    metadata={"unit": "tile"},
)
def warped_tile(context: dg.AssetExecutionContext) -> dg.MaterializeResult:
    keys = context.partition_key.keys_by_dimension
    z, x, y = (int(v) for v in keys["tile"].split("_"))
    dest = tile_path(keys["date"], z, x, y)

    stats = warp_window(z, x, y, on=keys["date"], dest=dest)
    context.log.info("tile %d/%d/%d built in %.1f s", z, x, y, stats.seconds)

    return dg.MaterializeResult(
        metadata={
            "path": dg.MetadataValue.path(str(dest)),
            "valid_px_pct": dg.MetadataValue.float(stats.valid_fraction * 100),
            "seconds": dg.MetadataValue.float(stats.seconds),
        }
    )


@dg.asset(partitions_def=GRID.get_partitions_def_for_dimension("date"), deps=[warped_tile])
def daily_mosaic(context: dg.AssetExecutionContext) -> dg.MaterializeResult:
    """One node depending on every tile of that date — the reduction."""
    present = list_built_tiles(context.partition_key)
    coverage = len(present) / len(TILES.get_partition_keys())
    if coverage < 0.995:
        raise dg.Failure(f"coverage {coverage:.3f} below threshold — not assembling")
    build_vrt_and_translate(present, mosaic_path(context.partition_key))
    return dg.MaterializeResult(metadata={"coverage": dg.MetadataValue.float(coverage)})

The metadata returned from each materialisation is worth attaching even when nothing reads it today. Dagster stores it against the partition, so valid_px_pct becomes a per-tile time series you can plot without instrumenting anything — a tile whose valid-pixel fraction drops from 0.98 to 0.4 on one date has a source problem, and that is visible in the asset catalogue rather than requiring a metric, a dashboard and an alert to have been set up in advance.

Parameter & Option Reference#

Verification & Testing#

Partition definitions are ordinary Python objects, so the grid can be tested without materialising anything.

import dagster as dg


def test_grid_covers_the_footprint() -> None:
    keys = set(TILES.get_partition_keys())
    expected = {f"12_{x}_{y}" for x, y in tiles_intersecting(SERVICE_AREA_WKT, 12)}
    assert keys == expected
    assert 500 < len(keys) < 5_000, "partition set is the wrong order of magnitude"


def test_one_tile_materialises(tmp_path) -> None:
    key = dg.MultiPartitionKey({"date": "2026-08-07", "tile": "12_2147_1398"})
    result = dg.materialize(
        [warped_tile], partition_key=key, resources={"scratch": tmp_path}
    )
    assert result.success
    meta = result.asset_materializations_for_node("warped_tile")[0].metadata
    assert meta["valid_px_pct"].value > 0


def test_reduction_refuses_an_incomplete_date(monkeypatch) -> None:
    monkeypatch.setattr("mymodule.list_built_tiles", lambda _: ["12_1_1"] * 10)
    with pytest.raises(dg.Failure, match="coverage"):
        dg.materialize([daily_mosaic], partition_key="2026-08-07")

The first test is the one that earns its place over time. A static partition set is computed from the footprint at import time, and footprints change — a service area gains a municipality, a coastline is re-cut. When that happens the partition set silently grows or shrinks, and every historical coverage figure changes meaning with it. Asserting the set against a freshly computed footprint makes the change fail in CI, where it can be reviewed, rather than appearing as an unexplained step in a coverage chart.

There is a second, quieter reason to prefer partitions for anything that runs on a schedule. A mapped fan-out records its history as runs, and a run is a poor unit to ask questions of: “did we build the coastal tiles on the fourth” turns into a search through logs, because the run knows it processed four hundred items but not which of them. A partitioned asset records its history as cells, so the same question is a lookup. Over a year of nightly builds that difference compounds into the whole difference between a pipeline whose past can be inspected and one whose past has to be reconstructed from artefacts in object storage — and reconstruction is exactly the work nobody has time for during an incident.

The cost side is worth stating plainly, because partitions are not free. Every materialisation writes an event to Dagster’s storage, so a grid of four thousand tiles times three hundred and sixty-five days is a million and a half rows a year in the event log. That is a small database by any standard, but it is not nothing, and it argues for partitioning at the deliverable rather than at the smallest addressable output — the same conclusion the catalogue-usability argument reaches from the other direction.

Common Pitfalls#

• Partitioning at the output zoom. A z14 national grid is millions of keys; partition at z10 and build the children inside one materialisation instead.
• Deriving the partition set from a hard-coded list. It drifts from the footprint, and the drift shows up as an unexplained coverage change months later.
• Using a run-level concurrency limit as the source protection. Runs and materialisations are different counters; the pool is the one that bounds simultaneous reads of an upstream service.
• Reducing with a plain dependency on the asset rather than the partitions. A date-partitioned reduction must actually wait on that date’s tiles, or it assembles whatever happens to exist.
• Returning no metadata. The materialisation record then says only “it happened”, which throws away the cheapest per-tile observability available.
• Treating a backfill as a re-run. Selecting the failed partitions is the whole advantage; a full backfill of a healthy grid is the mapped fan-out with extra steps.

Frequently Asked Questions#

Related#

• DAG design principles for spatial ETL — why the grid is the right unit
• Parametrizing spatial DAGs by tile index — the key each partition is named by
• Limiting DAG fan-out with concurrency groups — what the pool setting is doing
• Prefect vs Dagster for GIS workloads — choosing between the two models
• Collecting mapped results into a single mosaic — the reduction in detail

← DAG Design Principles for Spatial ETL