Dynamic Task Mapping for Tile Fan-Out

In short: compute the work list as a durable manifest first, map tasks over it with an explicit concurrency limit, allow partial failure, and fan back in through file references rather than through returned data. The number of tiles is not known until the source is inspected, so the graph has to be built at runtime — and everything that makes runtime graphs painful comes from skipping one of those four steps.

Fan-out is the natural shape of raster work. One scene becomes four hundred tiles; one national delivery becomes three hundred and fifty-six municipalities. Both orchestrators support this directly, and both will happily let you launch four thousand concurrent tasks that exhaust the worker pool, saturate an upstream service and return a result set too large to hold in memory. The discipline below is mostly about bounding things that are unbounded by default.

It is worth naming what makes this different from a generic map-reduce, because the differences are what the principles below are protecting. The item count is derived from geometry rather than from a row count, so it can be wrong by three orders of magnitude if a footprint or a zoom is wrong. The items differ enormously in cost, because a tile over a city is not a tile over the sea. The outputs are large binaries rather than small values, so the reduce step’s shape matters as much as the map step’s. And the upstream is usually a shared service with a real capacity limit rather than a database you own. Every one of those pushes in the same direction: bound things, make the work list inspectable, and never move pixels through the orchestrator.

Prerequisites & Architecture Baseline#

• Prefect ≥ 2.14 (.map()) or Dagster ≥ 1.5 (DynamicOut), both of which build the fan-out at runtime
• A concurrency limit configured at the orchestrator, per work pool or per tag — not only in the code
• Somewhere durable for the manifest: object storage or a table, not the flow's memory
• A per-item idempotency key, so a re-run of a partially-failed fan-out skips what succeeded
• Metrics per mapped item, aggregated — a fan-out of four hundred produces four hundred of everything
• A decision about what a partial failure means for the fan-in step

Core Principles#

1. The manifest is an artefact, not a variable. Write the list of tiles to storage before mapping over it. A manifest that exists only as a Python list disappears when the flow run is deleted, which makes “which tiles were in last Tuesday’s run?” unanswerable and makes a resumed run guess. It also makes the fan-out reviewable: a manifest with four hundred thousand entries is a bug you can see before it launches.

2. Concurrency is bounded twice, deliberately. The orchestrator’s limit bounds how many task runs exist; a semaphore or work-pool limit bounds how many touch a given resource. Both are needed, because a fan-out of four hundred tiles across sixteen workers still makes four hundred requests to the same upstream service unless something says otherwise.

3. Partial failure is the normal outcome. Of four hundred tiles, some will fail — that is what the tail of the cost distribution does. A fan-out that fails the whole flow when one item fails converts a 99.75% success into a zero, and re-running it repeats 399 successful tiles to retry one.

4. Fan back in by reference. The reduce step should receive paths, not arrays. Four hundred returned rasters is tens of gigabytes moving through the orchestrator’s result store; four hundred returned paths is a few kilobytes, and the merge reads them one at a time.

5. Size the fan-out to the work, not to the grid. Mapping over every tile in a bounding box includes the ocean. Filter the manifest against the source’s actual footprint before mapping, and a coastal scene’s fan-out halves — the same arithmetic used by skipping tiles with no new source data.

6. One mapped item should be worth a task. Orchestrators charge real overhead per task run — state transitions, logging, result persistence — typically tens to hundreds of milliseconds. Mapping over four hundred thousand items each taking 50 ms means the orchestration costs more than the work. Batch small items into chunks and map over the chunks.

Two of those principles interact in a way worth spelling out. Bounding concurrency and allowing partial failure sound independent, but together they decide how a fan-out degrades. With a tight limit and no partial failure, one bad tile cancels 411 in-flight or queued siblings, and the run’s cost is proportional to how far it got before failing. With a loose limit and partial failure allowed, the run completes but may have hammered an upstream service for twenty minutes. The combination you want — bounded, and tolerant — produces the behaviour people expect from a fan-out and almost never get by default.

Production Implementation#

The flow below inspects, writes a manifest, maps with a limit, and reduces through a VRT.

from __future__ import annotations

import json
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Optional, Sequence

from prefect import flow, task, get_run_logger
from prefect.concurrency.sync import concurrency


@dataclass(frozen=True)
class TileItem:
    z: int
    x: int
    y: int
    source_uri: str
    work_key: str


@task
def build_manifest(source_uri: str, zoom: int, manifest_path: Path) -> list[TileItem]:
    """Enumerate the tiles this scene actually covers, and persist the list."""
    footprint = footprint_of(source_uri)                 # header read, no pixels
    items = [
        TileItem(zoom, t.x, t.y, source_uri, work_key=tile_key(source_uri, zoom, t.x, t.y))
        for t in tiles_intersecting(footprint, zoom)     # not the bounding box: the footprint
    ]
    manifest_path.write_text(json.dumps([asdict(i) for i in items], indent=0))
    get_run_logger().info("manifest: %d tiles for %s", len(items), source_uri)
    if len(items) > 20_000:
        # A guard, not a limit: this many tiles from one scene means the zoom or
        # the footprint is wrong, and launching them would take hours to unwind.
        raise ValueError(f"implausible fan-out of {len(items)} tiles — check the zoom")
    return items


@task(retries=2, retry_delay_seconds=[20, 90])
def render_tile(item: TileItem, out_dir: Path) -> Optional[str]:
    """One tile. Returns a PATH, never an array."""
    if already_done(item.work_key):
        return str(out_dir / f"{item.z}_{item.x}_{item.y}.tif")
    # A named concurrency limit bounds requests to the SOURCE, independently of
    # how many task runs the orchestrator is willing to have in flight.
    with concurrency("scene-reads", occupy=1):
        dest = out_dir / f"{item.z}_{item.x}_{item.y}.tif"
        warp_window(item.source_uri, dest, item.z, item.x, item.y)
    mark_done(item.work_key)
    return str(dest)


@task
def reduce_to_mosaic(tile_paths: Sequence[Optional[str]], dest: Path) -> Path:
    """Fan-in by reference: a VRT over the paths, then one sequential translate."""
    ok = [p for p in tile_paths if p]
    missing = len(tile_paths) - len(ok)
    if missing:
        get_run_logger().warning("%d tile(s) missing from the mosaic", missing)
    run_gdal(["gdalbuildvrt", f"{dest}.vrt", *ok])
    run_gdal(["gdal_translate", "-co", "COMPRESS=DEFLATE", "-co", "TILED=YES",
              f"{dest}.vrt", str(dest)])
    return dest


@flow(name="tile-fanout")
def tile_scene(source_uri: str, zoom: int, scratch: Path) -> Path:
    items = build_manifest(source_uri, zoom, scratch / "manifest.json")
    # allow_failure means one bad tile does not cancel the other 411.
    results = render_tile.map(items, out_dir=scratch, return_state=True)
    paths = [r.result(raise_on_failure=False) if r.is_completed() else None for r in results]
    return reduce_to_mosaic(paths, scratch / "mosaic.tif")

Step-by-Step Walkthrough#

1. Inspect before enumerating. The footprint comes from the source’s header, and the tile list comes from the footprint — not from the bounding box. For a coastal or diagonal scene this alone can halve the fan-out.
2. Persist the manifest before mapping. It is the record of what the run intended to do, and it is what a resumed or re-driven run reads instead of recomputing. It also makes the count visible in the log before four hundred tasks appear.
3. Guard against an implausible fan-out. A wrong zoom level turns four hundred tiles into four hundred thousand. One comparison stops that before the orchestrator has to be told to cancel a run it has already started scheduling.
4. Give each item its own idempotency key. A re-run after a partial failure should skip completed tiles. Without the key, the second attempt repeats everything and the fan-out’s cost is proportional to how many times it has failed.
5. Bound concurrency at the resource, not only at the pool. concurrency("scene-reads") limits how many tasks read the source simultaneously, which is what the upstream service cares about. The work pool’s limit is about worker capacity, and the two numbers are rarely the same.
6. Collect states, not results. Asking for return_state=True lets the flow inspect each mapped run and treat failures as missing rather than fatal. This is where partial failure is either allowed or accidentally forbidden.
7. Reduce through a VRT. Building a virtual raster over the successful paths and translating once keeps the merge’s memory flat regardless of how many tiles there are.

Edge Cases & Failure Recovery#

A fan-out larger than the orchestrator can schedule. Prefect and Dagster both slow noticeably in the tens of thousands of mapped runs — state writes, UI queries and result persistence all scale with the count. Above a few thousand items, map over batches of tiles rather than tiles: one hundred tasks each handling forty tiles has the same parallelism and a hundredth of the bookkeeping.

Result storage filling with tile paths. Every mapped task persists its return value. Returning a path is a few dozen bytes; returning a numpy array is megabytes, times four hundred, per run, retained for the result store’s whole retention period. This is the single most common way a fan-out makes an orchestrator unhappy.

Partial failure that reaches the fan-in silently. A mosaic built from 398 of 412 tiles has holes and no error. Count what arrived against what the manifest listed, and decide explicitly: fail, warn, or publish with a coverage figure attached. Never let the reduce step silently accept fewer inputs than expected.

Retries multiplying the fan-out. Four hundred tiles with two retries each is up to twelve hundred task runs against the same upstream. The concurrency limit protects the service, but the budget — total wall-clock time — is what protects the run, and it has to account for the retry multiplier as described in timeout budgets and cancellation.

A manifest that goes stale between runs. Re-driving yesterday’s manifest against a republished source produces tiles from new pixels under yesterday’s keys. Store the source digest in the manifest and verify it before re-driving, so a changed source forces a fresh manifest rather than a silent mixture.

Scratch storage sized for one tile rather than for the fan-out. Sixteen concurrent tiles each writing a 300 MB intermediate need five gigabytes of scratch, and the failure arrives as a disk-full error in whichever tile happened to be writing when the last byte went — which is never the tile that caused it. Size scratch from concurrency times the largest expected intermediate, and clean up per item rather than at the end of the flow.

Uneven item cost starving the fan-out’s tail. Tile cost varies by an order of magnitude, so the last few tasks can take as long as the first three hundred. Ordering the manifest most-expensive-first — estimated from the source’s data density — keeps the workers busy to the end instead of trickling.

Configuration Reference#

The batch threshold is the setting most often discovered too late. A pipeline that maps over ten thousand small items works fine in testing at a hundred and then, at full scale, spends most of its wall-clock time in the orchestrator rather than in GDAL — visible as a run where the sum of task durations is a fraction of the run duration. That ratio is worth putting on a dashboard: when total task time drops below about half of run time, the fan-out has become bookkeeping, and batching is the fix.

Frequently Asked Questions#

Related#

• Fanning out Prefect tasks over a tile manifest — the map step in detail
• Collecting mapped results into a single mosaic — the fan-in step in detail
• Conditional branching in geospatial DAGs — deciding whether to fan out at all
• Limiting DAG fan-out with concurrency groups — the limits this relies on
• Idempotency keys in spatial ETL — what makes a re-run cheap

← Spatial Task Design & Dependency Mapping