Geospatial Orchestration Hub

Parametrizing Spatial DAGs by Tile Index

A spatial pipeline needs one parameter that names a unit of work exactly, and for anything rendered on a map that parameter is the tile index. Pass z/x/y rather than a bounding box, derive every path and cache key from it, and the run becomes addressable: an operator can re-run one tile, a cache can decide whether the work is already done, and a failure names a place rather than a task number. The whole benefit comes from choosing a canonical form early and never accepting anything else.

When to Use This Pattern

  • Output is a tile pyramid or a gridded product, so the natural unit of work is already a tile.
  • Re-running one unit must be cheap, which requires the unit to have a stable name.
  • More than one flow touches the same tiles — a build flow and a validation flow, say — and they must agree on what a tile is called.
  • Failures need to be locatable on a map, not merely in a task list.

Complete Working Example

The pattern has three parts: a canonical tile type, a work key derived from it, and a manifest that is written before anything fans out.

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass, asdict
from pathlib import Path

from prefect import flow, task, get_run_logger

BAD = "tile index must be canonical: 0 <= z <= 22 and 0 <= x,y < 2**z"


@dataclass(frozen=True, slots=True)
class Tile:
    """The only representation of a tile any task accepts."""

    z: int
    x: int
    y: int

    def __post_init__(self) -> None:
        if not (0 <= self.z <= 22 and 0 <= self.x < 2 ** self.z and 0 <= self.y < 2 ** self.z):
            raise ValueError(f"{BAD}: got z{self.z}/{self.x}/{self.y}")

    @property
    def path(self) -> str:
        # z/x/y with no leading zeros: one spelling, so two flows cannot disagree.
        return f"{self.z}/{self.x}/{self.y}"

    def parent(self) -> "Tile":
        return Tile(self.z - 1, self.x >> 1, self.y >> 1)


def work_key(tile: Tile, source_digest: str, recipe_version: str) -> str:
    """What this tile's output depends on, and nothing else."""
    material = f"{tile.path}|{source_digest}|{recipe_version}"
    return hashlib.sha256(material.encode()).hexdigest()[:16]


@task(timeout_seconds=300)
def plan(footprint_wkt: str, zoom: int, manifest: Path) -> list[Tile]:
    tiles = [Tile(zoom, x, y) for x, y in tiles_intersecting(footprint_wkt, zoom)]
    if not tiles:
        raise ValueError("empty manifest — footprint and zoom disagree")
    if len(tiles) > 20_000:
        raise ValueError(f"{len(tiles)} tiles at z{zoom} is implausible")
    manifest.write_text(json.dumps([asdict(t) for t in tiles]))
    get_run_logger().info("manifest: %d tiles at z%d", len(tiles), zoom)
    return tiles


@task(retries=2, retry_delay_seconds=[15, 60], timeout_seconds=900, tags=["tile"])
def build(tile: Tile, source_digest: str, recipe_version: str, out: Path) -> str:
    key = work_key(tile, source_digest, recipe_version)
    dest = out / f"{tile.path}.tif"
    if ledger_has(key):                       # already built from these exact inputs
        return str(dest)
    dest.parent.mkdir(parents=True, exist_ok=True)
    render_tile(tile, dest)                   # the only place z/x/y becomes geometry
    ledger_mark(key, tile.path)
    return str(dest)


@flow(name="tile-build")
def tile_build(footprint_wkt: str, zoom: int, source_digest: str,
               out: Path, recipe_version: str = "v4") -> int:
    tiles = plan(footprint_wkt, zoom, out / "manifest.json")
    states = build.map(tiles, source_digest=source_digest,
                       recipe_version=recipe_version, out=out, return_state=True)
    return sum(s.is_completed() for s in states)
One parameter, three derived namesThe tile index z twelve, x two one four seven, y one three nine eight produces a work key, an output path and a log line. All three are derived, so none can disagree.Tile(12, 2147, 1398)the only parameterwork keysha256(12/2147/1398 | src | v4) = 9f2c41ab77de0155output pathout/12/2147/1398.tiflog linetile 12/2147/1398 built in 4.1 sNothing downstream invents a name. Every identifier is a function of the one parameter the task was given.
Where the cache key is computed from a bounding box instead, two callers spelling the same extent differently produce two keys and the cache quietly halves its hit rate.

The __post_init__ check is doing more work than it looks. A tile index arriving from a manifest, a message queue or a manual re-run is a pair of integers with no unit attached, and the commonest corruption is not a wrong number but a right number at the wrong zoom — x=2147 is valid at z12 and out of range at z10. Validating at construction means the failure lands in the caller that produced the bad index rather than three tasks later in a renderer that silently returns an empty raster for a tile outside the world.

Parameter & Option Reference

Parameter Type Spatial notes
z int, 0–22 Zoom. Governs the fan-out width more than anything else: each level quadruples the count.
x, y int, 0 … 2**z - 1 Column and row in the scheme’s own axis order. Fix the origin once; a flipped y is the classic TMS-versus-XYZ bug.
source_digest str The content hash of the inputs. Putting it in the key is what makes a re-run after a source change actually rebuild.
recipe_version str Bumped when the rendering rule changes. Without it a styling fix leaves every cached tile stale.
manifest Path Written before the fan-out, so the width is inspectable and the run is reproducible.
tags=["tile"] list The handle a concurrency limit is applied to; see limiting DAG fan-out with concurrency groups.
The zoom parameter is the fan-out widthFor a fixed footprint the tile count quadruples with each zoom level, from twelve at zoom nine to about three thousand at zoom fourteen. A typo in the zoom is a typo in the parallelism.tasksz9 · 12z10 · 48z11 · 190z12 · 760z13 · 3.0kz14 · 12kOne footprint, six zooms. The guard in the plan stage exists because a single wrong digit moves three bars to the right.
The manifest guard is a cheap check against a costly typo: at z14 the same footprint schedules a thousand times the work it does at z9.

Verification & Testing

Three tests cover the parts that actually break: canonicity, key stability, and the axis convention.

import pytest


def test_out_of_range_index_is_rejected() -> None:
    Tile(10, 1023, 1023)                       # the last valid tile at z10
    with pytest.raises(ValueError, match="canonical"):
        Tile(10, 1024, 0)


def test_key_changes_with_source_and_recipe() -> None:
    t = Tile(12, 2147, 1398)
    base = work_key(t, "srcA", "v4")
    assert work_key(t, "srcA", "v4") == base            # stable
    assert work_key(t, "srcB", "v4") != base            # new source rebuilds
    assert work_key(t, "srcA", "v5") != base            # new recipe rebuilds
    assert work_key(t.parent(), "srcA", "v4") != base   # zoom is part of identity


def test_y_axis_matches_the_scheme(tmp_path) -> None:
    # A tile in the northern hemisphere must have a northern centroid.
    bounds = tile_bounds_wgs84(Tile(6, 33, 20))
    assert bounds.centroid_lat > 0, "y axis is flipped (TMS vs XYZ)"

The third test is the one worth insisting on. A flipped y axis produces output that is structurally perfect — every tile exists, every file is a valid raster, coverage is a hundred per cent — and geographically mirrored about the equator. No count-based check can see it, no schema validation can see it, and it typically surfaces when somebody opens the layer over a basemap weeks later. One assertion on a known tile’s latitude costs nothing and closes the whole category.

The two y conventions, side by sideUnder XYZ the row index increases southward from zero at the top. Under TMS it increases northward from zero at the bottom. The same integer names different ground.XYZ — y grows southy = 0 northy = 1y = 2 southTMS — y grows northy = 2 northy = 1y = 0 southboth produce a complete pyramidevery file present, every raster valid,coverage 100%, and one of them ismirrored about the equator.only a latitude assertion catches itDeclare the convention in the Tile type's docstring and test one known tile. Everything downstream then inherits it.
The convention is not a preference; it is part of the tile index’s meaning, and a pipeline that never states it has two.

Common Pitfalls

  • Passing a bounding box instead of an index. Two spellings of the same extent produce two cache keys, so the cache appears to work and hits half as often as it should.
  • Leaving the axis convention implicit. XYZ and TMS differ only in the direction of y, and the resulting error is invisible to every structural check.
  • Omitting the source digest from the work key. The pipeline then treats “already built” as permanent and never rebuilds after an upstream correction.
  • Omitting the recipe version. A styling or resampling change ships and nothing rebuilds, because the tile index and the source are both unchanged.
  • Fanning out before writing the manifest. The width is then only observable after the tasks are scheduled, which is exactly too late.
  • Zero-padding the path in one place and not another. 12/02147/1398 and 12/2147/1398 are different strings and the same tile, which is enough to duplicate an entire pyramid in object storage.

Frequently Asked Questions

Should the tile index be a string or a structured type?

Structured, with the string as a derived property. A string is easy to pass and impossible to validate, so a typo travels to whichever task first turns it into geometry. A frozen dataclass validates once at construction, hashes cleanly for use as a dictionary key, and still gives you the string wherever a path is wanted.

What about products that are not tiled?

Use whatever the data already partitions on — a scene identifier, a municipality code, a map sheet number — and apply the same three rules: one canonical spelling, a work key derived from it plus the inputs, and a manifest written before the fan-out. The tile index is the commonest instance of the pattern, not the pattern itself. DAG design principles for spatial ETL covers choosing that unit.

Does the parent tile belong in the key?

No. Deriving a parent is useful for building overviews, but including it in the key would make a tile’s identity depend on a tile it does not read. Overview levels are their own units of work with their own keys, and their inputs are the four children they actually consume.

How do I re-run a single tile?

Call the build task with the index directly, with the same source_digest and recipe_version the flow would have used. Because the key is a pure function of those three values, the re-run either finds the ledger entry and returns immediately or rebuilds exactly what the flow would have built — there is no third outcome, which is the property that makes manual intervention safe.

Is the ledger not just a cache?

It records what has been built rather than storing the output, so it survives a cache eviction and can be queried for coverage. The distinction matters when a source is corrected: a cache is invalidated by key, a ledger tells you which tiles were built from the bad input and therefore which ones to invalidate. Caching strategies for spatial tasks treats the storage side.

← DAG Design Principles for Spatial ETL

Related in this section