Choosing Tile Sizes for Raster Partitioning

Three tile sizes appear in a raster pipeline and they are routinely confused: the internal block a file is written in, the processing tile a task works on, and the delivered tile a client requests. They answer different questions — storage layout, memory and duration, and client experience — and setting them equal is a coincidence rather than a design. Choosing each from its own constraint takes an hour and removes most of the surprises about memory, throughput and egress that follow.

When to Use This Pattern#

• A new raster product is being designed and no size has been chosen yet.
• Tasks are running out of memory or taking wildly different times.
• Egress is higher than the output size suggests, which usually means the read granularity is wrong.
• A processing tile has been inherited from a tutorial and nobody has checked whether it fits.

Complete Working Example#

Compute each size from its own constraint, then verify by measurement.

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class SizePlan:
    internal_block: int      # how the file is written
    processing_tile: int     # what one task handles
    delivered_tile: int      # what a client asks for


def plan_sizes(bands: int, bytes_per_px: int, worker_mem_gb: float,
               concurrency: int, target_seconds: float,
               px_per_second: float, headroom: float = 1.3) -> SizePlan:
    # 1. Memory bound. A warp holds source, destination and one working copy.
    per_task_bytes = (worker_mem_gb * 1024 ** 3) / (concurrency * headroom)
    max_px = per_task_bytes / (3 * bands * bytes_per_px)
    mem_side = int(max_px ** 0.5)

    # 2. Duration bound: aim for a task of a few minutes.
    time_side = int((target_seconds * px_per_second) ** 0.5)

    processing = _round_to_power_of_two(min(mem_side, time_side))
    return SizePlan(
        internal_block=512,          # request-count vs waste, near-universal answer
        processing_tile=processing,  # the binding constraint of the two above
        delivered_tile=256,          # what map clients expect; not a processing choice
    )


def _round_to_power_of_two(n: int) -> int:
    p = 256
    while p * 2 <= n:
        p *= 2
    return p


plan = plan_sizes(bands=4, bytes_per_px=2, worker_mem_gb=8, concurrency=4,
                  target_seconds=300, px_per_second=1_800_000)
# SizePlan(internal_block=512, processing_tile=4096, delivered_tile=256)

The processing tile is then used consistently, and the delivered tile is a separate step:

@task(retries=2, timeout_seconds=900, tags=["tile"])
def process(window: Window, src_uri: str, dest: Path, plan: SizePlan) -> Path:
    with rasterio.open(src_uri) as src:
        data = src.read(window=window, boundless=True)
    warped = warp(data, window)
    with rasterio.open(
        dest, "w", driver="GTiff", tiled=True,
        blockxsize=plan.internal_block, blockysize=plan.internal_block,
        compress="deflate", predictor=2, **profile_from(warped),
    ) as dst:
        dst.write(warped)
    return dest


@task(timeout_seconds=1800)
def cut_delivery_tiles(mosaic: Path, plan: SizePlan, out: Path) -> int:
    """Delivered tiles come from the mosaic, not from the processing grid."""
    return cut(mosaic, size=plan.delivered_tile, out=out)

The min(mem_side, time_side) in the planner encodes something worth stating: the processing tile is bounded by two constraints and only one of them binds, and which one binds tells you what to change. If memory is the binding constraint, the fix is a larger worker or lower concurrency; if duration binds, the fix is accepting shorter tasks or optimising the operation. Computing both and taking the smaller makes that visible instead of leaving it as a number somebody picked.

Parameter & Option Reference#

Verification & Testing#

def test_processing_tile_fits_in_memory(plan) -> None:
    px = plan.processing_tile ** 2
    est_mb = 3 * px * BANDS * BYTES_PER_PX / 1e6
    assert est_mb * CONCURRENCY < WORKER_MB * 0.75, (
        f"{est_mb:.0f} MB per task x {CONCURRENCY} exceeds the worker"
    )


def test_typical_task_lands_in_the_target_band(sample_windows) -> None:
    times = [measure(process.fn, w) for w in sample_windows[:20]]
    p50 = statistics.median(times)
    assert 60 < p50 < 900, f"median task is {p50:.0f}s — outside the 1-15 min band"


def test_output_is_written_with_the_internal_block(tmp_path, plan) -> None:
    out = process.fn(WINDOW, SRC, tmp_path / "t.tif", plan)
    with rasterio.open(out) as ds:
        assert ds.profile["tiled"]
        assert ds.block_shapes[0] == (plan.internal_block, plan.internal_block)


def test_delivered_tile_is_independent_of_processing(plan) -> None:
    assert plan.delivered_tile == 256, "clients expect 256; processing size must not leak"


def test_edge_tiles_take_similar_time(sample_windows) -> None:
    interior = [measure(process.fn, w) for w in sample_windows if not w.on_edge]
    edges = [measure(process.fn, w) for w in sample_windows if w.on_edge]
    assert max(edges) < statistics.median(interior) * 3, "edge handling is not boundless"

The edge-tile test catches a specific and annoying asymmetry. Without boundless reads, a tile at the raster’s edge covers a partly-empty window and the code either special-cases it or reads a smaller array — either of which makes edge tiles behave differently from interior ones in memory, timing and output shape. Boundless reads make every task identical regardless of position, which removes a class of “only the coastal tiles fail” bug that is otherwise very tedious to track down.

The three sizes also have different revision cadences, which is a practical reason to keep them separate in code as the SizePlan does. The delivered tile is effectively frozen once clients exist, because changing it invalidates every cached tile in every browser and every proxy between you and them. The internal block changes only when the storage or access pattern changes, which is rare. The processing tile, by contrast, is expected to move: a new worker class, a change in concurrency, a source that gains bands. Holding all three in one constant means the cheap change drags the expensive ones with it, which is usually enough to stop anyone making the cheap change at all.

One more consideration applies when the operation needs a buffer — a focal filter, a hillshade, anything reading neighbouring pixels. The buffer is added to the processing tile on every side, so a 512-pixel tile with a 32-pixel buffer processes 576 pixels square and discards the frame, wasting about a quarter of the work. The same buffer on a 4096-pixel tile wastes three per cent. Where a buffer is required, it is a strong argument for the larger end of the workable band, and it is worth including in the memory arithmetic rather than discovering as an unexpected overshoot.

Common Pitfalls#

• One size for all three roles. The constraints are unrelated; equality is a coincidence and usually a costly one.
• Choosing the processing tile from the delivered tile. 256-pixel tasks last seconds and the run becomes orchestration.
• Non-power-of-two sizes. They stop aligning with overview levels, so a coarse read spans partial blocks.
• Ignoring band count and data type. A four-band 16-bit tile is eight times the memory of a single-band 8-bit one at the same dimensions.
• No boundless reads. Edge tiles then behave differently from interior ones in shape, memory and time.
• Skipping overviews on intermediates. Any coarse read of them becomes a full-resolution read.

Frequently Asked Questions#

Related#

• Partitioning strategies for spatial workloads — where this size becomes the partition
• Right-sizing workers for raster mosaic jobs — the memory side of the same arithmetic
• Cutting egress costs with COG range reads — why the internal block matters
• Handling skewed partitions in spatial workloads — when one tile is much heavier than the rest
• How to structure a DAG for raster processing — the stage this size applies to

← Partitioning Strategies for Spatial Workloads