Checkpointing Large Raster Mosaics

To make a multi-hour raster-mosaic build resumable, persist a manifest of completed windows keyed by content hash, write each window to a temporary GeoTIFF and atomically rename it into place, then verify the output with gdalinfo before recording it as done. Checkpointing large raster mosaics means a crash at window 8,000 of 10,000 resumes from window 8,001 instead of discarding hours of finished compute. This guide gives a working manifest-driven checkpoint loop, the atomic-write contract that keeps the mosaic consistent, and the verification step that prevents a half-written tile from being marked complete. It extends the State Management in Geospatial Flows section from tabular state into the raster domain, where a “record” is a pixel window rather than a row.

When to Use This Pattern#

Adopt manifest checkpointing when your mosaic build has any of these properties:

• The full build takes long enough that a mid-run failure is expensive — tens of minutes to many hours of resampling, warping, or band math.
• Output is assembled window by window (block-aligned reads/writes) rather than in a single in-memory pass.
• Inputs may change between runs, so you need to detect which windows are genuinely unchanged versus which must be recomputed.
• The process runs on preemptible or spot infrastructure where the worker can vanish at any moment.
• You want an auditable record of exactly which windows completed, for backfills and partial reruns.

If the entire mosaic fits comfortably in memory and rebuilds in seconds, checkpointing is overhead you do not need — just recompute. The pattern earns its keep once a restart-from-zero costs more than the bookkeeping.

There is a design choice hidden in the word “content” of content hash. The cheap version hashes input identity — file path, size, and modification time — which is nearly free and correctly detects a replaced or edited source. The expensive version hashes the actual pixel bytes of the input window, which survives a touch that changes mtime without changing data but forces a full read of every input on every run, defeating the point of skipping work. Most mosaic builds use the cheap identity hash and only fall back to byte hashing on inputs whose mtime is known to be unreliable, such as objects re-uploaded to storage that resets timestamps. The example below takes the identity approach so the skip check stays O(number of inputs) rather than O(pixels).

Complete Working Example#

The build walks a fixed grid of pixel windows over the destination mosaic. Before processing a window it consults a JSON manifest; a window is skipped only if its recorded content hash matches the hash of its current inputs. Each window is written to a temporary GeoTIFF and renamed atomically, then verified before the manifest is updated and flushed to disk.

import hashlib
import json
from dataclasses import dataclass, asdict
from pathlib import Path

import numpy as np
import rasterio
from rasterio.windows import Window
from pyproj import CRS


@dataclass(frozen=True)
class WindowKey:
    col_off: int
    row_off: int
    width: int
    height: int


def window_content_hash(src_uris: list[str], key: WindowKey) -> str:
    """Hash the input identity + window geometry so unchanged work is skippable."""
    h = hashlib.sha256()
    for uri in sorted(src_uris):
        stat = Path(uri).stat()
        h.update(uri.encode())
        h.update(str(stat.st_size).encode())
        h.update(str(int(stat.st_mtime)).encode())
    h.update(json.dumps(asdict(key), sort_keys=True).encode())
    return h.hexdigest()


class Manifest:
    """Persisted record of completed windows keyed by content hash."""

    def __init__(self, path: str) -> None:
        self.path = Path(path)
        self.done: dict[str, str] = {}
        if self.path.exists():
            self.done = json.loads(self.path.read_text())

    def is_done(self, key: WindowKey, content_hash: str) -> bool:
        return self.done.get(self._k(key)) == content_hash

    def mark(self, key: WindowKey, content_hash: str) -> None:
        self.done[self._k(key)] = content_hash
        tmp = self.path.with_suffix(".json.part")
        tmp.write_text(json.dumps(self.done, indent=2))
        tmp.replace(self.path)  # atomic manifest flush

    @staticmethod
    def _k(key: WindowKey) -> str:
        return f"{key.col_off}_{key.row_off}_{key.width}_{key.height}"


def build_mosaic(
    src_uris: list[str], dst_path: str, manifest_path: str,
    block: int = 1024, nodata: float = 0.0, dst_crs: str = "EPSG:3857",
) -> int:
    manifest = Manifest(manifest_path)
    processed = 0

    with rasterio.open(src_uris[0]) as ref:
        profile = ref.profile.copy()
        profile.update(
            driver="GTiff", crs=CRS.from_user_input(dst_crs), nodata=nodata,
            tiled=True, blockxsize=block, blockysize=block, compress="deflate",
        )
        full_w, full_h, count = ref.width, ref.height, ref.count

    dst_exists = Path(dst_path).exists()
    mode = "r+" if dst_exists else "w"
    with rasterio.open(dst_path, mode, **(profile if not dst_exists else {})) as dst:
        for row_off in range(0, full_h, block):
            for col_off in range(0, full_w, block):
                key = WindowKey(
                    col_off, row_off,
                    min(block, full_w - col_off), min(block, full_h - row_off),
                )
                chash = window_content_hash(src_uris, key)
                if manifest.is_done(key, chash):
                    continue  # already built from identical inputs

                data = _compose_window(src_uris, key, count, nodata)
                win = Window(key.col_off, key.row_off, key.width, key.height)
                dst.write(data, window=win)
                dst.flush()  # push this window to the file before recording it
                manifest.mark(key, chash)
                processed += 1
    return processed


def _compose_window(
    src_uris: list[str], key: WindowKey, count: int, nodata: float
) -> np.ndarray:
    """Read + reduce the window across all inputs (first-valid-wins here)."""
    win = Window(key.col_off, key.row_off, key.width, key.height)
    out = np.full((count, key.height, key.width), nodata, dtype="float32")
    for uri in src_uris:
        with rasterio.open(uri) as src:
            chunk = src.read(window=win, boundless=True, fill_value=nodata)
        mask = (out == nodata) & (chunk != nodata)
        out[mask] = chunk[mask]
    return out

The single-file r+ approach shown here is convenient because the mosaic is one addressable GeoTIFF, but it has a sharp edge: only one writer may hold the file open, so it does not parallelize across workers. For very large mosaics, write each window to its own temporary GeoTIFF and rename it into a tile directory rather than seeking within one file — the atomic-rename discipline is identical, many workers can write different tiles at once, and a final gdalbuildvrt or COG assembly step stitches them into the deliverable. This tile-per-file layout composes cleanly with the fan-out from parametrizing spatial DAGs by tile index, where the manifest simply records completed tiles instead of pixel windows.

Manifest growth is worth a moment of planning. A mosaic with a hundred thousand windows produces a manifest with a hundred thousand keys, which is still a few megabytes of JSON and loads in milliseconds — acceptable. Beyond that scale, switch the manifest store from a JSON file to a SQLite table or a small key-value store so lookups stay indexed and concurrent writers do not race on a single file. The interface stays the same: is_done(key, hash) and mark(key, hash). What must never change is the ordering guarantee — the manifest entry is written strictly after the pixel data is durable — because that ordering is the entire correctness argument for the resume. Wrapping the loop in a Prefect flow or Dagster job adds retries around each window, but the manifest — not the orchestrator’s task state — remains the source of truth for what is done, so a resume works even after a full server restart. That separation is why the pattern pairs naturally with the framework-level state discussed in Prefect vs Dagster for GIS Workloads.

Parameter & Option Reference#

Verification & Testing#

Never mark a window complete until the data on disk is provably valid. After the run, inspect the mosaic with the GDAL CLI to confirm dimensions, CRS, and nodata survived the write:

gdalinfo -stats build/mosaic.tif | grep -E "Size is|EPSG|NoData|Block"
# Size is 40960, 30720
# ... ID["EPSG",3857]]
# NoData Value=0
# Block=1024x1024

Simulate a crash to prove the resume works: run the build, kill it partway, and confirm the second invocation only processes the remaining windows. Because completed windows carry a content hash, a clean rerun does zero work:

def test_resume_skips_completed_windows(tmp_path):
    srcs = ["tests/data/a.tif", "tests/data/b.tif"]
    dst = str(tmp_path / "mosaic.tif")
    man = str(tmp_path / "manifest.json")

    first = build_mosaic(srcs, dst, man, block=512)
    assert first > 0                       # real work on the first pass
    second = build_mosaic(srcs, dst, man, block=512)
    assert second == 0                     # every window already recorded

    manifest = Manifest(man)
    assert len(manifest.done) == first     # manifest accounts for all windows

The content-hash comparison is exactly the idempotency guarantee described in generating idempotent keys for shapefile uploads — a window is redone only when its inputs actually change.

Common Pitfalls#

• Recording a window before it is durable. Calling manifest.mark() before dst.flush() (or before the atomic rename in the tile-per-file variant) lets a crash leave a window marked done but never written. Flush the pixels first, verify, then record.
• Hashing pixel values instead of input identity. Reading and hashing the full window’s data on every run defeats the purpose — the whole point is to skip reads. Hash file size, mtime, and window geometry so the check is cheap; fall back to full-content hashing only when mtime is unreliable.
• Non-atomic manifest writes. Writing the manifest in place means a crash mid-write corrupts the checkpoint and forces a full rebuild. Always write to a .part file and rename, as the Manifest.mark method does.
• Ignoring nodata on boundless reads. Edge windows extend past a source’s extent; without boundless=True and an explicit fill_value, rasterio raises or silently pads with zeros, seaming the mosaic. Match fill_value to the destination nodata.

Related: Anchor the work in State Management in Geospatial Flows, fan the build out with parametrizing spatial DAGs by tile index, reuse the same checkpoint idea for vectors in managing state for incremental shapefile updates, and skip redundant reprojections with caching reprojected rasters with content hashing.

← Back to State Management in Geospatial Flows