Checkpointing Large Raster Mosaics

A four-hour mosaic that restarts from zero after a spot instance is reclaimed has not failed once; it has failed twice, and the second failure is the one you chose. Checkpointing fixes it, but only if the checkpoint is verifiable — a resumed run that trusts a half-written block produces a mosaic with a corrupt stripe and no error anywhere. The pattern is one artefact per block, a digest written after the bytes, and a resume path that treats an unverifiable checkpoint as absent.

When to Use This Pattern#

• The job is longer than the interruption interval — spot reclamation, a nightly maintenance window, a deploy.
• Blocks are independently computable, so partial progress is genuinely reusable.
• Re-running is expensive in money or in a fixed publishing window, not merely slow.
• The output is assembled by reference — a VRT over blocks — rather than held in one array.

Complete Working Example#

The checkpoint is an object plus a sidecar digest. The digest is written second, which is what makes presence mean completeness.

from __future__ import annotations

import hashlib
import json
from pathlib import Path

from prefect import flow, task, get_run_logger


def block_key(run_id: str, block: Block, recipe: str) -> str:
    return f"ckpt/{run_id}/{recipe}/{block.z}_{block.x}_{block.y}"


@task(retries=2, timeout_seconds=900, tags=["block"])
def build_block(block: Block, run_id: str, recipe: str, scratch: Path) -> str:
    key = block_key(run_id, block, recipe)
    if (done := verified_checkpoint(key, scratch)) is not None:
        return done                                   # resume: nothing to do

    tmp = scratch / f"{key}.tif.part"
    tmp.parent.mkdir(parents=True, exist_ok=True)
    warp_block(block, tmp)                            # the expensive part

    digest = sha256_of(tmp)
    final = tmp.with_suffix("")                       # .tif
    tmp.rename(final)                                 # atomic within one filesystem
    # Sidecar written LAST: a block with a sidecar is complete by construction.
    final.with_suffix(".tif.sha256").write_text(
        json.dumps({"sha256": digest, "bytes": final.stat().st_size})
    )
    return str(final)


def verified_checkpoint(key: str, scratch: Path) -> str | None:
    obj, side = scratch / f"{key}.tif", scratch / f"{key}.tif.sha256"
    if not (obj.exists() and side.exists()):
        return None
    meta = json.loads(side.read_text())
    if obj.stat().st_size != meta["bytes"] or sha256_of(obj) != meta["sha256"]:
        obj.unlink(missing_ok=True)                   # torn write: treat as absent
        side.unlink(missing_ok=True)
        return None
    return str(obj)


@flow(name="mosaic", timeout_seconds=21600)
def mosaic(blocks: list[Block], run_id: str, scratch: Path, recipe: str = "v3") -> Path:
    paths = build_block.map(blocks, run_id=run_id, recipe=recipe, scratch=scratch)
    resumed = sum(1 for b in blocks if verified_checkpoint(block_key(run_id, b, recipe), scratch))
    get_run_logger().info("blocks: %d total, %d resumed from checkpoint", len(blocks), resumed)
    return assemble_vrt_and_translate([p.result() for p in paths], scratch / "mosaic.tif")

The digest check on read is easy to argue away as paranoia — the rename is atomic, so why verify? Because the rename is atomic only within one filesystem, and the moment scratch becomes object storage the guarantee changes shape: a multipart upload that fails between parts can leave a readable object of the wrong length, and some stores will happily serve it. Verifying costs a read of data you are about to read anyway, and it converts a silent corrupt stripe into a rebuilt block.

Parameter & Option Reference#

Verification & Testing#

The tests worth writing simulate the interruption rather than the happy path.

def test_resume_skips_verified_blocks(tmp_path, monkeypatch) -> None:
    built: list[str] = []
    monkeypatch.setattr("mymodule.warp_block", lambda b, p: (p.write_bytes(b"x" * 64),
                                                            built.append(b.key)))
    blocks = make_blocks(20)
    mosaic(blocks[:10], run_id="r1", scratch=tmp_path)     # first attempt
    built.clear()
    mosaic(blocks, run_id="r1", scratch=tmp_path)          # resume
    assert len(built) == 10, "resumed run rebuilt blocks it had already finished"


def test_sidecarless_block_is_rebuilt(tmp_path) -> None:
    key = block_key("r2", BLOCK, "v3")
    (tmp_path / f"{key}.tif").write_bytes(b"truncated")     # no sidecar
    assert verified_checkpoint(key, tmp_path) is None


def test_corrupt_block_is_rebuilt(tmp_path) -> None:
    key = write_good_checkpoint(tmp_path, "r3", BLOCK)
    (tmp_path / f"{key}.tif").write_bytes(b"different bytes entirely")
    assert verified_checkpoint(key, tmp_path) is None
    assert not (tmp_path / f"{key}.tif").exists(), "corrupt checkpoint was not cleaned up"


def test_recipe_change_does_not_resume(tmp_path) -> None:
    write_good_checkpoint(tmp_path, "r4", BLOCK, recipe="v3")
    assert verified_checkpoint(block_key("r4", BLOCK, "v4"), tmp_path) is None

The last test is the one people leave out. A checkpoint prefix that omits the recipe version lets a resumed run mix blocks built by two different processing rules into a single mosaic — a resampling change, a nodata fix, a different overview algorithm — and the result is a mosaic with a visible seam and a completely clean run history. Putting the recipe in the prefix makes the mixture impossible rather than merely unlikely.

One further property is worth designing for deliberately: a checkpoint should be cheap to discard. The temptation, once blocks are being written and verified, is to start reusing them across runs — the same tiles are being rebuilt every night, after all, so why throw them away? The answer is that a checkpoint reused across runs is no longer scratch, and everything that made it safe stops holding: it now needs an invalidation rule, a story about what happens when the source changes, and a decision about who owns it. That machinery already exists one layer up, keyed by content rather than by run, and it is the ledger. Keeping the checkpoint disposable is what keeps the two from becoming one confused thing.

The corollary is that the resumed-block count is a diagnostic and not a target. A pipeline whose runs resume half their blocks every night is a pipeline being interrupted every night, and the fix is upstream — a longer timeout, a less aggressive instance class, a smaller unit of work — rather than a better checkpoint. Checkpointing makes interruptions survivable; it does not make them acceptable, and a resumed count trending upward is the earliest signal that something about the platform has changed.

Common Pitfalls#

• Writing the digest before the data. A sidecar then proves nothing, and a resumed run reads whatever bytes happen to be there.
• Resuming across recipes. Without the recipe in the prefix, a processing change produces a seamed mosaic that passes every structural check.
• Checkpointing the whole mosaic. One enormous artefact means an interruption at ninety per cent still loses ninety per cent; the unit of checkpointing must be the unit of work.
• No lifecycle rule. Scratch that is never collected quietly becomes the largest line on the storage bill, especially when spot reclamation makes abandoned prefixes routine.
• Sharing a prefix between runs. Two runs then resume from each other’s partial work, which is only correct if they are computing the same thing — and if they are, they should have been one run.
• Treating a checkpoint as a record of completion. It is scratch. The ledger records what exists; see state management in geospatial flows.

Frequently Asked Questions#

Related#

• State management in geospatial flows — where checkpoints sit among the three kinds of state
• Resuming a failed mosaic from its last checkpoint — the recovery path
• Using spot instances for interruptible raster jobs — the interruption this defends against
• Collecting mapped results into a single mosaic — assembling by reference
• Caching reprojected rasters with content hashing — the digest, used for a different purpose

← State Management in Geospatial Flows