Geospatial Orchestration Hub

Resuming a Failed Mosaic From Its Last Checkpoint

Resuming is not “run it again and hope the cache helps”. It is a decision procedure: establish which blocks are genuinely reusable, rebuild exactly the gap, verify the set is complete, and only then assemble. Each of those four steps has a way of going wrong quietly, and the quiet failures all produce the same artefact — a mosaic that opens, renders, reports full coverage and is wrong somewhere nobody is looking.

When to Use This Pattern

  • A long mosaic run failed part-way and the window to republish is shorter than a full rebuild.
  • Checkpoints exist and are verifiable, as described in checkpointing large raster mosaics.
  • The failure was not caused by the data — a reclaimed instance, a full disk, a deploy — so the completed blocks are still trustworthy.
  • Somebody is waiting, which is what makes the difference between resuming and restarting worth the machinery.

Complete Working Example

The resume flow takes the same arguments as the original run. That is deliberate: if resuming needs different parameters, it is a different computation.

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

from prefect import flow, task, get_run_logger


@dataclass(frozen=True)
class ResumePlan:
    reusable: list[Block]
    rebuild: list[Block]
    discarded: list[tuple[Block, str]]      # block, why it was rejected


@task(timeout_seconds=900)
def plan_resume(blocks: list[Block], run_id: str, recipe: str, scratch: Path) -> ResumePlan:
    reusable, rebuild, discarded = [], [], []
    for b in blocks:
        key = block_key(run_id, b, recipe)
        state = inspect_checkpoint(key, scratch)
        if state.missing:
            rebuild.append(b)
        elif state.digest_mismatch:
            discarded.append((b, "digest mismatch")); rebuild.append(b)
        elif state.source_digest != current_source_digest(b):
            # The source changed while we were down. A reusable-looking block is stale.
            discarded.append((b, "source changed")); rebuild.append(b)
        else:
            reusable.append(b)

    log = get_run_logger()
    log.info("resume plan: %d reusable, %d to rebuild, %d discarded",
             len(reusable), len(rebuild), len(discarded))
    for b, why in discarded[:20]:
        log.warning("discarded %s: %s", b.key, why)
    return ResumePlan(reusable, rebuild, discarded)


@flow(name="mosaic-resume", timeout_seconds=21600)
def resume(blocks: list[Block], run_id: str, scratch: Path,
           recipe: str = "v3", coverage_floor: float = 1.0) -> Path:
    plan = plan_resume(blocks, run_id, recipe, scratch)

    # Rebuild only the gap. The same task the original run used, unchanged.
    states = build_block.map(plan.rebuild, run_id=run_id, recipe=recipe,
                             scratch=scratch, return_state=True)
    failed = [b for b, s in zip(plan.rebuild, states) if not s.is_completed()]

    present = len(plan.reusable) + len(plan.rebuild) - len(failed)
    coverage = present / len(blocks)
    if coverage < coverage_floor:
        raise RuntimeError(
            f"coverage {coverage:.4f} < {coverage_floor} "
            f"({len(failed)} block(s) still failing) — not assembling"
        )
    return assemble_vrt_and_translate(
        [checkpoint_path(run_id, b, recipe, scratch) for b in blocks],
        scratch / "mosaic.tif",
    )
The resume plan, before anything is rebuiltOf four hundred and twelve blocks, three hundred and one verify and are reusable, six are stale because their source changed, and one hundred and five were never built.412 blocks, inspected before any work starts301 reusable — digest verifies, source unchanged6 stale — the source was republished while we were down105 missing — never startedRebuild set = 111 blocks. The six stale ones are the whole reason the plan exists as a separate step.
A resume that only checks for presence would reuse those six and ship a mosaic mixing two versions of a source, with nothing to indicate it.

The staleness check is the part that distinguishes a resume from a retry. A checkpoint proves a block was built correctly from some input; it says nothing about whether that input is still current. When a run fails at midnight and is resumed at nine the next morning, a publisher may well have issued a correction in between — and a resume that reuses on presence alone will silently mix yesterday’s source with today’s. Comparing each checkpoint’s recorded source digest against the current one costs a head request per block and closes the gap entirely.

Parameter & Option Reference

Setting Typical Spatial notes
coverage_floor 1.0 for a resume A first attempt may tolerate 0.995; a resume should not, because the missing blocks are known and named.
Staleness check per block Compare the checkpoint’s stored source digest with the current one. A head request each, done concurrently.
Discard logging first 20, with reasons A resume that discards a hundred blocks is telling you something; unlogged, it just looks slow.
Same run_id required The checkpoint prefix is scoped to it. A new run id means a full rebuild, which is sometimes what you want.
Same recipe required Resuming across a recipe change produces a seamed mosaic; the prefix should make it impossible.
Retry budget 1 pass If blocks still fail after one resume, the cause is not transient and another pass will not help.
Three shapes a resume takesA clean resume rebuilds the gap. A resume with discards rebuilds more than expected because sources changed. When most blocks are discarded, a restart is the honest answer.what the plan says, and what it meansclean resume0 discardedrebuild the gapassemble, publishthe normal caseresume with discards6 discardedsources movedstill much cheaperthan a restartrestart instead280 discardeda bulk republishresume saves nothingand hides the reasonThe discard count is the number to look at first. It answers “is this still a resume?” before any time is spent.
Logging the plan before acting on it is what makes the third column visible; without it, that case just looks like a resume that took as long as a rebuild.

Verification & Testing

def test_resume_rebuilds_only_the_gap(tmp_path, monkeypatch) -> None:
    built: list[str] = []
    monkeypatch.setattr("mymodule.warp_block", lambda b, p: built.append(b.key) or write(p))
    blocks = make_blocks(50)
    for b in blocks[:30]:
        write_good_checkpoint(tmp_path, "r1", b)
    resume(blocks, run_id="r1", scratch=tmp_path)
    assert sorted(built) == sorted(b.key for b in blocks[30:])


def test_stale_checkpoint_is_discarded(tmp_path, monkeypatch) -> None:
    write_good_checkpoint(tmp_path, "r2", BLOCK, source_digest="old")
    monkeypatch.setattr("mymodule.current_source_digest", lambda b: "new")
    plan = plan_resume.fn([BLOCK], "r2", "v3", tmp_path)
    assert plan.reusable == [] and plan.rebuild == [BLOCK]
    assert plan.discarded[0][1] == "source changed"


def test_incomplete_resume_does_not_assemble(tmp_path, monkeypatch) -> None:
    monkeypatch.setattr("mymodule.warp_block", failing)
    with pytest.raises(RuntimeError, match="not assembling"):
        resume(make_blocks(10), run_id="r3", scratch=tmp_path)


def test_resume_is_itself_resumable(tmp_path) -> None:
    blocks = make_blocks(20)
    run_partial(resume, blocks, run_id="r4", scratch=tmp_path, stop_after=5)
    resume(blocks, run_id="r4", scratch=tmp_path)          # must not raise
    assert count_checkpoints(tmp_path, "r4") == 20

The last test is the one that catches a whole family of mistakes. A resume that cannot itself be resumed usually means a step somewhere has become order-dependent — a manifest that gets rewritten, a counter that gets incremented, a temporary directory that gets cleared on entry — and that dependency will surface the second time a run is interrupted, which is exactly the moment nobody has patience for it. Resuming should be an ordinary idempotent operation, not a special mode.

Two interruptions, two resumesThe first attempt completes two hundred blocks, the first resume adds one hundred and fifty before failing, and the second resume finishes the remaining sixty-two.412 blocks, three attempts, no reworkattempt 1200 blocksresume 1skipped150 blocksresume 2skipped62Total work done: 412 blocks, once each. A restart-on-failure policy would have done 762.The property that makes this hold is that resume is idempotent, not that it is clever.
Each attempt is the same flow with the same arguments. Nothing about the second resume knows it is the second.

It is worth being explicit about what the plan step buys, because the temptation is always to fold it into the build task and let each block decide for itself whether to skip. Doing so works, and it costs you the one thing that makes a resume operable: a statement, before any time is spent, about what this run is going to do. With the plan as a separate step an operator sees “301 reusable, 111 to rebuild, 6 discarded” within seconds of starting and can decide whether that is the run they wanted. With the decision distributed across four hundred tasks, the same information exists only as a pattern in the logs, and it arrives over the following half hour.

That visibility also changes what can be automated. A resume whose plan is a value can be gated — refuse automatically when discards exceed a quarter of the blocks, escalate to a person instead of quietly turning into a full rebuild at three in the morning. A resume whose decisions live inside the tasks has nowhere to put that rule, so the choice between resuming and restarting ends up being made by whoever notices the run is taking longer than it should.

Common Pitfalls

  • Reusing on presence alone. A block that exists is not a block that is current; check the source digest or accept mixing two versions of a source.
  • Resuming across a recipe change. The prefix should prevent it. If it does not, the seam will not be found by any automated check.
  • A coverage floor below one on a resume. The missing blocks are known by name at that point, so tolerating them is a choice to publish a hole deliberately.
  • Clearing scratch at the start of the flow. It makes the first resume work and the second one a full rebuild, and the behaviour is invisible until it happens.
  • Not logging the plan. Without discard counts, a resume that rebuilt everything is indistinguishable from one that rebuilt the gap, except by duration.
  • Treating a data-caused failure as resumable. If the run failed because a source is malformed, the completed blocks may be built from the same bad delivery; fix the source and restart.

Frequently Asked Questions

Should resuming be automatic?

For infrastructure failures, yes — a retry at the flow level with the same run id is exactly a resume, and it should be the default response to a reclaimed instance. For failures the pipeline raised itself, no: those mean the data disagreed with an expectation, and resuming past them repeats the disagreement with less information.

How long can a checkpoint be resumed from?

Until the lifecycle rule collects it, which should be a few days, and only while the source is unchanged. In practice the staleness check makes the second constraint bind first: a week-old checkpoint against a weekly source will be discarded anyway.

What if only the assembly failed?

Then every block is verifiable and the plan will report zero rebuilds, so the resume is just the assembly — usually minutes. That is the case where checkpointing pays for itself most obviously, because the expensive part is already done and the cheap part is what broke.

Does this work for vector pipelines?

The structure carries over, with the batch playing the part of the run and the staged table playing the part of the checkpoint. See managing state for incremental shapefile updates; the difference is that a vector merge is one transaction, so it either happened or it did not, and there is no partial state to inspect.

← State Management in Geospatial Flows

Related in this section