Using Spot Instances for Interruptible Raster Jobs

A spot instance is a discount in exchange for a promise you will not mind being interrupted, and a raster pipeline can honour that promise cheaply — the work is naturally divisible, idempotent when content-keyed, and resumable when checkpointed. What turns the discount into a saving is handling the reclamation notice properly, keeping the checkpoint interval well under the notice period, and being honest about which stages must not run on interruptible capacity at all.

When to Use This Pattern#

• The job is divisible into units of a few minutes, so an interruption loses minutes rather than hours.
• Checkpointing is already in place, as in checkpointing large raster mosaics.
• The deadline has slack — a nightly build that must finish by 06:00 and takes three hours has room; one that takes five and a half does not.
• Compute is a visible cost and the fleet is large enough for a sixty to eighty per cent discount to matter.

Complete Working Example#

Two parts: a handler that reacts to the reclamation notice, and a fleet split that keeps the uninterruptible stages safe.

from __future__ import annotations

import signal
import threading
import time
from pathlib import Path

import requests
from prefect import flow, task, get_run_logger

RECLAIM_URL = "http://169.254.169.254/latest/meta-data/spot/instance-action"
_draining = threading.Event()


def watch_for_reclamation(poll_seconds: int = 5) -> None:
    """Set the drain flag as soon as the notice appears. Runs in a daemon thread."""
    while not _draining.is_set():
        try:
            if requests.get(RECLAIM_URL, timeout=2).status_code == 200:
                _draining.set()
        except requests.RequestException:
            pass                                  # no notice yet: the endpoint 404s
        time.sleep(poll_seconds)


def _on_sigterm(_signum, _frame) -> None:
    _draining.set()                               # the orchestrator's own drain signal


signal.signal(signal.SIGTERM, _on_sigterm)
threading.Thread(target=watch_for_reclamation, daemon=True).start()


@task(retries=3, retry_delay_seconds=[5, 20, 60], timeout_seconds=900, tags=["tile"])
def build_block(block: Block, run_id: str, scratch: Path) -> str | None:
    if _draining.is_set():
        # Refuse new work rather than starting something that cannot finish.
        get_run_logger().info("draining: declining block %s", block.key)
        return None

    key = block_key(run_id, block, RECIPE)
    if (done := verified_checkpoint(key, scratch)) is not None:
        return done

    path = warp_block(block, scratch)             # ~90 s: well inside a 2-minute notice
    write_checkpoint(key, path, scratch)          # digest sidecar written last
    return path


@flow(name="mosaic-spot", timeout_seconds=21600, retries=2)
def mosaic(blocks: list[Block], run_id: str, scratch: Path) -> Path:
    states = build_block.map(blocks, run_id=run_id, scratch=scratch, return_state=True)
    built = [s.result() for s in states if s.is_completed() and s.result()]
    if len(built) < len(blocks):
        # A flow-level retry picks up the declined blocks on surviving workers.
        raise RuntimeError(f"{len(blocks) - len(built)} block(s) outstanding — retrying")
    return assemble(built, scratch / "mosaic.tif")

The fleet split is configuration rather than code, and it is what keeps the reduction safe:

# The wide, interruptible stage.
work_pool: spot-tiles
  instance_market: spot
  capacity_rebalance: true       # replace before reclamation where the platform offers it
  max_price: on-demand           # never pay more than the alternative
  tags: [tile]

# The single long reduction and the publish step: on-demand, small, always there.
work_pool: reduce-ondemand
  instance_market: on-demand
  tags: [reduce, publish]

Declining new work while draining is the part most implementations leave out, and it matters more than the checkpoint. Without it, a worker that has received its notice cheerfully starts a fresh ninety-second block with sixty seconds to live, guaranteeing that the work is lost and, worse, that the block’s slot in the concurrency limit is occupied by something that cannot succeed. With the drain flag, the worker finishes what it has, writes its checkpoint and exits cleanly, and every declined block is picked up elsewhere.

Parameter & Option Reference#

Verification & Testing#

Reclamation is testable by simulating the notice, which is the only way to find out whether the handler works before it matters.

def test_draining_declines_new_work(monkeypatch, tmp_path) -> None:
    _draining.set()
    try:
        assert build_block.fn(BLOCK, run_id="r1", scratch=tmp_path) is None
    finally:
        _draining.clear()


def test_sigterm_sets_the_drain_flag() -> None:
    _draining.clear()
    os.kill(os.getpid(), signal.SIGTERM)
    time.sleep(0.1)
    assert _draining.is_set()


def test_in_flight_block_still_checkpoints(tmp_path, monkeypatch) -> None:
    def slow_warp(block, scratch):
        _draining.set()                      # notice arrives mid-block
        return write_block(block, scratch)

    monkeypatch.setattr("mymodule.warp_block", slow_warp)
    build_block.fn(BLOCK, run_id="r2", scratch=tmp_path)
    assert verified_checkpoint(block_key("r2", BLOCK, RECIPE), tmp_path) is not None


def test_reduce_is_not_tagged_for_spot() -> None:
    assert "tile" not in reduce.tags, "the reduction would be scheduled on spot capacity"

Running a real reclamation drill once is worth more than all four tests together. Most platforms offer a way to trigger an interruption deliberately, and doing it during a nightly run — once, deliberately, while somebody is watching — answers questions the unit tests cannot: whether the drain propagates, whether the flow retry actually recovers the declined blocks, and how long the run takes when a third of the fleet disappears. Teams that have never run the drill usually find one thing that does not work, and they find it at three in the morning otherwise.

One consequence of the drain flag is worth designing around rather than discovering: a worker that declines work but stays alive will keep declining until the platform terminates it, and during that window it still holds whatever concurrency slots the orchestrator believes it owns. On most platforms the gap is short — a couple of minutes — and harmless. On a scheduler that reuses the node rather than terminating it, or where the notice is retracted, a permanently draining worker becomes a slot that never frees. Exiting the worker process once the in-flight block has checkpointed avoids the whole question and costs nothing, because the fleet is going to replace the node anyway.

The other operational habit worth adopting is logging reclamations as a metric rather than as an incident. Each one is expected — that is the deal — so an alert per reclamation trains everyone to ignore the channel. What deserves an alert is the rate: a fleet that normally sees two reclamations a night and suddenly sees thirty is telling you that the instance family is under pressure, and the correct response is a configuration change rather than an investigation of the pipeline. That distinction is only available if the ordinary case was counted rather than paged on.

Common Pitfalls#

• Starting new work while draining. The block cannot finish, its slot is wasted, and the loss was avoidable.
• Running the reduction on spot. One long sequential pass with no checkpointing is the worst possible interruptible workload.
• Units longer than the notice. Anything that cannot finish in the notice period loses its work every single time.
• Ignoring SIGTERM. Container schedulers signal before killing, and treating that as an ordinary crash discards a graceful exit you were offered.
• A hundred per cent spot fleet. A capacity event can then stall the run entirely; a small on-demand core keeps it moving.
• No max_price cap. Spot pricing can exceed on-demand during a crunch, which turns the whole exercise upside down.

Frequently Asked Questions#

Related#

• Cost optimization for spatial compute — where the price lever fits
• Checkpointing large raster mosaics — the prerequisite, in detail
• Resuming a failed mosaic from its last checkpoint — what happens after an interruption
• Right-sizing workers for raster mosaic jobs — smaller workers lose less when reclaimed
• Cancelling in-flight tile jobs cleanly — the signal handling, generalised

← Cost Optimization for Spatial Compute