Timeout Budgets and Cancellation for Geotasks

In short: pick one deadline for the flow, then derive every timeout beneath it — task, subprocess, HTTP read, database statement — from the time actually remaining rather than from independently chosen constants. Then make cancellation real: a Python task that “times out” while a gdalwarp child process keeps running has not stopped anything, it has only stopped watching.

Timeouts in spatial pipelines are usually a collection of numbers chosen at different times by different people: 30 seconds because that is the library default, 10 minutes because someone once saw a slow tile, no timeout at all on the PostGIS statement because nobody thought about it. The result is a pipeline whose actual worst-case runtime is the sum of its unbounded parts, discovered the night an upstream service starts hanging instead of failing. A budget replaces that with one number you choose deliberately and a set of derivations you can reason about.

There is a second reason budgets matter more in spatial work than in most pipelines, and it is about where the time goes. A tabular ETL step that hangs is usually waiting on a network call, and killing the Python process ends it. A spatial step that hangs is often waiting on a gdalwarp child that has already allocated ten gigabytes of resampling buffers, or on a PostGIS query grinding through a GiST index on ninety million geometries. In both cases the thing consuming resources is not the process your orchestrator is watching, and “the task timed out” is a statement about the watcher rather than about the work.

Prerequisites & Architecture Baseline#

• A stated deadline per flow — the time by which the output must exist, not a guess at how long it usually takes
• Python 3.11+, for asyncio.timeout() and exception groups if the pipeline is async
• subprocess calls that capture the child PID, since GDAL work is almost always out-of-process
• statement_timeout available on the PostGIS connection, set per session rather than globally
• Orchestrator-level timeouts configured: Prefect timeout_seconds on the task, Dagster max_runtime tags
• Metrics on time-to-completion per stage, because a budget you cannot measure against is a guess

Core Principles#

1. One deadline, many derived timeouts. The flow has a deadline. Each task computes its own limit as min(its own maximum, time remaining before the deadline). Nothing below the flow level owns an absolute number, which means a change to the deadline propagates automatically instead of requiring six edits in five files. It also means a task that starts late gets a shorter timeout, which is correct — the deadline has not moved.

2. Every layer’s timeout must be shorter than the layer above it. HTTP read timeout < subprocess timeout < task timeout < flow deadline. Invert any pair and the outer layer kills the inner one before it can report anything useful, so the failure arrives without a cause attached. This ordering is the single most valuable property of a budget, and the easiest to violate accidentally.

3. A timeout without cancellation is a lie. Python’s subprocess.run(timeout=…) raises TimeoutExpired and, crucially, does not kill the child unless you do. A gdalwarp that has consumed 12 GB of RAM keeps consuming it while the orchestrator marks the task failed and starts another one on the same worker. Cancellation must terminate the process group, wait, and escalate to SIGKILL.

4. Cancellation must leave the outputs clean. A killed gdal_translate leaves a truncated GeoTIFF that is a valid file with an invalid last block. Write to a temporary path and rename on success, so that a cancelled task leaves nothing rather than something plausible. The same discipline that makes retries safe makes cancellation safe.

5. Long operations need checkpoints, not just deadlines. A four-hour mosaic that is cancelled at 3h55m has produced nothing. Checkpointing lets cancellation preserve completed work, turning a deadline from a cliff into a boundary — see checkpointing large raster mosaics.

6. Budget the retries, not just the attempt. If a task has three retries and a ten-minute timeout, its worst case is thirty minutes plus backoff, not ten. Retry budgets and timeouts have to be designed together, which is why the exponential backoff policy takes a wall-clock budget rather than an attempt count.

The practical consequence of that nesting rule is worth spelling out, because it inverts a habit most engineers have. The instinct when a task times out too often is to raise its timeout. Under a budget, the correct first question is which layer fired — and if the layer that fired was the outermost one, raising it is the last thing to do, because it means an inner layer failed to bound something and the flow deadline caught the overrun. A pipeline where the flow deadline is the timeout that fires most often does not have a timeout that is too short; it has an unbounded operation somewhere inside, and finding it is the actual work.

Production Implementation#

The budget object below is passed down the call stack. Every layer asks it how long it may take, and gets an answer that accounts for the time already spent.

from __future__ import annotations

import os
import signal
import subprocess
import time
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Iterator, Optional, Sequence


class DeadlineExceeded(TimeoutError):
    """The flow's deadline passed before this step could start or finish."""


@dataclass
class Budget:
    """A shrinking allowance. Every layer derives its limit from this."""

    deadline_monotonic: float

    @classmethod
    def of(cls, seconds: float) -> "Budget":
        return cls(deadline_monotonic=time.monotonic() + seconds)

    def remaining(self) -> float:
        return max(0.0, self.deadline_monotonic - time.monotonic())

    def slice(self, want: float, reserve: float = 0.0) -> float:
        """`want` seconds, or whatever is left minus a reserve for cleanup."""
        available = self.remaining() - reserve
        if available <= 0:
            raise DeadlineExceeded("no time left in the budget")
        return min(want, available)


def run_gdal(
    argv: Sequence[str],
    budget: Budget,
    want_seconds: float = 900.0,
    grace_seconds: float = 10.0,
) -> subprocess.CompletedProcess:
    """Run a GDAL utility under the budget, and actually kill it on timeout.

    start_new_session puts the child in its own process group, so the signal
    reaches gdalwarp's own helper processes too — killing only the direct child
    leaves warp threads holding memory and file handles.
    """
    limit = budget.slice(want_seconds, reserve=grace_seconds + 5.0)
    proc = subprocess.Popen(
        argv,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        start_new_session=True,
    )
    try:
        out, err = proc.communicate(timeout=limit)
    except subprocess.TimeoutExpired:
        os.killpg(os.getpgid(proc.pid), signal.SIGTERM)   # ask the group to stop
        try:
            out, err = proc.communicate(timeout=grace_seconds)
        except subprocess.TimeoutExpired:
            os.killpg(os.getpgid(proc.pid), signal.SIGKILL)  # then insist
            out, err = proc.communicate()
        raise TimeoutError(
            f"{argv[0]} exceeded {limit:.0f}s and was terminated: "
            f"{err.decode('utf-8', 'replace')[-400:]}"
        )
    if proc.returncode != 0:
        raise RuntimeError(f"{argv[0]} failed: {err.decode('utf-8', 'replace')[-400:]}")
    return subprocess.CompletedProcess(argv, proc.returncode, out, err)


@contextmanager
def postgis_statement_timeout(conn, budget: Budget, want_seconds: float = 120.0) -> Iterator[None]:
    """Bound a PostGIS statement by the remaining budget, for this session only."""
    limit_ms = int(budget.slice(want_seconds) * 1000)
    with conn.cursor() as cur:
        cur.execute("SET LOCAL statement_timeout = %s", (limit_ms,))
        yield

Wiring it into a task is then a matter of threading one object through:

from prefect import flow, task


@task(retries=2)
def build_tile(tile, budget: Budget, dsn: str) -> str:
    tmp = f"/scratch/{tile.key}.tif.part"
    final = f"/scratch/{tile.key}.tif"

    run_gdal(
        ["gdalwarp", "-t_srs", "EPSG:3857", "-r", "bilinear",
         "-dstnodata", "-9999", "-co", "COMPRESS=DEFLATE",
         tile.source, tmp],
        budget=budget, want_seconds=600.0,
    )
    # Rename only after the child exited cleanly: a killed warp leaves a valid
    # file with an invalid final block, and .part never gets promoted.
    os.replace(tmp, final)
    return final


@flow(timeout_seconds=7200)
def build_mosaic(tiles) -> None:
    budget = Budget.of(7200 - 120)      # reserve two minutes for finalisation
    for tile in tiles:
        if budget.remaining() < 60:
            break                        # stop cleanly rather than being killed
        build_tile(tile, budget=budget, dsn=DSN)

Step-by-Step Walkthrough#

1. Choose the deadline from the requirement, not the runtime. “The mosaic must exist by 06:00” gives a deadline; “it usually takes 90 minutes” does not. The budget’s value comes from being tied to something real.
2. Reserve time for finalisation before deriving anything. The two minutes subtracted in Budget.of pay for the manifest write, the cache invalidation and the metrics flush. A budget that spends every second on work leaves the run unable to finish tidily.
3. Derive each limit with slice. A task asking for ten minutes when eight remain gets eight. This is what makes a late-starting run degrade gracefully instead of overrunning.
4. Put the child in its own process group. start_new_session=True plus os.killpg is what makes cancellation reach GDAL’s worker threads and any helper processes. Killing the direct child alone leaves them running.
5. Escalate the signal. SIGTERM, a short grace period, then SIGKILL. GDAL utilities handle SIGTERM reasonably and will close files; going straight to SIGKILL guarantees a truncated output.
6. Write to a temporary path and rename. The rename is the commit. A cancelled task leaves a .part file that the next run overwrites, and never a half-written GeoTIFF that looks complete to gdalinfo.
7. Check the budget before starting each unit. The if budget.remaining() < 60: break is what turns a hard flow timeout into a clean early stop with a known set of completed tiles.

Edge Cases & Failure Recovery#

A SIGTERM that GDAL ignores. Some GDAL operations — particularly those inside a long RasterIO call on a network-backed dataset — do not check for signals until the call returns, which may be minutes. The grace period must be long enough for the common case and the SIGKILL must be unconditional, because a process that ignores both is a process that will still hold its memory when the next task starts.

PostGIS statements that outlive the client. Cancelling the Python side does not cancel the query; the backend keeps working until it finishes or hits statement_timeout. Setting SET LOCAL statement_timeout inside the transaction is what actually bounds it. Without that, a cancelled task can leave a query holding locks that block the next run’s writes.

Nested budgets that double-count. If a sub-flow creates its own Budget.of(3600) instead of receiving the parent’s, the child can outlive the parent’s deadline. Pass the budget object down; never construct a second one inside a call tree.

Cancellation during the rename. os.replace is atomic on POSIX within one filesystem, so this is safe — but only within one filesystem. Writing the .part file to /tmp and renaming onto a network mount is a copy, not a rename, and can be interrupted halfway. Keep the temporary file on the same mount as its destination.

A deadline that arrives mid-batch, every night. Some pipelines are simply too large for their window, and the budget makes this visible by stopping early with a consistent number of tiles unbuilt. The temptation is to raise the deadline; the honest response is usually to change the shape of the work — more concurrency, coarser tiles, or an incremental run that only rebuilds what changed. A budget that is quietly extended every quarter has stopped being a constraint and become a record of drift.

Timeouts that fire during finalisation. The reserve exists precisely so the manifest write and cache invalidation are not themselves cancelled, but a reserve of two minutes against a finalisation step that takes three is worse than no reserve at all: the run stops halfway through publishing, leaving a manifest that references tiles that were never invalidated. Measure the finalisation step like any other, and size the reserve from that measurement.

A worker killed by the platform rather than by the budget. Spot reclamation and OOM kills give you no chance to clean up. The .part convention covers the output, and the idempotency ledger covers the state — its stale-claim reaper is what returns the work to the pool. A budget makes cancellation orderly; it cannot make every termination orderly.

Configuration Reference#

The one number that resists derivation is the grace period, because it depends on what the child is doing when the signal arrives. Ten seconds is enough for gdal_translate finishing a block write and far too little for a gdalwarp in the middle of a network read. If your workloads span both, measure: log the interval between SIGTERM and process exit for a week, take the p95, and use that. A grace period chosen this way costs a few seconds per cancellation and prevents the truncated outputs that an over-eager SIGKILL produces.

Frequently Asked Questions#

Related#

• Setting per-task timeouts for GDAL operations — sizing the subprocess limit from real runtimes
• Cancelling in-flight tile jobs cleanly — the signal, cleanup and ledger release in detail
• Exponential backoff for API rate limits — the retry budget that shares this deadline
• Checkpointing large raster mosaics — preserving work a cancellation would otherwise discard
• Idempotency keys in spatial ETL — the ledger a cancelled task must release

← Resilience & Failure Handling for GIS Pipelines