Exponential Backoff for API Rate Limits

In short: back off exponentially with full jitter, cap the delay, and bound the total retry budget by the deadline of the task rather than by an attempt count. For geospatial services the twist is that a single logical request often costs the server minutes of rendering, so the retry that helps a JSON API — fast, cheap, immediately re-issued — is precisely the one that turns a struggling WMS into an unavailable one.

Public spatial services are slow by nature and rate-limited by necessity. A GetFeature against a national WFS may stream 200 MB over four minutes. A GetMap at 4096×4096 makes the server rasterise real geometry. A geocoder enforces a hard requests-per-second ceiling written into its terms of use. When a pipeline with a hundred parallel tile workers hits any of these, naive retries do not recover the pipeline — they convert a transient 429 into a sustained outage, for you and for everyone else on the endpoint.

The difference is worth stating precisely, because it is what makes spatial retry policies unlike the ones in a web-services handbook. A retry against a stateless JSON API costs the server a few milliseconds of CPU; the marginal cost of being wrong about the schedule is negligible, so the industry advice — retry three times, double the delay — is fine. A retry against a rendering endpoint costs the server the entire render again: the geometry is re-read, the labels are re-placed, the image is re-encoded. Ten thousand tile workers retrying a 3-second render every second is not a retry policy, it is a load test the endpoint’s operator did not agree to. Everything below follows from that single asymmetry.

Prerequisites & Architecture Baseline#

• httpx 0.27+ or requests 2.31+ with an explicit connect and read timeout on every call — a retry policy without timeouts never triggers
• tenacity 8.2+, or your orchestrator's native retry configuration (Prefect retry_delay_seconds, Dagster RetryPolicy(backoff=Backoff.EXPONENTIAL))
• A concurrency limit on the tasks that call each external host — retry policy and concurrency are one design, not two
• The published rate limit for every third-party endpoint you call, recorded next to the client that calls it
• Metrics for attempts, retries and give-ups per host, so a change in the endpoint's behaviour is visible before it becomes an incident

Core Principles#

1. Exponential growth, with a cap. Delay attempt n by base × 2ⁿ, clamped at a ceiling — typically 60 s for interactive flows and 300 s for batch. Uncapped growth produces a fourth retry twenty minutes out, long after the task’s own deadline has passed, which wastes a worker slot without ever helping.

2. Full jitter, not fixed delay. A hundred tile workers that fail together and retry after exactly 8 s recreate the original stampede. Sampling the delay uniformly from [0, computed_delay] spreads the load, and it beats “equal jitter” in practice because it lets some workers retry early and confirm recovery cheaply.

3. Only retry what is retryable. 429, 502, 503, 504 and connection resets are worth another attempt. 400 and 404 are not — a malformed bounding box will be malformed on every retry. A GetMap that returns 200 with an XML ServiceException body is the geospatial special case: it looks successful to the HTTP layer and must be classified as an error by inspecting the content type.

4. Honour Retry-After when it is offered. A server that tells you when to come back knows more than your schedule does. Treat the header as a floor, take the maximum of it and your computed delay, and clamp the result to something sane so that a mistaken Retry-After: 86400 does not park a worker for a day.

5. Bound the budget by a deadline, not by a count. “Five attempts” means something different for a 200 ms geocode and a four-minute WFS stream. Give each task a wall-clock budget and stop retrying when the next delay would exceed it. This is what makes the retry policy compatible with timeout budgets and cancellation rather than in tension with them.

6. Retries are the first line, not the whole defence. When an endpoint is genuinely down, every worker discovering that independently costs one full retry ladder each. That is what a circuit breaker for external WMS services exists to prevent: retries handle the blip, the breaker handles the outage.

Production Implementation#

The client below wraps a single external host. It classifies responses the way spatial services actually behave, computes a jittered delay bounded by a deadline, and respects Retry-After when the server supplies one.

from __future__ import annotations

import random
import time
from dataclasses import dataclass
from typing import Optional

import httpx

RETRYABLE_STATUS = frozenset({408, 425, 429, 500, 502, 503, 504})


class ServiceExceptionError(RuntimeError):
    """An OGC service returned 200 with an XML ServiceException body."""


@dataclass(frozen=True)
class BackoffPolicy:
    base_seconds: float = 1.0
    cap_seconds: float = 60.0
    budget_seconds: float = 600.0     # total wall clock, not an attempt count
    retry_after_max: float = 300.0    # ignore an absurd server hint

    def delay_for(self, attempt: int, retry_after: Optional[float]) -> float:
        exponential = min(self.cap_seconds, self.base_seconds * (2 ** attempt))
        jittered = random.uniform(0.0, exponential)     # full jitter
        if retry_after is not None:
            # The server's hint is a floor, clamped so one bad header cannot
            # park a worker for the rest of the day.
            return max(jittered, min(retry_after, self.retry_after_max))
        return jittered


def _retry_after_seconds(response: httpx.Response) -> Optional[float]:
    raw = response.headers.get("Retry-After")
    if raw is None:
        return None
    try:
        return float(raw)                      # delta-seconds form
    except ValueError:
        return None                            # HTTP-date form: ignore, use our own


def fetch_with_backoff(
    client: httpx.Client,
    url: str,
    params: dict[str, str],
    policy: BackoffPolicy = BackoffPolicy(),
) -> bytes:
    """GET an OGC endpoint, retrying transient failures within a wall-clock budget."""
    started = time.monotonic()
    attempt = 0

    while True:
        retry_after: Optional[float] = None
        try:
            response = client.get(url, params=params, timeout=httpx.Timeout(10.0, read=180.0))
            if response.status_code in RETRYABLE_STATUS:
                retry_after = _retry_after_seconds(response)
                raise httpx.HTTPStatusError("retryable", request=response.request, response=response)
            response.raise_for_status()

            # The geospatial special case: HTTP 200 carrying an OGC exception report.
            content_type = response.headers.get("content-type", "")
            if "xml" in content_type and b"ServiceException" in response.content[:2048]:
                raise ServiceExceptionError(response.content[:400].decode("utf-8", "replace"))

            return response.content

        except (httpx.TransportError, httpx.HTTPStatusError, ServiceExceptionError):
            elapsed = time.monotonic() - started
            delay = policy.delay_for(attempt, retry_after)
            if elapsed + delay > policy.budget_seconds:
                raise                       # the budget, not the counter, ends it
            time.sleep(delay)
            attempt += 1

Step-by-Step Walkthrough#

1. Set the timeouts first. httpx.Timeout(10.0, read=180.0) gives a fast connect timeout and a generous read timeout, because a WFS legitimately takes minutes to stream a large feature collection. Without a read timeout, a hung connection blocks the worker forever and the retry policy never runs.
2. Classify the response before deciding anything. Retryable statuses, transport errors and OGC ServiceException bodies raise; everything else either returns or fails permanently. A 400 from a malformed BBOX must not consume the retry budget.
3. Read Retry-After at the moment of failure, not later. Only the response that carried the 429 knows what the server asked for, and only the delta-seconds form is worth parsing — the HTTP-date form is rare and easy to get wrong across clock skew.
4. Compute the delay from the attempt number, then jitter it. min(cap, base·2ⁿ) gives the ceiling for this attempt and random.uniform(0, ceiling) samples below it. The server hint, when present, raises the floor.
5. Check the budget before sleeping, not after. If the sleep would push total elapsed time past the budget, give up now and surface the failure — sleeping first and then giving up wastes exactly the time you were trying to protect.
6. Let the failure propagate to the orchestrator. The task raising after its budget is what promotes the payload to a dead-letter queue or opens the circuit breaker. Swallowing it here hides a real outage behind a partially empty result set.

Edge Cases & Failure Recovery#

200 OK with an exception body. OGC services routinely report failure with a 200 status and an XML ServiceException. Any policy that keys purely on status codes treats these as successes, writes an XML document where a GeoTIFF was expected, and fails three tasks downstream where the cause is unrecoverable. Sniff the content type at the boundary, as above.

Rate limits enforced per API key, not per host. Two flows sharing a key share a limit, so per-process concurrency caps do not bound the actual request rate. Where this matters, the limit belongs in a shared token bucket — a Redis counter or the orchestrator’s global concurrency limit — rather than in the HTTP client.

Retry storms after a deploy. Restarting a fleet re-issues every in-flight request at once. Full jitter alone does not fix this, because attempt zero is not delayed. Stagger worker start-up, or make the first attempt of a batch job sleep random.uniform(0, 5).

Slow success beats fast failure. A WFS that takes 400 seconds when healthy will be aborted by a 180-second read timeout and retried forever, each attempt costing the server the full render. Measure the healthy p99 for each endpoint and set the read timeout above it, then rely on the budget to bound the total.

A proxy or CDN that answers instead of the origin. Many national spatial data infrastructures sit behind a cache that serves a stale tile with a 200 when the origin is unhealthy. The pipeline sees success and writes month-old imagery into a fresh mosaic. There is no retry policy that fixes this, because nothing failed; the defence is to check the response’s Age and Last-Modified headers against the source metadata you expected, and to treat an implausibly old response as an error worth retrying with a cache-busting parameter.

Partial streams that look complete. A WFS response truncated by a proxy timeout is valid XML up to the truncation point and parses into a smaller-than-expected feature collection. Retrying is right, but only if you notice — compare the returned feature count against resultType=hits from a cheap prior request, or against the count the previous successful run produced, and raise when the delta is implausible. Silent under-fetching is far more damaging than a visible failure, because it propagates into every downstream aggregate.

Idempotency is a precondition. Retrying a GetMap is safe; retrying a Transaction that inserts features is not, unless the write carries an idempotency key. Never attach a retry policy to a write path that has not been made idempotent first.

That tail is the practical reason to treat these two patterns as one design. A retry ladder tuned for a five-second blip behaves reasonably during a five-second blip and badly during a two-hour maintenance window, because every worker independently walks the whole ladder before giving up, and new workers keep arriving with a fresh budget. The breaker adds the missing piece of shared knowledge: one worker’s discovery that the endpoint is down becomes every worker’s. Until that exists, the honest description of a retry policy is that it converts a short outage into a manageable delay and a long outage into a slower, more expensive version of the same outage.

Configuration Reference#

# Per-host retry configuration. Each endpoint gets its own block, because the
# right numbers follow from what the server does per request, not from a house style.
endpoints:
  national_wfs:
    base_seconds: 2.0
    cap_seconds: 120.0          # a slow renderer needs long gaps between attempts
    budget_seconds: 1800.0      # a 200 MB stream is worth half an hour of patience
    connect_timeout: 10.0
    read_timeout: 600.0         # healthy p99 is ~400 s; below that we retry a success
    concurrency: 2              # per-host task limit, enforced by the orchestrator

  tile_wms:
    base_seconds: 0.5
    cap_seconds: 30.0
    budget_seconds: 180.0       # a tile is cheap to abandon and cheap to redo
    connect_timeout: 5.0
    read_timeout: 60.0
    concurrency: 16

  geocoder:
    base_seconds: 1.0
    cap_seconds: 60.0
    budget_seconds: 300.0
    connect_timeout: 5.0
    read_timeout: 20.0
    concurrency: 1              # the published limit is 1 rps on a shared key
    honour_retry_after: true

The concurrency field is not decoration. Retry policy and concurrency limit are two halves of one control: doubling the workers doubles the request rate that reaches an already-struggling endpoint, so a retry ladder tuned at concurrency 2 misbehaves at concurrency 16. Change them together, and record the endpoint’s published limit next to them so the next person can see whether the numbers are derived or invented.

Frequently Asked Questions#

Related#

• Implementing retry logic for slow WFS endpoints — the long-stream case in detail
• Respecting Retry-After headers from geocoding APIs — when the server sets the pace
• Circuit breakers for external WMS services — what to do when retrying stops helping
• Idempotency keys in spatial ETL — the precondition for retrying a write
• Timeout budgets and cancellation for geotasks — where the retry budget comes from

← Resilience & Failure Handling for GIS Pipelines