Implementing Retry Logic for Slow WFS Endpoints

Retrying a slow Web Feature Service is not the same problem as retrying an API call, because the failure usually arrives after the server has already done most of the work. The fix is to stop treating one GetFeature as one unit: page the request with startIndex and count, retry each page independently against a read timeout derived from the endpoint’s healthy p99, and keep the page cursor in the task’s state so a resumed run continues from the last complete page instead of restarting a twenty-minute stream. Retry the page, never the dataset.

When to Use This Pattern#

• The full extract takes longer than about 60 seconds, which is where a single-shot request starts colliding with proxy and gateway timeouts you do not control.
• The service supports paging — WFS 2.0.0 with startIndex and count, or maxFeatures on a 1.1.0 server. Without paging, the pattern degrades to whole-request retries and the advice is to reduce the BBOX instead.
• Failures cluster at the end of the stream, the classic signature of a gateway timeout: successful small requests, consistent failures on large ones.
• The extract is a scheduled dependency — a nightly refresh that must be complete before downstream tasks fire — so an incomplete result is worse than a late one.

If the service returns the whole collection in under ten seconds, skip all of this. A plain exponential backoff around a single request is simpler, and simpler is what you want in a path that rarely fails.

Complete Working Example#

The reader below pages through a WFS 2.0.0 collection, retries each page with jittered backoff, and yields the raw GML/GeoJSON bytes for the caller to persist. The page cursor is returned to the caller rather than hidden inside the loop, so a task that dies mid-extract can resume.

from __future__ import annotations

import random
import time
from dataclasses import dataclass
from typing import Iterator, Optional

import httpx

# Healthy p99 for a 200 000-feature page on this service is ~150 s. The read
# timeout sits above it: below the healthy p99 we would retry successes forever.
PAGE_READ_TIMEOUT = 240.0
PAGE_CONNECT_TIMEOUT = 10.0


@dataclass(frozen=True)
class WfsPage:
    start_index: int
    count: int
    body: bytes


def _page_params(type_name: str, start_index: int, count: int, srs: str) -> dict[str, str]:
    return {
        "service": "WFS",
        "version": "2.0.0",
        "request": "GetFeature",
        "typeNames": type_name,
        "outputFormat": "application/json",
        "srsName": srs,               # never let the server choose: axis order bites
        "startIndex": str(start_index),
        "count": str(count),
        "sortBy": "gml_id",           # paging without a sort is not stable
    }


def fetch_page(
    client: httpx.Client,
    url: str,
    type_name: str,
    start_index: int,
    count: int,
    srs: str = "urn:ogc:def:crs:EPSG::25832",
    max_attempts: int = 5,
) -> bytes:
    """One page, retried with full jitter. Raises when the attempts are spent."""
    last_error: Optional[Exception] = None
    for attempt in range(max_attempts):
        try:
            response = client.get(
                url,
                params=_page_params(type_name, start_index, count, srs),
                timeout=httpx.Timeout(PAGE_CONNECT_TIMEOUT, read=PAGE_READ_TIMEOUT),
            )
            if response.status_code in (429, 500, 502, 503, 504):
                raise httpx.HTTPStatusError("retryable", request=response.request,
                                            response=response)
            response.raise_for_status()
            if b"ExceptionReport" in response.content[:2048]:
                # A WFS reports failure with HTTP 200 and an XML exception body.
                raise RuntimeError(response.content[:300].decode("utf-8", "replace"))
            return response.content
        except (httpx.TransportError, httpx.HTTPStatusError, RuntimeError) as exc:
            last_error = exc
            time.sleep(random.uniform(0.0, min(60.0, 2.0 * (2 ** attempt))))
    raise RuntimeError(f"page at startIndex={start_index} failed") from last_error


def paged_features(
    client: httpx.Client,
    url: str,
    type_name: str,
    page_size: int = 200_000,
    resume_from: int = 0,
) -> Iterator[WfsPage]:
    """Yield pages until the service returns fewer features than asked for."""
    start = resume_from
    while True:
        body = fetch_page(client, url, type_name, start, page_size)
        yield WfsPage(start_index=start, count=page_size, body=body)
        # A short page is the end of the collection — the only reliable terminator,
        # since numberMatched is optional and frequently absent.
        if body.count(b'"type":"Feature"') < page_size:
            return
        start += page_size

Parameter & Option Reference#

Verification & Testing#

Two properties are worth asserting: the pages tile the collection exactly once, and a resumed run produces the same set as an uninterrupted one.

def test_pages_cover_the_collection_once(wfs_stub, tmp_path) -> None:
    with httpx.Client() as client:
        pages = list(paged_features(client, wfs_stub.url, "topo:buildings", page_size=1000))

    ids = [fid for page in pages for fid in feature_ids(page.body)]
    assert len(ids) == len(set(ids)), "paging returned duplicates — check sortBy"
    assert len(ids) == wfs_stub.total_features

    # Resuming from page 2 must yield exactly the tail, with no gap at the seam.
    with httpx.Client() as client:
        tail = list(paged_features(client, wfs_stub.url, "topo:buildings",
                                   page_size=1000, resume_from=2000))
    tail_ids = [fid for page in tail for fid in feature_ids(page.body)]
    assert tail_ids == ids[2000:]

Against the real service, the cheap check is a count comparison before and after. resultType=hits returns the match count without the payload, and ogrinfo -so -al extract.gpkg reports what actually landed:

# What the server says it has, in one cheap request…
curl -s "$WFS?service=WFS&version=2.0.0&request=GetFeature&typeNames=topo:buildings&resultType=hits" \
  | grep -o 'numberMatched="[0-9]*"'

# …against what the pipeline wrote.
ogrinfo -so -al extract.gpkg | grep "Feature Count"

The cursor is what makes the whole thing resumable, and it needs somewhere durable to live. Writing it into the same transaction that persists the page keeps the two in step: if the page landed, the cursor advanced; if it did not, the cursor did not. Storing the cursor in the orchestrator’s task state instead is tempting and usually wrong, because task state is discarded when a flow run is deleted or a worker is replaced, and the cursor is then silently reset to zero on the next run — which looks exactly like a successful full re-extract until someone notices the egress bill.

Common Pitfalls#

• Paging without sortBy. The service is free to return rows in whatever order the query planner produces, and it may re-plan between pages. Without a stable sort you get duplicates at some seams and gaps at others, and the totals still look plausible.
• Letting the server pick the CRS. A WFS 2.0.0 that defaults to urn:ogc:def:crs:EPSG::4326 returns latitude first, while the same code path against a 1.1.0 endpoint returns longitude first. Requesting an explicit srsName and then validating the coordinate system before ETL is what stops a whole extract landing in the Gulf of Guinea.
• Treating a short page as an error. A page returning fewer features than requested is the end of the collection, not a truncation — but a page returning zero features when numberMatched said otherwise usually is a truncation. Distinguish them, or a single flaky page silently shortens the dataset.
• Retrying the extract instead of the page. A whole-extract retry after page five re-fetches pages zero to four at full cost, and it makes the endpoint’s load proportional to the square of its failure rate.
• Holding all pages in memory before writing. A 1.2 M-feature collection at 200 000 per page is several gigabytes of JSON. Persist each page as it arrives and let the loader stream from disk, the same discipline used for streaming large GeoPackage loads into PostGIS.

One more failure deserves its own line, because it is the one that survives every other defence: a service that answers a paged request with the first page regardless of startIndex. Some older deployments silently ignore parameters they do not implement, and the loop above then fetches the same 200 000 features forever, terminating only when the attempt budget runs out or the disk fills. The guard is one assertion at the seam — the first feature id of page n must differ from the first feature id of page n−1 — and it costs nothing compared to the afternoon it saves.

Frequently Asked Questions#

Related#

• Exponential backoff for API rate limits — the retry policy this recipe specialises
• Respecting Retry-After headers from geocoding APIs — when the server dictates the pace
• Circuit breakers for external WMS services — stopping the retries when the endpoint is genuinely down
• Timeout budgets and cancellation for geotasks — where the 240-second read timeout comes from

← Exponential Backoff for API Rate Limits