Geospatial Orchestration Hub

Implementing Retry Logic for Slow WFS Endpoints

Implementing retry logic for slow WFS endpoints requires wrapping OGC-compliant HTTP requests in an exponential backoff strategy that respects server-side spatial processing times and orchestration framework policies. In Python data pipelines, this means decoupling connection and read timeouts, applying jitter to progressive delays, and validating partial GML/GeoJSON payloads before marking a request successful. By combining urllib3 HTTP-level resilience with framework-native task resumption, you prevent transient network drops or thread exhaustion from cascading into full pipeline failures.

Why WFS Endpoints Require Specialized Retry Handling

Web Feature Service (WFS) endpoints operate fundamentally differently than standard REST APIs. A GetFeature request often triggers server-side spatial indexing, coordinate reference system (CRS) transformations, topology validation, and GML serialization before the first byte streams. This architecture introduces two failure modes that generic retry configurations consistently miss:

  • Silent Connection Drops: Reverse proxies (nginx, HAProxy) or cloud load balancers terminate idle connections during long-running spatial queries, returning 502 or 504 after 60–120 seconds.
  • Partial Payload Delivery: The server processes the query successfully but times out during XML/GML serialization, returning a truncated document that fails downstream schema validation.

To handle these, you must enforce progressive delays, validate payload integrity, and align retry policies with spatial query characteristics. This approach extends the core principles of Exponential Backoff for API Rate Limits, adapting jitter and server-aware scaling to geospatial workloads rather than simple rate-limit scenarios.

Production-Ready Implementation

The following implementation uses requests backed by urllib3.util.Retry for low-level HTTP resilience. It builds a reusable session, enforces explicit timeout separation, filters retryable status codes, and validates XML structure before returning.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import logging
from lxml import etree
from io import BytesIO

logger = logging.getLogger(__name__)

def create_wfs_session(max_retries: int = 4, backoff_factor: float = 1.0) -> requests.Session:
    """
    Builds a requests session with exponential backoff and jitter.
    Respects Retry-After headers and isolates connection/read timeouts.
    """
    session = requests.Session()
    retry_strategy = Retry(
        total=max_retries,
        backoff_factor=backoff_factor,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET", "POST"],
        respect_retry_after_header=True,
        raise_on_status=False,  # Let orchestration handle final failure
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session

def validate_wfs_payload(response: requests.Response) -> bool:
    """Checks for truncated GML/XML and valid HTTP 200."""
    if response.status_code != 200:
        return False
    try:
        # Fast parse to detect truncation or malformed XML
        parser = etree.XMLParser(recover=False, huge_tree=True)
        etree.parse(BytesIO(response.content), parser)
        return True
    except etree.XMLSyntaxError:
        logger.warning("Truncated or malformed WFS payload detected.")
        return False

def fetch_wfs_features(
    session: requests.Session,
    url: str,
    params: dict,
    timeout: tuple = (10.0, 120.0)
) -> bytes:
    """Executes request with retry and payload validation."""
    response = session.get(url, params=params, timeout=timeout)
    if not validate_wfs_payload(response):
        # Raise to trigger framework-level retry
        raise ValueError("Invalid or truncated WFS response payload.")
    return response.content

Framework Integration: Prefect & Dagster

Orchestration frameworks handle task-level resumption differently. Align HTTP-level retries with framework retry policies to avoid duplicate backoff calculations and connection pool starvation.

Prefect: Use the @task decorator with retries and retry_delay_seconds. Prefect’s retry mechanism operates at the task level, so set raise_on_status=False in urllib3 and let Prefect catch the ValueError from payload validation.

from prefect import task

@task(retries=3, retry_delay_seconds=[10, 30, 60])
def run_wfs_query_prefect(url: str, params: dict) -> bytes:
    session = create_wfs_session(max_retries=2)
    return fetch_wfs_features(session, url, params)

Dagster: Use the @op decorator with RetryPolicy. Dagster’s native policy supports exponential backoff and integrates cleanly with the HTTP retry layer.

from dagster import op, RetryPolicy
from dagster._core.definitions.retry_policy import Backoff

@op(
    retry_policy=RetryPolicy(max_retries=3, delay=2, backoff=Backoff.EXPONENTIAL)
)
def run_wfs_query_dagster(url: str, params: dict) -> bytes:
    session = create_wfs_session(max_retries=2)
    return fetch_wfs_features(session, url, params)

Tuning Timeouts and Backoff for Spatial Workloads

Generic API defaults fail for WFS because spatial operations are heavily CPU- and I/O-bound. Follow these tuning guidelines to stabilize throughput:

  • Separate Connect vs. Read Timeouts: Set connect_timeout low (5–10s) to fail fast on routing or DNS resolution issues. Set read_timeout high (60–180s) to accommodate server-side geometry processing. Refer to the official urllib3 Retry API Reference for precise backoff mechanics and jitter behavior.
  • Backoff Factor Selection: A backoff_factor of 1.0 yields delays of ~1s, 2s, 4s, 8s. Avoid aggressive multipliers (>2.0) that can starve connection pools during bulk spatial ETL.
  • Respect Retry-After: Enterprise WFS providers often return Retry-After headers during thread pool exhaustion. Enabling respect_retry_after_header=True prevents hammering degraded servers and reduces 429/503 loops.
  • Payload Validation is Mandatory: HTTP 200 does not guarantee a complete feature collection. Always parse the response against the OGC Web Feature Service 2.0 Specification schema or use lxml to catch truncation before downstream steps consume partial geometries.

Scaling Across the Pipeline

When orchestrating hundreds of spatial requests, HTTP retries alone are insufficient. Implement connection pool limits (pool_connections, pool_maxsize on HTTPAdapter), enforce idempotent request parameters, and add circuit breakers for endpoints that consistently return 5xx errors. For broader strategies on handling transient failures across geospatial ETL workflows, refer to Resilience & Failure Handling for GIS Pipelines. Properly implemented, this retry architecture transforms flaky spatial endpoints into reliable data sources without masking underlying infrastructure bottlenecks.