Configuring Failure Thresholds for WMS Endpoints

Pick a failure threshold for a WMS endpoint by deciding two numbers together: how many bad GetMap responses count as “unhealthy” and over what window you measure them. For flaky OGC servers, consecutive-failure counting trips too eagerly on isolated blips, while a sliding failure-rate window over the last N requests tracks sustained degradation far more faithfully. This page shows how to define what counts as a failure, size the window, and tune per-endpoint values, with a working pybreaker configuration wrapping a live tile fetch.

This is a companion how-to under Circuit Breakers for External WMS Services, which covers the broader state machine; here we drill into the single hardest knob to get right.

When to Use This Pattern#

Reach for deliberate threshold tuning when any of these hold:

• You call more than one WMS or WFS provider and they have visibly different reliability profiles.
• Your breaker trips during normal operation (false positives) or never trips during a real outage (false negatives).
• The endpoint returns HTTP 200 with truncated or malformed imagery that your current failure predicate ignores.
• You run many orchestration workers and need each to react to sustained failure rates, not one-off timeouts.
• Latency, not error codes, is your dominant failure mode and you want slow responses to count against health.

Two Counting Strategies#

Before any code, settle the counting model. The two dominant approaches trade sensitivity for stability.

Consecutive-failure counting trips after k failures land back-to-back. A single success resets the counter to zero. This is what the default pybreaker.CircuitBreaker(fail_max=k) implements. It is simple and cheap, but a WMS server that alternates success/failure under partial load can stay closed forever because each success wipes the tally. It also reacts slowly to a server that fails 40% of requests but never four times in a row.

Failure-rate counting over a sliding window trips when the failure ratio across the last N observed requests exceeds a threshold (say 50% of the last 20). This mirrors how production libraries such as Resilience4j and Istio model breaker health, and it captures intermittent degradation that consecutive counting misses. The cost is that you must retain a small ring buffer of recent outcomes and you need a minimum-samples floor so the rate is not computed from two requests.

For external WMS endpoints, prefer failure-rate counting. Spatial tile servers rarely fail cleanly; they wobble. A rate window over the last 20–50 requests, with a minimum of 10 samples before the rate is trusted, is a durable starting point.

What Counts as a Failure#

A breaker is only as good as its failure predicate. For WMS/WFS, count these as failures against the window:

• HTTP 5xx (500, 502, 503, 504) — server-side degradation, the canonical trip signal.
• Connect or read timeouts — a GetMap that streams a large extent and stalls is worse than a fast 500.
• Malformed tile payloads — HTTP 200 with a truncated PNG/TIFF, a body under a sane byte floor, or a Content-Type of application/vnd.ogc.se_xml (a WMS ServiceException masquerading as success).

Explicitly do not count HTTP 4xx (400, 401, 404). A malformed BBOX, an unknown layer, or a bad API key is a caller defect; counting it toward server health opens the breaker on inputs that will never succeed no matter how healthy the server is. This mirrors the exclusion guidance in the parent section and keeps the breaker measuring the provider, not your parameters.

Complete Working Example#

The example wraps a real GetMap call. It uses a custom sliding-window breaker so the failure-rate semantics are explicit, then feeds a strict failure predicate that inspects status, byte size, and content type.

from __future__ import annotations

import collections
import time
from dataclasses import dataclass, field

import requests


@dataclass
class SlidingWindowBreaker:
    """Failure-rate breaker over the last `window` observed requests."""

    window: int = 20
    min_samples: int = 10
    failure_rate: float = 0.5  # trip at 50% failures
    reset_timeout: float = 60.0  # seconds before a probe is allowed
    _outcomes: collections.deque = field(default_factory=lambda: collections.deque())
    _opened_at: float | None = None

    def allows_request(self) -> bool:
        if self._opened_at is None:
            return True
        if time.monotonic() - self._opened_at >= self.reset_timeout:
            return True  # half-open probe window (see the recovery guide)
        return False

    def record(self, failed: bool) -> None:
        self._outcomes.append(failed)
        while len(self._outcomes) > self.window:
            self._outcomes.popleft()
        if len(self._outcomes) < self.min_samples:
            return
        rate = sum(self._outcomes) / len(self._outcomes)
        if rate >= self.failure_rate:
            self._opened_at = time.monotonic()
        elif failed is False:
            self._opened_at = None  # healthy again, close the circuit


def is_wms_failure(resp: requests.Response) -> bool:
    """Failure predicate: 5xx, tiny bodies, or an OGC ServiceException."""
    if resp.status_code >= 500:
        return True
    ctype: str = resp.headers.get("Content-Type", "")
    if "se_xml" in ctype or "text/xml" in ctype:
        return True  # WMS error returned under HTTP 200
    if len(resp.content) < 1024:
        return True  # truncated or empty raster tile
    return False


def fetch_getmap(breaker: SlidingWindowBreaker, base_url: str, bbox: str) -> bytes:
    if not breaker.allows_request():
        raise RuntimeError("WMS breaker OPEN — endpoint marked unhealthy")

    params: dict[str, str] = {
        "SERVICE": "WMS",
        "VERSION": "1.3.0",
        "REQUEST": "GetMap",
        "LAYERS": "topo",
        "CRS": "EPSG:3857",
        "BBOX": bbox,
        "WIDTH": "512",
        "HEIGHT": "512",
        "FORMAT": "image/png",
    }
    try:
        resp = requests.get(base_url, params=params, timeout=(3.0, 15.0))
    except requests.exceptions.RequestException:
        breaker.record(failed=True)  # timeout / connection reset counts
        raise

    failed: bool = is_wms_failure(resp)
    breaker.record(failed=failed)
    if failed:
        raise RuntimeError(f"WMS GetMap failed: HTTP {resp.status_code}")
    return resp.content

Note the (3.0, 15.0) timeout tuple: a 3-second connect ceiling and a 15-second read ceiling. Omitting the read timeout is the single most common reason a breaker never trips — the request hangs instead of failing, so no failure is ever recorded. The predicate treats an XML Content-Type as failure because GeoServer and MapServer both emit a ServiceException body with a 200 status when a layer is temporarily unavailable.

Parameter & Option Reference#

Per-Endpoint Tuning#

A single global breaker averages together providers with unrelated health, so keep one breaker instance per host and tune each. A national mapping agency’s cached basemap might tolerate failure_rate=0.6 because occasional 503s under load are expected and cheap to retry. A slow dynamic-rendering WFS feeding a critical join might warrant failure_rate=0.35 and a longer reset_timeout so it is genuinely rested before probing. Store the tunables in per-endpoint config rather than code, and let the exponential backoff for API rate limits layer handle in-window transient 429s so the breaker only sees structural failure.

Verification & Testing#

Drive the breaker with a fake response sequence and assert the trip point. This unit sketch confirms a 50% rate over a 20-request window opens exactly when expected:

def test_breaker_trips_at_failure_rate() -> None:
    breaker = SlidingWindowBreaker(window=20, min_samples=10, failure_rate=0.5)

    # 9 clean requests: below min_samples, must stay closed
    for _ in range(9):
        breaker.record(failed=False)
    assert breaker.allows_request() is True

    # Now feed failures until the last-20 rate crosses 50%
    for _ in range(11):
        breaker.record(failed=True)
    assert breaker.allows_request() is False  # 11/20 failures -> OPEN


def test_four_xx_is_excluded() -> None:
    import types
    resp = types.SimpleNamespace(
        status_code=404, headers={"Content-Type": "image/png"}, content=b"x" * 2048
    )
    assert is_wms_failure(resp) is False  # client error, not server health

For a live smoke test, point fetch_getmap at a known-bad BBOX on a real GeoServer and confirm the XML ServiceException path records a failure, then curl the same endpoint with --max-time 15 to verify your read timeout matches the server’s slow-response ceiling.

Common Pitfalls#

• Counting 4xx as server failures. A bad LAYERS name or missing CRS opens the breaker permanently against inputs that can never succeed. Exclude all 4xx from the failure predicate.
• No read timeout. Without a read ceiling, a stalled GetMap hangs the worker and never records a failure, so the breaker stays closed through the entire outage.
• Consecutive counting on a wobbly server. A provider that alternates success and failure resets a consecutive counter every other request and never trips; use a sliding rate window instead.
• Sharing one breaker across endpoints. Averaging a healthy basemap with a failing WFS masks both signals; instantiate one breaker per host and store thresholds in per-endpoint config.

Related: Continue with half-open recovery for tile servers to tune the probe phase, layer this beneath exponential backoff for API rate limits, route tripped requests through dead-letter queues for failed geotasks, and keep deferred replays safe with idempotency keys in spatial ETL. All of these sit within Resilience & Failure Handling for GIS Pipelines.

← Back to Circuit Breakers for External WMS Services