Circuit Breakers for External WMS Services

In short: a circuit breaker turns one worker’s discovery that a map service is down into every worker’s knowledge, so the fleet stops paying full price for failures. For WMS and tile endpoints the breaker’s state must be shared across workers and scoped per endpoint, because the thing being protected is a single upstream renderer that all your workers point at.

Retries handle a blip. A breaker handles an outage. The distinction matters more for map services than for most APIs because of what a request costs the server: a GetMap at 2048×2048 over a vector layer with labels is seconds of CPU on the far side, and a fleet of sixty tile workers, each retrying five times with a jittered ladder, can hold a struggling renderer at its knees long after it would otherwise have recovered. The breaker exists to convert that from a self-sustaining outage into a short one.

Prerequisites & Architecture Baseline#

• Shared state that every worker can read and write: Redis 7+, or a small PostgreSQL table if Redis is not already in the stack
• One breaker instance per endpoint, keyed by host plus layer where the layers have independent backends
• A working retry policy underneath it — the breaker consumes the outcome of retries, it does not replace them
• A fallback path decided in advance: cached tiles, a lower-resolution basemap, or a clean failure. A breaker with no fallback only changes how the pipeline fails
• Metrics on state transitions, so "the breaker was open for 40 minutes last night" is answerable
• Per-endpoint concurrency limits, since the breaker's thresholds are meaningless if the request rate is unbounded

Core Principles#

1. The state must be shared, not per process. A breaker held in a module-level variable protects one worker process. With sixty workers that is sixty independent discoveries of the same outage, each costing a full retry ladder, which is roughly the situation the breaker was supposed to fix. Redis with a short TTL is the usual answer; the important property is that opening the breaker is visible to everyone within a second or two.

2. Scope the breaker to the failure domain, not to the code path. One breaker per host is right when a host is one renderer. It is wrong when a single hostname fronts several independent backends — a national portal where the orthophoto layer and the cadastral layer are different systems. Breaking on the hostname then takes out a healthy layer because a different one is unwell. Key the breaker on whatever unit actually fails together.

3. Count failures over a rolling window, not consecutively. “Five consecutive failures” never trips on an endpoint that fails 40% of the time, because a success keeps resetting the count — and a 40% failure rate is an outage by any practical definition. A rolling window (failures and successes in the last N seconds, with a minimum sample size) trips correctly and is barely harder to implement.

4. Open fast, close slowly. Opening on a 50% failure rate over thirty seconds costs you very little if you are wrong; the breaker half-opens shortly and finds out. Closing eagerly is the expensive mistake — a renderer that has just come back is fragile, and slamming sixty workers into it re-opens the breaker and delays real recovery. Require several consecutive successful probes before closing.

5. Half-open means one probe, not a trickle. In the half-open state exactly one request should be permitted through at a time. Allowing 10% of traffic through as a “gentle test” is still six concurrent renders against a service that just failed. The single-probe rule is what makes recovery detection cheap, and it needs a lock, not a probability. See half-open recovery for tile servers.

6. Decide what an open breaker returns. Failing fast is legitimate — it dead-letters the tile and the batch completes. Serving a stale cached tile is often better for a pipeline whose output is a basemap. Serving nothing but marking the tile as “not yet rendered” is best when a hole is more honest than stale data. The choice belongs to the pipeline, and falling back to cached tiles when a breaker opens covers the most common of the three.

It is worth being explicit about what the breaker is not protecting. It does not protect your pipeline from producing incomplete output — an open breaker still means tiles are not being rendered. It does not make the upstream service recover faster in any direct sense, though by removing load it often does so indirectly. And it does not replace a conversation with whoever operates the endpoint, if you are generating enough traffic to matter to them. What it does, precisely, is bound the cost of an outage: without it, the cost of a thirty-minute upstream failure is thirty minutes of your entire fleet’s capacity plus thirty minutes of their entire renderer’s capacity; with it, the cost is a handful of probes and a batch that finishes early with a known set of gaps.

Production Implementation#

The breaker below keeps its counters in Redis so the whole fleet shares one view. It uses a fixed-size rolling window of recent outcomes rather than a consecutive-failure counter, and it takes a lock for the half-open probe.

from __future__ import annotations

import time
from dataclasses import dataclass
from enum import Enum
from typing import Callable, TypeVar

import redis

T = TypeVar("T")


class State(str, Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class BreakerOpen(RuntimeError):
    """The endpoint is known-bad; the call was not attempted."""


@dataclass(frozen=True)
class BreakerConfig:
    window_seconds: int = 30
    min_samples: int = 20          # never trip on 2 of 3
    failure_ratio: float = 0.5
    cooldown_seconds: int = 30
    probes_to_close: int = 3


class SharedBreaker:
    """A breaker whose state lives in Redis, so all workers see it at once."""

    def __init__(self, r: redis.Redis, endpoint: str, cfg: BreakerConfig = BreakerConfig()):
        self.r = r
        self.cfg = cfg
        # Key on the failure domain: host + layer, not just host.
        self.k_state = f"cb:{endpoint}:state"
        self.k_ok = f"cb:{endpoint}:ok"
        self.k_fail = f"cb:{endpoint}:fail"
        self.k_probe = f"cb:{endpoint}:probe"
        self.k_closed_probes = f"cb:{endpoint}:probes_ok"

    def _record(self, key: str) -> None:
        # A counter with a TTL is a crude but adequate rolling window: it forgets
        # everything older than window_seconds without storing per-request state.
        pipe = self.r.pipeline()
        pipe.incr(key)
        pipe.expire(key, self.cfg.window_seconds)
        pipe.execute()

    def state(self) -> State:
        raw = self.r.get(self.k_state)
        if raw is None:
            return State.CLOSED
        value = raw.decode()
        if value == State.OPEN and self.r.ttl(self.k_state) <= 0:
            return State.HALF_OPEN
        return State(value)

    def _trip(self) -> None:
        self.r.set(self.k_state, State.OPEN.value, ex=self.cfg.cooldown_seconds)
        self.r.delete(self.k_ok, self.k_fail, self.k_closed_probes)

    def call(self, fn: Callable[[], T]) -> T:
        state = self.state()

        if state is State.OPEN:
            raise BreakerOpen(f"breaker open for {self.k_state}")

        if state is State.HALF_OPEN:
            # Exactly one probe at a time, fleet-wide. NX makes the lock atomic.
            if not self.r.set(self.k_probe, "1", nx=True, ex=30):
                raise BreakerOpen("another worker is probing")

        try:
            result = fn()
        except Exception:
            self._record(self.k_fail)
            if state is State.HALF_OPEN:
                self._trip()                    # one bad probe re-opens immediately
                self.r.delete(self.k_probe)
            else:
                ok = int(self.r.get(self.k_ok) or 0)
                fail = int(self.r.get(self.k_fail) or 0)
                total = ok + fail
                if total >= self.cfg.min_samples and fail / total >= self.cfg.failure_ratio:
                    self._trip()
            raise

        self._record(self.k_ok)
        if state is State.HALF_OPEN:
            self.r.delete(self.k_probe)
            if self.r.incr(self.k_closed_probes) >= self.cfg.probes_to_close:
                self.r.delete(self.k_state, self.k_closed_probes)   # fully closed
        return result

Step-by-Step Walkthrough#

1. Key the breaker on the failure domain. cb:{host}:{layer} when layers have independent backends, cb:{host} when they do not. Getting this wrong is the difference between a breaker that protects a fleet and one that takes out working layers because a neighbour is unwell.
2. Record every outcome, success and failure alike. The ratio is what trips the breaker, so successes are as important as failures. A counter with a TTL approximates a rolling window cheaply — it forgets the past without storing a per-request log.
3. Require a minimum sample before tripping. min_samples = 20 stops the breaker opening on the first three requests after a deploy, when two coincidental failures would otherwise look like a 67% failure rate.
4. Store the open state with the cool-down as its TTL. When the key expires, the breaker is half-open by construction — no timer, no background job, and every worker computes the same state from the same key.
5. Take a lock for the probe. SET … NX EX 30 grants exactly one worker permission to try. Everyone else is refused as though the breaker were open, which is exactly the behaviour you want.
6. Re-open on a single failed probe. A failed probe means the endpoint is still unwell; there is no value in a second opinion, and the cost of getting it is another render on a struggling server.
7. Close only after several clean probes. Three is a reasonable default. The counter is reset whenever the breaker trips, so a flapping endpoint never accumulates its way to closed.

Edge Cases & Failure Recovery#

Redis itself is unavailable. The breaker must fail open — that is, allow traffic — rather than block the pipeline because its coordination store is down. Wrap the Redis calls so any connection error is treated as “state unknown, proceed”, and alert on it. A breaker that halts the pipeline when its own infrastructure hiccups is a larger outage than the one it prevents.

A WMS that returns 200 with an exception body. The classic OGC behaviour: the HTTP layer sees success, so the breaker records a success and never trips while every tile in the mosaic is an XML error document. The breaker’s notion of failure must be the pipeline’s notion — content type checked, image decoded — not the transport’s. This is the same classification problem described in exponential backoff for API rate limits, and the breaker inherits whatever answer you gave there.

Slow success, not failure. Many map services degrade by getting slower rather than by erroring. If the read timeout is generous, the breaker sees successes and stays closed while throughput collapses. Feed latency into the breaker too: count a response slower than some multiple of the healthy p99 as a failure for breaker purposes, even though the pipeline keeps its tile.

A breaker that never closes because nothing probes it. If the pipeline finished its batch while the breaker was open, no work arrives to trigger a probe, and the breaker sits open until the next run — which then starts with a cold, pessimistic state. Either let the state key expire fully after a longer interval, or run a tiny scheduled probe flow whose only job is to exercise half-open transitions.

Two pipelines sharing one endpoint but not one breaker. A nightly mosaic flow and an on-demand preview service both call the same WMS, each with its own Redis prefix. Neither learns from the other’s failures, and the endpoint sees twice the load it should during an outage. If the failure domain is shared, the breaker key must be shared too — which means agreeing on a naming convention across teams, not just across modules.

Thundering herd on close. The instant the breaker closes, sixty workers resume simultaneously against a renderer that has served three probes. Ramp instead: allow a fraction of workers through for the first few seconds, or rely on the per-endpoint concurrency limit to do it for you. The limit is doing double duty here, which is another reason to treat concurrency and breaker configuration as one design.

Configuration Reference#

There is one setting not in the table because it is not a number: what an open breaker does. Failing fast pushes the tile into the dead-letter queue for a later re-drive, which is right for a batch pipeline whose output is consumed tomorrow. Serving a cached tile keeps a live basemap usable at the cost of freshness, which is right when someone is looking at a map right now. Both are defensible; choosing neither, and letting the breaker’s BreakerOpen exception propagate as an unhandled error, is what turns a protective mechanism into an outage of your own making.

Frequently Asked Questions#

Related#

• Configuring failure thresholds for WMS endpoints — choosing the window, ratio and sample floor
• Half-open recovery for tile servers — the probe protocol in detail
• Falling back to cached tiles when a breaker opens — what to serve instead
• Exponential backoff for API rate limits — the retry layer underneath the breaker
• Dead-letter queues for failed geotasks — where a fast failure lands

← Resilience & Failure Handling for GIS Pipelines