Configuring Failure Thresholds for WMS Endpoints

Breaker thresholds are derived, not chosen. Three numbers control when a breaker trips — the window length, the minimum sample size within it, and the failure ratio — and all three follow from two measurements you already have or can take in an afternoon: how many requests your fleet sends to this endpoint per minute, and what the endpoint’s healthy latency distribution looks like. Set the window so it normally contains the sample floor, set the floor high enough that noise cannot trip it, and set the ratio at the point where the remaining successes are no longer useful to you.

When to Use This Pattern#

• You are configuring a breaker for the first time and the defaults (5 consecutive failures, 30 s cool-down) do not fit — which they usually do not for map services.
• The breaker is tripping too often, taking out an endpoint that was merely having a bad minute.
• The breaker never trips even though everyone agrees the endpoint was down, which almost always means a consecutive-failure counter or a sample floor set too high for the request rate.
• Request rates differ wildly between flows — a nightly mosaic at 400 requests a minute and a preview service at 4 — so one set of numbers cannot serve both.

Complete Working Example#

Start by measuring. This query, against whatever store holds your request metrics, gives the two inputs the thresholds depend on:

from __future__ import annotations

import statistics
from dataclasses import dataclass


@dataclass(frozen=True)
class EndpointProfile:
    """Measured behaviour of one endpoint under normal conditions."""

    requests_per_minute: float
    latency_p50: float
    latency_p99: float
    baseline_failure_ratio: float     # what "healthy" already looks like


def derive_thresholds(profile: EndpointProfile) -> dict[str, float]:
    """Turn a measured profile into breaker settings.

    The rules encoded here:
      * the window must normally contain at least MIN_SAMPLES requests;
      * the sample floor must be large enough that baseline noise cannot reach
        the ratio by chance;
      * the ratio sits above the baseline failure rate with real headroom;
      * a response slower than 3x the healthy p99 counts as a failure, because a
        renderer that has slowed by that much is not serving the pipeline.
    """
    min_samples = max(20, round(4 / max(profile.baseline_failure_ratio, 0.01)))
    # Window long enough to collect min_samples at the observed rate, and never
    # shorter than 15 s — below that, a single slow request dominates the window.
    window_seconds = max(15.0, 60.0 * min_samples / max(profile.requests_per_minute, 1.0))
    failure_ratio = min(0.6, max(0.25, profile.baseline_failure_ratio * 5))
    slow_threshold = profile.latency_p99 * 3

    return {
        "window_seconds": round(window_seconds),
        "min_samples": min_samples,
        "failure_ratio": round(failure_ratio, 2),
        "slow_call_seconds": round(slow_threshold, 1),
        # Cool-down from the p99: a renderer needs at least one full request
        # cycle of quiet before a probe tells you anything.
        "cooldown_seconds": max(30, round(profile.latency_p99 * 4)),
    }


NIGHTLY_MOSAIC = EndpointProfile(
    requests_per_minute=400.0, latency_p50=0.8, latency_p99=4.2,
    baseline_failure_ratio=0.004,
)
PREVIEW_SERVICE = EndpointProfile(
    requests_per_minute=4.0, latency_p50=1.1, latency_p99=6.0,
    baseline_failure_ratio=0.02,
)

# {'window_seconds': 150, 'min_samples': 1000, ...} — clamped in practice, see below
print(derive_thresholds(NIGHTLY_MOSAIC))
print(derive_thresholds(PREVIEW_SERVICE))

The derivation deliberately produces different answers for the two profiles, and the difference is the point. The nightly mosaic sends a hundred times more requests, so it can afford a short window and still gather a large sample; the preview service cannot, and giving it the mosaic’s 15-second window would mean tripping on two failures out of three. Where the formula produces something impractical — a min_samples of 1000 for a very reliable endpoint — clamp it, but clamp it consciously and write down why.

Parameter & Option Reference#

Verification & Testing#

Threshold configuration is testable without an outage: replay a recorded outcome sequence through the breaker and assert where it trips.

import itertools


def replay(breaker, outcomes: list[bool]) -> list[str]:
    """Feed a sequence of successes/failures and record the state after each."""
    states = []
    for ok in outcomes:
        try:
            breaker.call(lambda: True if ok else (_ for _ in ()).throw(RuntimeError("boom")))
        except Exception:
            pass
        states.append(breaker.state().value)
    return states


def test_baseline_noise_does_not_trip(breaker) -> None:
    # 2% failures, evenly spread — this is what healthy looks like.
    noise = list(itertools.chain.from_iterable([[True] * 49 + [False] for _ in range(4)]))
    assert "open" not in replay(breaker, noise)


def test_real_outage_trips_within_one_window(breaker) -> None:
    outage = [True] * 30 + [False] * 30
    states = replay(breaker, outage)
    assert states[-1] == "open"
    # And it must trip promptly: no more than min_samples failures wasted.
    assert states.index("open") <= 30 + breaker.cfg.min_samples

In production, the number that tells you whether the thresholds are right is the count of breaker openings per week alongside the count of confirmed upstream incidents. If the breaker opened eleven times and there were two incidents, the ratio is too low or the sample floor too small. If it opened zero times and there were two incidents, the ratio is too high — or, far more likely, slow renders are not being counted as failures.

Reading that chart is also how you set the slow-call threshold itself. The gap between the two clusters is the useful signal: pick a value inside the gap, not at the edge of the healthy cluster. A threshold at six seconds — just above the healthy p99 — would classify the tail of the healthy distribution as failures and trip the breaker on a busy but working renderer. Twelve seconds sits in empty space, which means no healthy request can reach it and no degraded request can avoid it.

Common Pitfalls#

• Using consecutive failures. A “5 consecutive failures” rule never fires against an endpoint failing 40% of the time, because a success resets the counter roughly every other request. A rolling ratio with a sample floor is barely more code and is correct in the cases that matter.
• Copying thresholds between flows. The same endpoint hit at 400 req/min and at 4 req/min needs different windows. Configuration keyed only by endpoint, not by caller, is how a preview service ends up with a breaker that trips constantly.
• Not counting slow calls as failures. The dominant WMS failure mode is degradation, not errors. A breaker blind to latency stays closed through the outage that actually hurts you.
• A sample floor larger than the window can hold. If min_samples is 200 and the window only ever contains 40 requests, the ratio is never evaluated and the breaker is decorative. Derive them together, and assert the relationship in a test.
• Ignoring the baseline. Some public services simply return 502 for 3% of requests, forever. A ratio set at 0.05 against that baseline trips on a normal Tuesday. Measure the healthy failure rate before choosing a threshold above it.

Frequently Asked Questions#

Related#

• Circuit breakers for external WMS services — the breaker these numbers configure
• Half-open recovery for tile servers — what happens after the cool-down
• Exponential backoff for API rate limits — the retry layer whose outcomes feed these counters
• Measuring tile generation latency percentiles — where the p99 in the derivation comes from

← Circuit Breakers for External WMS Services