Geospatial Orchestration Hub

Half-Open Recovery for Tile Servers

Half-open is the state where a breaker asks “is it back yet?” — and the whole art is asking cheaply. Permit exactly one in-flight probe across the entire fleet, make that probe the least expensive request the endpoint can serve, require several consecutive successes before closing, and ramp traffic back rather than releasing sixty workers at once. A recovering tile server is fragile: it has cold caches, a cold connection pool and often a backlog to work through, and the fastest way to knock it over again is to treat “one success” as “recovered”.

When to Use This Pattern

  • A shared breaker exists and you are implementing or fixing its recovery path, which is where most breaker bugs live.
  • The endpoint is expensive per request — a rendering WMS, a tile server that reads from object storage on a miss — so the cost of probing matters.
  • Several workers share the breaker state, meaning “one probe” has to be enforced with a lock rather than assumed.
  • You have seen the breaker flap: open, close, open again within a minute. That is nearly always a recovery path that closes too eagerly.

Complete Working Example

The probe protocol below uses a Redis SET NX EX as the fleet-wide permit, a deliberately cheap probe request, and a consecutive-success counter that any failure resets.

from __future__ import annotations

import time
from dataclasses import dataclass
from typing import Optional

import httpx
import redis


@dataclass(frozen=True)
class ProbeConfig:
    lock_seconds: int = 20            # must exceed the probe's own timeout
    probes_to_close: int = 3
    probe_timeout: float = 8.0
    ramp_seconds: int = 30            # partial traffic for this long after closing


class HalfOpenProbe:
    def __init__(self, r: redis.Redis, endpoint: str, cfg: ProbeConfig = ProbeConfig()):
        self.r = r
        self.cfg = cfg
        self.k_permit = f"cb:{endpoint}:probe_permit"
        self.k_streak = f"cb:{endpoint}:probe_streak"
        self.k_ramp = f"cb:{endpoint}:ramp_until"

    def acquire(self) -> bool:
        """Exactly one worker fleet-wide gets True. NX is what makes this atomic."""
        return bool(self.r.set(self.k_permit, "1", nx=True, ex=self.cfg.lock_seconds))

    def release(self) -> None:
        self.r.delete(self.k_permit)

    def run(self, client: httpx.Client, base_url: str, layer: str) -> Optional[bool]:
        """Probe once. True = success, False = still failing, None = not our turn."""
        if not self.acquire():
            return None
        try:
            # The cheapest meaningful request: a 1x1 GetMap over a tiny extent.
            # GetCapabilities is cheaper still but is often served from a cache and
            # succeeds while the renderer itself is dead — it proves nothing.
            response = client.get(
                base_url,
                params={
                    "service": "WMS", "version": "1.3.0", "request": "GetMap",
                    "layers": layer, "styles": "",
                    "crs": "EPSG:3857",
                    "bbox": "0,0,100,100",       # 100 m square: no real work to render
                    "width": "1", "height": "1",
                    "format": "image/png",
                },
                timeout=httpx.Timeout(4.0, read=self.cfg.probe_timeout),
            )
            ok = (
                response.status_code == 200
                and response.headers.get("content-type", "").startswith("image/")
                and len(response.content) > 0
            )
        except httpx.HTTPError:
            ok = False
        finally:
            self.release()

        if not ok:
            self.r.delete(self.k_streak)        # any failure resets the streak
            return False

        streak = self.r.incr(self.k_streak)
        self.r.expire(self.k_streak, 300)
        if streak >= self.cfg.probes_to_close:
            self.r.delete(self.k_streak)
            # Closing starts a ramp window rather than releasing the full fleet.
            self.r.set(self.k_ramp, str(time.time() + self.cfg.ramp_seconds),
                       ex=self.cfg.ramp_seconds)
        return True

    def ramp_fraction(self) -> float:
        """0..1 — the share of workers allowed through during the ramp window."""
        raw = self.r.get(self.k_ramp)
        if raw is None:
            return 1.0
        remaining = float(raw) - time.time()
        if remaining <= 0:
            return 1.0
        return max(0.1, 1.0 - remaining / self.cfg.ramp_seconds)

Two choices in there are worth defending. The probe is a GetMap rather than a GetCapabilities because capabilities documents are almost always cached in front of the renderer — they will happily return 200 while the thing that draws maps is dead, which produces a breaker that closes into an outage. And the permit’s TTL must be longer than the probe’s own timeout: if the lock expires while a probe is still in flight, a second worker acquires it and you have two probes, which is the exact situation the lock exists to prevent.

One permit, six workersSix workers reach the half-open breaker. One acquires the permit and sends a probe. Five are refused as if the breaker were open. After three consecutive successful probes the breaker closes into a thirty-second ramp.worker 1worker 2worker 3worker 4worker 5worker 6SET NX permitone winner1×1 GetMap probe — 1 request100 m bbox, nothing to renderothers refused — BreakerOpenzero extra load on the renderer3 clean probes → closethen a 30 s ramp, not a flood
The refusal path is as important as the probe: five of six workers get an immediate exception and move on to other work instead of queueing behind a recovering server.
Releasing the fleet: step change against rampClosing the breaker with a step change puts the full fleet on a cold server at once. A thirty-second ramp raises the allowed fraction from ten per cent to one hundred per cent gradually.100%0close+15 s+30 sstep — 60 renders arrive on a cold cacheramp — the server warms as it goesThe ramp costs thirty seconds of reduced throughput and routinely saves a second outage.
Cold caches are the reason. A tile server that has been idle for two minutes has to re-read from object storage for almost every request it receives.

Parameter & Option Reference

Parameter Type Default Spatial notes
lock_seconds int 20 Must exceed probe_timeout. A lock that expires mid-probe permits a second probe, defeating the design.
probes_to_close int 3 Consecutive, and reset by any failure. Two is too eager for a renderer with cold caches; five is slow to recover.
probe_timeout float 8.0 Shorter than the normal read timeout. A probe that hangs for three minutes tells you nothing you did not already know.
probe request GetMap 1×1 — A tiny bbox at 1×1 pixels. Cheap for the server, and unlike GetCapabilities it exercises the renderer.
ramp_seconds int 30 Traffic is scaled from 10% to 100% across this window after closing.
streak TTL 300 s — A streak that stalls half-finished should expire rather than persist into the next incident.

Verification & Testing

The test that catches the real bug is the concurrent one: many workers, one probe.

import threading


def test_only_one_probe_escapes(fake_redis, probe_server) -> None:
    probe = HalfOpenProbe(fake_redis, endpoint="wms.example:ortho")
    barrier = threading.Barrier(12)
    results: list[Optional[bool]] = []
    lock = threading.Lock()

    def attempt() -> None:
        barrier.wait()                       # maximise the chance of a real race
        with httpx.Client() as client:
            outcome = probe.run(client, probe_server.url, layer="ortho")
        with lock:
            results.append(outcome)

    threads = [threading.Thread(target=attempt) for _ in range(12)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert probe_server.request_count == 1, "more than one probe reached the server"
    assert results.count(None) == 11          # the other eleven were refused


def test_failure_resets_the_streak(fake_redis, flapping_server) -> None:
    probe = HalfOpenProbe(fake_redis, endpoint="wms.example:ortho")
    with httpx.Client() as client:
        assert probe.run(client, flapping_server.ok_url, "ortho") is True
        assert probe.run(client, flapping_server.ok_url, "ortho") is True
        assert probe.run(client, flapping_server.fail_url, "ortho") is False
        # Two successes then a failure must NOT be one success away from closing.
        assert fake_redis.get("cb:wms.example:ortho:probe_streak") is None

In production, the metric worth graphing is time-in-state. A healthy recovery looks like a single open period followed by a half-open period of a minute or two and then closed. A pathological one looks like a comb — open, half-open, open, half-open — which means the probes are succeeding against something that is not really ready, and probes_to_close or the cool-down needs raising.

Healthy recovery against flappingThe healthy timeline shows one open period, a short half-open period with three probes, and then closed. The flapping timeline alternates between open and half-open repeatedly without ever closing.HEALTHY — probes_to_close = 3, cool-down 30 sopenhalf-open ×3closedFLAPPING — closes on one success, re-opens immediatelyopenopenopenopenThe comb pattern is diagnostic: the endpoint answers a probe but cannot carry the fleet. Raise probes_to_close, or the ramp.
Both timelines cover the same outage. The lower one spends most of it repeatedly re-discovering that the server is not ready.

There is a subtler reading of the comb pattern worth knowing. Sometimes the probes are not wrong at all — the server really can serve a 1×1 image, and really cannot serve the fleet. That is not a probe problem, it is a capacity problem, and no amount of tuning probes_to_close will fix it. The tell is that the failures after each close arrive within a second or two and are timeouts rather than errors. When you see that, the answer is to lower the per-endpoint concurrency limit permanently, not to make the breaker more patient: the endpoint is telling you, quite precisely, how much traffic it can take.

Common Pitfalls

  • Probing with GetCapabilities. It is usually served by a cache or a lightweight metadata path, so it succeeds while the renderer is still dead. The breaker closes, the fleet arrives, and the breaker re-opens ten seconds later — the classic flapping signature.
  • A permit TTL shorter than the probe timeout. The lock expires, another worker acquires it, and now two probes are in flight against a server that could not handle one. Assert lock_seconds > probe_timeout in configuration validation.
  • Closing on a single success. One probe proves the server can render a 1×1 image, not that it can carry sixty workers. Consecutive successes plus a ramp are what make the difference.
  • Releasing the whole fleet at once. Even after three clean probes, sixty concurrent renders against a cold cache is a load spike the server just proved it is sensitive to. Ramp, or lean on the per-endpoint concurrency limit to ramp for you.
  • No timeout on the probe. A probe that inherits the pipeline’s generous read timeout holds the permit for minutes, during which the breaker cannot re-evaluate anything. Probes should be short and decisive.

Frequently Asked Questions

Should the probe be a real tile from the batch?

It is tempting — a successful probe would then be useful work — but it makes the probe expensive and variable. A real tile might be a heavy render, so a slow response tells you about the tile rather than about the server. Keep the probe synthetic and cheap; the first real tile after the ramp is where useful work resumes.

What if the endpoint has no cheap request?

Some do not — every GetMap costs seconds regardless of extent. Then accept the cost and lengthen the cool-down instead, so probes are rarer. It is also worth asking the operator whether a health endpoint exists; many services have one that is not advertised in the capabilities document.

How does the ramp interact with the concurrency limit?

They multiply. If the per-endpoint limit is 16 and the ramp fraction is 0.25, four requests are in flight. That is usually what you want, and it means you can often skip an explicit ramp implementation by simply lowering the concurrency limit for the first minute after a close — see limiting DAG fan-out with concurrency groups.

Does anything need to happen if the probe never runs?

Yes — if the batch finished while the breaker was open, no worker arrives to probe and the breaker stays open into the next run. A tiny scheduled flow that does nothing but call run() every minute keeps recovery detection alive between batches, and costs one request a minute at most.

← Circuit Breakers for External WMS Services

Related in this section