Half-Open Recovery for Tile Servers

Half-open recovery for a tile server is the controlled test between “open” and “closed”: once the reset timeout elapses, the breaker lets a small number of probe tile requests through, and only closes fully after a run of successes. Get this phase wrong and a recovering tile cache either stays locked out far longer than needed or gets flooded the instant it comes back. This page covers success thresholds that prevent flapping, limiting concurrent probes, and jittering the reset timeout across workers so a fragile server is eased back online rather than stampeded.

It builds directly on Circuit Breakers for External WMS Services and pairs with the guide on configuring failure thresholds for WMS endpoints, which governs how the circuit opens in the first place.

When to Use This Pattern#

Tune the half-open phase when:

• Your breaker oscillates rapidly between open and closed — the classic flapping signature.
• A recovering GeoServer or tile cache falls over again the moment full traffic resumes, because a cold cache cannot absorb the restored load.
• Many workers share breaker state and all release traffic at the same instant when the timeout expires.
• You need confidence that “recovered” means a sustained run of good tiles, not one lucky GetMap.

The Half-Open State, Precisely#

When a breaker opens, it starts a reset_timeout clock and fast-fails every request in the meantime. When that clock expires, the breaker does not jump straight to closed — it enters half-open. In half-open, a bounded number of probe requests are admitted to the real tile server. Their outcomes decide the next transition:

1. A probe succeeds — increment a success counter.
2. Enough probes succeed in a row to reach the success threshold — transition to closed and resume full traffic.
3. Any probe fails — transition straight back to open and restart the reset timeout (often longer than the last one).

The success threshold is what separates a robust recovery from flapping. A threshold of one means a single fluke response reopens the floodgates; a cold tile cache then collapses under restored concurrency and the breaker trips again seconds later. Requiring three consecutive successful tile fetches before closing gives the server’s cache time to warm and confirms the recovery is real.

Complete Working Example#

This breaker implements an explicit half-open phase with a probe cap and a success threshold. It is deliberately self-contained so the state transitions are visible.

from __future__ import annotations

import enum
import random
import threading
import time
from dataclasses import dataclass, field


class State(enum.Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


@dataclass
class HalfOpenBreaker:
    fail_max: int = 5
    reset_timeout: float = 60.0
    jitter: float = 0.25          # +/- 25% spread on the reset window
    success_threshold: int = 3    # consecutive probe wins needed to close
    max_probes: int = 2           # concurrent probes allowed while half-open

    state: State = State.CLOSED
    _failures: int = 0
    _probe_successes: int = 0
    _probes_in_flight: int = 0
    _opened_at: float = 0.0
    _deadline: float = 0.0
    _lock: threading.Lock = field(default_factory=threading.Lock)

    def _jittered_timeout(self) -> float:
        spread = self.reset_timeout * self.jitter
        return self.reset_timeout + random.uniform(-spread, spread)

    def acquire(self) -> bool:
        """Return True if this call may hit the tile server."""
        with self._lock:
            now = time.monotonic()
            if self.state is State.OPEN:
                if now < self._deadline:
                    return False  # still resting
                self.state = State.HALF_OPEN
                self._probe_successes = 0
                self._probes_in_flight = 0
            if self.state is State.HALF_OPEN:
                if self._probes_in_flight >= self.max_probes:
                    return False  # cap concurrent probes
                self._probes_in_flight += 1
            return True

    def on_success(self) -> None:
        with self._lock:
            if self.state is State.HALF_OPEN:
                self._probes_in_flight = max(0, self._probes_in_flight - 1)
                self._probe_successes += 1
                if self._probe_successes >= self.success_threshold:
                    self.state = State.CLOSED
                    self._failures = 0
            else:
                self._failures = 0

    def on_failure(self) -> None:
        with self._lock:
            if self.state is State.HALF_OPEN:
                self._probes_in_flight = max(0, self._probes_in_flight - 1)
                self._trip(time.monotonic())
                return
            self._failures += 1
            if self._failures >= self.fail_max:
                self._trip(time.monotonic())

    def _trip(self, now: float) -> None:
        self.state = State.OPEN
        self._opened_at = now
        self._deadline = now + self._jittered_timeout()
        self._probe_successes = 0

Using it around a tile fetch keeps the transition logic out of the request code:

import requests


def probe_tile(breaker: HalfOpenBreaker, url: str, z: int, x: int, y: int) -> bytes:
    if not breaker.acquire():
        raise RuntimeError("tile breaker not accepting requests")
    try:
        resp = requests.get(f"{url}/{z}/{x}/{y}.png", timeout=(3.0, 12.0))
        if resp.status_code >= 500 or len(resp.content) < 512:
            raise RuntimeError(f"bad tile: HTTP {resp.status_code}")
    except Exception:
        breaker.on_failure()
        raise
    breaker.on_success()
    return resp.content

A single failing probe calls on_failure, which trips straight back to open from half-open — recovery must be proven, not assumed. Successful probes decrement the in-flight count and advance the success counter; only when three land does the state return to closed.

Parameter & Option Reference#

Jittering the Reset Timeout Across Workers#

The most damaging recovery bug in a distributed tile pipeline is synchronized probing. If twenty Prefect or Dagster workers all open their breakers at the same second and share an identical 60-second reset_timeout, all twenty release probes simultaneously when it expires — a thundering herd that knocks over the very server they were protecting. Adding random.uniform(-spread, spread) to each worker’s timeout spreads the probe wave across a window (here, 45–75 seconds) so the tile server sees a trickle it can absorb. This is the same reasoning behind jitter in exponential backoff for API rate limits, applied to the recovery boundary rather than the retry interval. If workers share breaker state in Redis, prefer a shared probe token (only one worker probes at a time) over independent per-worker timeouts.

Verification & Testing#

Assert the flapping guard and the trip-back-from-half-open behavior directly:

def test_requires_success_threshold_to_close() -> None:
    b = HalfOpenBreaker(fail_max=1, reset_timeout=0.0, success_threshold=3)
    b.on_failure()                      # trips -> OPEN, deadline in the past
    assert b.acquire() is True          # moves to HALF_OPEN
    b.on_success()
    b.on_success()
    assert b.state is State.HALF_OPEN   # only 2 wins, not closed yet
    b.acquire(); b.on_success()
    assert b.state is State.CLOSED


def test_probe_failure_reopens() -> None:
    b = HalfOpenBreaker(fail_max=1, reset_timeout=0.0)
    b.on_failure()
    b.acquire()                         # HALF_OPEN
    b.on_failure()                      # a failed probe
    assert b.state is State.OPEN

In staging, restart a GeoServer container and watch the breaker logs: you should see one probe admitted, a short warm-up, and a clean transition to closed only after the third good tile — never a burst of concurrent probes at the instant the timeout fires.

Common Pitfalls#

• Success threshold of one. A single fluke tile closes the breaker and full traffic collapses a cold cache; require a run of successes.
• Unbounded probes in half-open. Admitting every waiting request the moment the timeout expires re-overloads the server; cap concurrent probes with max_probes.
• Identical timeouts across workers. Without jitter, distributed workers probe in lockstep and stampede a recovering tile server; spread the reset window per worker or gate probing with a shared token.
• Not resetting on probe failure. Leaving the breaker half-open after a failed probe lets more requests leak through to a still-broken server; trip straight back to open.

Related: Set the trip conditions with configuring failure thresholds for WMS endpoints, borrow the jitter idea from exponential backoff for API rate limits, divert requests rejected during recovery into dead-letter queues for failed geotasks, and de-duplicate flushed retries with idempotency keys in spatial ETL — all part of Resilience & Failure Handling for GIS Pipelines.

← Back to Circuit Breakers for External WMS Services