Geospatial Orchestration Hub

Falling Back to Cached Tiles When a Breaker Opens

When a breaker opens, the pipeline has three honest options: fail the tile, serve the last good tile, or emit a placeholder that says “not available”. Serving the cached tile is usually right for a basemap someone is looking at now, and usually wrong for an analytical product where old pixels silently corrupt a result. Whichever you choose, the non-negotiable part is that the output carries its own provenance — an age, a source and a flag — so that nothing downstream mistakes a two-day-old tile for a fresh one.

When to Use This Pattern

  • The output is consumed visually — a slippy map, a printed atlas, a preview — where a slightly stale tile is far better than a grey square.
  • A tile cache already exists with reliable timestamps, whether that is S3 object metadata, a mbtiles file or a filesystem tree.
  • Staleness has a defensible budget. Imagery from last week is fine for a road basemap and unacceptable for a flood-extent layer; the budget is a product decision, not an engineering one.
  • The consumer can see the provenance. If nothing downstream can distinguish fresh from stale, do not do this — take the failure instead.

The pattern is wrong for analytical pipelines that compute over the pixels. A zonal statistic that silently mixes yesterday’s and today’s imagery produces a number nobody can reproduce, and the failure is invisible because the output looks completely normal.

Three things an open breaker can returnFailing gives the consumer nothing but is unambiguous. Serving a stale tile with provenance gives a usable map and a visible age. A placeholder gives a legible gap. Serving stale data without provenance is the only option that is never acceptable.strategyconsumer seescan they tell?fail the tilenothing — a gapyes, unambiguouslystale tile + provenancea usable map, aged 6 hyes, from the headerplaceholder tilea legible “no data” squareyes, visiblystale, no provenancea map that looks currentno — never do this
Three of these are defensible choices for different products. The fourth is the one that turns an upstream outage into a data-quality incident weeks later.

Complete Working Example

The fallback below reads from an S3-backed tile cache, enforces a staleness budget, and returns the tile together with its provenance so the writer can record it.

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Optional

import boto3
from botocore.exceptions import ClientError


@dataclass(frozen=True)
class TileResult:
    """A tile plus the truth about where it came from."""

    data: bytes
    source: str                 # "live" | "cache" | "placeholder"
    generated_at: datetime
    age: timedelta

    @property
    def is_stale(self) -> bool:
        return self.source != "live"


# One budget per layer, because staleness tolerance is a property of the data,
# not of the pipeline. Flood extents go stale in minutes; road networks do not.
STALENESS_BUDGET: dict[str, timedelta] = {
    "ortho": timedelta(days=30),
    "roads": timedelta(days=7),
    "flood_extent": timedelta(hours=2),
    "traffic": timedelta(minutes=10),
}

PLACEHOLDER_PNG = b"\x89PNG\r\n\x1a\n..."      # a 256x256 "unavailable" tile


def cached_tile(
    s3, bucket: str, layer: str, z: int, x: int, y: int
) -> Optional[TileResult]:
    """The last good tile for this coordinate, if it is inside the budget."""
    key = f"{layer}/{z}/{x}/{y}.png"
    try:
        obj = s3.get_object(Bucket=bucket, Key=key)
    except ClientError as exc:
        if exc.response["Error"]["Code"] in ("NoSuchKey", "404"):
            return None
        raise

    generated_at = obj["LastModified"]
    if generated_at.tzinfo is None:
        generated_at = generated_at.replace(tzinfo=timezone.utc)
    age = datetime.now(timezone.utc) - generated_at

    budget = STALENESS_BUDGET.get(layer, timedelta(hours=24))
    if age > budget:
        # Outside the budget the cached tile is not a fallback, it is misinformation.
        return None

    return TileResult(data=obj["Body"].read(), source="cache",
                      generated_at=generated_at, age=age)


def fetch_tile(breaker, client, s3, bucket, layer: str, z: int, x: int, y: int) -> TileResult:
    """Live tile when the breaker allows it, else the best honest alternative."""
    try:
        data = breaker.call(lambda: render_tile(client, layer, z, x, y))
        return TileResult(data=data, source="live",
                          generated_at=datetime.now(timezone.utc), age=timedelta(0))
    except BreakerOpen:
        pass                                  # expected: fall through to the cache

    fallback = cached_tile(s3, bucket, layer, z, x, y)
    if fallback is not None:
        return fallback

    return TileResult(data=PLACEHOLDER_PNG, source="placeholder",
                      generated_at=datetime.now(timezone.utc), age=timedelta(0))

Whatever writes the tile onward must carry the provenance with it. For an HTTP tile service that means response headers; for a pipeline writing to object storage it means object metadata, which is what lets a later audit answer “which tiles in this mosaic were stale?” without re-deriving anything:

def put_tile(s3, bucket: str, key: str, result: TileResult) -> None:
    s3.put_object(
        Bucket=bucket, Key=key, Body=result.data, ContentType="image/png",
        Metadata={
            "tile-source": result.source,                       # live | cache | placeholder
            "tile-generated-at": result.generated_at.isoformat(),
            "tile-age-seconds": str(int(result.age.total_seconds())),
        },
        # A stale tile must not be cached downstream as though it were fresh.
        CacheControl="max-age=60" if result.is_stale else "max-age=86400",
    )

The staleness budget deserves more thought than it usually gets, because it is the only part of this design that encodes a judgement about the world rather than about software. A useful way to arrive at a number is to ask what decision the tile supports and how quickly that decision would change: a road basemap supports navigation over a network that changes on the scale of months, so a week is generous and safe; a flood extent supports evacuation decisions over a phenomenon that changes hourly, so two hours is already at the edge. Written that way, the budget is defensible to whoever asks, and it survives the reorganisation that loses the original author.

Budget follows the phenomenon, not the pipelineTraffic changes over minutes and carries a ten-minute budget. Flood extent changes hourly with a two-hour budget. Roads change over months with a seven-day budget. Orthophotos change over years with a thirty-day budget.layerhow fast it changesbudgettrafficminutes10 minflood_extenthours2 hroadsmonths7 dorthoyears30 d
Read down the middle column and the right column follows almost mechanically — which is the sign that the budget is a property of the data rather than a preference.

Parameter & Option Reference

Parameter Type Default Spatial notes
STALENESS_BUDGET per layer varies A property of the phenomenon. Traffic goes stale in minutes; orthophotos in months. One global budget is always wrong for something.
default budget 24 h — For layers not listed. Deliberately short, so an unconfigured layer fails safe rather than serving last year’s tile.
tile-source metadata str required The single most important field. Without it “which tiles are stale?” is unanswerable.
CacheControl on stale max-age=60 — Prevents a CDN pinning a stale tile for a day after the outage ends.
placeholder tile 256×256 PNG — Visibly different from both a real tile and a transparent one, so a gap reads as a gap.
cache lookup timeout 2 s — The fallback path must be fast. A slow cache turns an open breaker into a slow pipeline instead of a fast one.

Verification & Testing

Test the budget boundary and the provenance, not the happy path.

def test_stale_beyond_budget_is_not_served(s3_stub, frozen_clock) -> None:
    s3_stub.put("flood_extent/12/2100/1300.png", b"old", age=timedelta(hours=3))
    # The flood budget is 2 h, so a 3 h tile must be refused outright.
    assert cached_tile(s3_stub, "tiles", "flood_extent", 12, 2100, 1300) is None


def test_fallback_is_labelled(s3_stub, open_breaker, frozen_clock) -> None:
    s3_stub.put("roads/12/2100/1300.png", b"cached", age=timedelta(hours=6))
    result = fetch_tile(open_breaker, None, s3_stub, "tiles", "roads", 12, 2100, 1300)
    assert result.source == "cache"
    assert result.is_stale
    assert timedelta(hours=5) < result.age < timedelta(hours=7)


def test_placeholder_when_nothing_cached(s3_stub, open_breaker) -> None:
    result = fetch_tile(open_breaker, None, s3_stub, "tiles", "roads", 12, 9, 9)
    assert result.source == "placeholder"

After an incident, the audit query is what makes the fallback defensible. With provenance in object metadata, listing the affected tiles is a scan rather than an investigation:

# Which tiles in this layer were served from cache during the outage window?
aws s3api list-objects-v2 --bucket tiles --prefix roads/12/ \
  --query 'Contents[].Key' --output text |
  xargs -n1 -P8 -I{} sh -c \
    'aws s3api head-object --bucket tiles --key {} \
       --query "Metadata.\"tile-source\"" --output text | grep -q cache && echo {}'
What a mosaic looks like with fallback in forceA grid of tiles where most are live, a contiguous group is served from cache inside the staleness budget, and a small number are placeholders because no cached tile existed or it was beyond budget.live — rendered nowcache — inside the budgetplaceholder — nothing usableEvery square carries its owntile-source metadata, so thispicture is queryable afterwards.
The map stays usable through the outage, and the record of which pixels were stale survives it — which is the difference between a fallback and a cover-up.

Common Pitfalls

  • No staleness budget. “Serve whatever is in the cache” eventually serves a tile from two years ago, and nobody notices because a tile is a tile. The budget is what makes the fallback bounded.
  • Losing the provenance at the next hop. The pipeline records tile-source: cache and the publishing step copies the bytes without the metadata. Everything downstream then believes the tile is fresh. Provenance has to survive every copy, which usually means asserting it in the publishing step.
  • Caching the stale tile downstream with a long TTL. A CDN that picks up a fallback tile with max-age=86400 will keep serving it for a day after the endpoint recovered. Shorten CacheControl for stale responses.
  • Falling back on the analytical path. A statistic computed over silently mixed dates is unreproducible and looks perfectly normal. Analytical tasks should fail and land in the dead-letter queue instead.
  • A slow cache lookup. If reading the fallback takes as long as the live request would have, the breaker has saved the upstream but not your throughput. Keep the fallback path fast and bounded, and treat a cache timeout as a placeholder.

Frequently Asked Questions

Where should the staleness budget live?

Next to the layer definition, not in the fallback code. It is a statement about the data — how fast the phenomenon changes — and the person who knows the answer is whoever owns the layer. Putting it in a config file the data owner can read and edit is worth more than any amount of cleverness in the code path.

Should the placeholder be transparent?

Usually not. A transparent tile is indistinguishable from a genuinely empty area, so a gap in coverage reads as ocean. A subtle diagonal hatch or a light grey square with “unavailable” is legible at a glance and photographs badly enough that nobody publishes it by accident.

Does the fallback count as a success for the breaker?

No, and this matters. The live call failed, and the breaker must record that failure regardless of what the fallback produced. Counting a fallback as a success is how a breaker stays closed through an outage — the pipeline “succeeds” on every tile while serving nothing but cache.

How does this interact with tile-cache invalidation?

Carefully. A tile that was served from cache during an outage must be re-rendered once the endpoint recovers, or the stale pixels persist indefinitely. Emit the affected coordinates into the same invalidation path described in invalidating tile caches after a source update, and treat the outage as an update event.

← Circuit Breakers for External WMS Services

Related in this section