Data-Quality SLOs for Spatial Pipelines

In short: a pipeline’s users care about the output, not the process. An SLO on run success rate promises nothing they can use; an SLO on “95% of the populated area is no more than 48 hours old” promises something they can plan around, and it is measurable from the ledger without any new instrumentation. Three objectives cover most spatial products: freshness, coverage and validity.

The distinction matters because a spatial pipeline can be perfectly healthy and produce unusable output. Every run succeeds, throughput is normal, no task fails — and one region has not been updated in six weeks because its source stopped publishing. Process metrics are all green throughout. An SLO defined on the data closes that gap, and it changes the conversation with consumers from “the pipeline ran” to “here is what you can rely on”.

There is a second, quieter benefit. Writing down what the output should look like forces a conversation that most spatial platforms never have: what, precisely, is being promised, to whom, about which area. Teams routinely discover during that conversation that two consumers have incompatible expectations of the same layer, or that a region everyone assumed was covered has never been in scope. Those findings arrive whether or not the objective is ever alerted on, and they are frequently worth more than the monitoring.

Prerequisites & Architecture Baseline#

• A ledger recording per-tile or per-partition build times and outcomes — the SLO is computed from it, not from metrics
• An agreed definition of the serving area: which tiles or regions the objective actually covers
• A weighting decision — is a tile over open sea worth as much as one over a city?
• Prometheus or equivalent for the error-budget burn rate, with a recording rule so the SLI is computed once
• A stated measurement window: 28 or 30 days, rolling, agreed with whoever consumes the data
• Someone who can decide what happens when the budget is exhausted, before it is
• A weight per unit of the serving area — population, request counts, or an explicit priority band
• A published statement of the objective, its weighting and its window, somewhere consumers can read it

Core Principles#

1. Measure the output, not the process. Run success rate is an internal signal. The SLI is computed from the artefacts: how old is the oldest tile in the serving area, what share of the area exists, what fraction of features are valid. Those are the numbers a consumer would check themselves.

2. Weight by what matters, not by area. Ninety-five per cent of a national tile pyramid is uninhabited, so an unweighted coverage SLO can be met while every city is stale. Weight by population, by request volume, or by an explicit priority band — and state the weighting, because it is where the objective’s honesty lives.

3. Freshness is per layer, not per pipeline. A road basemap may tolerate a week; a flood extent tolerates hours. One objective across every layer either over-promises on the slow ones or is unachievable for the fast ones. Group layers by their phenomenon’s rate of change.

4. The error budget is the useful half. “99% freshness” is a threshold nobody negotiates with. “1% of 30 days is 7.2 hours of staleness we can spend” is a quantity that makes a decision — whether to do a risky deployment, whether to defer a backfill — concrete and shared.

5. An objective needs an owner and a consequence. If nothing changes when the budget is exhausted, the SLO is a dashboard panel with ambitions. The consequence need not be dramatic; “we stop shipping new features and fix the source pipeline” is enough, provided it is agreed in advance.

6. Start with one objective per product. Three well-understood SLIs across a whole platform beat thirty per-layer objectives nobody can recite. Freshness is usually the right first one, because it is the property consumers notice and the one that degrades silently.

Weighting deserves more attention than it usually receives, because it is where an objective becomes either honest or decorative. An unweighted coverage figure over a national pyramid is dominated by sea, forest and mountain, so a pipeline can lose every city and still report 98%. Weighting by population inverts that, and it introduces its own bias — rural staleness becomes nearly invisible, which is a real trade-off that rural users would notice. There is no neutral choice, and the useful discipline is to publish the weighting alongside the target so that whoever reads the promise can see whose experience it describes.

Production Implementation#

The SLI is a query, computed on a schedule and exported as a gauge so the error budget can be tracked with ordinary tooling.

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable

import psycopg
from prometheus_client import Gauge

SLI = Gauge("spatial_sli_ratio", "Data-quality SLI", ("layer", "objective"))
BUDGET = Gauge("spatial_slo_budget_remaining", "Error budget remaining, 0-1",
               ("layer", "objective"))


@dataclass(frozen=True)
class Objective:
    layer: str
    name: str                 # "freshness" | "coverage" | "validity"
    target: float             # 0.95 means 95% of weighted area must comply
    window_days: int = 30
    max_age_hours: float = 48.0


FRESHNESS_SQL = """
SELECT
  -- The SLI: weighted share of the serving area meeting the age threshold.
  -- Weighting by population is what stops an ocean-heavy pyramid flattering itself.
  sum(t.weight) FILTER (
      WHERE t.built_at > now() - (%(max_age_hours)s * interval '1 hour'))
  / NULLIF(sum(t.weight), 0) AS sli
FROM   tile_ledger t
WHERE  t.layer = %(layer)s
  AND  t.in_serving_area                     -- not every tile is promised
"""


def compute_freshness(conn: psycopg.Connection, obj: Objective) -> float:
    with conn.cursor() as cur:
        cur.execute(FRESHNESS_SQL,
                    {"layer": obj.layer, "max_age_hours": obj.max_age_hours})
        (sli,) = cur.fetchone()
    return float(sli or 0.0)


def budget_remaining(sli: float, target: float) -> float:
    """1.0 = untouched, 0.0 = exhausted, negative = the objective is missed."""
    allowed_failure = 1.0 - target
    if allowed_failure <= 0:
        return 1.0 if sli >= 1.0 else 0.0
    return max(-1.0, 1.0 - (1.0 - sli) / allowed_failure)


def publish(conn: psycopg.Connection, objectives: Iterable[Objective]) -> None:
    for obj in objectives:
        sli = compute_freshness(conn, obj) if obj.name == "freshness" else compute_other(conn, obj)
        SLI.labels(layer=obj.layer, objective=obj.name).set(sli)
        BUDGET.labels(layer=obj.layer, objective=obj.name).set(
            budget_remaining(sli, obj.target)
        )


OBJECTIVES = [
    Objective("ortho", "freshness", target=0.95, max_age_hours=24 * 30),
    Objective("roads", "freshness", target=0.99, max_age_hours=24 * 7),
    Objective("flood_extent", "freshness", target=0.99, max_age_hours=2),
    Objective("parcels", "validity", target=0.999),
    Objective("ortho", "coverage", target=0.995),
]

Step-by-Step Walkthrough#

1. Define the serving area explicitly. in_serving_area is a column, not an assumption. A pyramid covers the globe; the promise covers the part someone actually looks at, and stating which part is half the work of a defensible objective.
2. Attach a weight to every unit. Population is the usual choice for a public basemap; request counts are better where you have them. The weight is what makes the number mean something to a consumer.
3. Compute the SLI as a ratio of weighted units. Share of weighted area meeting the threshold — not an average age, which one very stale region can hide inside a healthy mean.
4. Convert to an error budget. The budget is the fraction of the allowance still unspent, and it is what makes the objective actionable rather than merely observable.
5. Set the objectives per layer, from the phenomenon. Two hours for a flood extent and a month for orthophotos are both correct, because they are properties of the world rather than of the pipeline.
6. Export both numbers. The SLI for the dashboard and the budget for the alert. Alerting on the SLI itself produces a page the moment a threshold is crossed; alerting on burn rate gives warning first.
7. Recompute on a schedule, not per request. These are aggregate queries over a large table. Once every few minutes is ample, and it keeps the dashboard’s cost independent of how many people are looking at it.

Edge Cases & Failure Recovery#

A serving area that changes. Adding a region to the promise changes the denominator, so the SLI moves for reasons unrelated to the pipeline. Version the serving-area definition and record which version each measurement used, or a boundary change looks like a regression and consumes budget it did not cause.

Weighting that nobody agreed to. Population weighting is defensible and it is a choice with consequences: it makes rural staleness nearly invisible. Whatever the weighting, it belongs in the published objective, so a consumer in a rural area knows the promise was never about them.

An objective that has never been missed. That usually means the target is too loose, not that the pipeline is exceptional. An SLO that cannot be breached provides no signal and no budget to spend. Tighten it until it is occasionally uncomfortable, which is where the information is.

Budget exhausted by one incident. A single bad week can consume a month’s allowance, after which the objective is missed regardless of what happens next. That is the intended behaviour and it is worth explaining in advance, because the first time it happens the instinct is to reset the window.

Freshness measured on the wrong timestamp. built_at is when the pipeline wrote the tile; the consumer cares when the source was captured. For imagery those differ by days or weeks. Where the distinction matters, carry the source’s acquisition time through and measure against that instead — the number will be worse and it will be the true one.

A ledger that is itself incomplete. The SLI is computed from the ledger, so a tile that was built but never recorded counts as stale, and one recorded but never built counts as fresh. Both directions are possible if the ledger write and the artefact write are not in the same transaction or the same ordered sequence. Before trusting an objective, verify a sample of the ledger against the object store — a hundred rows is enough to find a systematic discrepancy.

An SLO on a layer nobody uses. Objectives cost attention to maintain. A layer with no consumers does not need a promise, and removing its objective is a legitimate simplification rather than a retreat.

Configuration Reference#

The window length is the setting most often chosen carelessly. Too short and every incident breaches the objective, which trains everyone to ignore it; too long and a serious degradation is diluted into invisibility for weeks. Thirty days rolling is a reasonable default because it matches how most people think about a month, and because it means a single bad night costs roughly three per cent of the budget — noticeable, survivable, and cumulative if it keeps happening. Whatever the choice, it should be the same across objectives, or comparing their budgets becomes an exercise in arithmetic nobody performs.

Frequently Asked Questions#

Related#

• Defining freshness SLOs for tile layers — the freshness objective in detail
• Monitoring geometry validity rates as an SLO — the validity objective
• Visualizing tile coverage gaps on a geomap — localising a coverage breach
• Prometheus metrics for raster throughput — the process metrics these complement
• Spatial validation & sync tasks — where the validity numbers originate

← Observability & Monitoring for Geospatial Pipelines