Geospatial Orchestration Hub

Exporting Per-Tile Metrics Without Cardinality Blowups

Per-tile detail is a legitimate need and a metric label is the wrong place to put it. A tile identifier has millions of values, and every distinct label combination is a separate time series that Prometheus must index, store and query. The answer is to route each question to the system built for it: aggregates and rates to metrics with bounded labels, individual slow tiles to traces with a sampling rule, and the full per-tile record to a table you can query with SQL. Each is cheap in its own medium and ruinous in the others.

When to Use This Pattern

  • Someone has asked for per-tile timing and the obvious implementation would add a tile_id label.
  • Prometheus has become slow or expensive and the cause is suspected to be a spatial label.
  • The question is really “which tiles are slow?”, which is a trace or a table question rather than a metric one.
  • You need per-tile data for a report — coverage, freshness, cost — which is a warehouse job, not a monitoring one.

Complete Working Example

The exporter aggregates into bounded buckets and, separately, records the individual outliers that actually need attention.

from __future__ import annotations

import json
import time
from dataclasses import dataclass
from typing import Optional

import psycopg
from prometheus_client import Counter, Histogram

# BOUNDED labels only. Every value here is enumerable from the domain.
TILE_SECONDS = Histogram(
    "tile_generation_seconds", "Tile generation wall-clock",
    ("layer", "zoom", "density_band"),
    buckets=(0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256, float("inf")),
)
TILES = Counter(
    "tile_generation_total", "Tiles generated",
    ("layer", "zoom", "density_band", "outcome"),
)

# The trick that recovers most of the lost detail: bucket the tile by a property
# that explains its cost, rather than identifying it. Five bands, not a million ids.
DENSITY_BANDS = ((0.02, "sparse"), (0.15, "low"), (0.45, "medium"), (0.80, "high"))


def density_band(nonzero_fraction: float) -> str:
    for threshold, name in DENSITY_BANDS:
        if nonzero_fraction <= threshold:
            return name
    return "dense"


@dataclass(frozen=True)
class TileRecord:
    """The full per-tile row — for the table, never for a label."""

    layer: str
    z: int
    x: int
    y: int
    seconds: float
    pixels: int
    outcome: str
    worker: str


SLOW_MULTIPLE = 4.0          # relative to the band's rolling median


def record_tile(
    conn: psycopg.Connection, rec: TileRecord, band_median: float, tracer=None
) -> None:
    """Three destinations, each getting what it is good at."""
    band = density_band(nonzero_fraction_of(rec))

    # 1. Metrics: aggregates, bounded labels, cheap forever.
    TILE_SECONDS.labels(layer=rec.layer, zoom=str(rec.z), density_band=band).observe(rec.seconds)
    TILES.labels(layer=rec.layer, zoom=str(rec.z), density_band=band,
                 outcome=rec.outcome).inc()

    # 2. Traces: only the outliers, so the sampling stays affordable.
    if tracer is not None and rec.seconds > band_median * SLOW_MULTIPLE:
        with tracer.start_as_current_span("slow_tile") as span:
            span.set_attribute("tile.z", rec.z)
            span.set_attribute("tile.x", rec.x)
            span.set_attribute("tile.y", rec.y)
            span.set_attribute("tile.seconds", rec.seconds)
            span.set_attribute("tile.density_band", band)

    # 3. The table: every tile, queryable with SQL, retained on its own schedule.
    with conn.cursor() as cur:
        cur.execute(
            "INSERT INTO tile_runs (layer, z, x, y, seconds, pixels, outcome, worker, ran_at) "
            "VALUES (%s,%s,%s,%s,%s,%s,%s,%s, now())",
            (rec.layer, rec.z, rec.x, rec.y, rec.seconds, rec.pixels, rec.outcome, rec.worker),
        )

The per-tile questions then become SQL, which is what they were all along:

-- Which tiles are consistently slow, over the last week?
SELECT z, x, y, count(*) AS runs,
       round(avg(seconds)::numeric, 1) AS avg_seconds,
       round(max(seconds)::numeric, 1) AS worst
FROM   tile_runs
WHERE  ran_at > now() - interval '7 days' AND layer = 'ortho'
GROUP  BY z, x, y
HAVING avg(seconds) > 60
ORDER  BY avg_seconds DESC
LIMIT  25;

-- And where are they? The answer is usually a shape, not a list.
SELECT ST_Extent(ST_TileEnvelope(z, x, y)) AS slow_region
FROM   tile_runs
WHERE  ran_at > now() - interval '7 days' AND seconds > 120;
Route each question to the system built for itMetrics carry aggregates with bounded labels at thousands of series. Traces carry the slow outliers at a small sampled volume. A table carries every tile, queryable with SQL and retained separately.one tile finishes4.2 M times a weekmetrics — layer × zoom × density_bandrates, ratios, percentiles · ~3 000 series · queried in millisecondstraces — the slow 0.3%full tile identity and timing, sampled · 12 000 spans a weektable — every tile, one row eachSQL, spatial joins, its own retention · cheap in a database
The same event, three destinations. Putting all three questions into the metrics layer is what produces a million series and answers none of them well.

The density band is the part of this worth stealing even if you take nothing else. It recovers most of what a per-tile label would have told you, for five series instead of a million, because the question people are usually asking is not “how long did tile 12/2100/1300 take” but “why are some tiles slow”. A label that names the reason answers that directly and stays bounded forever; a label that names the tile answers it only after a human joins it to something else. Finding the right explanatory property takes one analysis — density, terrain class, source scheme, band count — and it pays back every time someone opens the dashboard.

Identify the tile, or explain itA tile identifier requires a human to join the metric to something else before it explains anything. A density band explains the cost directly at five values.LABEL IDENTIFIES THE TILEtile_id — 6.2 M values“12/2100/1300 took 94 s”…and now a human has to find out whyLABEL EXPLAINS THE TILEdensity_band — 5 values“dense tiles take 94 s, sparse take 3 s”…which is the answer, not the starting pointThe identifier belongs in the table, where the join it requires is one line of SQL rather than a monitoring problem.
Choosing an explanatory label rather than an identifying one is the whole technique. The cardinality saving is a side effect of asking the better question.

Parameter & Option Reference

Choice Value Spatial notes
density_band 5 values The trick: a bounded label that explains cost without identifying the tile.
tile identity in metrics never Millions of values. This is the rule the whole page exists to state.
SLOW_MULTIPLE 4.0 Relative to the band’s median, not absolute — otherwise dense tiles are always “slow”.
trace sample rate outliers only Head-based sampling on a threshold keeps volume proportional to the problem.
table retention 90 days Rows are small; a spatial index makes the interesting queries fast.
worker label in the table Useful per-tile, ruinous as a metric label on an autoscaled fleet.

Verification & Testing

Assert the cardinality bound directly — it is the property that matters and it is easy to check.

def test_metric_cardinality_is_bounded() -> None:
    layers, zooms, bands, outcomes = 12, 21, 5, 3
    assert layers * zooms * bands <= 2_000          # histogram label sets
    assert layers * zooms * bands * outcomes <= 5_000

    # And no metric may carry anything tile-identifying.
    for metric in (TILE_SECONDS, TILES):
        assert not {"tile_id", "x", "y", "key", "path"} & set(metric._labelnames)


def test_density_band_is_total_and_bounded() -> None:
    bands = {density_band(f / 100) for f in range(0, 101)}
    assert bands == {"sparse", "low", "medium", "high", "dense"}


def test_only_outliers_are_traced(fake_tracer) -> None:
    median = 8.0
    for seconds in (7.0, 9.0, 12.0, 40.0):
        record_tile(conn, make_record(seconds=seconds), band_median=median,
                    tracer=fake_tracer)
    assert fake_tracer.span_count == 1, "only the 40 s tile should be traced"

Once the series exist, one query keeps the cardinality honest over time. Running it monthly catches the label somebody added in a hurry, which is how nearly every cardinality incident actually starts:

# Series count per metric name — the number to watch, not the ingest rate.
topk(20, count by (__name__)({__name__=~"tile_.*|raster_.*|gdal_.*"}))
How a cardinality incident actually beginsSeries count stays flat for three months, then rises steeply in the fourth after a well-meaning label addition. The rise is gradual enough that nobody notices for weeks.seriesMayJunJulAug“just adding a source label”bounded labels, flat for monthsNothing broke on the day of the change. Six weeks later the queries were timing out and the cause was a month old.
The slow onset is why a monthly series-count check is worth more than an ingest-rate alert: the damage is done long before anything is visibly wrong.

Common Pitfalls

  • A tile identifier as a label. The canonical mistake, in all its forms: tile_id, x and y as separate labels, an object key, a file path. Any of them makes cardinality proportional to the data.
  • A source URI as a label. Bounded in a demo with three sources and unbounded in production with four thousand scenes. Use the scheme, or a small source_group.
  • A worker or pod label on an autoscaled fleet. Every new pod is a new series, and they never expire while the retention holds. Useful in the table, ruinous in a metric.
  • Absolute slow-tile thresholds. “Slower than sixty seconds” traces every dense tile and none of the anomalies. Comparing against the band’s own median is what makes the outlier definition meaningful.
  • Assuming Prometheus will tell you. It degrades gradually — slower queries, then slower ingest, then failures — over weeks. A periodic series-count check is the only reliable early warning.

Frequently Asked Questions

What if I genuinely need a per-tile time series?

Then you need a time-series database designed for high cardinality, or a table. Prometheus is explicitly not built for it, and working around that produces a system that is expensive and still slow. A tile_runs table with a spatial index answers per-tile questions faster than Prometheus would, and it supports spatial joins that Prometheus cannot express at all.

How do I pick the density bands?

From the same measurement that sized the histogram buckets. Compute the non-zero pixel fraction for a sample of tiles, look at where the latency distribution separates, and put the boundaries there. Five bands is usually plenty; the goal is for latency within a band to be roughly unimodal, which is what makes the band a useful explanatory label.

Does the table become expensive?

Far less than the equivalent metrics. A row is about a hundred bytes, so four million tiles a week is under half a gigabyte — trivial for PostgreSQL, and partitionable by month with a simple drop for retention. The same information as metric labels would be millions of series with a sample every scrape interval, which is orders of magnitude larger.

Should the trace and the table share an identifier?

Yes — put the trace id in the table row. That is what lets “this tile was slow last Tuesday” lead directly to the trace showing why, which is the single most useful join in this whole arrangement. Correlating logs and traces for one tile run covers the identifier plumbing.

Is this over-engineering for a small pipeline?

The table alone is not — it is one insert and it answers most questions. The traces are worth adding when “why was this slow?” stops being answerable by reading the code. The discipline that matters at every size is the negative one: keep unbounded values out of metric labels, which costs nothing and prevents the only failure here that is genuinely hard to undo.

← Prometheus Metrics for Raster Throughput

Related in this section