Prometheus Metrics for Raster Throughput

In short: measure raster work in pixels and bytes rather than in files, because a “file” varies by three orders of magnitude and a pixel does not. Keep labels bounded — layer, zoom, resampling, outcome, never tile id — and size histogram buckets from measured latency rather than from the client library’s defaults, which are tuned for web requests and useless for a four-minute warp.

Throughput is the number a raster pipeline is judged on, and it is easy to measure in a way that means nothing. “Tiles per minute” rises when the work gets easier and falls when it gets harder, without either being a change in the pipeline. “Megapixels per second” is stable across a job whose tiles differ hundredfold in size, which makes it comparable between nights, between regions and between worker types — and comparability is the entire point of a metric.

There is a second reason the unit matters, beyond comparability. Capacity planning for a raster pipeline is an exercise in pixels per worker-second: how many megapixels can one worker produce, how many megapixels does tomorrow’s backlog contain, therefore how many worker-hours are needed. Every step of that arithmetic works in pixels, and none of it works in tiles — a backlog of “forty thousand tiles” is not a quantity of work until you know which tiles. Choosing the metric’s unit is therefore also choosing whether capacity questions can be answered from the dashboard or require a bespoke analysis every time.

Prerequisites & Architecture Baseline#

• prometheus_client ≥ 0.20, and a decision about push versus pull for short-lived task processes
• Prometheus ≥ 2.45, or a compatible remote-write endpoint
• A label vocabulary agreed in advance and written down, with a stated maximum cardinality per label
• Latency measurements from a real run, to size histogram buckets before the first deploy
• A scrape or push interval matched to task duration — a 15-second scrape misses a 4-second task entirely
• Grafana ≥ 10 or equivalent, since the metrics are only worth what the dashboard shows
• An agreed retention: high-resolution series for a fortnight, downsampled for a year, so week-on-week comparisons remain possible without unbounded storage
• A named owner for the label vocabulary, because the cardinality discipline survives exactly as long as someone is reviewing additions to it

Core Principles#

1. Count pixels, not files. A tile is not a unit of work in any meaningful sense: a 256×256 tile and an 8192×8192 window are both “one tile”. raster_pixels_processed_total divided by time is a throughput figure that stays comparable when the tiling scheme changes, when a new region with denser imagery arrives, and when someone doubles the tile size to reduce overhead.

2. Every label must have a bounded, known cardinality. layer has a dozen values, zoom has twenty, resampling has six, outcome has three. tile_id has millions, and adding it creates millions of time series that will make the metric unusable and Prometheus unhappy. The rule is that you should be able to state the maximum number of values a label can take, from the domain, before you add it.

3. Histogram buckets come from measurement. The default buckets in most client libraries top out around ten seconds, which puts every raster operation in the +Inf bucket and makes every percentile meaningless. Measure a real run, then choose buckets that bracket the observed p50 through p99 with a few above it.

4. Distinguish work from waiting. A task that takes four minutes may have spent thirty seconds warping and three and a half minutes waiting for an object store. Those are different problems with different fixes, and one histogram cannot tell them apart. Instrument the phases separately, even coarsely.

5. Errors are a label on the work, not a separate metric. raster_tiles_total{outcome="ok|failed|skipped"} gives the rate, the ratio and the total from one series family. A separate raster_errors_total cannot answer “what fraction failed?” without a second query and an assumption that the two metrics cover the same population.

6. Short-lived processes need a push gateway or an exporter. A task that runs for ninety seconds and exits will never be scraped on a fifteen-second interval reliably. Either push on completion, or have a long-lived exporter read the pipeline’s own state — the approach alerting on dead-letter queue growth uses.

The same argument applies to vector work with a different unit. “Features per second” is closer to useful than “files per second”, but a feature can be a point or a coastline with a million vertices, so the stable unit is vertices — or, where that is awkward to count, bytes of geometry. The test for a good unit is whether its rate stays flat when the workload’s composition changes and the pipeline does not. Anything that fails that test will generate incidents that resolve themselves, and a team that has seen a few of those stops trusting the dashboard.

Production Implementation#

The instrumentation below is deliberately small: three metric families, four labels, and buckets measured rather than defaulted.

from __future__ import annotations

import time
from contextlib import contextmanager
from typing import Iterator

from prometheus_client import Counter, Gauge, Histogram

# Label vocabulary, fixed and documented. Maximum cardinality:
#   layer 12 × zoom 20 × resampling 6 × outcome 3 = 4 320 series. Acceptable.
LABELS = ("layer", "zoom", "resampling", "outcome")

TILES = Counter(
    "raster_tiles_total",
    "Tiles processed, by outcome",
    LABELS,
)

PIXELS = Counter(
    "raster_pixels_processed_total",
    "Output pixels written — the unit that stays comparable across tilings",
    ("layer", "zoom", "resampling"),
)

# Buckets measured from a week of real runs: p50 ~9 s, p95 ~48 s, p99 ~140 s.
# The library default tops out at 10 s, which would put 40% of tiles in +Inf.
WARP_SECONDS = Histogram(
    "raster_warp_duration_seconds",
    "Wall-clock of one warp, excluding source fetch",
    ("layer", "zoom", "resampling"),
    buckets=(1, 2, 5, 10, 20, 40, 80, 160, 320, float("inf")),
)

FETCH_SECONDS = Histogram(
    "raster_source_fetch_seconds",
    "Time spent waiting for the source, separate from the warp",
    ("layer", "source_scheme"),          # scheme, not URI: bounded cardinality
    buckets=(0.25, 0.5, 1, 2, 5, 10, 30, 60, 120, float("inf")),
)

INFLIGHT = Gauge(
    "raster_tiles_inflight",
    "Tiles currently being processed by this worker",
    ("layer",),
)


@contextmanager
def observe_tile(layer: str, zoom: int, resampling: str) -> Iterator[dict]:
    """Times the phases separately and records the outcome exactly once."""
    ctx: dict = {"pixels": 0, "outcome": "failed"}
    labels = {"layer": layer, "zoom": str(zoom), "resampling": resampling}
    INFLIGHT.labels(layer=layer).inc()
    started = time.monotonic()
    try:
        yield ctx
        ctx["outcome"] = ctx.get("outcome_override", "ok")
    finally:
        INFLIGHT.labels(layer=layer).dec()
        WARP_SECONDS.labels(**labels).observe(time.monotonic() - started)
        PIXELS.labels(**labels).inc(ctx["pixels"])
        TILES.labels(outcome=ctx["outcome"], **labels).inc()


def process_tile(item, layer: str) -> None:
    with observe_tile(layer, item.z, item.resampling) as ctx:
        fetch_start = time.monotonic()
        source = fetch_source(item.source_uri)
        FETCH_SECONDS.labels(
            layer=layer, source_scheme=scheme_of(item.source_uri)
        ).observe(time.monotonic() - fetch_start)

        if already_done(item.work_key):
            ctx["outcome_override"] = "skipped"
            return

        width, height = warp_window(source, item)
        ctx["pixels"] = width * height          # the unit that matters

Step-by-Step Walkthrough#

1. Write the label vocabulary down before the first metric. Four labels with stated maxima gives a computable series count. Adding a fifth “just for debugging” is how a metric becomes a cardinality incident, and it is far easier to refuse at review time than to remove later.
2. Make outcome a label, not a metric. ok, failed and skipped on one counter means the failure ratio is a single expression, and it means “skipped” is visible — which is the number that tells you the cache and the skip branch are working.
3. Separate fetch from warp. Two histograms rather than one is the difference between “tiles are slow” and “the object store is slow”. They usually have different owners and always have different fixes.
4. Label the source by scheme, not by URI. s3, https, file is three values. The URI is unbounded, and it is the single most common cardinality mistake in spatial pipelines.
5. Measure buckets from a real run. The library defaults assume web-request latencies. A histogram whose top bucket is below your p50 reports percentiles that are pure interpolation, and the resulting dashboard is confidently wrong.
6. Record the outcome exactly once, in a finally. A metric incremented on the happy path only under-counts precisely when things are going badly, which is when the numbers matter.
7. Track in-flight work as a gauge. It is the fastest way to see that a worker has stalled: throughput falls to zero while in-flight stays at eight.

Edge Cases & Failure Recovery#

Short-lived processes that are never scraped. A task running for ninety seconds on a fifteen-second scrape interval is sometimes observed and sometimes not, so counters appear to jump and rates are wrong. Either push to a gateway on completion, or aggregate in a long-lived process. This is the single most common reason raster metrics look implausible.

Counters that reset on every task. A fresh process starts every counter at zero, and Prometheus interprets a decrease as a counter reset — which it is, but the semantics are wrong when the “reset” is one worker of many. Aggregating in a sidecar or gateway that persists across tasks avoids inventing throughput at every task boundary.

A label that is bounded in theory and not in practice. zoom looks like twenty values until someone adds a per-scene pyramid with zooms up to 22 and a second grid with its own numbering. Assert the bound in code — a lookup that raises on an unexpected value — rather than trusting the domain to stay as it was.

Histograms that outlive their calibration. Buckets sized for a 2-vCPU worker are wrong on a 16-vCPU one, and the percentiles quietly degrade as everything shifts into fewer buckets. Re-measure after any change in worker class, and treat bucket boundaries as configuration that has an owner.

Aggregation across workers that hides a bad one. Fleet-wide megapixels per second looks fine while one worker does nothing, because the others compensate. Keep a worker label only if the fleet is small and stable; otherwise rely on the in-flight gauge and on per-worker up to catch stalls.

A skipped tile counted as throughput. When the skip branch is working well, most tiles are skipped, and a pixel counter that increments for skips reports a throughput the pipeline is not achieving. Increment PIXELS only on real work — the outcome label on the tile counter is where skips are visible — or the capacity arithmetic above silently assumes a fleet several times more productive than it is.

Metrics that exist and are never looked at. The most common failure of all. A metric earns its place by appearing on a dashboard or in an alert; one that does neither is cost without benefit, and pruning it is a legitimate maintenance activity rather than a loss.

Configuration Reference#

There is one more decision that is not a setting: what the metrics are for. Throughput metrics answer capacity questions — is the pipeline keeping up, does it need more workers, did the last change help. They are poor at answering correctness questions, because a pipeline can process pixels at full speed while producing the wrong ones. Keeping that boundary clear stops the dashboard from accumulating metrics that look like data quality and are not; those belong with the data-quality SLOs, measured from the output rather than from the process.

Frequently Asked Questions#

Related#

• Instrumenting gdalwarp with Prometheus counters — measuring an out-of-process GDAL step
• Measuring tile generation latency percentiles — sizing buckets and reading the result
• Exporting per-tile metrics without cardinality blowups — when you genuinely need per-tile detail
• Building a raster pipeline Grafana dashboard — what to do with these series
• Data-quality SLOs for spatial pipelines — the correctness questions these metrics cannot answer

← Observability & Monitoring for Geospatial Pipelines