Prometheus Metrics for Raster Throughput

Instrument raster warp, mosaic, and tiling stages with the prometheus_client library using Prometheus metrics for raster throughput: Counters for cumulative pixels and bytes, Histograms for per-tile duration, and Gauges for in-flight jobs — with labels restricted to a bounded set so a per-tile pyramid never explodes your time-series database. This guide gives the exact metric definitions, the pull-versus-push decision for batch geotasks, and a bucket configuration tuned to the multi-second reality of raster work.

The direct recommendation: define three Counters (pixels, bytes, tiles), one Histogram per stage with explicit raster-scale buckets, and one Gauge for concurrency. Label only on operation, output format, and target CRS — never on tile coordinates. Scrape long-lived workers; push from short-lived batch jobs via a Pushgateway. Everything else in this page is the reasoning and the code behind those four sentences.

Prerequisites & Architecture Baseline#

• prometheus-client==0.20.0 installed in the worker environment
• rasterio==1.3.10 (or GDAL 3.8+ bindings) for reading raster dimensions and CRS
• A Prometheus server (2.50+) able to reach worker /metrics endpoints, or a pushgateway (1.9+) for batch jobs
• Workers expose a stable HTTP port for scraping, or write to the Pushgateway on completion
• Metric names agreed with the platform team and namespaced with a raster_ prefix
• A label allowlist documented so instrumentation reviews can reject unbounded labels

Raster throughput monitoring assumes you already have the orchestration in place — the DAG design principles for spatial ETL section covers structuring the warp-and-tile stages that this page instruments.

Core Concepts#

1. Counter for monotonic volume. A Counter only increases, which fits cumulative quantities: total pixels reprojected, total bytes read, total tiles written. You query the derivative with PromQL rate() to get throughput per second. Never model a rate directly as a Gauge — you lose the ability to re-aggregate across time windows and to survive scrape gaps.

2. Histogram for duration distribution. A Histogram buckets observations into cumulative counts, letting you compute latency percentiles server-side with histogram_quantile(). For raster stages the bucket bounds are the whole game: buckets designed for sub-second HTTP calls are useless when a mosaic runs for ninety seconds. Each stage — warp, mosaic, tile — gets its own Histogram because their duration profiles differ by an order of magnitude.

3. Gauge for in-flight concurrency. A Gauge goes up and down, which fits the count of jobs currently executing. Increment on task entry, decrement in a finally block so a crash does not leave the Gauge stuck high. Saturation of this Gauge against a known worker-pool ceiling is your earliest warning of a throughput stall.

4. Label hygiene as a first-class constraint. Every distinct label-value combination is a separate stored time series. Bounded dimensions — operation name, output driver (GTiff, COG, PNG), target EPSG from a known projection set — are safe. Unbounded dimensions — tile x/y, feature ids, file paths, full PROJ strings — are forbidden. The cost of a bad label is a Prometheus server that falls over, so this is a hard rule, not a style preference.

5. Collection topology: pull versus push. Prometheus prefers to scrape a stable endpoint. Long-running tile servers satisfy that. Short-lived batch warp jobs that start, run, and exit between scrapes do not — their metrics never get read. Those jobs push their final counters and histogram observations to a Pushgateway, which holds them for the next scrape.

Production Implementation#

The module below instruments a three-stage raster pipeline. It defines the metric family once at import time, wraps each stage, and preserves nodata and CRS through the actual raster work so the numbers describe correct output.

import time
import rasterio
from rasterio.warp import reproject, Resampling, calculate_default_transform
from pyproj import CRS
from contextlib import contextmanager
from prometheus_client import Counter, Histogram, Gauge

# --- Metric family: labels are bounded dimensions only ---
PIXELS = Counter(
    "raster_pixels_processed_total",
    "Cumulative pixels written, by stage and target CRS",
    labelnames=("stage", "dst_epsg"),
)
BYTES_READ = Counter(
    "raster_bytes_read_total",
    "Cumulative source bytes read, by stage",
    labelnames=("stage",),
)
STAGE_SECONDS = Histogram(
    "raster_stage_seconds",
    "Wall-clock duration of a raster stage",
    labelnames=("stage", "dst_epsg"),
    # Buckets tuned to multi-second raster work, not sub-second HTTP.
    buckets=(0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128),
)
IN_FLIGHT = Gauge(
    "raster_jobs_in_flight",
    "Raster jobs currently executing, by stage",
    labelnames=("stage",),
)

@contextmanager
def track(stage: str, dst_epsg: str):
    """Time a stage, count concurrency, and never leak the Gauge on error."""
    IN_FLIGHT.labels(stage=stage).inc()
    start = time.perf_counter()
    try:
        yield
    finally:
        STAGE_SECONDS.labels(stage=stage, dst_epsg=dst_epsg).observe(
            time.perf_counter() - start
        )
        IN_FLIGHT.labels(stage=stage).dec()

def warp_stage(src_path: str, dst_path: str, dst_crs: CRS) -> int:
    with rasterio.open(src_path) as src:
        src_crs: CRS = src.crs
        d_epsg = str(dst_crs.to_epsg())
        BYTES_READ.labels(stage="warp").inc(src.width * src.height * src.count *
                                            src.dtypes[0].__len__())
        with track("warp", d_epsg):
            transform, width, height = calculate_default_transform(
                src_crs, dst_crs, src.width, src.height, *src.bounds
            )
            profile = src.profile.copy()
            profile.update(crs=dst_crs, transform=transform,
                           width=width, height=height, nodata=src.nodata)
            with rasterio.open(dst_path, "w", **profile) as dst:
                for band in range(1, src.count + 1):
                    reproject(
                        source=rasterio.band(src, band),
                        destination=rasterio.band(dst, band),
                        src_crs=src_crs, dst_crs=dst_crs,
                        src_nodata=src.nodata, dst_nodata=src.nodata,
                        resampling=Resampling.bilinear,
                    )
            written = width * height * src.count
    PIXELS.labels(stage="warp", dst_epsg=d_epsg).inc(written)
    return written

For the mosaic and tile stages you reuse track() and PIXELS with stage="mosaic" or stage="tile", keeping the metric family flat and the label set identical. The companion how-to on instrumenting gdalwarp with Prometheus counters drills into the subprocess variant where the warp runs as an external gdalwarp invocation rather than in-process rasterio.

Step-by-Step Walkthrough#

1. Declare metrics at module scope. The Counter, Histogram, and Gauge objects are created once when the module imports, registering into the default registry. Creating them inside a function would re-register on every call and raise a duplicate-timeseries error.

2. Choose labels from bounded sets. stage is one of three constant strings; dst_epsg comes from your projection allowlist. Both are enumerable, so total series count stays in the low hundreds regardless of how many tiles you process.

3. Count bytes before the work. BYTES_READ.inc() fires on the source dimensions so the counter reflects input volume even if the warp later fails partway.

4. Wrap the work in track(). The context manager increments the concurrency Gauge, times the body, records the Histogram observation, and decrements the Gauge in finally — so an exception mid-warp still cleans up.

5. Preserve correctness inside the timed body. nodata flows into the output profile and into every reproject call, and the target CRS is set on the profile. The metrics only mean something if the output they describe is spatially correct.

6. Increment pixel volume on success. PIXELS.inc(written) fires after the write completes, so the counter tracks delivered pixels, not attempted ones. rate(raster_pixels_processed_total[5m]) then gives live pixels-per-second per stage and CRS.

Edge Cases & Failure Recovery#

Batch job exits before scrape. A one-shot warp container finishes in eight seconds; Prometheus scrapes every fifteen. Its metrics are never read. Fix: push to a Pushgateway on completion with a stable grouping key, and remember the Pushgateway holds the last value until overwritten — stale values must be deleted when a job group retires.

Cardinality creep from a well-meaning label. Someone adds output_path to help debugging; series count quietly grows unbounded. Detection: alert on count({__name__=~"raster_.*"}) crossing a threshold. Fix: remove the label; move path-level detail to structured logs as described in structured logging for geospatial flows.

Gauge stuck high after a crash. A worker is killed mid-warp and IN_FLIGHT never decrements, so saturation looks permanent. The finally block prevents this within a process, but a hard kill still leaks. Fix: on worker restart, reset the Gauge to zero; for scraped workers a process restart clears the registry anyway.

Histogram top-bucket saturation. A pathological 200-second mosaic lands above the 128-second top bucket, so histogram_quantile cannot resolve the true p99. Fix: extend the bucket list or, better, split the mosaic stage — chronically saturated buckets mean the stage is too coarse. The tile-latency percentiles guide covers bucket selection in depth.

Counter reset on redeploy. A worker restart resets Counters to zero, which looks like a throughput cliff. This is expected and rate() handles counter resets automatically — do not “fix” it by making the counter a Gauge, which would break re-aggregation.

Configuration Reference#

# raster_metrics.yaml — tunables for the instrumentation layer
metrics:
  namespace_prefix: "raster_"
  labels:
    allowed: ["stage", "dst_epsg", "output_driver"]   # bounded sets only
    forbidden: ["tile_x", "tile_y", "z", "feature_id", "path", "proj4"]
  histogram_buckets:
    warp:   [0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128]  # seconds
    mosaic: [1, 2, 4, 8, 16, 32, 64, 128, 256]        # mosaics run longer
    tile:   [0.05, 0.1, 0.25, 0.5, 1, 2, 4]           # per-tile is fast
collection:
  mode: "scrape"            # scrape | push
  scrape_port: 9105
  scrape_interval_hint: "15s"
  pushgateway:
    url: "http://pushgateway:9091"
    use_for: "short_lived_batch"
    grouping_keys: ["job", "worker_id"]
    delete_group_on_complete: true   # avoid stale pushed values

The forbidden-label list is worth enforcing in code, not just documentation — a small check that raises when a metric is created with a banned label name catches the mistake at import rather than in production. This baseline slots into the wider Observability & Monitoring for Geospatial Pipelines telemetry design, and the Prometheus client library docs document every metric type in full.

Frequently Asked Questions#