Geospatial Orchestration Hub

Building a Raster Pipeline Grafana Dashboard

To build a raster pipeline Grafana dashboard, write three PromQL queries — pixels processed per second, warp-duration p95, and in-flight mosaic jobs — bind each to a panel, template the whole board by pipeline and zoom, then ship the JSON model and provisioning YAML in your repo. This walkthrough assembles exactly that dashboard from empty file to provisioned artifact, so a raster ingest and mosaic pipeline becomes observable in one screen.

The dashboard here is deliberately narrow: it monitors a GDAL-based reproject-and-mosaic pipeline. It is the concrete counterpart to the broader Grafana dashboards for GIS workflows guide, and it consumes the metrics defined in instrumenting gdalwarp with Prometheus counters.

When to Use This Pattern

Reach for a dedicated raster dashboard when:

  • You run a warp or mosaic pipeline whose throughput is measured in pixels or tiles, not rows.
  • Warp latency has a heavy upper distribution you need to watch at p95 rather than as an average.
  • Multiple mosaic jobs run concurrently and you need to see how many are in flight before workers OOM.
  • You want the dashboard reviewed and deployed as code, not hand-clicked per environment.
  • You template one board across several raster pipelines instead of maintaining copies.

Complete Working Example

Start with the three queries. Each is scoped by the $pipeline and $zoom template variables so a single board serves the whole raster fleet.

# Panel 1 — pixels processed per second (throughput), summed per pipeline
sum by (pipeline) (
  rate(raster_pixels_processed_total{pipeline=~"$pipeline"}[5m])
)

# Panel 2 — gdalwarp duration p95, from a histogram, per pipeline
histogram_quantile(
  0.95,
  sum by (le, pipeline) (
    rate(gdalwarp_duration_seconds_bucket{pipeline=~"$pipeline"}[5m])
  )
)

# Panel 3 — in-flight mosaic jobs (a gauge read directly, never rate())
sum by (pipeline) (
  mosaic_jobs_in_flight{pipeline=~"$pipeline", zoom=~"$zoom"}
)

The instrumentation feeding these lives in the worker. Here is the minimal Python that produces all three metrics with the labels the queries expect, so the dashboard has something real to read.

import time
from typing import Iterator
import rasterio
from rasterio.warp import calculate_default_transform, reproject, Resampling
from pyproj import CRS
from prometheus_client import Counter, Histogram, Gauge

PIXELS = Counter(
    "raster_pixels_processed_total",
    "Pixels warped/reprojected",
    ["pipeline", "zoom"],
)
WARP_SECONDS = Histogram(
    "gdalwarp_duration_seconds",
    "Wall-clock seconds per warp job",
    ["pipeline"],
    buckets=(0.5, 1, 2, 5, 10, 30, 60, 120, 300),
)
IN_FLIGHT = Gauge(
    "mosaic_jobs_in_flight",
    "Concurrent mosaic jobs",
    ["pipeline", "zoom"],
)

def warp_tile(src_path: str, dst_path: str, pipeline: str, zoom: int,
              dst_crs: str = "EPSG:3857") -> int:
    """Reproject one raster to dst_crs, emitting Prometheus metrics."""
    IN_FLIGHT.labels(pipeline=pipeline, zoom=str(zoom)).inc()
    start: float = time.perf_counter()
    try:
        with rasterio.open(src_path) as src:
            target: CRS = CRS.from_user_input(dst_crs)
            transform, width, height = calculate_default_transform(
                src.crs, target, src.width, src.height, *src.bounds
            )
            profile = src.profile.copy()
            profile.update(
                crs=target, transform=transform, width=width, height=height,
                nodata=src.nodata if src.nodata is not None else 0,
            )
            with rasterio.open(dst_path, "w", **profile) as dst:
                for band in range(1, src.count + 1):
                    reproject(
                        source=rasterio.band(src, band),
                        destination=rasterio.band(dst, band),
                        src_crs=src.crs, dst_crs=target,
                        src_nodata=src.nodata, dst_nodata=profile["nodata"],
                        resampling=Resampling.bilinear,
                    )
            pixels: int = width * height
            PIXELS.labels(pipeline=pipeline, zoom=str(zoom)).inc(pixels)
            return pixels
    finally:
        WARP_SECONDS.labels(pipeline=pipeline).observe(
            time.perf_counter() - start
        )
        IN_FLIGHT.labels(pipeline=pipeline, zoom=str(zoom)).dec()

The try/finally matters: the IN_FLIGHT gauge must decrement and the histogram must observe even when a warp raises, otherwise a failing job leaves a phantom in-flight count that never clears and the panel lies. Note that nodata is always set explicitly — an unset nodata silently blends edge pixels into mosaics, a defect no throughput panel will reveal.

Assembling the Provisioned Dashboard JSON

With metrics flowing, bind the queries to panels in a JSON model. Below is a trimmed model showing the three panels, the two template variables, and the data source reference. Real exports include layout gridPos and field configs; those are elided for readability but the structure is complete enough to provision.

{
  "title": "Raster Pipeline",
  "uid": "raster-pipeline",
  "editable": false,
  "templating": {
    "list": [
      {
        "name": "pipeline",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "geo-prom" },
        "query": "label_values(raster_pixels_processed_total, pipeline)",
        "includeAll": true,
        "multi": true
      },
      {
        "name": "zoom",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "geo-prom" },
        "query": "label_values(mosaic_jobs_in_flight{pipeline=~\"$pipeline\"}, zoom)",
        "includeAll": true,
        "sort": 3
      }
    ]
  },
  "panels": [
    {
      "title": "Pixels / sec",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "geo-prom" },
      "targets": [
        {
          "expr": "sum by (pipeline) (rate(raster_pixels_processed_total{pipeline=~\"$pipeline\"}[5m]))",
          "legendFormat": "{{pipeline}}"
        }
      ]
    },
    {
      "title": "Warp duration p95",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "geo-prom" },
      "fieldConfig": { "defaults": { "unit": "s" } },
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum by (le, pipeline) (rate(gdalwarp_duration_seconds_bucket{pipeline=~\"$pipeline\"}[5m])))",
          "legendFormat": "p95 {{pipeline}}"
        }
      ]
    },
    {
      "title": "Mosaic jobs in-flight",
      "type": "stat",
      "gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "geo-prom" },
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              { "color": "green", "value": null },
              { "color": "orange", "value": 8 },
              { "color": "red", "value": 16 }
            ]
          }
        }
      },
      "targets": [
        {
          "expr": "sum by (pipeline) (mosaic_jobs_in_flight{pipeline=~\"$pipeline\", zoom=~\"$zoom\"})",
          "legendFormat": "{{pipeline}}"
        }
      ]
    }
  ],
  "schemaVersion": 39
}

Provision it with a file provider so it deploys with the stack rather than being clicked in:

# grafana/provisioning/dashboards/raster.yml
apiVersion: 1
providers:
  - name: raster-pipeline
    orgId: 1
    folder: GIS Pipelines
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

Mount the JSON at /var/lib/grafana/dashboards/raster-pipeline.json, and Grafana loads it on startup. Because editable and allowUiUpdates are both false, the committed file stays the single source of truth.

Laying Out the Grid and Templating by Zoom

Grafana positions panels with gridPos, a 24-column grid where w is width in columns and h is height in eight-pixel rows. The three panels above sit on one row at x: 0, x: 8, and x: 16, each eight columns wide, which fills the width evenly. When you add the CRS-correctness and memory panels from the parent guide, drop them onto a second row by setting y: 8. Keeping throughput on the top row and correctness or resources below mirrors how an operator reads the board top-to-bottom: is it moving, then is it healthy.

The $zoom variable is what makes this board scale across a tiling pipeline. A mosaic job at zoom 6 processes a handful of coarse tiles, while zoom 14 fans out into thousands of fine tiles with very different throughput and memory profiles. Templating the in-flight panel by zoom lets an operator isolate a single zoom level during an incident — for example, confirming that only high-zoom jobs are backing up because they exhaust worker memory faster. Because the zoom variable is populated by label_values(mosaic_jobs_in_flight{pipeline=~"$pipeline"}, zoom), it only ever offers zoom levels that the currently selected pipeline actually emits, so the dropdown never shows irrelevant options.

To load the board without restarting Grafana during development, drop the JSON into the provisioned path and trigger a reload, or import it once through the UI to preview before committing:

# Copy the model into the provisioned directory; Grafana picks it up
# within updateIntervalSeconds (30s) without a restart
cp raster-pipeline.json /var/lib/grafana/dashboards/raster-pipeline.json
# Confirm Grafana loaded it (requires an API token)
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/dashboards/uid/raster-pipeline \
  | python -c "import sys, json; print(json.load(sys.stdin)['dashboard']['title'])"

Once the board is committed and provisioning is green, treat any change as a code change: edit the JSON, open a pull request, and let review catch a broken query before it reaches the on-call view.

Parameter & Option Reference

Parameter Type Default Spatial notes
pipeline variable query includeAll, multi Regex-matched in every panel via =~"$pipeline"
zoom variable query includeAll, sort: 3 Numeric sort keeps zoom 0..18 in order
rate() window duration 5m Widen to 10m for spiky mosaic batches
histogram buckets seconds 0.5..300 Cover fast tiles and slow full-scene warps
in-flight thresholds int 8 / 16 Amber/red mapped to worker concurrency limits
warp p95 unit string s Set panel unit so axis reads seconds, not raw floats
updateIntervalSeconds int 30 Grafana reload cadence for the provisioned file

Verification & Testing

Confirm the metrics exist before trusting a panel. Scrape the worker’s /metrics endpoint and check each series is present with the expected labels:

# Confirm all three metric families are exposed with labels
curl -s http://localhost:8000/metrics | grep -E \
  'raster_pixels_processed_total|gdalwarp_duration_seconds_bucket|mosaic_jobs_in_flight'

Then validate each PromQL expression against the Prometheus HTTP API — if these return data, the panels will too:

# p95 warp duration should return a numeric value, not empty
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.95, sum by (le, pipeline) (rate(gdalwarp_duration_seconds_bucket[5m])))' \
  | python -c "import sys, json; print(json.load(sys.stdin)['data']['result'])"

Finally, assert the instrumentation contract in a unit test so a refactor can’t silently drop a label the dashboard depends on:

from prometheus_client import REGISTRY

def test_warp_emits_labelled_metrics(tmp_path) -> None:
    warp_tile("fixtures/small.tif", str(tmp_path / "out.tif"),
              pipeline="ingest", zoom=10, dst_crs="EPSG:3857")
    pixels = REGISTRY.get_sample_value(
        "raster_pixels_processed_total",
        {"pipeline": "ingest", "zoom": "10"},
    )
    assert pixels is not None and pixels > 0
    # in-flight must return to zero after the job completes
    inflight = REGISTRY.get_sample_value(
        "mosaic_jobs_in_flight",
        {"pipeline": "ingest", "zoom": "10"},
    )
    assert inflight == 0.0

Common Pitfalls

  • Charting rate() over the in-flight gauge. Concurrency is a level, not a flow. Read mosaic_jobs_in_flight directly with sum by (pipeline); wrapping it in rate() produces meaningless near-zero values.
  • Dropping the le label before histogram_quantile. Aggregate with sum by (le, pipeline) or the p95 panel flat-lines. This is the number-one cause of “the latency panel shows nothing”.
  • Forgetting nodata in the warp. The throughput panel will look healthy while mosaics develop black seams. Correctness never shows up on a pixels-per-second chart — validate it upstream via validating coordinate systems before ETL.
  • Zoom variable sorted lexically. Without sort: 3 the dropdown reads 0, 1, 10, 2, which misleads during an incident. Use numeric sort.
  • Histogram buckets that miss the tail. If your slowest full-scene warps take four minutes but the top bucket is 60, everything above a minute collapses into +Inf and p95 reads a meaningless ceiling. Size the top bucket above your worst observed warp — the (0.5 .. 300) range here covers a five-minute reproject. Revisit the buckets whenever you add a larger source raster.
  • Committing the volatile id field. Grafana stamps a numeric id and rising version on every export. Leave them in and each save produces a noisy diff and, worse, a provisioning conflict if two environments assign different ids. Strip them before committing so the uid alone identifies the board.

Related: This board pairs with the metric definitions in instrumenting gdalwarp with Prometheus counters, the latency methodology in measuring tile generation latency percentiles, and the correctness alert in alerting on CRS validation failures in Grafana. See the parent Grafana dashboards for GIS workflows guide for layout principles.

← Back to Grafana Dashboards for GIS Workflows

Related in this section