Geospatial Orchestration Hub

Building a Raster Pipeline Grafana Dashboard

A raster pipeline’s dashboard is nine panels in four rows, and the value is in which nine. The top row says whether to keep reading; the second compares throughput against the same time last week; the third reports what the output actually looks like; the fourth puts the failures on a map. Each panel carries its own description of what healthy means, thresholds that agree with that description, and a query that someone can paste into Prometheus or psql unchanged. Everything else goes into a collapsed row at the bottom.

When to Use This Pattern

  • A raster pipeline is in production and the current dashboard is either absent or an accumulation.
  • More than one person is on call, so the dashboard has to be readable by someone who did not build it.
  • The metrics and tables already exist — this recipe assembles them rather than defining them.
  • You want the dashboard in version control, which is the only way its history is reviewable.

Complete Working Example

The provisioning file and the dashboard JSON live in the repository; Grafana reads them at start-up and on change.

# /etc/grafana/provisioning/dashboards/raster.yaml
apiVersion: 1
providers:
  - name: raster-pipelines
    folder: Pipelines
    type: file
    disableDeletion: false
    # The UI becomes read-only for these, which is the point: changes go through
    # the repository, so a panel that moved has a commit explaining why.
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

The nine panels, with the queries that back them. Row one is the only row that needs to be read every day:

{
  "title": "Raster pipeline — $layer",
  "time": { "from": "now-2d", "to": "now" },
  "refresh": "1m",
  "templating": { "list": [
    { "name": "layer", "type": "query", "datasource": "Prometheus",
      "query": "label_values(raster_pixels_processed_total, layer)" }
  ]},
  "panels": [
    { "gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
      "title": "Did the last run finish, and when?",
      "description": "Green if a run completed within the last 26 h. Red means the schedule stopped, which no other panel detects.",
      "type": "stat",
      "targets": [{ "expr": "time() - max(raster_run_completed_timestamp{layer=\"$layer\"})" }],
      "fieldConfig": { "defaults": { "unit": "s", "thresholds": { "steps": [
        { "color": "green", "value": null }, { "color": "red", "value": 93600 }]}}}},

    { "gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
      "title": "What share of tiles failed?",
      "description": "Healthy is under 0.5%. Above 2% the mosaic will have visible gaps.",
      "type": "stat",
      "targets": [{ "expr":
        "sum(rate(raster_tiles_total{layer=\"$layer\",outcome=\"failed\"}[1h])) / sum(rate(raster_tiles_total{layer=\"$layer\"}[1h]))" }],
      "fieldConfig": { "defaults": { "unit": "percentunit", "thresholds": { "steps": [
        { "color": "green", "value": null }, { "color": "orange", "value": 0.005 },
        { "color": "red", "value": 0.02 }]}}}},

    { "gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
      "title": "Is the dead-letter queue growing?",
      "description": "The 24 h delta. Sustained growth means a failure class nobody has triaged.",
      "type": "stat",
      "targets": [{ "expr": "delta(geotask_dlq_open_entries{pipeline=\"$layer\"}[24h])" }],
      "fieldConfig": { "defaults": { "thresholds": { "steps": [
        { "color": "green", "value": null }, { "color": "orange", "value": 50 }]}}}},

    { "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
      "title": "Is it keeping up? (Mpx/s, vs. last week)",
      "description": "Healthy is 40-60 Mpx/s on the 12-worker pool. Below 25 the run misses 06:00.",
      "type": "timeseries",
      "targets": [
        { "expr": "sum(rate(raster_pixels_processed_total{layer=\"$layer\"}[5m]))/1e6",
          "legendFormat": "now" },
        { "expr": "sum(rate(raster_pixels_processed_total{layer=\"$layer\"}[5m] offset 1w))/1e6",
          "legendFormat": "last week" }]},

    { "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
      "title": "Where is the time going? (warp duration)",
      "description": "A heatmap, not percentiles: this workload is bimodal and a p95 sits in the valley.",
      "type": "heatmap",
      "targets": [{ "expr":
        "sum by (le) (rate(raster_warp_duration_seconds_bucket{layer=\"$layer\"}[10m]))",
        "format": "heatmap" }]}
  ]
}
Nine panels, four rows, one screenRow one holds three stat panels. Row two holds a throughput time series and a duration heatmap. Row three holds coverage and freshness. Row four holds the failure map spanning the full width.last run finishedfailure sharedead-letter growthMpx/s vs last weekwarp duration heatmaptile coverage %oldest tile agegeomap — failed tiles, last 24 h
The grid positions are part of the design. Nine panels fit one screen at this layout; a tenth pushes the map below the fold and it stops being looked at.

Parameter & Option Reference

Panel Query source Threshold Spatial notes
last run finished Prometheus gauge 26 h Catches a stopped schedule, which no throughput panel detects.
failure share ratio of counters 0.5% / 2% A ratio, not a count — the count is meaningless without the denominator.
dead-letter growth delta(...[24h]) +50 The trend, never the depth.
Mpx/s vs last week rate + offset 1w 25 / 40 The comparison is what makes the number a judgement.
warp duration histogram buckets — A heatmap, because the distribution is bimodal.
tile coverage PostGIS 99.5% From the ledger, not from metrics — coverage is a property of the output.
oldest tile age PostGIS per layer The freshness number, which differs by layer by orders of magnitude.
failure map PostGIS geomap — Aggregated server-side, fitted to the layer’s extent.

Verification & Testing

Dashboards are JSON, so they can be linted and their queries can be tested before anyone sees them.

import json
from pathlib import Path


def test_every_panel_has_a_description() -> None:
    dash = json.loads(Path("dashboards/raster.json").read_text())
    missing = [p["title"] for p in dash["panels"] if not p.get("description")]
    assert not missing, f"panels without a description: {missing}"


def test_thresholds_agree_with_descriptions() -> None:
    """A description saying 'below 25' must have a threshold at 25."""
    dash = json.loads(Path("dashboards/raster.json").read_text())
    panel = next(p for p in dash["panels"] if "keeping up" in p["title"])
    steps = panel["fieldConfig"]["defaults"]["thresholds"]["steps"]
    assert any(s.get("value") == 25 for s in steps)


def test_queries_are_valid(prometheus_client) -> None:
    dash = json.loads(Path("dashboards/raster.json").read_text())
    for panel in dash["panels"]:
        for target in panel.get("targets", []):
            if "expr" not in target:
                continue
            expr = target["expr"].replace("$layer", "ortho")
            # A parse error here is a broken panel that would otherwise be
            # discovered by whoever opens the dashboard during an incident.
            assert prometheus_client.parse(expr), f"invalid: {expr}"


def test_panels_fit_one_screen() -> None:
    dash = json.loads(Path("dashboards/raster.json").read_text())
    visible = [p for p in dash["panels"] if not p.get("collapsed")]
    bottom = max(p["gridPos"]["y"] + p["gridPos"]["h"] for p in visible)
    assert bottom <= 24, "the map has been pushed below the fold"

The last test is the one that stops the dashboard growing. Grafana’s grid is 24 units tall on a typical screen, so asserting that the visible panels fit within it turns “we should keep this focused” from a good intention into a build failure. Anyone adding a tenth panel then has to decide which of the nine it replaces, or put it in the collapsed row — which is exactly the conversation the assertion is there to force.

The comparison series is what makes the panel readableA throughput line alone shows a value that could be normal or alarming. The same line with last week's overlaid immediately shows it is thirty per cent below normal for this time of night.WITHOUT THE COMPARISON“31 Mpx/s.” Is that fine?WITH offset 1w“31, against 45 last week.” No, it is not.One extra query target, no extra storage, and the panel stops requiring someone to remember what normal was.
The dashed line costs one line of JSON. It is the highest return per character available anywhere in a Grafana dashboard.
Row one: three panels, three failures nothing else catchesThe last-run panel catches a stopped schedule. The failure-share panel catches a run that completed badly. The dead-letter growth panel catches a slow degradation that neither of the others sees.panelthe failure only it detectslast run finishedtime since completionthe schedule stopped entirelyevery other panel goes flat and greenfailure shareratio, last houra run that finished badlythroughput can look normal throughoutdead-letter growth24 h deltaa slow, weeks-long degradationinvisible to any single-run panel
Three panels is the minimum because these are three genuinely different failures. Dropping any one of them leaves a mode with no detector at all.

Common Pitfalls

  • Panels titled with metric names. raster_pixels_processed_total requires the reader to translate. The title should be the question, and the metric name belongs in the query where it is needed.
  • Thresholds that disagree with the description. The text says 25 and the colour changes at 40, so the panel is amber when the description says it is fine. People resolve the contradiction by ignoring both.
  • A tenth panel. The map goes below the fold and stops being looked at, which removes the one panel that distinguishes this from a generic pipeline dashboard. Collapse or replace.
  • Percentiles where the distribution is bimodal. The warp-duration panel is a heatmap for a reason. A p95 line on this workload sits between the two modes and describes no tile that ever ran.
  • Editing in the UI with allowUiUpdates: true. Changes accumulate untracked and diverge from the repository, and the next provisioning run either loses them or entrenches them. Keep the UI read-only.
  • A refresh interval faster than the data. A nightly pipeline refreshed every thirty seconds runs its SQL panels thousands of times a day to display a number that changes once.

Frequently Asked Questions

Where does the coverage panel's query come from?

The pipeline’s own ledger, not from metrics. count(*) FILTER (WHERE built_at > now() - interval '24 hours') / count(*) over the expected tile set, grouped by layer. Coverage is a property of the output and the metrics only describe the process — a pipeline can process pixels at full speed and still be missing a region, which is precisely the case this panel exists to catch.

Should each layer get its own dashboard?

No — that is what the $layer variable is for. One dashboard with a variable stays consistent; eleven copies drift within a quarter, and the drift is invisible until someone compares two of them during an incident. If a layer genuinely needs different panels, that is a signal about the layer, not about dashboards.

What goes in the collapsed row?

Everything that is useful during an investigation and noise the rest of the time: per-zoom breakdowns, source-scheme splits, worker-level in-flight, GDAL peak-memory distribution. Collapsed rows cost nothing when closed and are one click away, which is the right price for detail that is needed monthly.

How do I test a dashboard change before merging?

Point a local Grafana at the branch’s provisioning directory with a read-only copy of the datasources. The four assertions above catch structural problems in CI; a local render catches the ones that only appear visually, like a panel whose legend covers the data. Both together take under a minute and prevent the class of change that is only discovered when someone opens the dashboard under pressure.

Should alerts be defined in the same file?

Yes, in Grafana’s unified alerting format alongside the dashboard, referencing the same queries. Keeping them together is what stops the alert and the panel diverging — and divergence between them is the failure that produces an alert nobody can corroborate from the dashboard it supposedly came from.

← Grafana Dashboards for GIS Workflows

Related in this section