Geospatial Orchestration Hub

Alerting on CRS Validation Failures in Grafana

A CRS validation failure is one of the few spatial errors worth paging on, because the alternative to catching it at the boundary is discovering it in a published product weeks later. Two rules cover it: a rate rule that fires when validation rejections climb above the normal trickle, and a new-source rule that fires the first time a source that has always been clean starts failing. Route both to whoever owns the data, not to whoever owns the pipeline — the fix is almost always a conversation with the publisher.

When to Use This Pattern

  • A validation boundary exists and emits a metric per outcome, as described in spatial validation & sync tasks.
  • Sources are outside your control, so a declaration can change without warning.
  • Wrong projections have reached production before, which is the experience that makes this alert obviously worth having.
  • There is someone to route to — an alert with no owner is a notification, not an alert.

Complete Working Example

The metric comes first: one counter, with the verdict as a label and the source group bounded.

from prometheus_client import Counter

CRS_CHECKS = Counter(
    "crs_validation_total",
    "CRS validation outcomes at the ingest boundary",
    # source_group, not source_uri: a dozen values, not four thousand.
    ("layer", "source_group", "verdict"),
)


def validate_and_count(path, layer: str, source_group: str) -> CrsCheck:
    check = check_crs(path)
    CRS_CHECKS.labels(layer=layer, source_group=source_group,
                      verdict=check.verdict.value).inc()
    return check

The rules live beside the dashboard, in Grafana’s unified alerting format, so they are provisioned from the same repository:

# provisioning/alerting/crs.yaml
apiVersion: 1
groups:
  - orgId: 1
    name: spatial-validation
    folder: Pipelines
    interval: 5m
    rules:
      - title: CRS validation failures climbing
        condition: threshold
        for: 15m            # ride out a single bad delivery
        annotations:
          summary: "{{ $labels.source_group }} is failing CRS validation ({{ $values.A }}/h)"
          description: >
            Normal is under 2/h. Above 20/h a source has almost certainly changed
            its declared projection. Do NOT reproject to compensate — check with
            the publisher. Runbook: /runbooks/crs-validation
        labels:
          severity: page
          team: data-stewardship          # the data owner, not the platform team
        data:
          - refId: A
            relativeTimeRange: { from: 3600, to: 0 }
            model:
              expr: |
                sum by (layer, source_group) (
                  rate(crs_validation_total{verdict!="ok"}[1h])
                ) * 3600
          - refId: threshold
            model:
              type: threshold
              conditions: [{ evaluator: { type: gt, params: [20] } }]

      - title: A clean source started failing CRS validation
        condition: newly-failing
        for: 10m
        annotations:
          summary: "{{ $labels.source_group }} failed CRS validation for the first time in 30 d"
          description: >
            This source has been clean for a month. A first failure is usually a
            changed export configuration upstream, and is worth a message today.
        labels:
          severity: page
          team: data-stewardship
        data:
          - refId: newly-failing
            model:
              expr: |
                (sum by (source_group) (increase(crs_validation_total{verdict!="ok"}[1h])) > 0)
                unless
                (sum by (source_group) (increase(crs_validation_total{verdict!="ok"}[30d] offset 1h)) > 0)
What the rule is watching forValidation verdicts are almost entirely fine for six days. On the seventh, out-of-area rejections rise from near zero to sixty an hour when one source changes its declared projection.per hourMonThuSunthreshold — 20/h, held for 15 minthe source re-exported in EPSG:4258Six days of two an hour, then sixty. The threshold sits well above the noise and well below the signal.
The gap between the baseline and the incident is what makes this alert quiet and reliable. Where a metric has no such gap, it is a dashboard panel rather than an alert.

Two rules rather than one is a deliberate redundancy, and each catches something the other misses. The rate rule detects a large, ongoing problem — a national source that re-exported everything in a different projection — but it is blind to a small source whose entire delivery is twelve features, because twelve failures an hour never reaches twenty. The new-source rule catches exactly that case: it does not care about volume, only about a source that was clean and is not any more. Between them, a change in any publisher’s behaviour produces a page within an hour, which is the property worth having.

Why two rules rather than oneThe rate rule detects large-volume failures and misses a small source entirely. The new-source rule detects any first failure regardless of volume. Together they cover both.scenariorate rulenew-source rulenational source re-exportedfiresfiressmall municipality, 12 featuresmisses — under 20/hfireschronically messy sourcefires on a spikesilent — correctlyThe third row is why the new-source rule uses a thirty-day lookback rather than simply “any failure”.
Neither rule alone covers the middle row, and the middle row is the delivery most likely to reach production unnoticed.

Parameter & Option Reference

Setting Value Spatial notes
threshold 20/h From a fortnight of history, roughly ten times the baseline.
for 15 m Rides out one bad delivery; a real declaration change persists.
new-source lookback 30 d Long enough to cover a monthly publisher. Shorten for daily sources.
severity page Rare, unambiguous and time-sensitive — the profile a page is for.
team data stewardship The fix is a conversation with the publisher, not a code change.
source_group bounded A dozen values. source_uri in an alert label is the same cardinality mistake as in a metric.
runbook link in the annotation Including the instruction not to reproject, which is the tempting wrong response.

Verification & Testing

Alert rules are testable, and the second test below is the one that stops this becoming a nuisance.

# crs_alerts_test.yaml — promtool test rules crs_alerts_test.yaml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # A steady, healthy trickle: two an hour.
      - series: 'crs_validation_total{layer="ortho",source_group="national",verdict="out_of_area"}'
        values: '0+0.033x600'
    alert_rule_test:
      - eval_time: 300m
        alertname: CRS validation failures climbing
        exp_alerts: []          # must NOT fire on the baseline

  - interval: 1m
    input_series:
      # A source that changed projection: one a minute.
      - series: 'crs_validation_total{layer="ortho",source_group="national",verdict="out_of_area"}'
        values: '0+1x120'
    alert_rule_test:
      - eval_time: 60m
        alertname: CRS validation failures climbing
        exp_alerts:
          - exp_labels: { severity: page, team: data-stewardship, source_group: national }

Once the alert has fired for real, the follow-up is a query rather than a guess. The validation record says which sources, which verdict and — because the boundary recorded the bounds — what the coordinates actually looked like, which is what turns the conversation with the publisher into a specific one:

SELECT source_group, verdict, declared_crs, count(*) AS features,
       min(bounds_minx) AS minx, max(bounds_maxx) AS maxx
FROM   crs_validation_log
WHERE  checked_at > now() - interval '6 hours' AND verdict <> 'ok'
GROUP  BY source_group, verdict, declared_crs
ORDER  BY features DESC;
Who should receive this alertBoth CRS rules route to the data stewardship team because the fix is upstream. The platform team receives a notification but is not paged, since there is nothing for them to change.rate climbing> 20/h for 15 minclean source failingfirst time in 30 ddata stewardshippaged — they own the sourceplatform teaminformed, not pagedPaging the platform team for a source’s projection change produces a handover before anything can be done,which is the slowest possible path to the only useful action: asking the publisher.
Routing is part of the alert’s design, not an afterthought. An alert that reaches someone who cannot act on it is an alert that will be silenced.

Recording the bounds alongside the verdict is what makes the follow-up query useful rather than merely confirmatory. A verdict tells you the check failed; the observed coordinate range tells you how, and the two forms are diagnostic on sight — coordinates in the hundreds of thousands under a declaration of EPSG:4326 are UTM metres, and a latitude beyond ninety is a swapped axis pair. Publishers respond considerably faster to “your file declares 4326 and contains coordinates at 598000, 6643000” than to “your file failed our validation”.

Common Pitfalls

  • Alerting on the count rather than the rate. A cumulative counter grows forever, so a threshold on it fires once and never resolves. The alert wants rate or increase over a window.
  • A threshold with no gap below it. If the healthy baseline is 15/h and the threshold is 20/h, the alert fires on ordinary variance. Measure first; if there is no clear gap, this is a dashboard panel rather than an alert.
  • source_uri as a label. Same cardinality problem as in any metric, with the added consequence that the alert produces one notification per source rather than one per incident.
  • Routing to the platform team. They cannot fix a publisher’s projection. The page should reach whoever can talk to the source’s owner, and the platform team should see it without being woken.
  • A runbook that says “reproject to correct it”. Reprojecting a wrongly-declared source produces confidently wrong coordinates. The runbook must say to quarantine and ask, and saying so explicitly in the annotation is worth the characters.
  • Alerting on every verdict equally. undeclared is a broken delivery and outside_expected is often a stale expectation. Splitting them by severity keeps the page rare and the ticket useful.

Frequently Asked Questions

Why page rather than ticket?

Because the failure is silent downstream and expensive to unwind. A wrongly-projected source that loads successfully will propagate into tiles, statistics and published products, and each hour of delay adds to what has to be recalled. That combination — silent, expensive, and fixable only by someone else — is what justifies waking a person for what is technically a data-quality issue.

What if the source is right and our expectation is wrong?

That is the outside_expected verdict, and it should be a ticket rather than a page for exactly this reason. A delivery that legitimately expanded to a new region is a configuration update on your side, and it can wait until morning. Keeping the two verdicts on different severities is what makes the page trustworthy.

How do I stop this firing during a backfill?

A backfill of historical data may legitimately include sources with older declarations. Add a silence scoped to the backfill’s layer label for its duration, rather than lowering the threshold — a silence expires and a lowered threshold does not, and a threshold quietly lowered during a backfill six months ago is a common reason an alert stops working.

How often should this alert actually fire?

A handful of times a year, in a mature pipeline with stable publishers. If it fires monthly, either the threshold is too tight or one source is genuinely unreliable and deserves its own handling — a per-source threshold, or a conversation about the export process. If it has never fired in two years, verify it still works: point a test source at the boundary with a deliberately wrong declaration and confirm the page arrives. An alert nobody has seen fire is an alert nobody knows is broken.

Should the alert include the coordinates?

Include the bounds in the annotation if they fit, because they turn the message into a diagnosis: “declared EPSG:4326, coordinates 598000…612000” tells the reader immediately that the file holds UTM metres. Anything longer belongs behind the runbook link — an alert that scrolls is an alert that gets skimmed.

← Grafana Dashboards for GIS Workflows

Related in this section