Alerting on CRS Validation Failures in Grafana

To alert on CRS validation failures, record the ratio of features arriving without a defined projection or with an unexpected EPSG code, then fire a Prometheus alert when that ratio crosses a threshold and route it by severity. This page builds the recording rule, the alert expression, and the Alertmanager routing that together turn silent coordinate-system corruption into a page before bad geometries reach your warehouse.

Coordinate-system defects are the quietest failure in spatial ETL: a shapefile with a missing .prj, a WFS response defaulting to EPSG:4326 when you expected EPSG:3857, or a feature whose geometry is fine but whose projection is undeclared. Throughput dashboards stay green while your data silently corrupts. This alert closes that gap, and it complements the correctness panel on the parent Grafana dashboards for GIS workflows board.

When to Use This Pattern#

Add a CRS-failure alert when:

• You ingest spatial data from sources you don’t fully control (WFS, uploaded shapefiles, third-party feeds).
• A wrong or missing projection would corrupt downstream joins, tiling, or area calculations.
• You already emit crs_validation_checks_total and crs_validation_failures_total from a fail-fast validation task.
• You want a page on a sustained failure rate, not a one-off transient reject.
• Your team runs Grafana with Prometheus and Alertmanager rather than a hosted APM.

Complete Working Example#

The alert rests on two counters emitted by the validation task: total checks and total failures, both labelled by pipeline and epsg. Here is the fail-fast validation that produces them.

from typing import Optional
import geopandas as gpd
from pyproj import CRS
from prometheus_client import Counter

CHECKS = Counter(
    "crs_validation_checks_total",
    "Features whose CRS was validated",
    ["pipeline", "epsg"],
)
FAILURES = Counter(
    "crs_validation_failures_total",
    "Features rejected for CRS reasons",
    ["pipeline", "reason"],
)

def validate_crs(gdf: gpd.GeoDataFrame, pipeline: str,
                 expected_epsg: int = 3857) -> gpd.GeoDataFrame:
    """Reject features with a missing or unexpected CRS; emit metrics."""
    declared: Optional[CRS] = gdf.crs
    epsg_label: str = str(declared.to_epsg()) if declared else "none"
    CHECKS.labels(pipeline=pipeline, epsg=epsg_label).inc(len(gdf))

    if declared is None:
        # No .prj / undefined projection — the classic silent-corruption case
        FAILURES.labels(pipeline=pipeline, reason="missing_crs").inc(len(gdf))
        raise ValueError(f"{pipeline}: features arrived with no defined CRS")

    actual_epsg: Optional[int] = declared.to_epsg()
    if actual_epsg != expected_epsg:
        FAILURES.labels(pipeline=pipeline, reason="unexpected_epsg").inc(len(gdf))
        raise ValueError(
            f"{pipeline}: expected EPSG:{expected_epsg}, got EPSG:{actual_epsg}"
        )
    return gdf

The two failure reasons — missing_crs and unexpected_epsg — are the exact conditions the brief asks about, and keeping them as a reason label lets you route or silence each independently. The validation itself follows the fail-fast contract described in validating coordinate systems before ETL; the alert simply watches its metrics.

Next, precompute the failure ratio with a recording rule so both the alert and the dashboard panel read one cheap series instead of re-dividing counters on every evaluation:

# prometheus/rules/crs_validation.rules.yml
groups:
  - name: crs_validation_recording
    interval: 30s
    rules:
      - record: pipeline:crs_validation:failure_ratio
        expr: |
          sum by (pipeline) (rate(crs_validation_failures_total[5m]))
          /
          clamp_min(sum by (pipeline) (rate(crs_validation_checks_total[5m])), 1e-9)

The clamp_min(..., 1e-9) guard stops a divide-by-zero NaN when a pipeline is idle — without it the alert evaluates against NaN and either never fires or flaps unpredictably during quiet windows.

The alert rule then watches that recorded ratio and fires when it stays above threshold long enough to be real rather than a transient blip:

# prometheus/rules/crs_validation.alerts.yml
groups:
  - name: crs_validation_alerts
    rules:
      - alert: CRSValidationFailureRateHigh
        expr: pipeline:crs_validation:failure_ratio > 0.02
        for: 10m
        labels:
          severity: page
          domain: spatial
        annotations:
          summary: "High CRS-validation failure rate on {{ $labels.pipeline }}"
          description: >
            {{ $labels.pipeline }} is rejecting
            {{ $value | humanizePercentage }} of features for CRS reasons
            (missing projection or unexpected EPSG) over the last 5m.
            Check the upstream source for a dropped .prj or a changed default CRS.

      - alert: CRSValidationFailureSpike
        expr: pipeline:crs_validation:failure_ratio > 0.20
        for: 2m
        labels:
          severity: page
          domain: spatial
        annotations:
          summary: "CRS-validation failures spiking on {{ $labels.pipeline }}"
          description: >
            Over 20% of features rejected on {{ $labels.pipeline }} — likely an
            upstream feed switched projection or lost its CRS entirely.

Two alerts, two sensitivities: a sustained 2%-over-10-minutes catches slow degradation, while a 20%-over-2-minutes spike catches a source that suddenly lost its projection. Both use for: so a single bad batch does not page anyone.

Finally, route the alerts in Alertmanager so the spatial domain reaches the right responders:

# alertmanager/config.yml
route:
  receiver: default
  group_by: [alertname, pipeline]
  routes:
    - matchers:
        - domain = "spatial"
        - severity = "page"
      receiver: geo-oncall
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 2h

receivers:
  - name: default
    slack_configs:
      - channel: "#alerts"
  - name: geo-oncall
    slack_configs:
      - channel: "#geo-oncall"
        title: "{{ .CommonAnnotations.summary }}"
        text: "{{ .CommonAnnotations.description }}"

The group_by: [alertname, pipeline] collapses per-EPSG noise into one notification per pipeline, so a feed that lost its projection pages once, not once per affected feature batch.

Why a Ratio and a Fail-Fast Reject, Not a Retry#

The design decisions in this alert are deliberate and worth making explicit, because the intuitive alternatives all fail in production.

Alerting on a ratio rather than a raw failure count normalises across volume. A pipeline ingesting ten thousand features an hour will accrue more absolute rejects than one ingesting a hundred, even at identical data quality. If you alert on crs_validation_failures_total, you either page constantly on the busy pipeline or never on the quiet one. The ratio failures / checks means “2%” carries the same meaning everywhere, so one threshold governs the whole fleet.

Rejecting fast rather than reprojecting on the fly is the second decision. It is tempting to auto-correct a missing CRS by assuming a default, but a guessed projection is worse than a rejected feature: it produces geometries that are subtly, silently wrong — offset by hundreds of metres, or landing in the wrong hemisphere — and no downstream check will catch them. Fail-fast validation surfaces the problem at ingest, where an operator can fix the source, rather than letting corrupt coordinates propagate into tiles, spatial joins, and area calculations. This is the same contract enforced across spatial validation and sync tasks.

Not retrying a CRS failure is the third. Backoff and retry are correct for transient faults — a throttled WFS, a briefly unavailable tile server — but a wrong or missing projection is a data-contract violation that will reproduce identically on every retry. Routing it through a retry loop wastes worker time and delays the page. CRS failures belong on an alert, and the offending payloads belong in a dead-letter store for triage, not in a retry queue.

Because the two counters are labelled by reason, you can also break the alert down by failure mode in the panel legend: a rising missing_crs share usually means an upstream export dropped its .prj sidecar, while a rising unexpected_epsg share means a source silently changed its default projection. The two point to different fixes, and keeping them distinct shortens the investigation.

Parameter & Option Reference#

Verification & Testing#

Confirm the recording rule resolves before wiring the alert. Query the recorded series directly:

# The recorded ratio should return a value per active pipeline
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=pipeline:crs_validation:failure_ratio' \
  | python -c "import sys, json; print(json.load(sys.stdin)['data']['result'])"

Check the alert transitions through pending to firing under load with promtool, and unit-test the validation contract so the metric labels the alert depends on cannot silently disappear:

import geopandas as gpd
from shapely.geometry import Point
from prometheus_client import REGISTRY
import pytest

def test_missing_crs_increments_failure() -> None:
    gdf = gpd.GeoDataFrame(
        {"id": [1]}, geometry=[Point(0, 0)], crs=None
    )
    with pytest.raises(ValueError, match="no defined CRS"):
        validate_crs(gdf, pipeline="ingest", expected_epsg=3857)
    failures = REGISTRY.get_sample_value(
        "crs_validation_failures_total",
        {"pipeline": "ingest", "reason": "missing_crs"},
    )
    assert failures is not None and failures >= 1

You can also validate a rule file statically before deploying, which catches YAML and expression errors in CI:

promtool check rules prometheus/rules/crs_validation.alerts.yml

For a full behavioural test, promtool test rules lets you assert that the alert actually fires at the intended failure ratio and stays silent below it. Feeding a synthetic series where failures climb past 2% and holding it for the for: duration proves the alert transitions from inactive through pending to firing exactly when expected — the kind of regression guard that stops a well-meaning threshold tweak from silently disabling the page:

# crs_validation_test.yml — run with: promtool test rules crs_validation_test.yml
rule_files:
  - crs_validation.rules.yml
  - crs_validation.alerts.yml
tests:
  - interval: 1m
    input_series:
      - series: 'crs_validation_checks_total{pipeline="ingest"}'
        values: '0+100x20'
      - series: 'crs_validation_failures_total{pipeline="ingest"}'
        values: '0+5x20'
    alert_rule_test:
      - eval_time: 15m
        alertname: CRSValidationFailureRateHigh
        exp_alerts:
          - exp_labels:
              severity: page
              domain: spatial
              pipeline: ingest

Here failures grow at 5 per minute against 100 checks per minute — a steady 5% ratio that clears the 2% threshold, so the alert should be firing by the fifteen-minute mark.

Common Pitfalls#

• Alerting on the raw failure counter, not the ratio. A busy pipeline naturally accrues more absolute failures than a quiet one. Alert on failure_ratio so the threshold means the same thing at any volume.
• No for: clause. Firing on a single evaluation pages on one bad batch. Always require the condition to hold for minutes so transient rejects self-resolve silently.
• Divide-by-zero during idle windows. Without clamp_min on the denominator, an idle pipeline yields NaN and the alert behaves unpredictably. Guard the ratio in the recording rule.
• Treating a wrong EPSG as retryable. A projection mismatch is a data-contract violation, not a transient fault — do not route it through exponential backoff for API rate limits. Reject fast and page; retrying re-ingests the same broken projection.
• Grouping alerts too finely. Routing per epsg label floods the channel when one feed drops its projection across many batches. Group by [alertname, pipeline] so a single source problem pages once, and read the per-EPSG breakdown from the dashboard panel rather than from a wall of notifications.

Related: This alert watches the metrics from validating coordinate systems before ETL and its parent spatial validation and sync tasks section, surfaces on the correctness panel from building a raster pipeline Grafana dashboard, and pairs with structured logging for geospatial flows for post-page triage. See Prometheus documentation on alerting rules for rule syntax details.

← Back to Grafana Dashboards for GIS Workflows