Observability & Monitoring for Geospatial Pipelines

Watching a spatial pipeline is not the same job as watching a web service: Observability & Monitoring for Geospatial Pipelines means correlating per-tile metrics, cross-boundary traces, and per-feature logs against SLOs written on geometry validity and CRS conformance, not latency alone. A raster warp that finishes fast but silently drops a nodata mask is a green dashboard hiding a broken product. This section treats spatial telemetry as its own discipline, with instrumentation designed around the shapes of the data — tiles, coordinate reference systems, feature collections, and band arrays — rather than generic request counters bolted onto GDAL calls.

Geospatial workloads break the assumptions baked into most monitoring stacks. Cardinality explodes when engineers reflexively label every metric with a tile z/x/y triple. Traces fracture at the seam where a Python orchestrator hands a job to a gdalwarp subprocess. Log volume balloons when a task emits one line per feature across a million-row GeoPackage. And the metric that actually matters — did the output stay spatially correct — never appears on a default Grafana panel. The pages below build a telemetry layer that answers the questions a GIS platform team actually asks during an incident.

Core Observability Layers for Spatial Systems#

Effective spatial monitoring separates concerns into distinct signal types, each answering a different class of question. Treating them as one undifferentiated “logging” problem is the root cause of most unusable GIS dashboards.

1. Instrumentation at the task boundary. Every spatial operation — a reprojection, a mosaic, a topology check — is wrapped so it emits a counter, a duration sample, and a structured event. The wrapper lives close to the GDAL or GeoPandas call so it captures the real work, not orchestrator overhead. This is where band counts, pixel totals, and feature counts become numbers rather than guesses.

2. A metrics store for aggregatable numbers. Prometheus scrapes counters and histograms that answer “how much” and “how fast” across the fleet. Metrics are cheap to store and fast to query, but they are low-cardinality by design: they tell you throughput collapsed by 40% at 14:00, not which specific tile failed. The companion guide on Prometheus metrics for raster throughput covers the counter and histogram design that keeps this layer sustainable.

3. A trace collector for causal ordering. Distributed traces stitch a single logical job — fetch, warp, tile, upload — into one waterfall, even when the steps cross process, container, and database boundaries. Spans carry the CRS and extent as attributes so a slow trace is diagnosable without cross-referencing logs. OpenTelemetry tracing for spatial tasks explains context propagation through orchestrator task graphs.

4. A log store for high-cardinality detail. Structured logs capture the per-task specifics — source EPSG, target EPSG, feature count, invalid-geometry count — that would blow up a metrics database if modeled as labels. Logs are queried during incidents, not scraped every fifteen seconds. See structured logging for geospatial flows for the field schema that keeps these searchable.

5. A visualization and alerting layer. Grafana unifies the three signal types into panels keyed to spatial SLOs, and evaluates alert rules against them. The discipline here is choosing panels that surface geometry-validity regressions, not just CPU graphs. Grafana dashboards for GIS workflows walks through the panel and alert design.

These layers map cleanly onto the orchestration concepts in Geospatial Orchestration Architecture & Fundamentals: the task boundary you instrument is the same task boundary you designed for state and dependency management.

Key Design Constraints Imposed by Geospatial Data#

Metric cardinality from tiles and CRS codes#

A time series in Prometheus is uniquely identified by its metric name plus the full set of label key/value pairs. Label a tile-generation counter with z, x, and y and a single global tile pyramid at zoom 14 alone implies over 268 million label combinations — every one a distinct, indexed time series. The database will out-of-memory long before that. CRS codes are subtler: EPSG:4326, EPSG:3857, and a handful of UTM zones are safe labels because the set is small and bounded, but a proj_string label carrying full PROJ pipelines is unbounded. The rule is that a label value must come from a small, enumerable set known in advance. Zoom level (0–22) is a fine label; tile coordinates are not.

Tracing across raster and vector task boundaries#

Spatial pipelines constantly cross execution seams. A Python task calls subprocess to run gdalwarp, hands a path to a Dagster op running in another container, then triggers a PostGIS spatial join. Each hop is an opportunity to lose trace context, and a trace that stops at the subprocess boundary tells you nothing about where the time went. Instrumentation must propagate the trace and span identifiers across these gaps — through environment variables into subprocesses, through headers into HTTP calls, and through orchestrator metadata between tasks — so the reprojection, the mosaic, and the database write appear as child spans of one parent.

Log volume from per-feature processing#

A vector task that logs one line per feature turns a routine 5-million-feature ingest into 5 million log lines. At scale this bankrupts the log store and buries the three lines that matter. Geospatial logging must aggregate: emit one structured event per task with feature_count, invalid_geometry_count, and crs_transform fields, and reserve per-feature logging for a sampled debug path or the specific features that failed validation. The unit of logging is the task, not the geometry.

SLOs defined on spatial correctness#

The defining constraint of this discipline is that latency SLOs are insufficient. A pipeline can hit every timing target while producing garbage: self-intersecting polygons, features silently dropped during a datum shift, or rasters written in the wrong CRS. Spatial SLOs must be expressed on correctness — percentage of output geometries that are valid under the OGC simple-features rules, percentage of tasks whose output CRS matches the declared target, percentage of expected features that survived the transform. These become first-class metrics, alerted on with the same seriousness as an error rate. This connects directly to Resilience & Failure Handling for GIS Pipelines, where the same correctness signals gate retries and dead-letter routing.

The Four Companion Guides in This Section#

Prometheus metrics for raster throughput#

Raster processing is where throughput monitoring earns its keep. Warp, mosaic, and tiling stages move enormous pixel volumes, and a regression there is invisible until a downstream tile server starts serving stale data. The Prometheus metrics for raster throughput guide instruments these stages with the prometheus_client library: Counters for bytes and pixels processed, Histograms for per-tile duration, and Gauges for in-flight jobs.

The hard part is label hygiene. The guide shows how to keep cardinality bounded by labeling on zoom level and output format rather than tile coordinates, and it weighs the pushgateway-versus-pull decision for batch geotasks that exit before Prometheus can scrape them. It closes with a bucket configuration tuned to the multi-second durations typical of raster work — the default prometheus_client buckets top out too low to resolve a slow mosaic.

OpenTelemetry tracing for spatial tasks#

When a pipeline spans a Prefect flow, a GDAL subprocess, and a PostGIS query, only distributed tracing reconstructs the causal chain. OpenTelemetry tracing for spatial tasks sets up the OTel SDK inside orchestrated tasks, attaches spatial attributes — source and target CRS, feature count, tile extent — to spans, and propagates context so nothing fractures at a boundary.

Two focused how-tos build on it: propagating trace context across Prefect tasks tackles the orchestrator-specific problem of carrying a trace id through task submission, and tracing PostGIS query spans in spatial ETL instruments the database layer so a slow ST_Intersects shows up as its own span.

Grafana dashboards for GIS workflows#

Numbers no one looks at are not observability. Grafana dashboards for GIS workflows turns the metrics and traces into panels a platform team reads during an incident: throughput over time, latency percentiles by zoom level, in-flight job saturation, and CRS-conformance ratios.

Its two companion pages get concrete. Building a raster pipeline Grafana dashboard assembles the panel JSON and PromQL for a warp-and-tile workflow, and alerting on CRS validation failures in Grafana wires an alert rule to the correctness SLO so a datum-shift regression pages someone.

Structured logging for geospatial flows#

Logs are the high-cardinality backstop the metrics layer deliberately avoids. Structured logging for geospatial flows defines a JSON log schema whose fields — task id, source EPSG, target EPSG, feature count, invalid-geometry count, bounding box — are indexable and correlatable with trace ids.

The companion how-to, logging feature counts and EPSG codes per task, shows the exact emit-once-per-task pattern that keeps volume sane while preserving the fields an on-call engineer needs to reconstruct what a task actually did to the data.

Implementation Patterns & Code Scaffold#

The following scaffold shows a single instrumented spatial task emitting all three signal types from one wrapper. It is deliberately minimal but complete: a Prometheus histogram and counter, an OpenTelemetry span carrying CRS attributes, and a structured log line — all around one rasterio reprojection.

import logging
import time
import json
import rasterio
from rasterio.warp import calculate_default_transform, reproject, Resampling
from pyproj import CRS
from prometheus_client import Counter, Histogram
from opentelemetry import trace

logger = logging.getLogger("spatial")
tracer = trace.get_tracer("geospatial.pipeline")

# Low-cardinality labels only: bounded CRS set + operation name.
WARP_SECONDS = Histogram(
    "raster_warp_seconds",
    "Wall-clock duration of a raster reprojection",
    labelnames=("src_epsg", "dst_epsg"),
    buckets=(0.5, 1, 2, 5, 10, 30, 60, 120),
)
PIXELS_TOTAL = Counter(
    "raster_pixels_reprojected_total",
    "Cumulative pixels written by warp",
    labelnames=("dst_epsg",),
)

def warp_raster(src_path: str, dst_path: str, dst_crs: CRS) -> dict[str, int]:
    with rasterio.open(src_path) as src:
        src_crs: CRS = src.crs
        # Bounded labels: EPSG codes come from a known projection set.
        s_epsg = str(src_crs.to_epsg())
        d_epsg = str(dst_crs.to_epsg())

        with tracer.start_as_current_span("warp_raster") as span:
            span.set_attribute("gis.src_crs", s_epsg)
            span.set_attribute("gis.dst_crs", d_epsg)
            span.set_attribute("gis.width", src.width)
            span.set_attribute("gis.height", src.height)

            transform, width, height = calculate_default_transform(
                src_crs, dst_crs, src.width, src.height, *src.bounds
            )
            profile = src.profile.copy()
            profile.update(crs=dst_crs, transform=transform,
                           width=width, height=height, nodata=src.nodata)

            start = time.perf_counter()
            with rasterio.open(dst_path, "w", **profile) as dst:
                for band in range(1, src.count + 1):
                    reproject(
                        source=rasterio.band(src, band),
                        destination=rasterio.band(dst, band),
                        src_crs=src_crs, dst_crs=dst_crs,
                        src_nodata=src.nodata, dst_nodata=src.nodata,
                        resampling=Resampling.bilinear,
                    )
            elapsed = time.perf_counter() - start

    pixels = width * height
    WARP_SECONDS.labels(src_epsg=s_epsg, dst_epsg=d_epsg).observe(elapsed)
    PIXELS_TOTAL.labels(dst_epsg=d_epsg).inc(pixels)
    logger.info(json.dumps({
        "event": "warp_complete", "src_epsg": s_epsg, "dst_epsg": d_epsg,
        "pixels": pixels, "warp_seconds": round(elapsed, 3),
    }))
    return {"pixels": pixels}

Note what the scaffold deliberately does: it labels metrics only on the bounded EPSG set, preserves nodata through the reprojection so correctness is not silently lost, attaches CRS and dimensions to the span rather than logging them separately, and emits exactly one structured log line for the whole task. Every one of those choices is a defense against a specific failure mode covered below.

Failure Modes & Operational Guardrails#

Cardinality explosion from tile labels. Symptom: Prometheus memory climbs steadily until the scrape endpoint times out, and count({__name__=~"tile.*"}) returns millions. Cause: a z/x/y or feature_id label. Mitigation: drop the offending label, keep only zoom level, and add a CI check that rejects new metrics whose label values are not from an allowlist.

Trace fracture at the subprocess boundary. Symptom: traces end abruptly at a gdalwarp or ogr2ogr call and the reprojection time is invisible. Cause: OTel context is not passed into the child process. Mitigation: inject the trace context into the subprocess environment and start a span in the wrapper around the subprocess so the external tool’s runtime is captured even without in-process instrumentation.

Silent CRS mismatch. Symptom: every latency panel is green but downstream layers misalign on the map. Cause: an output written in the source CRS because a transform parameter was dropped. Mitigation: assert the output CRS after every write, emit a crs_conformance metric, and alert when it drops below 100%. This is a correctness SLO, not a timing one.

nodata loss during warp. Symptom: reprojected rasters show black borders or corrupted edge statistics. Cause: src_nodata/dst_nodata omitted so fill pixels become valid data. Mitigation: make nodata a required parameter in the instrumented wrapper and log it as a field so its absence is visible in the log store.

Log-volume denial of service. Symptom: the log pipeline lags hours behind and ingest costs spike. Cause: per-feature logging in a large vector task. Mitigation: aggregate to one event per task; sample per-feature detail; cap log rate per task with a guard.

Histogram bucket saturation. Symptom: histogram_quantile returns the top bucket bound for p99 and tail latency looks flat and wrong. Cause: buckets that top out below real durations. Mitigation: tune buckets to observed maxima, as the measuring tile generation latency percentiles guide details.

Toolchain & Dependency Matrix#

Frequently Asked Questions#