Observability & Monitoring for Geospatial Pipelines

Observability for a spatial pipeline has to answer one question that general practice does not ask: where. A tabular pipeline’s failures are distributed by chance; a spatial pipeline’s failures cluster at projection edges, along coastlines, in dense terrain and in one municipality whose export tool changed. That correlation is the most useful diagnostic signal available, and it is invisible to any monitoring stack that treats location as an inconvenient high-cardinality attribute rather than as the thing being measured.

The second difference is what “healthy” means. A run that completes, on time, with no failed tasks, can produce output that is a month stale over half the country because one source stopped publishing. Process telemetry is green throughout. Observability that stops at the process — run success, task duration, throughput — describes a machine rather than a product, and the gap between the two is where the incidents that reach users actually live.

Core Architectural Layers#

Four signals, each answering a question the others cannot, plus one that turns measurement into a promise.

1. Metrics — is it getting worse? Prometheus metrics for raster throughput covers the aggregate view: pixels rather than files as the unit, bounded label vocabularies, and histogram buckets sized from real latencies rather than from a library’s web-request defaults.

2. Traces — where did the time go for this one? OpenTelemetry tracing for spatial tasks is where high cardinality belongs: the tile index, the extent, both CRSs. It is also the only signal that carries location alongside timing, which is what turns a performance question into a map.

3. Logs — what exactly happened? Structured logging for geospatial flows makes the spatial facts fields rather than prose, so a run can be reconstructed by query rather than by reading. Under sixteen-way concurrency that is the difference between legible and useless.

4. Dashboards — can someone else read this? Grafana dashboards for GIS workflows arranges the signals in the order the questions are asked, and includes the one panel that distinguishes a spatial dashboard from any other: a map.

5. Objectives — is it good enough? Data-quality SLOs for spatial pipelines turns measurement of the output into a promise with an error budget, which is the only one of the five that a consumer would recognise as being about them.

Key Design Constraints Imposed by Geospatial Data#

Location is high-cardinality and indispensable. A tile index has millions of values, so it cannot be a metric label — and it is exactly what explains an outlier. The resolution is to route it: bounded explanatory labels in metrics, the full index in traces and logs, and the complete record in a table. Pipelines that put it everywhere have a cardinality incident; ones that put it nowhere have a mystery.

Work is not uniform. Tile runtimes span two orders of magnitude within one job, so histogram buckets, latency thresholds and percentile alerts all need measurement rather than defaults. A p95 over a bimodal distribution describes no tile that ever ran.

The process and the product fail independently. A pipeline can be perfectly healthy and produce stale or incomplete output, and the reverse is also true. Two families of signal are needed, and conflating them means the dashboard is green during the incident that matters.

Failures correlate geographically. Errors cluster at projection edges, at datum boundaries, in dense terrain and in single municipalities. Any signal that discards location discards the fastest available diagnosis, which is why a map panel earns its place ahead of most time series.

Volume scales with the fan-out, not with the traffic. A web service’s telemetry volume tracks its request rate, which grows gradually. A tiling pipeline’s tracks the product of scenes and tiles per scene, so a decision to render one more zoom level quadruples the spans, the log lines and the ledger rows overnight. Telemetry budgets that were comfortable at zoom 14 are not at zoom 16, and the change that caused it looks like a routine configuration edit.

Everything is out of process. GDAL runs as a subprocess, PostGIS runs on another machine, and the object store is somewhere else entirely. Instrumentation happens at boundaries — start, end, exit code, what was asked, what was produced — because there is nothing inside those processes to hook.

Topic Deep-Dives#

Prometheus metrics for raster throughput#

Metrics answer the trend questions, and the two decisions that make them useful are the unit and the label vocabulary. Pixels rather than files, because a file varies by three orders of magnitude and a pixel does not; and a label set whose maximum cardinality you can state before adding it.

Three recipes cover the practice: instrumenting gdalwarp measures an opaque subprocess from its boundary; measuring tile latency percentiles sizes histogram buckets from real data rather than from defaults that stop at ten seconds; and exporting per-tile metrics without cardinality blowups explains how to keep the detail without the million time series.

OpenTelemetry tracing for spatial tasks#

Traces answer “where did the time go for this tile”, which no aggregate can. The span boundaries should follow failure modes — fetch, warp, write, register — and the attributes should carry the spatial facts, including the extent, which is what makes a slow-trace query into a map.

Propagating trace context across Prefect tasks is the plumbing that stops a fan-out becoming four hundred orphan traces. Tracing PostGIS query spans names the spatial operation rather than the SQL verb and attaches the plan where it is worth having. And sampling traces for high-volume tile pipelines keeps every error and outlier while dropping the identical fast ones.

Structured logging for geospatial flows#

Logs are the most voluminous signal and the one most often wasted, because spatial events carry many facts and prose absorbs all of them invisibly. Events get names, facts get fields, and the vocabulary is shared with the traces so one set of queries serves both.

Logging feature counts and EPSG codes per task covers the two fields that reconstruct a vector run and catch a silent join failure. Correlating logs and traces for one tile run covers the two identifiers that turn four independent systems into one investigation.

Grafana dashboards for GIS workflows#

Dashboards are documents with an audience, and the ordering is the design: is anything wrong, is it keeping up, is the output complete, where is the problem. Four rows on one screen, each narrowing the search.

Building a raster pipeline dashboard gives the nine panels and the provisioning that keeps them in version control. Alerting on CRS validation failures is one of the few spatial alerts worth paging on. And visualizing tile coverage gaps on a geomap is the panel that turns a coverage percentage into a place.

Data-quality SLOs for spatial pipelines#

Objectives are the layer that measures the product rather than the machine. Freshness, coverage and validity cover most spatial products, each computed from the pipeline’s own ledger and each expressed with an error budget so the number leads to a decision.

Defining freshness SLOs for tile layers derives the target from how fast the phenomenon changes and alerts on burn rate rather than on the threshold. Monitoring geometry validity rates as an SLO turns the validation boundary’s outcomes into a promise, and finds a publisher’s changed export twelve days before the quarantine queue does.

How the signals divide the work#

The division between the four signals is not a matter of taste, and getting it wrong is the source of most of the expense and most of the frustration in a spatial observability stack. Each signal has a cardinality it tolerates and a question it answers, and using one to answer another’s question is what produces both the million-series Prometheus incident and the “we have traces and still cannot tell whether throughput is falling” complaint.

Metrics are cheap because they are aggregates: a bounded set of label combinations, sampled at a fixed interval, retained for a year. That makes them ideal for trends and alerts and useless for anything about an individual tile. Traces are the opposite: expensive per unit, sampled, retained for a fortnight, and able to carry any attribute you like — which is why the tile index, the extent and both CRSs belong there. Logs sit between the two in cost and above both in flexibility, which is why they attract everything and why the discipline of naming events and limiting volume matters most for them. And a table — the ledger, the validation log, the dead-letter queue — is the fourth destination that people forget they already have: it holds per-unit facts indefinitely, joins spatially, and costs a fraction of what the equivalent metric labels would.

The practical rule that follows is worth stating as a sentence anyone can apply at a review: if the answer to your question is a number over time, it is a metric; if it is about one unit of work, it is a trace or a log; if it is about where, it is a table with a geometry column. Almost every observability design error in a spatial pipeline is a violation of that sentence.

Adopting this without doing all of it#

Nobody builds five layers at once, and the order that works is not the order they are listed. Start with metrics on the unit that stays comparable — pixels for raster, vertices for vector — because that is a day’s work and it makes every capacity conversation concrete. Add a freshness objective next, computed from the ledger you already have, because it is the fastest route to discovering whether the process signals have been describing a healthy machine producing stale output. Those two together are a complete and defensible stack for a pipeline with one or two consumers.

Structured logging comes third, and it is worth doing before traces because it is cheaper to adopt incrementally: configure a JSON renderer and a trace-context processor, and every existing log line improves without a single call site changing. Traces come fourth, when investigations have started taking longer than the incidents. Dashboards come whenever a second person joins the on-call rotation, because that is the moment the knowledge in one person’s head stops being sufficient.

Implementation Patterns & Code Scaffold#

One tile, four signals, two shared identifiers. The instrumentation is deliberately thin — everything interesting is in what is recorded rather than in how.

from __future__ import annotations

import time

import structlog
from opentelemetry import trace

from .metrics import TILES, PIXELS, WARP_SECONDS
from .vocab import tile_fields          # one place that owns the field names

tracer = trace.get_tracer("geospatial.pipeline")
log = structlog.get_logger()


def process_tile(item, conn) -> None:
    with tracer.start_as_current_span("tile") as span:
        for key, value in tile_fields(item).items():
            span.set_attribute(key, value)
        span.set_attribute("work.key", item.work_key)

        # The two identifiers that join everything: bound once, carried everywhere.
        structlog.contextvars.bind_contextvars(
            work_key=item.work_key, trace_id=current_trace_id(), **tile_fields(item)
        )
        labels = {"layer": item.layer, "zoom": str(item.z), "resampling": item.resampling}
        started = time.monotonic()
        outcome = "failed"

        try:
            if ledger_has(conn, item.work_key):
                outcome = "skipped"
                log.info("tile_skipped", reason="digest_unchanged")
                return

            with tracer.start_as_current_span("warp"):
                width, height = warp_window(item)

            # Pixels, not tiles: the unit that stays comparable across tilings.
            PIXELS.labels(**labels).inc(width * height)
            record_build(conn, item.work_key, trace_id=current_trace_id())
            outcome = "ok"
            log.info("tile_written", pixels=width * height)

        except Exception as exc:
            log.error("tile_failed", failure_class=classify(exc), exc_info=exc)
            dead_letter(conn, item, trace_id=current_trace_id())
            raise
        finally:
            WARP_SECONDS.labels(**labels).observe(time.monotonic() - started)
            TILES.labels(outcome=outcome, **labels).inc()
            structlog.contextvars.clear_contextvars()

Reading that scaffold, the striking thing is how little of it is observability code. Four metric calls, two spans, one context binding and two writes to the ledger — and the rest is the work. That ratio is the target: instrumentation that is visible enough to review and thin enough that nobody is tempted to remove it when the function is edited. Instrumentation that grows into a wrapper framework, with decorators that also handle retries and timeouts, is instrumentation that will eventually hide a bug, and the bug will be in the observability layer where nobody looks for it.

The other thing worth noticing is where the identifiers appear. work.key on the span, work_key and trace_id bound to the logger, and trace_id written to the ledger and the dead-letter row — four appearances of two values, none of them repeated at a call site. That is what makes the correlation reliable rather than aspirational: it is established once, in one place, and every subsequent record inherits it whether or not anyone remembered.

Failure Modes & Operational Guardrails#

A tile identifier in a metric label. Detection: series count growing with the data rather than with the label vocabulary; queries slowing over weeks. Mitigation: an explanatory label such as a density band instead of an identifying one, with the identifier in traces and a table.

Histogram buckets from a library default. Detection: most observations in the +Inf bucket; percentiles that do not move when latency does. Mitigation: measure one real run, then set boundaries that bracket p50 to p99.

Traces that stop at a task boundary. Detection: root spans per run in the hundreds; no scene-level view. Mitigation: carry the trace context as an ordinary argument across every task, pool and subprocess boundary.

Logs that cannot be queried. Detection: investigations that involve grep and a text editor. Mitigation: events with names, facts as fields, and a shared vocabulary with the traces.

A dashboard nobody but its author can read. Detection: a colleague cannot answer the four questions in a minute. Mitigation: one panel per question, descriptions saying what healthy looks like, and deletion of everything else.

Green process metrics over a stale product. Detection: a data-quality objective that nobody computes. Mitigation: a freshness SLI from the ledger, weighted, with an error budget and a burn-rate alert.

Every one of those six has the same structure — a detection signal that is cheap to compute and a mitigation that is a design decision rather than a parameter. That is worth noticing because the instinct when observability is inadequate is to add more of it, and in five of the six cases adding more makes things worse. A cardinality problem is not solved by more metrics; a dashboard nobody can read is not improved by another panel; a stale product is not detected by a finer-grained process metric. The fix is nearly always to move information to the destination that was built for it.

The sixth case — green process metrics over a stale product — is the one worth checking first in any pipeline that has never been audited this way, because it is both the most common and the most consequential. The test takes ten minutes: compute the age of the oldest tile in the serving area, and compare it against what everyone believes the freshness to be. A substantial gap between the two is extremely common, and the fact that no existing signal was reporting it is the whole argument for the objectives layer.

Toolchain & Dependency Matrix#

Frequently Asked Questions#

Related#

• Prometheus metrics for raster throughput — the aggregate view
• OpenTelemetry tracing for spatial tasks — one unit of work in detail
• Structured logging for geospatial flows — what happened, queryably
• Grafana dashboards for GIS workflows — arranging the signals for a reader
• Data-quality SLOs for spatial pipelines — measuring the product rather than the machine