OpenTelemetry Tracing for Spatial Tasks

In short: a trace answers “where did the time go for this unit of work”, which is exactly the question metrics cannot answer and logs answer only by hand. In a spatial pipeline the unit is a tile or a partition, the spans are the phases that have genuinely different failure modes — fetch, warp, write, register — and the attributes are the spatial facts that explain an outlier: zoom, extent, CRS, pixel count, source scheme.

Metrics tell you that p99 tile latency doubled. Traces tell you that it doubled because the source fetch went from two seconds to ninety on tiles whose imagery moved to a different bucket. That is the division of labour, and it is worth respecting: a pipeline that tries to answer “where did the time go” from metrics ends up with a cardinality problem, and one that tries to answer “how is throughput trending” from traces ends up sampling away the answer.

There is a specifically geospatial reason traces earn their keep here, beyond the general one. A spatial pipeline’s slow cases are usually correlated with where the work was, and a trace is the only signal that carries location alongside timing. Once geo.bbox is on the span, “show me the slow traces” becomes “show me the slow area”, and the answer to a performance question is frequently a map — a mountain range, a coastline, one municipality’s denser imagery. No metric can express that, because the location is precisely the high-cardinality attribute metrics must exclude, and no log query will draw it without a lot of work.

Prerequisites & Architecture Baseline#

• opentelemetry-sdk ≥ 1.24 and the OTLP exporter, with a collector endpoint that is reachable from workers
• A backend that can query by attribute: Tempo, Jaeger, or a vendor — searching traces by tile.z is the point
• A sampling decision made deliberately, because a fan-out of four hundred tiles per scene produces a lot of spans
• Context propagation across process and task boundaries, which orchestrators do not do for you by default
• An attribute vocabulary, so tile.z does not coexist with zoom and tile_zoom
• Agreement on what a trace's root is: one run, one scene, or one tile
• A retention decision for traces, which is usually far shorter than for metrics — a fortnight is generous for an investigation aid
• Clock synchronisation across workers, since span timestamps come from the machine that produced them

Core Principles#

1. The trace’s root is the unit people ask about. For a tiling pipeline that is usually one scene, with a span per tile beneath it — small enough to be readable and large enough to show the fan-out. Making the whole nightly run one trace produces something with forty thousand spans that no UI will render; making each tile its own root loses the relationship between them.

2. Span boundaries follow failure modes, not function calls. Fetch, warp, write and register fail differently, are owned differently and are fixed differently. Those are the spans worth having. A span per Python function produces a trace that is technically complete and practically unreadable.

3. Attributes carry the spatial facts. tile.z, tile.x, tile.y, raster.pixels, crs.source, crs.target, resampling, source.scheme. Traces are where high cardinality belongs — this is precisely the data that must stay out of metric labels, and putting it here is what makes the two systems complementary rather than redundant.

4. Sampling must be decided by the time it matters. Head-based sampling decides at the root, before anything interesting has happened, so a one-per-cent rate discards ninety-nine per cent of the slow tiles too. Tail-based sampling decides after the trace completes and can keep every error and every outlier — at the cost of a collector that buffers.

5. Context does not propagate itself. A Prefect task boundary, a process-pool submission and a subprocess invocation all lose the current span unless you carry the context across explicitly. A trace that stops at the first boundary is a span, not a trace.

6. Record the failure on the span, not only in the log. span.record_exception() and a status of ERROR make the failure searchable alongside its timing and attributes. That is what lets a query find “every trace where the warp failed on a lanczos tile above zoom 14”, which no log search will do as cleanly.

Choosing the span boundaries is the decision that determines whether anyone ever looks at these traces again. Too coarse and the trace repeats what the orchestrator already showed. Too fine and it becomes a call graph: technically complete, hundreds of spans deep, and unreadable in the ninety seconds someone has during an incident. The useful test is ownership — if two adjacent operations would be investigated by different people or fixed in different systems, they are different spans; if they would always be looked at together, they are one.

Production Implementation#

The instrumentation below is a decorator plus a small set of attribute helpers, so the spatial vocabulary is defined once rather than at every call site.

from __future__ import annotations

import functools
from contextlib import contextmanager
from typing import Any, Callable, Iterator, Optional

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("geospatial.pipeline")


# One place where the attribute names are decided, so nothing else invents its own.
def tile_attributes(item) -> dict[str, Any]:
    return {
        "tile.z": item.z,
        "tile.x": item.x,
        "tile.y": item.y,
        "tile.layer": item.layer,
        "crs.source": item.src_crs,
        "crs.target": item.dst_crs,
        "raster.resampling": item.resampling,
        "source.scheme": item.source_uri.split(":", 1)[0],
        # The extent is high-cardinality and exactly the kind of thing a trace
        # should carry: it is what makes a slow tile findable on a map afterwards.
        "geo.bbox": ",".join(f"{v:.5f}" for v in item.bounds),
    }


@contextmanager
def phase(name: str, **attributes: Any) -> Iterator[trace.Span]:
    """One span per phase, with the failure recorded on the span itself."""
    with tracer.start_as_current_span(name) as span:
        for key, value in attributes.items():
            span.set_attribute(key, value)
        try:
            yield span
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)[:200]))
            raise


def traced_tile(fn: Callable) -> Callable:
    @functools.wraps(fn)
    def wrapper(item, *args, **kwargs):
        with tracer.start_as_current_span("tile") as span:
            for key, value in tile_attributes(item).items():
                span.set_attribute(key, value)
            result = fn(item, *args, **kwargs)
            # Attributes known only after the work: put them on the same span.
            if isinstance(result, dict) and "pixels" in result:
                span.set_attribute("raster.pixels", result["pixels"])
            return result

    return wrapper


@traced_tile
def render_tile(item, scratch) -> dict:
    with phase("fetch", **{"source.scheme": item.source_uri.split(":", 1)[0]}):
        local = fetch_source(item.source_uri, scratch)

    with phase("warp", **{"raster.resampling": item.resampling}) as span:
        width, height = warp_window(local, item, scratch)
        span.set_attribute("raster.output_width", width)
        span.set_attribute("raster.output_height", height)

    with phase("write"):
        dest = promote_output(item, scratch)

    with phase("register"):
        mark_done(item.work_key)

    return {"pixels": width * height, "path": dest}

Step-by-Step Walkthrough#

1. Define the attribute vocabulary in one function. tile_attributes is the only place that decides whether the key is tile.z or zoom. Without it, three modules invent three conventions and no query matches everything.
2. Make the root the scene, not the run. A scene’s trace has hundreds of spans, which every backend renders. A run’s trace has tens of thousands, which none of them do usefully.
3. Split spans by failure mode. Fetch, warp, write and register can each fail independently and are each owned by someone different. That is the test for whether a boundary deserves a span.
4. Set the attributes you know at the start, and the rest at the end. Output dimensions are only known after the warp; attaching them to the same span rather than a child keeps the trace flat and readable.
5. Record exceptions on the span. record_exception plus an ERROR status makes failures searchable with their timing and spatial attributes attached — the query that finds “failed warps at high zoom” depends on it.
6. Carry the extent as an attribute. It is high-cardinality by nature, which is fine here and forbidden in metrics, and it is what lets a slow-trace query become a map.
7. Keep the decorator thin. Everything it does is bookkeeping; the work stays in the function. A decorator that also handles retries or timeouts becomes the place bugs hide.

Edge Cases & Failure Recovery#

Context lost at a task boundary. Prefect and Dagster run tasks in separate contexts, and a naive instrumentation produces one root span per task with no parent. The fix is to inject the trace context into the task’s parameters and extract it on the other side — covered in propagating trace context across Prefect tasks.

Context lost at a process-pool boundary. The same problem with a different mechanism: pool workers are separate processes with their own context. Pass the serialised context as an argument, since anything relying on a thread-local will silently produce orphans.

Traces that dwarf the work. A fan-out of four hundred tiles across three hundred scenes is 120 000 spans a night. Without sampling that is a real cost in collector CPU and storage, and the signal-to-noise ratio is poor — most of those spans are identical and uninteresting. Sampling traces for high-volume tile pipelines covers the strategies.

Attributes that should have been metrics. Recording every tile’s duration as a span attribute and then trying to compute a p95 from traces is slow, expensive and — with sampling — wrong. Durations belong in a histogram; traces explain individual cases.

A collector that drops silently. OTLP exporters buffer and drop when the collector is unreachable, usually with a log line nobody reads. Monitor the exporter’s own dropped-span counter, or the first sign of trouble is an investigation that finds no traces for the interesting hour.

Clock skew across workers. Spans carry timestamps from the machine that produced them, so a worker whose clock is a second fast produces a child span that appears to start before its parent. Most backends render this as a negative duration or a span floating outside its parent, and it is confusing enough to derail an investigation. Run NTP on workers and treat a negative duration in a trace as an infrastructure signal rather than an application bug.

Spans without an end. A start_span without a matching end — easy with manual span management — produces a trace that never completes and is discarded by tail-based samplers. Use the context-manager form everywhere; the explicit API is for cases the context manager genuinely cannot express.

Configuration Reference#

Two of those rows are worth revisiting periodically rather than setting once. The attribute vocabulary tends to drift as services are added, and a fortnightly grep for attribute keys across the codebase costs nothing and catches zoom creeping in beside tile.z. And the sampling configuration should be revisited whenever the fan-out width changes materially, because a rate tuned for four hundred tiles per scene is either wasteful or blind at four thousand.

Frequently Asked Questions#

Related#

• Propagating trace context across Prefect tasks — keeping one trace across task boundaries
• Tracing PostGIS query spans in spatial ETL — instrumenting the database side
• Sampling traces for high-volume tile pipelines — keeping the interesting traces and dropping the rest
• Exporting per-tile metrics without cardinality blowups — the division of labour with metrics
• Structured logging for geospatial flows — the third signal, and how to join it to these

← Observability & Monitoring for Geospatial Pipelines