OpenTelemetry Tracing for Spatial Tasks

OpenTelemetry tracing for spatial tasks turns an opaque geospatial DAG into a single stitched trace, where each fetch, reproject, validate, and load step becomes a span carrying EPSG codes, feature counts, and tile coordinates. This guide instruments a Prefect or Dagster flow with the OpenTelemetry Python SDK, propagates context across task and subprocess boundaries, attaches spatial attributes to every span, and exports everything over OTLP to a collector — so you can answer “which tile, in which CRS, made this run slow?” from one waterfall instead of grepping seven log streams.

The direct recommendation: wrap each orchestrated task in an explicit span, standardise a small set of spatial attribute keys (geo.epsg, geo.feature_count, geo.geometry_type, geo.tile.z/x/y, geo.bbox), propagate the trace context manually through any boundary the SDK cannot auto-instrument (subprocess GDAL calls, process pools, Prefect task hand-offs), and sample with a parent-based ratio sampler so high-volume tile runs stay affordable. Tracing complements the counters and gauges described in the parent Observability & Monitoring for Geospatial Pipelines section: metrics tell you that throughput dropped, traces tell you where and why.

Prerequisites & Architecture Baseline#

• Python 3.11+ with opentelemetry-sdk==1.27.0 and opentelemetry-api==1.27.0
• opentelemetry-exporter-otlp-proto-grpc==1.27.0 for OTLP export
• An OpenTelemetry Collector reachable at otel-collector:4317 (gRPC OTLP)
• prefect==2.19 or dagster==1.8 as the orchestrator
• geopandas==1.0.1, rasterio==1.3.10, pyproj==3.6.1 in the worker image
• GDAL 3.8 CLI tools (gdalwarp, gdal_translate) on the worker PATH
• A backend that ingests OTLP from the collector (Tempo, Jaeger, or an OTLP-compatible vendor)

The mental model is a linear-with-fan-out spatial pipeline. A root span opens when the flow starts; child spans nest under it for each logical stage. Where work forks into a ProcessPoolExecutor or shells out to gdalwarp, the context must be carried by hand because the SDK’s thread-local context does not cross those boundaries. The diagram below shows the resulting waterfall for a single tile.

Before writing any instrumentation, decide what a “unit of work” is in your pipeline, because that decision governs span granularity. In a tile pipeline the natural unit is a single tile passing through fetch, reproject, validate, and load; in a batch vector pipeline it may be a feature collection or an administrative region. Spans should mirror the same boundaries your orchestrator already draws around tasks, so that the trace and the flow-run graph line up one-to-one. When they diverge — a single Prefect task internally doing three unrelated spatial operations, for instance — the trace becomes harder to read than the code, which defeats the purpose. Keep the span tree a faithful, slightly coarser projection of the task graph and the instrumentation stays maintainable as the pipeline grows.

Core Concepts#

1. Tracer and span lifecycle. A Tracer (obtained from a named TracerProvider) mints spans. Every span has a trace_id shared across the whole flow and a unique span_id; a child records its parent’s span_id so the backend can reconstruct the tree. Always open spans with a context manager so the end time and status are recorded even on exceptions.

2. Spatial span attributes. Attributes are typed key-value pairs attached to a span. For geospatial work, standardise on a namespaced set: geo.epsg (int), geo.feature_count (int), geo.geometry_type (str), geo.tile.z/x/y (int), and geo.bbox (a string like "minx,miny,maxx,maxy", since attribute values cannot be nested lists). These are exactly the fields you filter and group by when hunting a slow CRS or a hot tile.

3. Context propagation. The active span lives in a thread-local Context. Auto-instrumentation carries it within one process and one thread. Across a Prefect task boundary, a ProcessPoolExecutor worker, or a subprocess GDAL call, you must serialise the context into a carrier dict with inject() and rebuild it with extract() on the other side — the technique detailed in the companion page on propagating trace context across Prefect tasks.

4. Span status and events. Set span.set_status(Status(StatusCode.ERROR)) and span.record_exception(exc) on failure so the waterfall highlights the failing stage. Use span.add_event("geometry_repaired", {...}) for point-in-time notes such as a ST_MakeValid fix, which is cheaper than a full child span.

5. Sampling. A sampler decides, at root-span creation, whether a trace is recorded. A ParentBased(TraceIdRatioBased(ratio)) sampler keeps sampling decisions consistent for a whole trace and lets you record, say, 5% of high-volume tile runs while a downstream tail sampler in the collector keeps every errored trace.

6. OTLP export. A BatchSpanProcessor buffers finished spans and flushes them to an OTLPSpanExporter pointed at the collector. Batching is essential: a per-tile flush would add network latency to the very hot path you are measuring. The collector, not the worker, is where you should concentrate routing and enrichment logic — it can add resource attributes, drop noisy spans, apply tail-based sampling, and fan out to multiple backends without a worker redeploy.

7. Resource attributes. Distinct from span attributes, resource attributes describe the producer of the telemetry and are attached once at provider construction: service.name, service.version, deployment.environment, and worker identifiers. In a fleet of raster workers, stamping service.instance.id on the resource lets you tell whether one bad node is responsible for a run of slow reproject spans, rather than the work itself being expensive. Resource attributes cost nothing per span because they are sent once per export batch, so they are the right home for anything that is constant across a worker’s lifetime.

Production Implementation#

The module below configures the SDK once, exposes a helper to open spatial spans with standard attributes, and instruments a four-stage flow. The reproject stage shells out to gdalwarp, so it injects the trace context into the child process environment. Read it as three layers: a one-time provider setup (configure_tracing), a reusable attribute helper (set_spatial_attrs) that centralises the namespaced key names so they never drift between stages, and the instrumented stage functions themselves. Centralising the attribute keys in one helper is not cosmetic — it is what keeps geo.epsg from becoming geo.crs in one task and epsg_code in another, a divergence that quietly breaks every dashboard query built on those keys.

import os
import subprocess
from typing import Optional

import geopandas as gpd
from pyproj import CRS

from opentelemetry import trace, context as otel_context
from opentelemetry.propagate import inject
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider, Span
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.trace import Status, StatusCode


def configure_tracing(service_name: str = "spatial-pipeline", ratio: float = 0.05) -> None:
    """Install a global TracerProvider with ratio sampling and OTLP export."""
    resource = Resource.create({"service.name": service_name})
    provider = TracerProvider(
        resource=resource,
        sampler=ParentBased(root=TraceIdRatioBased(ratio)),
    )
    exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
    provider.add_span_processor(BatchSpanProcessor(exporter, max_export_batch_size=512))
    trace.set_tracer_provider(provider)


tracer = trace.get_tracer("geo.orchestration")


def set_spatial_attrs(
    span: Span,
    *,
    epsg: Optional[int] = None,
    feature_count: Optional[int] = None,
    geometry_type: Optional[str] = None,
    tile: Optional[tuple[int, int, int]] = None,
    bbox: Optional[tuple[float, float, float, float]] = None,
) -> None:
    """Attach the standard geospatial attribute set to a span."""
    if epsg is not None:
        span.set_attribute("geo.epsg", epsg)
    if feature_count is not None:
        span.set_attribute("geo.feature_count", feature_count)
    if geometry_type is not None:
        span.set_attribute("geo.geometry_type", geometry_type)
    if tile is not None:
        z, x, y = tile
        span.set_attribute("geo.tile.z", z)
        span.set_attribute("geo.tile.x", x)
        span.set_attribute("geo.tile.y", y)
    if bbox is not None:
        span.set_attribute("geo.bbox", ",".join(f"{v:.6f}" for v in bbox))


def fetch_features(wfs_url: str, tile: tuple[int, int, int]) -> gpd.GeoDataFrame:
    with tracer.start_as_current_span("fetch_wfs") as span:
        gdf = gpd.read_file(wfs_url)  # returns EPSG:4326 features
        set_spatial_attrs(
            span,
            epsg=gdf.crs.to_epsg(),
            feature_count=len(gdf),
            geometry_type=str(gdf.geom_type.iloc[0]) if len(gdf) else "EMPTY",
            tile=tile,
        )
        return gdf


def reproject_raster(src_path: str, dst_path: str, dst_epsg: int = 3857) -> str:
    with tracer.start_as_current_span("reproject") as span:
        span.set_attribute("geo.epsg", dst_epsg)
        # Carry the active trace context into the GDAL child process.
        carrier: dict[str, str] = {}
        inject(carrier)
        child_env = {**os.environ, **{f"OTEL_CARRIER_{k}": v for k, v in carrier.items()}}
        result = subprocess.run(
            ["gdalwarp", "-t_srs", CRS.from_epsg(dst_epsg).to_wkt(), src_path, dst_path],
            env=child_env,
            capture_output=True,
            text=True,
        )
        if result.returncode != 0:
            span.set_status(Status(StatusCode.ERROR, result.stderr[:200]))
            raise RuntimeError(f"gdalwarp failed: {result.stderr}")
        return dst_path


def validate_geometries(gdf: gpd.GeoDataFrame) -> gpd.GeoDataFrame:
    with tracer.start_as_current_span("validate") as span:
        invalid = (~gdf.geometry.is_valid).sum()
        span.set_attribute("geo.invalid_count", int(invalid))
        if invalid:
            span.add_event("geometry_repair", {"repaired": int(invalid)})
            gdf["geometry"] = gdf.geometry.make_valid()
        set_spatial_attrs(span, feature_count=len(gdf), epsg=gdf.crs.to_epsg())
        return gdf


def run_tile(wfs_url: str, src_raster: str, dst_raster: str, tile: tuple[int, int, int]) -> None:
    with tracer.start_as_current_span("spatial_pipeline") as root:
        set_spatial_attrs(root, tile=tile, epsg=4326)
        gdf = fetch_features(wfs_url, tile)
        reproject_raster(src_raster, dst_raster, dst_epsg=3857)
        validate_geometries(gdf)

Step-by-Step Walkthrough#

1. Configure once at process start. configure_tracing() builds a TracerProvider bound to a service.name resource, wires a ParentBased(TraceIdRatioBased(0.05)) sampler, and attaches a BatchSpanProcessor. Call it in your worker entrypoint, never per-task — a fresh provider per task orphans spans into separate traces.

2. Open the root span. run_tile starts spatial_pipeline as the current span and stamps the tile coordinates plus the input CRS. Because it uses start_as_current_span, every span opened underneath it automatically nests as a child through the thread-local context.

3. Instrument the fetch. fetch_features reads a WFS response into a GeoDataFrame, then records geo.epsg, geo.feature_count, and geo.geometry_type. Reading these off the actual gdf.crs and geometry column — not a hard-coded assumption — is what makes the trace trustworthy when an upstream service silently returns EPSG:3857.

4. Cross the subprocess boundary. reproject_raster calls inject(carrier) to serialise the current traceparent into a dict, forwards it through the child environment, and runs gdalwarp. A wrapper inside the GDAL worker rebuilds the context so the reproject span links back to the root rather than starting a detached trace.

5. Record repairs as events. validate_geometries counts invalid geometries, adds a geometry_repair event when it runs make_valid(), and stores geo.invalid_count. Events keep the fix visible without inflating the span count on clean tiles.

6. Export in batches. Finished spans queue in the BatchSpanProcessor and flush to the collector over OTLP gRPC. The collector then fans them out to your tracing backend and can apply tail-based sampling to retain all error traces.

A subtle but important detail in this flow is what happens on the exception path. Because each span is opened with start_as_current_span as a context manager, an exception raised inside reproject_raster — a gdalwarp non-zero exit, say — still closes the span with an end timestamp before it propagates. The explicit span.set_status(Status(StatusCode.ERROR, ...)) call additionally flags the span red in the backend and records the truncated stderr, so the waterfall visually surfaces the failing stage without you cross-referencing logs. This is the payoff of tracing over ad-hoc timing: the failure, its stage, its CRS context, and its duration all arrive as one correlated record. When a run does fail, the same trace_id becomes the join key to pull the matching structured log lines, closing the loop between the two signals.

Edge Cases & Failure Recovery#

• Orphaned subprocess spans. If gdalwarp work shows up as its own root trace, the carrier never reached the child or extract() ran before configure_tracing(). Detection: filter for traces with a single span named reproject. Fix: initialise the provider first inside the child, then extract() and use start_as_current_span(context=ctx).

• Attribute cardinality explosion. Putting a raw per-feature id or a full WKT geometry in an attribute balloons backend storage and slows queries. Detection: backend rejects spans or ingestion cost spikes. Fix: keep attributes to bounded-cardinality summaries (counts, EPSG codes, tile indices); push per-feature detail to logs correlated by trace_id, as covered in structured logging for geospatial flows.

• Lost spans on crash. A hard SIGKILL (OOM during a raster mosaic) drops the BatchSpanProcessor buffer. Detection: traces truncated mid-flow. Fix: lower max_export_batch_size/schedule_delay_millis on memory-hungry stages, and register provider.shutdown() (or force_flush()) in a finally block so graceful exits flush.

• Sampling hides the slow tile. Ratio sampling at 5% may drop the exact pathological tile you are chasing. Detection: known-slow runs absent from the backend. Fix: force-sample suspect tiles by seeding the trace from a parent context marked sampled, or move to tail-based sampling in the collector keyed on span duration.

• Clock skew across workers. Distributed workers with drifting clocks render nonsensical waterfalls (children starting before parents). Detection: negative gaps in the waterfall. Fix: run NTP on all workers; rely on the collector, not worker wall-clocks, for ordering where possible.

• Duplicate providers dropping spans. Calling configure_tracing() more than once per process installs a second TracerProvider; the SDK warns and keeps the first, so spans created against the later tracer vanish. Detection: a subset of stages consistently missing from otherwise complete traces. Fix: guard provider setup so it runs exactly once per process, and initialise it before any tracer handle is captured at module import time.

• Blocking export on the hot path. Swapping the BatchSpanProcessor for a SimpleSpanProcessor in production makes every span end block on a network round trip to the collector, inflating the very durations you are trying to measure. Detection: span durations that balloon uniformly when the collector is slow or unreachable. Fix: keep SimpleSpanProcessor for tests only and always batch in production.

Configuration Reference#

# opentelemetry tracing config for a high-volume tile pipeline
service:
  name: spatial-pipeline          # becomes resource service.name
exporter:
  otlp:
    endpoint: otel-collector:4317 # gRPC OTLP; use 4318 for HTTP/protobuf
    insecure: true                # false + TLS certs in production
    timeout_ms: 10000
batch_processor:
  max_export_batch_size: 512      # spans per export; lower on OOM-prone stages
  schedule_delay_millis: 5000     # max buffer age before a flush
  max_queue_size: 2048            # dropped-span guard under burst load
sampling:
  strategy: parentbased_traceidratio
  ratio: 0.05                     # 5% of root traces recorded head-side
  tail_sampling_in_collector:     # keep every error/slow trace regardless
    error_traces: keep
    latency_threshold_ms: 3000
attributes:
  namespace: geo                  # geo.epsg, geo.feature_count, geo.tile.z/x/y
  max_bbox_precision: 6           # decimal places when serialising bbox string

Tune ratio down as tile volume climbs; a pipeline emitting millions of tiles a day can sit at 0.01 head sampling and still keep all failures through collector-side tail sampling. Keep service.name stable so traces from different deployments group cleanly. These knobs pair naturally with the DAG shape decisions in DAG design principles for spatial ETL: fewer, coarser tasks mean fewer spans and cheaper tracing. Treat the sampling ratio and the batch-processor sizing as the two levers you revisit whenever tracing cost or worker memory becomes a concern — lowering the ratio cuts backend ingest volume, while shrinking max_export_batch_size bounds the memory a single worker holds in flight. Neither change touches instrumentation code, so both can ship as configuration on a struggling fleet without a redeploy.

Frequently Asked Questions#