Geospatial Orchestration Hub

Propagating Trace Context Across Prefect Tasks

OpenTelemetry’s current span lives in a context variable, and a context variable does not survive a Prefect task boundary, a process-pool submission or a subprocess launch. Each of those produces a fresh root span instead of a child, so a fan-out of four hundred tiles becomes four hundred unrelated traces. The fix is mechanical: serialise the context into a carrier at the parent, pass the carrier as an ordinary argument, and extract it in the child before the first span starts. Three functions, applied at every boundary the work crosses.

When to Use This Pattern

  • Traces stop at the first task boundary, so every mapped tile is its own root and the scene-level view does not exist.
  • Work crosses processes — a ProcessPoolExecutor, a subprocess, a separate worker — which is normal for GDAL-heavy pipelines.
  • A trace should span services, for example from the orchestrator into a tile-rendering service behind an HTTP call.
  • You are adopting tracing and want the propagation right before four hundred orphan traces a night become normal.

Complete Working Example

The carrier is a plain dictionary, which is exactly why this works across every boundary: Prefect can pass it, pickle can serialise it, and an environment variable can hold its JSON.

from __future__ import annotations

import json
import os
from typing import Any, Optional

from opentelemetry import context, trace
from opentelemetry.propagate import extract, inject
from prefect import flow, task

tracer = trace.get_tracer("geospatial.pipeline")


def current_carrier() -> dict[str, str]:
    """Serialise the active span context into a plain dict (W3C traceparent)."""
    carrier: dict[str, str] = {}
    inject(carrier)               # {'traceparent': '00-<trace_id>-<span_id>-01'}
    return carrier


def attach_carrier(carrier: Optional[dict[str, str]]):
    """Make the caller's span the parent of whatever we start next."""
    if not carrier:
        return context.get_current()
    return extract(carrier)


@task
def render_tile(item, carrier: Optional[dict[str, str]] = None) -> str:
    # The carrier is an ordinary task argument, so Prefect passes it like any other.
    parent = attach_carrier(carrier)
    with tracer.start_as_current_span("tile", context=parent) as span:
        span.set_attribute("tile.z", item.z)
        span.set_attribute("tile.x", item.x)
        span.set_attribute("tile.y", item.y)
        return warp_window(item)


@flow
def render_scene(items, scene_id: str) -> list[str]:
    with tracer.start_as_current_span("scene") as span:
        span.set_attribute("scene.id", scene_id)
        span.set_attribute("scene.tiles", len(items))
        carrier = current_carrier()          # captured INSIDE the span
        # Every mapped run receives the same carrier and becomes a child of `scene`.
        return render_tile.map(items, carrier=carrier)

Crossing a process boundary uses the same carrier, passed as part of the job rather than through a thread-local:

from concurrent.futures import ProcessPoolExecutor
from dataclasses import dataclass


@dataclass(frozen=True)
class PoolJob:
    item: Any
    carrier: dict[str, str]          # travels with the job, survives pickling


def pool_worker(job: PoolJob) -> str:
    """Runs in another process, with no inherited context at all."""
    parent = extract(job.carrier)
    with tracer.start_as_current_span("warp", context=parent) as span:
        span.set_attribute("tile.z", job.item.z)
        return warp_window(job.item)


def run_pool(items, max_workers: int = 4) -> list[str]:
    with tracer.start_as_current_span("pool") as span:
        carrier = current_carrier()
        jobs = [PoolJob(item, carrier) for item in items]
        with ProcessPoolExecutor(max_workers=max_workers) as pool:
            return list(pool.map(pool_worker, jobs))


def run_subprocess(argv: list[str]) -> None:
    """A subprocess inherits the environment, so the carrier rides there."""
    env = os.environ.copy()
    env["TRACEPARENT"] = current_carrier().get("traceparent", "")
    subprocess.run(argv, env=env, check=True, start_new_session=True)
    # The child extracts it with extract({"traceparent": os.environ["TRACEPARENT"]}).
Four hundred roots, or one traceWithout a carrier, every mapped task starts its own trace and the relationship between them is lost. With the carrier passed as an argument, all of them are children of one scene span.NO CARRIER — 412 unrelated traces…406 more rootsNo scene view, no fan-out shape, no way to ask “what else was slow in this run?”CARRIER PASSED AS AN ARGUMENTscene · 412 tilesOne trace, and the slow tile is visible in the context of the 411 that were not.
The carrier is one string. Everything the lower row makes possible — fan-out shape, sibling comparison, a scene-level duration — follows from passing it.

The traceparent string itself is worth understanding, because knowing what is in it makes every propagation bug diagnosable by inspection. It is four hyphen-separated fields: a version, a 32-character trace id, a 16-character span id, and two flags characters where 01 means sampled and 00 means not. A carrier whose trace id is all zeros means it was captured with no active span; one ending in -00 means the sampler decided not to record this trace, which is a legitimate state that looks identical to a bug in a trace UI. Printing the carrier once, during development, saves a great deal of guessing later.

Reading a traceparentThe traceparent contains a version, a trace id, a parent span id and flags. An all-zero trace id means no active span was present. Flags of zero-zero mean the trace was not sampled.004bf92f3577b34da6a3ce929d0e0e473600f067aa0ba902b701versiontrace id — shared by every spanparent span idflagstrace id all zeros → the carrier was captured with no active span. Move current_carrier() inside the with block.flags 00 → not sampled. Propagation is working; the sampler chose not to record this one.
Those two cases account for almost every “propagation is broken” report, and they are distinguishable at a glance once you know where to look.

Parameter & Option Reference

Mechanism Carrier Spatial notes
Prefect task argument dict Simplest and most explicit. Serialises with the task’s other parameters.
Process pool field on the job dataclass Context variables do not cross a process. The carrier must be part of the payload.
Subprocess TRACEPARENT env var The W3C convention; many libraries pick it up automatically.
HTTP call request headers inject(headers) — the case OpenTelemetry’s auto-instrumentation already handles.
Message queue message attributes Same shape: inject on publish, extract on consume.
capture point inside the parent span current_carrier() outside the with block captures nothing and produces silent orphans.

Verification & Testing

The assertion that matters is that the child’s trace id equals the parent’s — everything else follows from it.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter


def test_child_shares_the_parent_trace(prefect_harness) -> None:
    exporter = InMemorySpanExporter()
    trace.set_tracer_provider(TracerProvider())
    trace.get_tracer_provider().add_span_processor(SimpleSpanProcessor(exporter))

    render_scene(items=[tile(0), tile(1), tile(2)], scene_id="S2_TEST")

    spans = exporter.get_finished_spans()
    scene = next(s for s in spans if s.name == "scene")
    tiles = [s for s in spans if s.name == "tile"]

    assert len(tiles) == 3
    assert {s.context.trace_id for s in tiles} == {scene.context.trace_id}
    assert {s.parent.span_id for s in tiles} == {scene.context.span_id}


def test_carrier_captured_outside_the_span_is_empty() -> None:
    """The most common bug: capturing before the parent span exists."""
    carrier = current_carrier()             # no active span here
    assert "traceparent" not in carrier or carrier["traceparent"].endswith("-00")


def test_pool_worker_attaches(monkeypatch) -> None:
    with tracer.start_as_current_span("pool") as parent:
        job = PoolJob(item=tile(0), carrier=current_carrier())
    # Simulate the other process: no ambient context at all.
    context.attach(context.Context())
    pool_worker(job)
    child = last_finished_span()
    assert child.context.trace_id == parent.context.trace_id

In production, the check is a query rather than a test: count root spans per run. A tiling flow that produces one root per scene is propagating correctly; one that produces hundreds is not, and the number is a direct measure of how many boundaries are leaking.

# Root spans per minute. A steady rise here after a deploy means a boundary
# started losing context — usually a new task or a new pool.
sum(rate(traces_spanmetrics_calls_total{parent_span_id=""}[10m]))
Four boundaries, one carrier, four vehiclesA Prefect task boundary carries the context as an argument. A process pool carries it on the job payload. A subprocess carries it in an environment variable. An HTTP call carries it in a header.boundarywhat carries the traceparentPrefect taskan ordinary keyword argumentprocess poola field on the pickled jobsubprocessthe TRACEPARENT environment variableHTTP calla request header — usually automatic
Only the last one is handled for you. The other three each need three lines, and each one that is missed silently halves the trace.

Common Pitfalls

  • Capturing the carrier outside the parent span. current_carrier() called before the with block returns an empty or sampled-out carrier, so every child becomes a root. It fails silently and looks exactly like no propagation at all.
  • Assuming the pool inherits context. contextvars are per-process. A pool worker starts with nothing, and code that relies on the ambient context produces orphans without any error.
  • Propagating into a task that starts no span. The carrier arrives and is never used, which is harmless but misleading — the argument suggests the boundary is instrumented when it is not.
  • Forgetting the sampling flag. The traceparent ends in -01 when sampled and -00 when not. Propagating a not-sampled context is correct — the child respects the decision — but debugging it looks like broken propagation, so check the flag before assuming.
  • Using a global carrier variable. With concurrent tasks in one process, a module-level carrier is overwritten by whichever task ran last, and spans attach to arbitrary parents. Pass it explicitly, always.

Frequently Asked Questions

Does Prefect have built-in OpenTelemetry support?

Recent versions emit their own telemetry, and it improves steadily — but it instruments Prefect’s view of the run, not the spans you create inside a task. The two coexist happily, and the carrier approach above works regardless of what the orchestrator does, which is worth something given how quickly that landscape moves.

What if the carrier makes the task signature ugly?

Put it in the item rather than in the signature. A TileItem with a carrier field travels as one argument, is serialised alongside everything else, and keeps the task’s parameters about the work. That is also what makes the process-pool case identical to the task case, which is a small simplification worth having.

Should the carrier be logged?

The trace id, yes — that is what joins a log line to its trace, and it is the single most useful field a structured log can carry. The full traceparent is unnecessary noise. See correlating logs and traces for one tile run for the join.

Does propagation work across a retry?

Yes, and it is worth thinking about what you want. A retried task that reuses the carrier appears as a second child of the same parent, which is usually right — the retry is part of the same unit of work. Starting a fresh trace per attempt loses that relationship and makes “how many attempts did this tile take?” a log question rather than a trace one.

How much overhead does this add?

Effectively none. The carrier is a string of about sixty characters, injection and extraction are string operations, and the span itself is a few hundred bytes buffered by the batch processor. Against a tile that takes seconds to render, the instrumentation is unmeasurable.

← OpenTelemetry Tracing for Spatial Tasks

Related in this section