Sampling Traces for High-Volume Tile Pipelines

Head-based sampling decides whether to keep a trace before anything has happened, which is exactly the wrong moment: a one-per-cent rate keeps one per cent of the errors and one per cent of the slow tiles along with one per cent of the boring ones. Tail-based sampling decides after the trace completes, so it can keep every error, every outlier and a small sample of everything else. For a pipeline producing a hundred thousand tile traces a night, that is the difference between an unaffordable firehose and a searchable record of everything interesting.

When to Use This Pattern#

• Trace volume is high enough to notice — a fan-out of hundreds per scene, hundreds of scenes per night.
• The interesting traces are rare, which for a healthy pipeline they always are.
• Storage or vendor cost is the binding constraint, and the instinct is to reduce it by lowering the head sample rate.
• A collector is already in the path, since tail-based sampling requires one that can buffer complete traces.

Complete Working Example#

Tail sampling lives in the collector, not in the application. The policies are evaluated in order and a trace is kept if any of them matches.

# otel-collector.yaml — the whole sampling decision, in one place.
processors:
  tail_sampling:
    # Traces are buffered until this long after the last span, then judged.
    # It must exceed the longest realistic trace, or long traces are cut short.
    decision_wait: 120s
    num_traces: 200000              # in-memory buffer; sized from rate × decision_wait
    expected_new_traces_per_sec: 400
    policies:
      # 1. Every error, always. These are the traces someone will look for.
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      # 2. Every slow trace. The threshold is the p99, not a round number.
      - name: slow-tiles
        type: latency
        latency: { threshold_ms: 140000 }

      # 3. Anything touching a source we are currently suspicious of.
      - name: watched-layers
        type: string_attribute
        string_attribute:
          key: tile.layer
          values: [ortho_experimental, flood_extent]

      # 4. Everything at high zoom, where the work is densest and least uniform.
      - name: deep-zoom
        type: numeric_attribute
        numeric_attribute: { key: tile.z, min_value: 15, max_value: 22 }

      # 5. A baseline sample of the ordinary, so the healthy shape stays visible.
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 0.5 }

exporters:
  otlp/tempo:
    endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo]

On the application side the only requirement is that everything is sampled in, so the collector has something to judge:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ALWAYS_ON
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# ALWAYS_ON at the application, sampling at the collector. A head sampler here
# would discard traces before the collector could see whether they mattered.
provider = TracerProvider(sampler=ALWAYS_ON)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True),
        max_queue_size=8192,            # a fan-out produces spans in bursts
        max_export_batch_size=1024,
        schedule_delay_millis=2000,
    )
)
trace.set_tracer_provider(provider)

The policy order in that configuration is not significant to the collector — a trace matching any policy is kept — but it is significant to whoever reads it next. Listing them from most specific to least makes the intent legible: errors first because they are non-negotiable, then outliers, then the deliberate exceptions, then the baseline that exists only for comparison. A configuration that opens with a probabilistic policy reads as though sampling is primarily about volume, which is how the error policy ends up being removed in a cost-cutting exercise six months later.

Parameter & Option Reference#

Verification & Testing#

The properties worth checking are that errors always survive and that the buffer is not silently dropping.

def test_every_error_trace_is_retained(collector_harness) -> None:
    for i in range(200):
        emit_trace(error=(i % 50 == 0), duration_ms=800)
    collector_harness.flush()

    kept = collector_harness.exported_traces()
    assert sum(1 for t in kept if t.has_error) == 4          # all four errors
    # …and the ordinary ones are mostly gone.
    assert sum(1 for t in kept if not t.has_error) < 20


def test_slow_traces_survive_regardless_of_error(collector_harness) -> None:
    emit_trace(error=False, duration_ms=180_000)
    collector_harness.flush()
    assert len(collector_harness.exported_traces()) == 1


def test_decision_wait_exceeds_the_longest_trace(trace_durations) -> None:
    longest = max(trace_durations)
    assert DECISION_WAIT_SECONDS > longest * 1.2, (
        "traces longer than decision_wait are judged on partial data"
    )

In production, the collector’s own metrics are the ones to watch. Two of them decide whether the sampling is working or quietly discarding what it was meant to keep:

# Traces dropped because the buffer was full — should be zero.
rate(otelcol_processor_tail_sampling_sampling_trace_dropped_too_early[10m])

# The split between policies. If "baseline" dominates, the thresholds are too high;
# if "errors" dominates, something is wrong with the pipeline, not the sampling.
sum by (policy) (rate(otelcol_processor_tail_sampling_count_traces_sampled[10m]))

Reviewing the policy split periodically is worth more than tuning any single threshold. If the baseline policy accounts for most of what is stored, the specific policies are set too tightly and are catching nothing — the sampling has quietly become probabilistic with extra configuration. If the error policy dominates, the sampling is fine and the pipeline is not. Either way the split is a one-line query and it says more about the system than the total volume does.

Common Pitfalls#

• Leaving a head sampler in the application. It discards traces before the collector can judge them, so the tail policies operate on a random one per cent and keep one per cent of the errors. ALWAYS_ON at the SDK is the whole point.
• A decision_wait shorter than the longest trace. Long traces are judged on the spans that arrived in time and then completed afterwards, producing truncated traces that look like a propagation bug.
• Sizing the buffer from the average rate. Fan-outs are bursty. A buffer that holds the average comfortably will overflow during the burst, dropping the traces from the busiest scenes.
• Setting the latency threshold to a round number. “Over sixty seconds” keeps every dense tile and no anomalies. Derive it from the measured p99, per operation if the populations differ.
• Forgetting the baseline policy. With only error and latency policies, the stored traces are all pathological and there is nothing normal to compare them against. Half a per cent of ordinary traces is cheap and makes the rest legible.

Frequently Asked Questions#

Related#

• OpenTelemetry tracing for spatial tasks — the spans being sampled
• Propagating trace context across Prefect tasks — why complete traces are a precondition
• Measuring tile generation latency percentiles — where the latency threshold comes from
• Exporting per-tile metrics without cardinality blowups — the division of labour this assumes

← OpenTelemetry Tracing for Spatial Tasks