Alerting on Dead-Letter Queue Growth

Alert on the rate at which a dead-letter queue grows and on the appearance of new failure classes — never on its absolute depth. A queue holding 340 entries is meaningless without knowing whether that is Tuesday’s normal or the result of a source changing its projection at 02:00. Export three gauges from the queue (open depth by class, arrival rate, and the age of the oldest unresolved entry), alert when arrivals exceed the drain rate for long enough to matter, and page immediately on a class that has never been seen before.

When to Use This Pattern#

• A dead-letter queue already exists and is written to by the pipeline. Alerting on an empty design is a no-op.
• The pipeline runs unattended — nightly, hourly — so nobody would otherwise notice a steady 3% failure rate until a downstream consumer complains.
• Failures are heterogeneous: several sources, several failure classes, so a single count hides the interesting change.
• You have Prometheus or an equivalent that can scrape a gauge and evaluate a rule over a window. A daily digest email is a weaker version of the same idea and is much better than nothing.

The reason depth alone fails as a signal is worth being concrete about. Consider a tiling pipeline that dead-letters roughly forty tiles a day from a flaky upstream and drains them each night. Its depth oscillates between zero and about sixty, so any threshold below sixty pages every afternoon and any threshold above it misses a genuine regression that adds thirty a day for a fortnight — by the time depth crosses sixty the pipeline has been quietly broken for two weeks. The information is entirely in how the number moves, and none of it is in where the number is.

Complete Working Example#

The exporter is a small periodic job. It runs three queries and sets three gauge families; everything else — thresholds, windows, routing — lives in the alert rules, where it can be changed without a deploy.

from __future__ import annotations

import time

import psycopg
from prometheus_client import Gauge, start_http_server

DLQ_DEPTH = Gauge(
    "geotask_dlq_open_entries",
    "Unresolved dead-letter entries",
    ["failure_class", "pipeline"],
)
DLQ_OLDEST = Gauge(
    "geotask_dlq_oldest_seconds",
    "Age of the oldest unresolved entry, in seconds",
    ["pipeline"],
)
DLQ_ARRIVALS = Gauge(
    "geotask_dlq_arrivals_1h",
    "Entries written in the last hour",
    ["failure_class", "pipeline"],
)

DEPTH_SQL = """
SELECT failure_class, count(*)
FROM   geometry_dlq
WHERE  resolved_at IS NULL
GROUP  BY failure_class
"""

OLDEST_SQL = """
SELECT COALESCE(EXTRACT(EPOCH FROM now() - min(failed_at)), 0)
FROM   geometry_dlq
WHERE  resolved_at IS NULL
"""

ARRIVALS_SQL = """
SELECT failure_class, count(*)
FROM   geometry_dlq
WHERE  failed_at > now() - interval '1 hour'
GROUP  BY failure_class
"""


def collect(conn: psycopg.Connection, pipeline: str = "parcels") -> None:
    """Refresh every gauge. Cheap enough to run once a minute on an indexed table."""
    with conn.cursor() as cur:
        # Clear stale label sets first: a class that drops to zero must report zero,
        # not keep its last value forever. A missing series looks like "no data",
        # which most alert routes treat as OK — the wrong default here.
        DLQ_DEPTH.clear()
        DLQ_ARRIVALS.clear()

        cur.execute(DEPTH_SQL)
        for failure_class, count in cur.fetchall():
            DLQ_DEPTH.labels(failure_class=failure_class, pipeline=pipeline).set(count)

        cur.execute(OLDEST_SQL)
        DLQ_OLDEST.labels(pipeline=pipeline).set(cur.fetchone()[0])

        cur.execute(ARRIVALS_SQL)
        for failure_class, count in cur.fetchall():
            DLQ_ARRIVALS.labels(failure_class=failure_class, pipeline=pipeline).set(count)


def main(dsn: str, interval_seconds: int = 60) -> None:
    start_http_server(9109)
    with psycopg.connect(dsn) as conn:
        while True:
            collect(conn)
            conn.rollback()      # keep the snapshot fresh; this is a read-only loop
            time.sleep(interval_seconds)

The rules are where the judgement lives. These three cover the cases that matter, and deliberately leave out “depth above N”:

groups:
  - name: geotask-dlq
    rules:
      # 1. Arrivals outpacing drains for an hour. This is the workhorse alert:
      #    it fires on a pipeline that is failing steadily, which is invisible
      #    to run-success dashboards.
      - alert: DeadLetterQueueGrowing
        expr: |
          sum by (pipeline) (delta(geotask_dlq_open_entries[1h])) > 50
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "{{ $labels.pipeline }} dead-letter queue grew by {{ $value }} in an hour"

      # 2. A class nobody has seen before. Almost always a real change upstream:
      #    a new CRS, a new geometry type, a schema edit.
      - alert: NewDeadLetterClass
        expr: |
          sum by (pipeline, failure_class) (geotask_dlq_arrivals_1h) > 0
          unless
          sum by (pipeline, failure_class) (geotask_dlq_arrivals_1h offset 7d) > 0
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "new failure class {{ $labels.failure_class }} on {{ $labels.pipeline }}"

      # 3. Nothing is draining. Depth may be flat and still be wrong if the
      #    oldest entry keeps ageing — that means the drain flow is broken.
      - alert: DeadLetterQueueStale
        expr: geotask_dlq_oldest_seconds > 7 * 24 * 3600
        for: 1h
        labels: { severity: ticket }
        annotations:
          summary: "oldest unresolved dead letter on {{ $labels.pipeline }} is over a week old"

Two details in those rules repay a second look. delta(...[1h]) > 50 measures the change in depth, which nets arrivals against drains — a pipeline adding 200 entries an hour while the drain removes 190 is not an emergency, and this expression says so, where a raw arrival-rate alert would not. And the new-class rule uses unless … offset 7d rather than a threshold, because the interesting property of a new failure class is not how many entries it has but that it exists at all: one entry with failure_class="unknown" from a source that has run cleanly for a year is worth more attention than four hundred timeouts.

What none of these rules do is tell you the pipeline is fine. A queue can stay flat and empty because the pipeline stopped running altogether, which is a different alert on a different signal — run freshness — and the two are complementary. It is worth writing that pairing down somewhere visible, because a team that has just built queue alerting tends to assume it now covers silence too.

Parameter & Option Reference#

Verification & Testing#

Alert rules are code and deserve tests. promtool evaluates them against a synthetic series, which is far more reliable than waiting for a real incident.

# dlq_alerts_test.yaml — run with: promtool test rules dlq_alerts_test.yaml
rule_files: [dlq_alerts.yaml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # A steady climb of 2 entries a minute for two hours.
      - series: 'geotask_dlq_open_entries{pipeline="parcels",failure_class="crs_mismatch"}'
        values: '0+2x120'
    alert_rule_test:
      - eval_time: 90m
        alertname: DeadLetterQueueGrowing
        exp_alerts:
          - exp_labels: { severity: ticket, pipeline: parcels }

  - interval: 1m
    input_series:
      # A sawtooth: rises to 60, drained to 0, repeatedly. Must NOT alert.
      - series: 'geotask_dlq_open_entries{pipeline="parcels",failure_class="upstream_timeout"}'
        values: '0+1x60 0+1x60'
    alert_rule_test:
      - eval_time: 115m
        alertname: DeadLetterQueueGrowing
        exp_alerts: []

The second test is the important one. Most first attempts at this alert fire on the healthy sawtooth, and the fastest way to find that out is a test rather than a week of pages.

Common Pitfalls#

• Alerting on absolute depth. It fires every afternoon on a healthy queue and stays silent through four days of monotonic growth if the starting point was low. The derivative is what carries the information.
• Forgetting to clear stale label sets. A failure_class that stops occurring keeps its last gauge value forever unless the exporter clears its labels, so a resolved problem alerts indefinitely and everyone learns to ignore the rule.
• One alert for every class. Ten classes times three rules is thirty alerts, most of which are duplicates of the same incident. Aggregate with sum by (pipeline) for the growth rule and reserve per-class alerting for the new-class rule.
• No alert on the drain itself. If the drain flow dies, depth stops falling but also stops rising sharply — it can look almost normal. The oldest-entry gauge is what makes that visible, and it is the rule people most often skip.
• Scraping the queue table with an unindexed query. count(*) WHERE resolved_at IS NULL on a large table without the partial index is a sequential scan every minute. Index it, as in storing failed geometries in a PostGIS dead-letter queue.

Frequently Asked Questions#

Related#

• Dead-letter queues for failed geotasks — the queue being measured
• Reprocessing dead-letter geotasks safely — the drain whose absence the staleness rule detects
• Prometheus metrics for raster throughput — the exporter conventions this reuses
• Data-quality SLOs for spatial pipelines — where queue depth becomes a user-visible promise

← Dead-Letter Queues for Failed Geotasks