Masking Sensitive Coordinates in Task Logs

Logs are the least controlled copy of a pipeline’s data. They are shipped off the worker, retained for months, indexed for full-text search and readable by a much wider group than the database ever is — and a task that logs its inputs will log coordinates. The control that works is a processor in the logging configuration rather than discipline at the call sites, because the call site that leaks is the one somebody adds at two in the morning while debugging.

When to Use This Pattern#

• Any layer contains precise locations tied to people — addresses, meter readings, vehicle traces, incident reports.
• Logs leave the worker, which they do the moment there is more than one worker.
• Debug logging gets enabled during incidents, which is exactly when the most detailed values are printed.
• A retention policy exists and is longer than anybody’s memory of what was logged.

Complete Working Example#

A structlog processor that rounds coordinates, hashes identifiers and refuses to emit raw geometry, installed once.

from __future__ import annotations

import hashlib
import re
from typing import Any

import structlog

COORD_KEYS = {"lat", "lon", "latitude", "longitude", "x", "y", "centroid"}
ID_KEYS = {"address_id", "meter_id", "household_id", "subject_id"}
GEOM_KEYS = {"geom", "geometry", "wkt", "wkb"}

# 3 decimals ≈ 110 m: a street, not a doorstep. Set per environment, not per call.
PRECISION = 3
_PEPPER = os.environ["LOG_HASH_PEPPER"].encode()

# Catches coordinates embedded in free text, which structured keys cannot reach.
_COORD_IN_TEXT = re.compile(r"(-?\d{1,3}\.\d{5,})")


def _round_coord(value: Any) -> Any:
    try:
        return round(float(value), PRECISION)
    except (TypeError, ValueError):
        return value


def _pseudonymise(value: Any) -> str:
    digest = hashlib.blake2b(str(value).encode(), key=_PEPPER, digest_size=8)
    return f"id:{digest.hexdigest()}"


def scrub(_logger, _method, event: dict) -> dict:
    for key, value in list(event.items()):
        low = key.lower()
        if low in GEOM_KEYS:
            # Never a rounded geometry: a rounded polygon is still a polygon.
            event[key] = f"<{type(value).__name__} suppressed>"
        elif low in COORD_KEYS:
            event[key] = _round_coord(value)
        elif low in ID_KEYS:
            event[key] = _pseudonymise(value)
    if isinstance(event.get("event"), str):
        event["event"] = _COORD_IN_TEXT.sub(
            lambda m: f"{float(m.group(1)):.{PRECISION}f}", event["event"]
        )
    return event


structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        scrub,                                   # before rendering, after context merge
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

Using it requires nothing of the task, which is the point:

log = structlog.get_logger()


@task(retries=2, tags=["tile"])
def geocode_delivery(row: dict) -> Point:
    point = geocode(row["address"])
    # Both of these are scrubbed on the way out; the call site does not have to know.
    log.info("geocoded", address_id=row["address_id"], lat=point.y, lon=point.x)
    return point

The keyed hash matters more than it looks. An unkeyed hash of an address identifier is trivially reversible when the identifier space is small — a few million households is a few seconds of brute force — so the pseudonym would protect nothing. A keyed hash with a per-environment pepper is not reversible without the key, and because the key is stable within the environment, two log records about the same household still carry the same pseudonym and can still be correlated. That combination, unlinkable outside and linkable inside, is exactly what a debugging workflow needs.

Parameter & Option Reference#

Verification & Testing#

def test_structured_coordinates_are_rounded(cap_structlog) -> None:
    log.info("geocoded", lat=51.5074231, lon=-0.1277653)
    rec = cap_structlog[-1]
    assert rec["lat"] == 51.507 and rec["lon"] == -0.128


def test_identifier_is_pseudonymised_and_stable(cap_structlog) -> None:
    log.info("a", household_id=4419023)
    log.info("b", household_id=4419023)
    a, b = cap_structlog[-2:]
    assert a["household_id"].startswith("id:") and a["household_id"] == b["household_id"]
    assert "4419023" not in a["household_id"]


def test_geometry_is_suppressed_not_rounded(cap_structlog) -> None:
    log.info("loaded", geom=Polygon(HOUSE_OUTLINE))
    assert "suppressed" in cap_structlog[-1]["geom"]


def test_coordinate_in_a_message_is_caught(cap_structlog) -> None:
    log.info(f"failed at 51.5074231, -0.1277653")
    assert "51.5074231" not in cap_structlog[-1]["event"]
    assert "51.507" in cap_structlog[-1]["event"]


def test_pepper_differs_between_environments() -> None:
    assert os.environ["LOG_HASH_PEPPER"] != STAGING_PEPPER, (
        "staging and production logs can be joined on the pseudonym"
    )

The last test is the one that turns a pseudonym into an actual control. If staging and production share a pepper, an identifier that appears in a staging log — where retention is longer, access is broader and the data is often a copy of production — can be joined straight back to the production record. Separate peppers per environment cost one secret each and remove the whole linkage, and the property is worth asserting because a shared pepper is what happens when somebody copies a configuration file.

A word on where the processor sits in the chain, because the ordering is not arbitrary. It runs after merge_contextvars, so anything bound to the logging context — a household identifier bound once at the top of a flow and carried through twenty log lines — is scrubbed on every one of them rather than only where it was written. It runs before the renderer, so it still sees typed values: a float it can round and a Polygon it can recognise, rather than a string it would have to parse. Moving it either side of those two boundaries produces a processor that appears to work and misses most of what it is for.

There is also a question of what not to scrub, and it is worth answering deliberately rather than by omission. Tile indices at moderate zoom, EPSG codes, feature counts, source names, durations and coverage figures are all safe and all genuinely useful, and a scrubbing rule so broad that it removes them makes the logs useless and the rule the first thing somebody disables. The list of sensitive keys should be short, explicit and reviewed when a new layer arrives — which is the same moment the layer’s precision classification is decided, so the two belong in the same conversation.

Common Pitfalls#

• Scrubbing at the call site. It works until somebody adds a log line during an incident, which is the line that matters.
• Rounding geometry instead of suppressing it. A rounded polygon still traces a boundary and still identifies a property.
• An unkeyed hash. A small identifier space makes it reversible in seconds; use a keyed hash with a pepper.
• A shared pepper across environments. Staging logs then join to production records, which is worse than not pseudonymising at all.
• Ignoring free text. Key-based rules cannot see a coordinate interpolated into a message.
• Placing the processor after the renderer. By then the record is a string and the structured fields are gone.

Frequently Asked Questions#

Related#

• Security boundaries for spatial data — why the log boundary is the leakiest of the three
• Structured logging for geospatial flows — where this processor is installed
• Logging feature counts and EPSG codes per task — what is safe to log freely
• Exporting per-tile metrics without cardinality blowups — the same problem in the metrics path
• Scoping object storage credentials per tile job — the credential boundary

← Security Boundaries for Spatial Data