Caching Strategies for Spatial Tasks

In short: a spatial cache key is a content hash of everything that determines the output — source bytes, source CRS, target CRS, resampling method, nodata value, and the exact extent — not a filename or a timestamp. Get the key right and reuse is safe and automatic; get it wrong and the cache serves last week’s projection under this week’s name, which is a much worse outcome than no cache at all.

Caching is the highest-leverage optimisation available to most spatial pipelines, because the work is expensive, deterministic and frequently repeated: the same tile is reprojected every night from a source that changed once a quarter. It is also the optimisation with the worst failure mode. A cache miss costs time; a cache hit on the wrong key costs correctness, silently, in a way that surfaces weeks later in a derived product nobody thinks to blame on caching.

The reason this bites harder in spatial work than elsewhere is that so many of the inputs are invisible in the obvious identifiers. A tabular transform is usually a function of a table and a query, both of which appear in any reasonable cache key by accident. A raster transform is a function of a scene, two coordinate reference systems, a resampling kernel, a nodata sentinel, an output grid and a compression choice — and the conventional identifier for all of that is a filename that mentions perhaps two of them. The gap between what a name says and what a result depends on is where every spatial caching bug lives.

Prerequisites & Architecture Baseline#

• Storage that suits the artefact: object storage for tiles and rasters, a database table for small derived features
• A content digest for every source — an ETag, a checksum header, or a hash you compute once on ingest
• pyproj ≥ 3.6, pinned, since CRS serialisation participates in the key
• A stated eviction policy, even if it is "never" — an unbounded cache is a storage incident waiting for a quiet week
• Metrics on hit rate, miss rate and bytes served, per layer
• A decision about whether a cache miss may block the request or must fall through to a slower path

Core Principles#

1. The key is a function of the inputs, not of the name. Filenames lie: parcels_2026.gpkg is republished in place, ortho.tif is regenerated with different resampling. A digest over the actual bytes plus the actual parameters cannot lie, and it is the same digest an idempotency key uses — the two mechanisms differ in what they do with the answer, not in how they compute it.

2. Every parameter that changes the pixels belongs in the key. Target CRS, resampling method, nodata value, output dtype, compression, and the extent. Omitting resampling is the classic error: a switch from nearest to cubic produces visibly different rasters, and a cache blind to it serves the old pixels indefinitely under a name that promises the new ones.

3. Normalise before hashing. EPSG:3857, epsg:3857 and the equivalent WKT2 string all describe one CRS and must produce one key. Run every CRS through pyproj’s authority lookup, sort dictionaries, and pin the library — otherwise a routine dependency upgrade silently invalidates the entire cache and the next run rebuilds a continent.

4. Invalidation follows the source, not the clock. A TTL is a guess about how often the source changes; the source itself knows. Where the upstream publishes an ETag, a Last-Modified or a version number, key on that and the cache is exactly as fresh as the data. TTLs are a fallback for sources that offer nothing, not a default.

5. Cache the expensive intermediate, not the cheap final. The reprojected raster is worth caching; the PNG rendered from it in 40 ms usually is not. Look at where the time actually goes before deciding what to store — cached artefacts cost storage and complexity forever, so they should be earning it.

6. A stale hit must be impossible, and a stale miss must be cheap. These pull in opposite directions and the resolution is asymmetric: make the key conservative, so anything uncertain is a miss, and make misses fast by keeping the cache close to the compute. A cache that occasionally recomputes something it could have reused is merely slower; one that occasionally reuses something it should have recomputed is wrong.

There is a useful discipline hiding in that comparison. Write down, as a list, every argument the compute function takes and every environment value it reads. Then mark each one as either “affects the output” or “does not”. The first group is the key; the second group is configuration. Anything you cannot confidently place — does the GDAL version affect the output? does the thread count? — is exactly where a cache bug will eventually live, and the honest response is either to include it and accept the extra misses, or to pin it so tightly that it cannot vary. What does not work is leaving it unresolved and hoping.

Production Implementation#

The cache below is content-addressed on object storage. Reads are a single HEAD; writes go to a temporary key and are renamed, so a partially-written object is never visible under a valid key.

from __future__ import annotations

import base64
import hashlib
import json
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Mapping, Optional

from botocore.exceptions import ClientError
from pyproj import CRS

CACHE_VERSION = 3        # bump to invalidate everything deliberately


@dataclass(frozen=True)
class RasterOp:
    """Every input that decides the output pixels."""

    source_digest: str          # ETag or checksum of the source scene
    src_crs: str
    dst_crs: str
    resampling: str
    nodata: Optional[float]
    dtype: str
    bounds: tuple[float, float, float, float]
    resolution: float

    def canonical(self) -> Mapping[str, Any]:
        payload = dict(asdict(self))
        # Collapse equivalent CRS spellings so a source that switches to WKT2
        # does not silently invalidate every cached tile it feeds.
        payload["src_crs"] = CRS.from_user_input(self.src_crs).to_authority()
        payload["dst_crs"] = CRS.from_user_input(self.dst_crs).to_authority()
        # Round the bounds: float noise in the 12th decimal is not a new extent.
        payload["bounds"] = tuple(round(v, 6) for v in self.bounds)
        return payload


def cache_key(op: RasterOp, namespace: str = "warp") -> str:
    blob = json.dumps(op.canonical(), sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(f"{namespace}:v{CACHE_VERSION}:{blob}".encode()).digest()
    token = base64.urlsafe_b64encode(digest[:15]).decode().rstrip("=")
    # Two-level prefix keeps object-store listings and any filesystem mirror sane.
    return f"{namespace}/{token[:2]}/{token[2:4]}/{token}.tif"


def get_or_compute(s3, bucket: str, op: RasterOp, compute) -> tuple[str, bool]:
    """Returns (key, was_hit). `compute` is called only on a miss."""
    key = cache_key(op)
    try:
        s3.head_object(Bucket=bucket, Key=key)
        return key, True
    except ClientError as exc:
        if exc.response["Error"]["Code"] not in ("404", "NoSuchKey"):
            raise

    local: Path = compute(op)
    staging = key + ".part"
    s3.upload_file(str(local), bucket, staging, ExtraArgs={
        "Metadata": {
            # Provenance on the object itself: what produced it, and from what.
            "cache-version": str(CACHE_VERSION),
            "source-digest": op.source_digest,
            "resampling": op.resampling,
            "dst-crs": op.dst_crs,
        }
    })
    s3.copy_object(Bucket=bucket, Key=key,
                   CopySource={"Bucket": bucket, "Key": staging})
    s3.delete_object(Bucket=bucket, Key=staging)
    return key, False

Step-by-Step Walkthrough#

1. Collect the inputs as a frozen structure. RasterOp makes the key’s dependencies explicit and reviewable. Anyone adding a parameter to the operation has to decide, in the same commit, whether it belongs in the key.
2. Normalise the CRS through pyproj. to_authority() collapses spellings. Without it, the first source that starts emitting WKT2 invalidates every tile it feeds, and the resulting rebuild looks like an unexplained cost spike.
3. Round the bounds before hashing. Floating-point extents computed two different ways differ in the last decimal place. Rounding to six decimals — sub-millimetre in projected metres — removes a source of spurious misses without ever merging genuinely different extents.
4. Prefix with a namespace and a version. The namespace separates operations that share inputs but produce different outputs. The version is the deliberate invalidation lever, used when the hashing rule itself changes.
5. Shard the key path. warp/a3/f0/a3f0…tif keeps object-store prefixes and any filesystem mirror from developing directories with a million entries, which degrades listing performance badly on some backends.
6. Check with HEAD, not GET. Existence is all the read path needs, and a HEAD is a fraction of the cost. The consumer fetches the object by key afterwards if it actually needs the bytes.
7. Write through a temporary key. An interrupted upload leaves …​.part, never a truncated object under a valid key. The copy-then-delete is the commit point, and it is the object-store equivalent of the write-and-rename discipline used everywhere else in the pipeline.

Edge Cases & Failure Recovery#

A source that changes its ETag without changing its content. Multipart re-uploads and some CDN configurations do this, and the cache rebuilds everything for no reason. Where the store offers a content checksum (x-amz-checksum-sha256), prefer it; where it does not, hashing the first and last megabyte plus the size is a cheap approximation that is stable across re-uploads.

Two operations that are genuinely identical but keyed differently. A pipeline that computes bounds in two places, with different float arithmetic, produces two keys for one output and halves the hit rate. The rounding above fixes most of it; the rest is fixed by computing the bounds once and passing them, rather than recomputing per step.

A cache that outlives its correctness. CACHE_VERSION is the answer, but only if it is actually bumped. Any change to what the compute function does — a GDAL upgrade that alters resampling, a change in how nodata is filled — must bump it. Tying the version to a value the compute step also reads makes this harder to forget.

Unbounded growth. Content-addressed caches never overwrite, so every source revision leaves its predecessors behind forever. A lifecycle rule that expires objects untouched for ninety days handles it on object storage; on a filesystem you need an eviction job, and it needs to run before the disk fills rather than after.

A concurrent miss stampede. A hundred workers all miss the same key at once and all compute the same tile. For expensive operations, take a short lock on the key — the same SET NX EX pattern as a half-open breaker probe — so one computes and the rest wait for the result.

An artefact that is correct but unusable. A cached GeoTIFF written without internal tiling or overviews is a perfectly valid cache hit that makes every downstream read slower than recomputing a properly structured one would have been. The cache stores whatever the compute step produced, so the structure of the artefact is decided upstream — which means “add TILED=YES and COMPRESS=DEFLATE” is a caching decision even though it appears nowhere near the cache code.

The cache is faster than the source but not fast enough. A hit that takes 800 ms from a distant region is barely better than recomputing locally. Cache locality matters: keep the cache in the same region as the compute, and measure hit latency alongside hit rate, because a high hit rate on a slow cache can be a net loss.

Configuration Reference#

The remaining decision is where the cache lives relative to the compute, and it is worth making explicitly rather than by default. Object storage in the same region is the usual answer: durable, effectively unbounded, and a HEAD is a few milliseconds. A local disk cache in front of it helps when the same tiles are read repeatedly within one run, which is common for overlapping mosaic windows. What rarely pays is a cache in a different region from the workers — the transfer cost and the latency together can exceed the compute you were avoiding, and the hit-rate dashboard will look excellent the whole time.

Frequently Asked Questions#

Related#

• Caching reprojected rasters with content hashing — the raster case end to end
• Invalidating tile caches after a source update — propagating a source change through the pyramid
• Idempotency keys in spatial ETL — the same digest, a different question
• Cutting egress costs with COG range reads — reducing what a miss costs in the first place
• Skipping tiles with no new source data — avoiding the work rather than caching it

← Spatial Task Design & Dependency Mapping