Geospatial Orchestration Hub

Caching Reprojected Rasters with Content Hashing

Reprojection is the most cacheable operation in a raster pipeline: deterministic, expensive, and run repeatedly against sources that change far less often than the pipeline does. Content-address the output on a digest of the source bytes plus every warp parameter, look it up with a single HEAD before doing any work, and the nightly run stops re-warping the ninety per cent of scenes that did not change. The whole recipe is thirty lines; the care goes into computing a source digest cheaply and into making sure nothing that changes the pixels is missing from the key.

When to Use This Pattern

  • The same scenes are reprojected on a schedule — nightly mosaics, weekly refreshes — while the underlying imagery changes quarterly or less.
  • A warp costs minutes, so a lookup that costs milliseconds is obviously worth it.
  • The source exposes something you can hash cheaply: an ETag, a checksum header, or a stable version identifier.
  • Storage is cheaper than compute, which for object storage against GDAL time is essentially always.

Complete Working Example

The cheap part of this is the digest. Reading a 4 GB scene to hash it defeats the purpose, so use what the store already knows, and fall back to a sampled hash only when it does not.

from __future__ import annotations

import base64
import hashlib
import json
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Optional

import boto3
from botocore.exceptions import ClientError
from pyproj import CRS

WARP_CACHE_VERSION = 4


def source_digest(s3, bucket: str, key: str) -> str:
    """Identify the source's content without reading it.

    Preference order: a real content checksum, then the ETag, then a sampled
    hash. The last is a fallback, not a default — it reads 2 MB rather than 4 GB
    but it can be fooled by a change confined to the middle of the file.
    """
    head = s3.head_object(Bucket=bucket, Key=key, ChecksumMode="ENABLED")
    if "ChecksumSHA256" in head:
        return "sha256:" + head["ChecksumSHA256"]
    etag = head["ETag"].strip('"')
    if "-" not in etag:                     # a single-part upload: ETag is the MD5
        return "md5:" + etag
    # Multipart ETags depend on part size, so they change on re-upload. Sample
    # the head, the tail and the size instead, which does not.
    size = head["ContentLength"]
    head_bytes = s3.get_object(Bucket=bucket, Key=key, Range="bytes=0-1048575")["Body"].read()
    tail_bytes = s3.get_object(
        Bucket=bucket, Key=key, Range=f"bytes={max(0, size - 1048576)}-{size - 1}"
    )["Body"].read()
    sampled = hashlib.sha256(head_bytes + tail_bytes + str(size).encode()).hexdigest()
    return "sampled:" + sampled


@dataclass(frozen=True)
class WarpSpec:
    source_digest: str
    dst_crs: str = "EPSG:3857"
    resampling: str = "bilinear"
    nodata: Optional[float] = -9999.0
    resolution: Optional[float] = None
    compress: str = "DEFLATE"

    def key(self) -> str:
        payload = {
            "digest": self.source_digest,
            # Authority form, so an upstream switch to WKT2 does not re-key everything.
            "dst": CRS.from_user_input(self.dst_crs).to_authority(),
            "resampling": self.resampling,
            "nodata": self.nodata,
            "resolution": self.resolution,
            "compress": self.compress,
        }
        blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
        digest = hashlib.sha256(f"warp:v{WARP_CACHE_VERSION}:{blob}".encode()).digest()
        token = base64.urlsafe_b64encode(digest[:15]).decode().rstrip("=")
        return f"warp/{token[:2]}/{token[2:4]}/{token}.tif"


def warp_cached(s3, cache_bucket: str, src_uri: str, spec: WarpSpec, scratch: Path) -> str:
    key = spec.key()
    try:
        s3.head_object(Bucket=cache_bucket, Key=key)
        return key                                    # hit: no GDAL, no I/O
    except ClientError as exc:
        if exc.response["Error"]["Code"] not in ("404", "NoSuchKey"):
            raise

    out = scratch / "warp.tif"
    argv = [
        "gdalwarp", "-t_srs", spec.dst_crs, "-r", spec.resampling,
        "-dstnodata", str(spec.nodata),
        "-co", f"COMPRESS={spec.compress}", "-co", "TILED=YES",
        "-co", "BLOCKXSIZE=512", "-co", "BLOCKYSIZE=512",
        "-wm", "512",
    ]
    if spec.resolution:
        argv += ["-tr", str(spec.resolution), str(spec.resolution)]
    subprocess.run(argv + [src_uri, str(out)], check=True, start_new_session=True)

    staging = key + ".part"
    s3.upload_file(str(out), cache_bucket, staging, ExtraArgs={"Metadata": {
        "source-digest": spec.source_digest,
        "resampling": spec.resampling,
        "dst-crs": spec.dst_crs,
        "cache-version": str(WARP_CACHE_VERSION),
    }})
    s3.copy_object(Bucket=cache_bucket, Key=key,
                   CopySource={"Bucket": cache_bucket, "Key": staging})
    s3.delete_object(Bucket=cache_bucket, Key=staging)
    return key
How to identify a 4 GB source without reading itA content checksum header costs one HEAD request and is exact. A single-part ETag is also one HEAD and is exact. A sampled hash costs two range reads of one megabyte each and can miss a change confined to the middle of the file.methodbytes readdetects every change?ChecksumSHA256 header0 — one HEADyessingle-part ETag0 — one HEADyessampled head + tail + size2 MB of 4 GBno — middle edits missedA multipart ETag is deliberately absent: it changes on re-upload of identical bytes, so it produces false misses.
Ask the store for a checksum before falling back. Many pipelines reach straight for the sampled hash and inherit its blind spot for no reason.

The WarpSpec is deliberately the only place warp parameters are named. When a new option is added — a mask band, an alpha channel, a different creation option — it has to go into the dataclass to be used, and going into the dataclass puts it in front of whoever writes the key() method. That coupling is the point: it converts “remember to update the cache key” from a habit into a step the type system nudges you toward. It is not airtight, since a parameter could still be passed around the spec, but it removes the most common path to a stale hit.

The failure a missing key field producesTwo warps differing only in resampling map to the same key when resampling is omitted from it. The second warp is never run and the first warp's pixels are served under both names.warp A · nearestrun in Marchwarp B · cubicrequested in Juneone cache keyJune serves March'snearest-neighbour pixelsNothing errors. The June run is faster than expected, which is usually taken as good news.
The tell is a cache hit rate that goes up after a parameter change. That is the shape of this bug, and it is worth an alert of its own.

Parameter & Option Reference

Parameter Type Default Spatial notes
WARP_CACHE_VERSION int 4 Bump on any change to the warp itself, including a pinned GDAL upgrade that alters resampling output.
resampling str bilinear In the key, always. nearest and cubic produce visibly different rasters from identical input.
nodata float -9999.0 In the key. It changes every edge pixel of the result and is frequently omitted by accident.
resolution float source In the key when set, since -tr changes the output grid entirely.
-co TILED=YES — on The cached artefact is read many times; an untiled GeoTIFF makes every windowed read slow.
BLOCKXSIZE int 512 Matches typical read windows. In the key indirectly via compress and creation options if you vary them.
-wm MB 512 Bounds GDAL’s warp memory so the miss path has predictable peak usage.

Verification & Testing

Two assertions matter: identical inputs hit, and any parameter change misses.

def test_identical_inputs_hit(s3_stub, scratch) -> None:
    spec = WarpSpec(source_digest="sha256:abc")
    first = warp_cached(s3_stub, "cache", "s3://src/scene.tif", spec, scratch)
    calls_before = s3_stub.gdal_calls
    second = warp_cached(s3_stub, "cache", "s3://src/scene.tif", spec, scratch)
    assert first == second
    assert s3_stub.gdal_calls == calls_before, "a hit must not invoke gdalwarp"


def test_every_parameter_changes_the_key() -> None:
    base = WarpSpec(source_digest="sha256:abc")
    variants = [
        WarpSpec(source_digest="sha256:def"),
        WarpSpec(source_digest="sha256:abc", dst_crs="EPSG:25832"),
        WarpSpec(source_digest="sha256:abc", resampling="cubic"),
        WarpSpec(source_digest="sha256:abc", nodata=0.0),
        WarpSpec(source_digest="sha256:abc", resolution=10.0),
    ]
    keys = {base.key()} | {v.key() for v in variants}
    assert len(keys) == 6, "some parameter is missing from the key"


def test_equivalent_crs_spellings_share_a_key() -> None:
    a = WarpSpec(source_digest="sha256:abc", dst_crs="EPSG:3857")
    b = WarpSpec(source_digest="sha256:abc", dst_crs="epsg:3857")
    assert a.key() == b.key()

The second test is the one to write first and keep forever: it fails loudly the day someone adds a warp parameter and forgets the key. Against real storage, the object’s own metadata makes an audit trivial, which matters when someone asks why a particular tile looks the way it does:

aws s3api head-object --bucket cache --key warp/a3/f0/a3f0…tif \
  --query 'Metadata' --output table
# source-digest, resampling, dst-crs and cache-version, straight from the artefact.
What the cache does to a nightly runWithout the cache every night warps all four hundred scenes. With it, the first night warps four hundred and subsequent nights warp only the thirty-eight that changed.NO CACHE — 400 warps every nightMon 400Tue 400Wed 400Thu 400CONTENT-ADDRESSED CACHEMon 400Tue 12Wed 38Thu 6The bars after Monday are the source actually changing. Everything else was work the pipeline was repeatingbecause it had no way to know it had already done it.
Monday still costs full price, and it always will — a content-addressed cache has no warm-up strategy beyond running once.

Monday’s full-price run is worth planning for rather than absorbing. A cold cache on a national dataset can be several times the normal nightly cost, which is fine once and unwelcome as a surprise. The two practical mitigations are to warm the cache incrementally — run the previous night’s scene list against the new key before the change takes effect, spreading the rebuild over a week — and to schedule any deliberate CACHE_VERSION bump for a weekend rather than the night before a delivery. Both are scheduling decisions rather than code, and both are easy to skip and then regret.

Common Pitfalls

  • Keying on a multipart ETag. It depends on the upload’s part size, so a re-upload of identical bytes changes it and the cache rebuilds for nothing. Prefer a checksum header; fall back to sampling.
  • Omitting nodata from the key. It is easy to forget because it often has a default, and it changes every edge pixel of the warped result. A cache blind to it will serve the wrong edges indefinitely.
  • Hashing the whole source every run. Reading 4 GB to decide whether to warp 4 GB saves nothing. Use metadata the store already has.
  • Writing directly to the final key. An interrupted upload leaves a truncated object that every subsequent run treats as a valid hit. The .part-then-copy sequence costs one API call and removes the possibility.
  • Forgetting TILED=YES on the cached artefact. The object is read far more often than it is written, and an untiled GeoTIFF makes every windowed read pull the whole file — turning a cache hit into a slow one.

Frequently Asked Questions

Is a sampled digest safe enough?

For imagery that is written once and never edited in place — which describes most published scenes — yes. It is unsafe where a producer patches a file’s interior and keeps the size identical, which does happen with some in-place raster editing workflows. If your sources are edited rather than republished, insist on a real checksum or accept the cost of a full hash.

Should the cache be in the same bucket as the source?

Separate buckets are better. It keeps lifecycle rules independent — sources may be archived to cold storage while the cache stays hot — and it makes the cache’s cost visible as its own line rather than blended into source storage. Same region, though, always.

What about caching the mosaic rather than the tiles?

Cache both, at different granularities. The tiles are what change independently, so they get the highest hit rate; the mosaic is what consumers ask for, and caching it saves the assembly step on the many nights when nothing changed. Keying the mosaic on the sorted list of its tiles’ keys makes it invalidate automatically when any tile does.

Does this replace the idempotency ledger?

No. The cache says “here is a result you can reuse”; the ledger says “this work has been done”. A cache entry may be evicted without breaking anything, and a ledger entry may not. Pipelines that write side effects need both, as described in idempotency keys in spatial ETL.

← Caching Strategies for Spatial Tasks

Related in this section