Caching Reprojected Rasters with Content Hashing

To cache a reprojection with content hashing, build a deterministic key from a hash of the source raster (or its stored checksum) combined with the source and target EPSG codes, the resampling method and the target resolution — then store the warped output under that key so an identical warp is skipped and only a novel one recomputes. This page gives a self-contained function that builds the key, persists the result, and verifies correctness with gdalinfo band checksums.

When to Use This Pattern

Apply content-hashed reprojection caching when:

The same scenes are re-warped to the same target CRS across repeated runs (a nightly tiling batch, a re-triggered flow).
Reprojection is a measurable cost — large scenes, expensive resampling, or per-run egress pulling the source repeatedly.
You need reproducibility: the same input and parameters must always resolve to the same cached artifact.
Multiple workers may warp the same source and you want deduplicated, shared results in object storage.
You want warps to invalidate automatically when the source pixels, target EPSG, resampling method or resolution change — with no manual purge.

Complete Working Example

The function hashes the source raster’s GDAL checksum together with the warp parameters, checks object storage for that key, and warps only on a miss. It writes to a temporary key and renames atomically so concurrent workers cannot leave a torn object.

import hashlib
import json
from pathlib import PurePosixPath
from typing import Optional

import rasterio
import pyproj
from osgeo import gdal

gdal.UseExceptions()

CACHE_PREFIX: str = "s3://cache/warp"


def reproject_cache_key(
    src_path: str,
    source_epsg: int,
    target_epsg: int,
    resampling: str,
    resolution: float,
) -> str:
    """Deterministic key: source checksum + EPSG pair + resampling + resolution + PROJ."""
    with rasterio.open(src_path) as src:
        # GDAL stores a per-band checksum; reading it is cheap and content-sensitive.
        checksum: int = src.checksum(1)
    material: dict[str, object] = {
        "checksum": checksum,
        "source_epsg": source_epsg,
        "target_epsg": target_epsg,
        "resampling": resampling,
        "resolution": resolution,
        "proj_version": pyproj.proj_version_str,  # PROJ bump invalidates the entry
    }
    blob: str = json.dumps(material, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()


def object_exists(uri: str) -> bool:
    """Cheap existence check via GDAL VSI stat — no full download."""
    vsi_path: str = "/vsis3/" + uri.removeprefix("s3://")
    return gdal.VSIStatL(vsi_path) is not None


def cached_reproject(
    src_path: str,
    source_epsg: int,
    target_epsg: int,
    resampling: str = "bilinear",
    resolution: float = 30.0,
    nodata: Optional[float] = None,
) -> str:
    key: str = reproject_cache_key(src_path, source_epsg, target_epsg, resampling, resolution)
    dst_uri: str = f"{CACHE_PREFIX}/{target_epsg}/{key}.tif"

    if object_exists(dst_uri):
        return dst_uri  # HIT: identical warp already materialized, skip it

    # MISS: warp to a temp key, then rename atomically so readers never see a partial.
    tmp_uri: str = f"{CACHE_PREFIX}/{target_epsg}/{key}.tmp.tif"
    warp_opts = gdal.WarpOptions(
        srcSRS=f"EPSG:{source_epsg}",
        dstSRS=f"EPSG:{target_epsg}",
        resampleAlg=resampling,
        xRes=resolution, yRes=resolution,
        dstNodata=nodata,             # carry nodata through the reprojection
        format="COG",
    )
    gdal.Warp(tmp_uri, src_path, options=warp_opts)
    gdal.Rename(tmp_uri, dst_uri)     # atomic publish of the finished artifact
    return dst_uri


if __name__ == "__main__":
    out = cached_reproject(
        "s3://imagery/scene.tif",
        source_epsg=4326, target_epsg=3857,
        resampling="bilinear", resolution=30.0, nodata=0.0,
    )
    print(f"reprojected artifact at: {out}")

Because the key folds the checksum and the exact parameters, calling cached_reproject twice with the same source and options resolves the second call to a store existence check and an immediate return — no warp, no re-fetch of the source pixels. Change the target_epsg, resampling or resolution and the key changes, forcing a fresh warp.

Parameter & Option Reference

Parameter	Type	Default	Spatial notes
`src_path`	`str`	—	Source raster URI; its band checksum, not its name, drives the key
`source_epsg`	`int`	—	Declared source CRS; must match the file’s real CRS or the warp is wrong
`target_epsg`	`int`	—	Destination CRS; part of the key and the store path
`resampling`	`str`	`bilinear`	Use `nearest` for categorical/land-cover data to avoid interpolating classes
`resolution`	`float`	`30.0`	Target `xRes`/`yRes` in CRS units; changing it changes the pixels and the key
`nodata`	`float \| None`	`None`	Carried through the warp; dropping it corrupts masked areas
`proj_version`	`str`	auto	Read from `pyproj.proj_version_str`; ensures a PROJ upgrade invalidates entries

Verification & Testing

First confirm the cache hit produces a byte-faithful artifact by comparing GDAL band checksums between a fresh warp and the cached one:

# Checksum of the cached artifact
gdalinfo -checksum /vsis3/cache/warp/3857/<key>.tif | grep Checksum
# Checksum=  51234

# Warp the source again to a scratch file and compare
gdalwarp -t_srs EPSG:3857 -r bilinear -tr 30 30 \
  /vsis3/imagery/scene.tif /vsimem/fresh.tif
gdalinfo -checksum /vsimem/fresh.tif | grep Checksum
# Checksum=  51234   # identical -> the cached warp matches a fresh one

Then assert the key logic in a unit test: identical inputs share a key, and any changed parameter diverges.

def test_key_is_deterministic_and_parameter_sensitive() -> None:
    args = dict(src_path="s3://imagery/scene.tif", source_epsg=4326,
                target_epsg=3857, resampling="bilinear", resolution=30.0)
    k1 = reproject_cache_key(**args)
    k2 = reproject_cache_key(**args)
    assert k1 == k2                                  # deterministic -> cache hit

    k3 = reproject_cache_key(**{**args, "resampling": "nearest"})
    assert k1 != k3                                  # changed method -> miss

    k4 = reproject_cache_key(**{**args, "resolution": 10.0})
    assert k1 != k4                                  # changed resolution -> miss

If the two gdalinfo -checksum values differ while the key matched, your PROJ or GDAL version drifted between the cached run and the fresh one — confirm pyproj.proj_version_str is in the key so the mismatch forces a recompute instead of a stale hit.

Common Pitfalls

Hashing the filename instead of the content. Two scenes published under the same name hash identically and collide. Always fold in the GDAL band checksum (or the object ETag) so different pixels yield different keys.
Omitting the PROJ version. Reprojection results depend on the PROJ release and its datum-shift grids; a PROJ upgrade that changes a transform will serve stale artifacts unless proj_version is part of the key.
Interpolating categorical rasters. Defaulting to bilinear on land-cover or classification rasters blends discrete class codes into meaningless averages. Use nearest for categorical data — and because resampling is in the key, switching it correctly forces a recompute.
Dropping nodata through the warp. Omitting dstNodata lets fill values bleed into valid pixels at the reprojected edges. Always pass nodata so masked areas survive, matching the CRS discipline in validating coordinate systems before ETL.

Related: This how-to is the concrete case behind caching strategies for spatial tasks, cuts the re-warp egress described in cost optimization for spatial compute, and slots into a warp step chained per how to chain GDAL tasks in Prefect. See the GDAL warp documentation for the resampling algorithms.

← Back to Caching Strategies for Spatial Tasks

When to Use This Pattern#

Complete Working Example#

Parameter & Option Reference#

Verification & Testing#

Common Pitfalls#

When to Use This Pattern

Complete Working Example

Parameter & Option Reference

Verification & Testing

Common Pitfalls