Caching Strategies for Spatial Tasks

Caching Strategies for Spatial Tasks turn a reprojection, spatial join or resample from a cost you pay on every run into one you pay once, by keying cached artifacts on a content-addressed hash of the input geometry or raster plus its CRS and the exact task parameters — and invalidating that key the instant source data, resampling method, or the PROJ version changes. Done right, an unchanged tile pipeline re-runs in seconds by reading object storage instead of re-warping gigabytes.

The trap with spatial caching is that the obvious key — a filename or a task-run ID — is wrong. Two runs with the same filename can hold different pixels; the same input warped with nearest versus bilinear produces different outputs; and the same coordinates transformed under two PROJ releases can differ at the sub-metre level. A correct cache key is derived from content and parameters, not identity. This guide shows how to build content-addressed keys, wire them into Prefect’s cache_key_fn and Dagster’s asset caching, persist the resulting artifacts to object storage, and invalidate correctly when any input the output depends on shifts. For the reprojection case specifically, the companion caching reprojected rasters with content hashing walks through the warp key end to end.

Prerequisites & Architecture Baseline#

Confirm the stack exposes the caching hooks and hashing primitives this guide relies on.

• prefect>=2.19 (for cache_key_fn and cache_expiration) or dagster>=1.7 (for code-versioned assets)
• rasterio>=1.3.9 / gdal>=3.6 for checksums and reprojection
• geopandas>=0.14 and pyproj>=3.6 — and you can read pyproj.proj_version_str for PROJ pinning
• A content-addressable store: object storage (s3://, gs://, az://) or a Prefect result store configured
• Source rasters/vectors expose a stable checksum or ETag you can read without downloading the whole file
• Task parameters (CRS, resampling, resolution) are explicit function arguments, not hidden globals
• The PROJ and GDAL versions are pinned in the worker image (see the note on invalidation below)

If parameters are buried in environment variables or global config, surface them as arguments first — a cache key can only be correct if every input that changes the output is visible to the hash function.

Core Concepts#

Five concepts define correct spatial caching. Each answers a question the naive filename-key approach gets wrong.

1. Content-addressed keys, not identity keys#

The cache key is a hash of what the input contains and how it is processed, never of what it is called. For a raster, hash the file’s checksum (or its bytes); for a vector, hash the canonical WKB of the geometries plus the attribute values that matter. Fold in the source CRS, the target CRS, and every parameter that alters the output — resampling method, target resolution, snap grid, nodata. Two runs that produce byte-identical output must produce the same key; two runs that produce different output must not collide. This is what makes the cache safe to share across workers and across flow runs.

2. CRS and PROJ version are part of the input#

A coordinate transform is only reproducible for a fixed PROJ version and datum-shift grid set. PROJ 9.x can return transforms that differ from 8.x at the centimetre-to-metre level because the pipeline selection or the bundled grids changed. If your key ignores the PROJ version, a worker-image upgrade silently serves stale reprojected artifacts that no longer match a fresh warp. Read pyproj.proj_version_str and include it in the key so a PROJ bump cleanly invalidates every affected entry. The same logic applies to the GDAL version for raster resampling.

3. Persist artifacts to object storage, keyed by hash#

The cache value is the expensive artifact itself — a reprojected COG, a spatial-join result, a rasterized layer — stored at a path derived from the key, e.g. s3://cache/reproject/<hash>.tif. Because the key is content-addressed, the store is naturally deduplicated: identical inputs land on the same object. Persisting to object storage (rather than a worker’s local disk) means the cache survives scale-to-zero and spot reclaims, which is what makes it pay off under the elasticity patterns in cost optimization for spatial compute.

4. Invalidation follows the dependency, not a clock#

A spatial artifact is stale the moment any input it was derived from changes: the source raster is re-published, the resampling method changes, the target resolution changes, or PROJ/GDAL is upgraded. Because all of those are folded into a content-addressed key, invalidation is automatic — a changed input produces a different key and therefore a cache miss, without any manual purge. Time-based expiration (cache_expiration) is a coarse safety net for inputs you cannot hash cheaply (a live WFS feed), not the primary mechanism.

5. Framework hooks: Prefect cache_key_fn vs Dagster asset caching#

Prefect and Dagster expose caching differently. Prefect computes a key per task run via cache_key_fn(context, parameters) and short-circuits the run if the key already has a persisted result. Dagster caches at the asset level using code_version plus upstream data versions, re-materializing only when a version changes. Both reduce to the same discipline: make the key a pure function of content and parameters. Which you pick usually follows your broader Prefect versus Dagster for GIS workloads decision.

Here is how a raster’s checksum, CRS and parameters collapse into a single cache key and drive the hit/miss path:

Production Implementation#

The example below builds a content-addressed key for a spatial task and wires it into Prefect’s cache_key_fn. The key folds the source checksum, source and target CRS, resampling method, resolution, and the PROJ and GDAL versions — so any change that would alter the output produces a miss.

import hashlib
import json
from typing import Any, Callable

import rasterio
import pyproj
from osgeo import gdal
from prefect import task, flow
from prefect.context import TaskRunContext


def spatial_cache_key(context: TaskRunContext, parameters: dict[str, Any]) -> str:
    """Derive a content-addressed key from inputs + parameters + toolchain versions."""
    src_path: str = parameters["src_path"]
    # Read the source checksum cheaply — a stored band checksum, not a full re-hash.
    with rasterio.open(src_path) as src:
        checksum: int = src.checksum(1)          # GDAL per-band checksum
        source_crs: str = src.crs.to_string()    # never assume; read the real CRS

    key_material: dict[str, Any] = {
        "checksum": checksum,
        "source_crs": source_crs,
        "target_crs": parameters["target_epsg"],
        "resampling": parameters["resampling"],
        "resolution": parameters["resolution"],
        "nodata": parameters.get("nodata"),
        # Toolchain versions: a bump here must invalidate every derived artifact.
        "proj_version": pyproj.proj_version_str,
        "gdal_version": gdal.__version__,
    }
    blob: str = json.dumps(key_material, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()


@task(
    cache_key_fn=spatial_cache_key,
    cache_expiration=None,          # invalidation is content-driven, not time-driven
    persist_result=True,            # persist the artifact to the configured result store
)
def reproject_raster(
    src_path: str,
    target_epsg: int,
    resampling: str = "bilinear",
    resolution: float = 30.0,
    nodata: float | None = None,
) -> str:
    dst_path: str = f"s3://cache/reproject/{target_epsg}/{hashlib.sha256(src_path.encode()).hexdigest()}.tif"
    warp_opts = gdal.WarpOptions(
        dstSRS=f"EPSG:{target_epsg}",
        resampleAlg=resampling,
        xRes=resolution, yRes=resolution,
        dstNodata=nodata,               # never drop nodata through a warp
        format="COG",
    )
    gdal.Warp(dst_path, src_path, options=warp_opts)
    return dst_path


@flow
def reproject_batch(sources: list[str], target_epsg: int = 3857) -> list[str]:
    # Re-running with unchanged sources hits the cache: no warp, just a store read.
    return [reproject_raster.submit(s, target_epsg).result() for s in sources]

Step-by-Step Walkthrough#

1. Read the real content and CRS. spatial_cache_key opens the source and reads GDAL’s per-band checksum plus the actual src.crs — never a filename and never an assumed CRS. The checksum is cheap because GDAL stores it; you are not re-hashing the whole file on every lookup.
2. Fold in every output-affecting parameter. Target CRS, resampling algorithm, resolution and nodata all change the output pixels, so all of them enter key_material. Omitting any one causes silent collisions where different outputs share a key.
3. Fold in the toolchain versions. pyproj.proj_version_str and gdal.__version__ join the key so a worker-image upgrade that changes transform results cleanly invalidates the cache instead of serving stale artifacts.
4. Hash canonically. json.dumps(..., sort_keys=True) gives a deterministic serialization regardless of dict ordering, and SHA-256 turns it into the stable key. The same inputs always yield the same hex digest across workers.
5. Let Prefect short-circuit. With cache_key_fn set and persist_result=True, Prefect checks the key before running: on a hit it loads the persisted artifact and skips the warp entirely; on a miss it runs gdal.Warp and persists the result under the key.
6. Store by content. The output lands in object storage under a path derived from the input, so the cache survives scale-to-zero and is shared across the whole worker pool — not stranded on one worker’s local disk.

Edge Cases & Failure Recovery#

Filename reused for new content. A publisher overwrites scene.tif with fresh pixels under the same name. Detection: downstream outputs look stale despite a “new” source. Fix: key on the band checksum or object ETag, never the path — a content-addressed key produces a miss the moment the bytes change.

PROJ upgrade serves stale reprojections. The worker image bumps PROJ 8→9 and cached warps no longer match a fresh transform. Detection: a re-warped tile differs from the cached one at the sub-metre level. Fix: include pyproj.proj_version_str in the key (as above) so the bump invalidates every affected entry automatically. This is the caching analogue of the CRS checks in validating coordinate systems before ETL.

Resampling method silently changed. A refactor flips a default from nearest to bilinear; categorical land-cover values get interpolated into nonsense. Detection: class counts drift after a code change. Fix: the resampling parameter is in the key, so the change forces a recompute rather than reusing the wrong-algorithm artifact.

Cache write races between parallel workers. Two workers miss the same key simultaneously and both warp and write. Detection: duplicate compute and a possible torn object. Fix: write to a temporary key and atomically rename to the final key; the second writer’s rename is idempotent. Pair with idempotency keys in spatial ETL for the database-write equivalent.

Unbounded cache growth. Every parameter permutation persists forever and object-storage costs creep up. Detection: the cache bucket grows without bound. Fix: apply an object-storage lifecycle rule to expire entries not read in N days, and reserve cache_expiration for inputs you cannot hash cheaply.

Configuration Reference#

Frequently Asked Questions#