Cost Optimization for Spatial Compute

Cost Optimization for Spatial Compute means matching cloud instances, worker pools and I/O patterns to how raster and vector jobs actually consume memory, CPU and network — so you stop paying for provisioned RAM that a mosaic touches for ninety seconds and never again. The single biggest lever is recognizing that raster peak memory is non-linear in tile count, then choosing spot workers, windowed reads and scale-to-zero pools around that fact.

Most geospatial cloud bills are not driven by compute hours at all — they are driven by over-provisioning for the worst-case tile and by egress pulling Cloud-Optimized GeoTIFFs and tiles back out of object storage. A worker sized for the largest mosaic in the batch sits mostly idle; a task that reads a full scene into RAM forces a memory tier three sizes larger than a windowed reader needs. This guide walks through the four cost centers that matter for orchestrated pipelines — instance selection, worker-pool elasticity, memory-bounded I/O, and data movement — and shows where Prefect and Dagster give you the hooks to control each. For the mechanical sizing calculation, the companion right-sizing workers for raster mosaic jobs walks through the byte arithmetic in detail.

Prerequisites & Architecture Baseline#

Before tuning spend, confirm your stack exposes the levers this guide uses. The versions below are the ones the code was validated against.

• rasterio>=1.3.9 (GDAL 3.6+) — windowed reads and block-aligned I/O
• gdal>=3.6 with COG driver — for Cloud-Optimized GeoTIFF range reads
• prefect>=2.19 or dagster>=1.7 — for work-pool concurrency and scale-to-zero
• Workers run in a pool you can scale independently of the scheduler (Kubernetes, ECS, or a process pool with a queue)
• Object storage (s3://, gs://, az://) holds source rasters, and you can read egress metering per bucket
• A metrics sink (Prometheus or the orchestrator's own run metadata) records peak RSS per task
• Retry-safe task design: any task that a spot reclaim can kill mid-run is idempotent (see the note on preemption below)

If any box is unchecked, fix that first — cost tuning on a stack where workers cannot scale to zero, or where tasks are not retry-safe, produces fragile savings that evaporate on the first reclaim.

Core Principles#

Six principles govern spatial compute cost. Each maps to a concrete decision you make in code or work-pool config.

1. Raster peak memory is non-linear in mosaic footprint#

A mosaic’s peak resident memory is not the sum of its input file sizes on disk — GeoTIFFs are compressed on disk and decompressed in RAM. Peak memory tracks width × height × bands × dtype_size of the decoded array plus any intermediate warp buffers. Doubling the linear extent of a mosaic quadruples pixel count and therefore quadruples peak RAM. This is why a naive “read everything, then write” mosaic explodes: a 40000×40000 three-band uint16 array is 9.6 GB before you allocate a single output buffer. The instance tier you need is set by this peak, not by the average.

2. Instance tier follows the memory ceiling, not the CPU count#

Spatial compute is usually memory-bound, not CPU-bound. gdalwarp and rasterio.merge will saturate a single core long before they saturate eight, but they will happily allocate tens of gigabytes. Pick the instance by dividing peak array bytes by a safe utilization fraction (0.6–0.7, leaving headroom for GDAL block cache and the OS), then take the smallest tier that clears it. Over-CPU’d, under-RAM’d instances (compute-optimized c-family) are a common and expensive mismatch for mosaic work; memory-optimized (r-family) or general-purpose (m-family) tiers fit better.

3. Windowed and block-aligned reads cap memory independent of scene size#

The decisive cost move is refusing to hold a whole scene in RAM. Reading a raster in rasterio windows aligned to the file’s internal block/tile size means peak memory is a function of the window, not the scene. A 40000×40000 scene processed in 2048×2048 windows never allocates more than a few hundred megabytes for the read buffer — collapsing the required instance tier from r6i.4xlarge to m6i.large. This directly enables the earlier scale-to-zero and spot strategies, because small workers are cheap and abundant. The right-sizing companion gives the window-size math.

4. Spot/preemptible workers are free money for retry-safe geotasks#

Batch raster and vector jobs are the ideal spot workload: they are throughput-oriented, not latency-sensitive, and a reclaimed task simply reruns. Spot and preemptible instances run 60–90% cheaper than on-demand. The one hard requirement is idempotency — a task killed mid-write must not leave a half-written COG or a partially committed PostGIS batch. Pair spot workers with the idempotency patterns from idempotency keys in spatial ETL and with checkpointing so a reclaim resumes rather than restarts the whole mosaic — see checkpointing large raster mosaics.

5. Right-size the pool, then scale it to zero between runs#

A worker pool sized for peak concurrency during a nightly batch should not be running at 3 p.m. when nothing is queued. Orchestrator work pools let you cap concurrency (so you never spin up more workers than the object store’s egress budget tolerates) and drain to zero when the queue empties. The savings are the entire idle window — often 20+ hours a day for a batch pipeline. This is the difference between paying for a steady-state fleet and paying only for the hours a job actually runs.

6. Egress is a first-class cost, not an afterthought#

Pulling COGs and tiles out of object storage across a region or internet boundary is metered per gigabyte and often dwarfs compute. Three moves cut it: keep workers in the same region as the bucket (same-region egress is usually free); use COG range reads so you fetch only the byte ranges a window needs instead of whole files; and cache reprojected or resampled outputs so a re-run reads the cache rather than re-fetching and re-warping — the pattern in caching strategies for spatial tasks.

The relationship between mosaic size, peak memory and instance tier is worth visualizing, because it is where sizing decisions go wrong:

Production Implementation#

The following Prefect flow ties the principles together: a memory-bounded windowed mosaic task, deployed to a spot work pool with capped concurrency that drains to zero. The task estimates its own peak memory before running so an oversized job fails fast rather than OOM-killing an undersized worker.

import math
from pathlib import Path
from typing import Iterator

import numpy as np
import rasterio
from rasterio.windows import Window
from rasterio.io import DatasetReader
from prefect import flow, task, get_run_logger
from prefect.task_runners import ThreadPoolTaskRunner

# Safe fraction of instance RAM we let a single task's read buffer occupy.
MEMORY_UTILIZATION: float = 0.65


def estimate_peak_bytes(width: int, height: int, bands: int, dtype: str) -> int:
    """Decoded array size for one full read — the number that sets the tier."""
    dtype_size: int = np.dtype(dtype).itemsize
    return width * height * bands * dtype_size


def block_windows(src: DatasetReader, block: int = 2048) -> Iterator[Window]:
    """Yield block-aligned windows so peak memory is bounded by `block`, not scene size."""
    for row_off in range(0, src.height, block):
        for col_off in range(0, src.width, block):
            w: int = min(block, src.width - col_off)
            h: int = min(block, src.height - row_off)
            yield Window(col_off, row_off, w, h)


@task(retries=3, retry_delay_seconds=10)  # retry-safe: reclaim just reruns
def mosaic_windowed(src_path: str, dst_path: str, instance_ram_gb: float) -> str:
    logger = get_run_logger()
    # Read only band count / dtype metadata first — never open the pixels yet.
    with rasterio.open(src_path) as src:
        peak = estimate_peak_bytes(src.width, src.height, src.count, src.dtypes[0])
        budget = int(instance_ram_gb * 1e9 * MEMORY_UTILIZATION)
        if peak > budget:
            logger.info(
                "Full-scene read of %.1f GB exceeds %.1f GB budget; using windows.",
                peak / 1e9, budget / 1e9,
            )
        profile = src.profile.copy()
        # COG output: tiled + compressed, so downstream reads are range-friendly.
        profile.update(driver="GTiff", tiled=True, blockxsize=512,
                       blockysize=512, compress="deflate", nodata=src.nodata)
        with rasterio.open(dst_path, "w", **profile) as dst:
            for window in block_windows(src):
                data: np.ndarray = src.read(window=window)  # bounded by block size
                dst.write(data, window=window)
    return dst_path


@flow(task_runner=ThreadPoolTaskRunner(max_workers=4))
def build_mosaic(sources: list[str], out_dir: str, instance_ram_gb: float = 8.0) -> list[str]:
    outputs: list[str] = []
    for src in sources:
        dst = str(Path(out_dir) / (Path(src).stem + "_cog.tif"))
        outputs.append(mosaic_windowed.submit(src, dst, instance_ram_gb))
    return [o.result() for o in outputs]

Deploy this to a work pool whose concurrency is capped (so egress stays inside budget) and which scales to zero when idle:

# prefect.yaml — spot work pool, capped concurrency, scale-to-zero
deployments:
  - name: nightly-mosaic
    entrypoint: flows/mosaic.py:build_mosaic
    work_pool:
      name: spot-raster-pool
      job_variables:
        # Kubernetes worker; nodeSelector targets a spot node group.
        node_selector:
          eks.amazonaws.com/capacityType: SPOT
        # 8 GB request matches the windowed memory ceiling, not the scene size.
        memory: "8Gi"
        cpu: "2"
    schedule:
      cron: "0 2 * * *"   # 2 a.m. batch; pool idles the rest of the day
    concurrency_limit: 6  # never fan out past the egress budget

Step-by-Step Walkthrough#

1. Estimate before you allocate. estimate_peak_bytes computes the decoded size of a full read from metadata only — width × height × bands × dtype_size. This is the number that would set the instance tier if you read the whole scene, and it is computed before any pixel is touched, so an oversized job is visible in logs immediately.
2. Compare against the memory budget. The task multiplies the instance RAM by MEMORY_UTILIZATION (0.65) to leave headroom for GDAL’s block cache and the OS. If the full-scene read exceeds that budget, the log flags it — but the windowed path handles it regardless.
3. Read in block-aligned windows. block_windows yields 2048×2048 tiles aligned to the source. Because src.read(window=...) only decodes the requested window, peak memory is bounded by the window, not the scene — this is what lets an 8 GB m6i.large process a scene that a full read would demand 128 GB for.
4. Write a COG output. The output profile sets tiled=True with 512×512 blocks and deflate compression, and preserves nodata. Tiled output means downstream tasks can do range reads instead of full-file fetches, cutting egress on every subsequent read.
5. Cap concurrency and target spot. The deployment pins the worker to a spot node group and caps concurrency at 6, so you never fan out past what your object store’s egress budget tolerates. The retries=3 on the task makes spot reclaims transparent — a killed task reruns on the next available spot node.
6. Scale to zero between runs. The cron schedule runs the pool at 2 a.m.; the rest of the day the pool has no queued work and drains to zero, so you pay for roughly two hours of compute instead of twenty-four.

Edge Cases & Failure Recovery#

Spot reclaim mid-write leaves a truncated COG. If a task is killed while writing, the partial .tif on object storage is corrupt. Detection: a downstream rasterio.open raises RasterioIOError or gdalinfo reports an unexpected size. Fix: write to a temporary key and atomically rename/copy to the final key only after the writer context closes, so readers never see a half-written object. Combine with the idempotency-key pattern so a rerun overwrites cleanly.

Windowed reads thrash when the window is not block-aligned. If your window size does not match the file’s internal block size, GDAL re-reads overlapping blocks and both I/O and egress balloon. Detection: egress-per-scene far exceeds the scene’s on-disk size. Fix: query src.block_shapes and align window dimensions to a multiple of the internal block size before iterating.

Compute-optimized instance OOM-kills the mosaic. A c-family instance chosen for its core count has too little RAM for even a modest full-scene read. Detection: the worker is killed with exit code 137 (OOM) despite low CPU utilization. Fix: switch to m- or r-family and size by the memory ceiling from the right-sizing companion, not by CPU.

Cross-region egress silently doubles the bill. Workers in us-west-2 reading a bucket in us-east-1 pay per-gigabyte egress on every scene. Detection: egress charges appear on the bucket even for internal reads. Fix: colocate the work pool and bucket in one region; if a multi-region read is unavoidable, cache the reprojected result close to the workers using caching strategies for spatial tasks.

Scale-to-zero cold starts stall latency-sensitive branches. A pool that scaled to zero takes minutes to pull a GDAL image and register a worker. Detection: the first task of a run waits on worker provisioning. Fix: keep a single warm worker for interactive or SLA-bound flows and reserve scale-to-zero for the nightly batch pool; route work with the conditional patterns in conditional branching in geospatial DAGs.

Configuration Reference#

Frequently Asked Questions#