Cost Optimization for Spatial Compute
In short: a spatial pipeline’s bill is usually three things — memory headroom nobody measured, egress nobody looked at, and rebuilds of work that did not change. None of them is fixed by a cheaper instance type. Measure the per-unit peak, read only the bytes you need, and make “unchanged” mean “not rebuilt”, and the same pipeline costs a fraction of what it did without going any slower.
Geospatial compute has an unusual cost profile. The work is bursty, memory-hungry in short spikes, and reads from object storage in patterns that are either very efficient or catastrophically wasteful depending on details invisible in the code. That combination punishes the standard optimisation instinct — pick a cheaper instance — because the instance was never the expensive part.
What is expensive is the gap between what a worker is provisioned for and what it uses. A fleet sized for the largest scene runs at fifteen per cent memory utilisation for the other ninety-nine per cent of the time, and that idle headroom is billed at full price all night. Beside it sits egress: a pipeline that reads whole GeoTIFFs to extract a window can move a hundred times the bytes it needs, and object storage charges for every one of them. And underneath both is the rebuild problem, which is the largest and the least visible, because a pipeline that rebuilds everything nightly looks exactly like one that rebuilds what changed.
It is worth naming the reason these three go unnoticed for so long. All of them are invisible in the only place most teams look, which is whether the pipeline finished on time. Idle headroom makes a run more reliable, not less; whole-file reads make it slower but not wrong; and rebuilding everything nightly is the most robust behaviour available. Each is what a careful engineer would choose when uncertain, and each is charged for every night thereafter. Cost optimisation in a spatial pipeline is largely the work of replacing three reasonable defaults with three measurements.
Prerequisites & Architecture Baseline
Core Principles
1. Size from the measured peak, not from the largest possible input. The peak is a product of per-unit memory and concurrency, both of which you control. The largest possible input is a property of the world.
2. Bytes read is a bigger lever than instance price. A range read against a cloud-optimised GeoTIFF moves the window you asked for. A naive read moves the file.
3. The cheapest work is the work that is skipped. A content-keyed pipeline that rebuilds one and a half per cent of its tiles on a normal night is sixty times cheaper than one that rebuilds all of them, and produces the same product.
4. Interruptible capacity is nearly free if the job can be resumed. Spot pricing is a discount for accepting interruption, and checkpointing is what converts that discount into savings rather than into rework.
5. Idle is the default failure mode. A fan-out bounded by a source limit leaves workers waiting; scaling the fleet to the fan-out rather than to the limit pays for idleness at full rate.
6. Attribute before optimising. A bill without tags is a single number, and every conversation about it is speculation.
Production Implementation
Instrumenting for cost is mostly instrumenting for memory and bytes, and both are cheap to add.
from __future__ import annotations
import resource
import time
from prefect import task, get_run_logger
from prometheus_client import Histogram
TILE_PEAK_MB = Histogram(
"tile_peak_memory_mb", "Peak RSS per tile task",
buckets=(64, 128, 256, 512, 1024, 2048, 4096, 8192),
labelnames=("layer",),
)
TILE_BYTES_READ = Histogram(
"tile_source_bytes_read", "Bytes fetched from the source per tile",
buckets=(1e5, 1e6, 1e7, 1e8, 1e9), labelnames=("layer",),
)
@task(retries=2, timeout_seconds=900, tags=["tile"])
def build_tile(tile: Tile, layer: str, src_uri: str, dest: str) -> str:
before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
t0 = time.perf_counter()
# Window read, not whole-file read: the single largest egress decision.
with rasterio.open(src_uri) as src:
window = src.window(*tile.bounds_in(src.crs))
data = src.read(window=window, boundless=True)
bytes_read = data.nbytes
write_tile(warp(data, tile), dest)
peak_mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
TILE_PEAK_MB.labels(layer=layer).observe(peak_mb)
TILE_BYTES_READ.labels(layer=layer).observe(bytes_read)
get_run_logger().info("tile %s: %.0f MB peak, %.1f MB read, %.1f s",
tile.path, peak_mb, bytes_read / 1e6, time.perf_counter() - t0)
return dest
With the histogram in place, the fleet size follows from arithmetic rather than from caution:
def worker_memory_gb(p99_peak_mb: float, concurrency: int, headroom: float = 1.3) -> float:
"""Size the worker from the measured p99, not from the worst case ever seen."""
return (p99_peak_mb * concurrency * headroom) / 1024
The ru_maxrss measurement has a subtlety worth knowing, because getting it wrong produces confident nonsense. It reports the peak for the whole process since it started, not for the current task, so in a worker that runs tasks in the same process the value only ever rises. Taking a difference, as the example does, gives the marginal peak, which is what you want for sizing; taking the absolute value gives the high-water mark of the largest task the worker has ever run, which will size the fleet for the worst case forever. Where tasks run in separate processes the absolute value is correct, which is one reason a process pool makes this measurement easier to trust.
The bytes-read histogram earns its place for a different reason: it is the only cheap way to detect a reader that has quietly stopped doing window reads. Reading a window from a cloud-optimised GeoTIFF over HTTP depends on a chain of conditions — the file has internal tiling, the server honours range requests, the environment permits partial reads, the library is configured to use them — and any link breaking degrades silently to a full-file read. Nothing errors. The tile is still correct. The only observable symptom is that the pipeline moves a hundred times more data than it did last month, which is invisible in a log and obvious in a histogram.
The same instrumentation answers the question that always follows a bill: which layer is expensive? Labelling both histograms by layer costs one dimension and turns “compute went up” into “the orthophoto layer’s p99 memory doubled after the source moved to sixteen-bit”. That is a sentence somebody can act on, and it is not reconstructible after the fact from anything else the pipeline records.
Step-by-Step Walkthrough
- Tag everything. Fleet, bucket, prefix, by pipeline. Until the bill is attributable, every optimisation is a guess.
- Instrument peak memory and bytes read per unit. One run produces both distributions.
- Resize from the p99, not the maximum. Handle the tail with a separate larger work pool rather than by inflating the whole fleet.
- Convert sources to cloud-optimised formats or read them with range requests. This is usually the single largest saving.
- Make unchanged work skippable with content-derived keys, and log the skip rate every run.
- Move interruptible work to spot capacity once checkpointing is in place, and not before.
Step three deserves an expansion, because the phrase “route the tail” hides the interesting part. The routing decision has to be made before the memory is allocated, which means it has to come from metadata rather than from experience: a header read gives width, height, band count and data type, and their product is a good enough estimate of the peak to decide which pool a unit belongs in. That estimate costs a single range request against the source and is available before any pixels are touched. Pipelines that instead discover the size by running out of memory pay for the discovery twice — once in the failed attempt and once in the retry — and usually respond by enlarging the whole fleet.
Step five has a similar hidden requirement. “Make unchanged work skippable” is only meaningful if the pipeline can tell what changed, and for most sources that means computing a digest rather than trusting a timestamp. A publisher that rewrites every file nightly with identical content — which is extremely common, because their export pipeline has the same rebuild problem — will defeat any timestamp-based skip and none of the savings will materialise. Digesting the content is what makes the skip real, and it is the step most often left out when the numbers disappoint.
Edge Cases & Failure Recovery
One scene is ten times larger than the rest. Do not size the fleet for it. Route it by a size guard to a larger work pool, so the ninety-nine per cent case runs on cheap workers and the outlier runs on an expensive one for twenty minutes.
The source is not cloud-optimised. Converting once and caching the result is almost always cheaper than reading the original repeatedly, and the conversion is itself a pipeline that can run on interruptible capacity.
Egress within the same region is free — and the pipeline is not in that region. Co-locating workers with the data is a configuration change with a large and immediate effect, and it is the first thing to check when egress dominates.
Spot reclamation happens mid-mosaic. With checkpoints this costs the current block. Without them it costs the run, and the savings are erased by the third reclamation.
A skip rate that suddenly drops to zero. Something has entered the work key that changes every run — a timestamp, an ephemeral path — and the pipeline is now rebuilding everything while appearing to be incremental. This is worth alerting on.
The deadline is soft and nobody said so. This is the most valuable edge case to discover, because slack is what every cheap option is bought with. Interruptible capacity, deferred rebuilds, off-peak scheduling and smaller fleets all trade time for money, and a product whose real deadline is “before the analysts arrive” rather than “by 04:00” has hours of slack nobody has spent. Asking the question explicitly, once, tends to be worth more than any individual optimisation.
Two pipelines share a fleet and one of them is the expensive one. Without per-pipeline tags the shared bill is attributed by guesswork, usually to whichever pipeline is most visible rather than whichever is most costly. Separate work pools cost nothing to create and make the attribution automatic.
Configuration Reference
| Lever | Typical saving | Spatial notes |
|---|---|---|
| Size from p99 + route the tail | 40–60% of compute | Needs a per-unit memory histogram and a size guard on the header. |
| Range reads / cloud-optimised sources | up to 90% of egress | The largest single lever when sources are large and windows are small. |
| Content-keyed skipping | 90%+ of a nightly rebuild | Requires a ledger; see state management. |
| Interruptible capacity | 60–80% of instance price | Only with checkpointing, and only for work with slack in its deadline. |
| Co-location with the data | all cross-region egress | A configuration change, and often the fastest win available. |
| Overview levels built once | proportional to pyramid depth | Overviews change only when their children do; keying them separately stops needless rebuilds. |
Frequently Asked Questions
Who should own this?
The team that runs the pipeline, reviewed by whoever receives the bill. Cost work delegated entirely to a platform team tends to produce instance-level changes, because that is the layer a platform team can see; the three levers here all live in the pipeline’s own code and are invisible from outside it.
Where should I start?
With attribution, then with the skip rate. A pipeline rebuilding everything nightly has a saving available that dwarfs every other lever, and finding out takes one query against the ledger: how many units were rebuilt, and how many of those had a changed source.
Do these levers ever conflict with each other?
Occasionally, and the conflict is worth anticipating. Interruptible capacity works best with small units, because a reclamation loses at most one; right-sizing works best with large units, because per-task overhead is amortised. Batching for orchestration overhead pushes one way and checkpoint granularity pushes the other. The resolution is usually to keep the orchestrated unit large and the checkpoint unit small — a task that builds two hundred tiles and writes a checkpoint after each — which satisfies both without compromise.
Is a cheaper instance family never the answer?
Occasionally — ARM instances run GDAL well and cost meaningfully less, and it is a genuine saving. It is just much smaller than the three levers above, and it should not be the first thing tried because it changes the runtime environment for everything at once.
How do I know whether egress is a problem?
Compare bytes read to bytes needed. The histogram in the example gives the first; the second is the window size times the tile count. A ratio above about three says the reads are not being served as windows, and the fix is the format or the reader, not the network.
Does compression help?
For storage and egress, yes, and for compute it depends: heavier codecs trade CPU for bytes. For tiles served directly to clients the codec is a product decision rather than a cost one. For intermediates, a fast codec is nearly always right because they are written once and read once.
Should overviews be rebuilt with their tiles?
No — key them separately, on the children they actually consume. A pipeline that rebuilds the whole pyramid because one z14 tile changed is doing about a third more work than it needs to, and the fix is one extra key rather than any new machinery.
How often should the sizing be revisited?
Whenever a source changes shape, and otherwise quarterly. Sources gain bands, change bit depth and grow resolution without announcement, and each of those moves the memory distribution. A p99 that has drifted upward for a month is a fleet running closer to its limit than anyone realises, which is a reliability problem before it is a cost one.
Is it worth optimising a pipeline that costs very little?
Usually not in money, and sometimes in habit. The levers here — measuring the peak, reading windows, keying by content — are the same practices that make a pipeline scale and recover well, so applying them early on a cheap pipeline is mostly free and the alternative is discovering all three at once when it stops being cheap.
What about the cost of the control plane itself?
Small in money and occasionally large in overhead per task; see benchmarking orchestrator overhead for small geotasks. It becomes a cost question only when the units are so small that bookkeeping outweighs work, and the fix there is a coarser unit.
Related
- Right-sizing workers for raster mosaic jobs — the memory lever in detail
- Using spot instances for interruptible raster jobs — the price lever, and its prerequisite
- Cutting egress costs with COG range reads — the bytes lever
- State management in geospatial flows — what makes skipping possible
- Caching reprojected rasters with content hashing — the same idea applied to intermediates
- Measuring tile generation latency percentiles — the histograms this borrows