Async Execution for Heavy GIS Tasks

In short: asyncio is the right tool for the parts of a spatial pipeline that wait — fetching four hundred tiles, calling a geocoder, listing an object store — and the wrong tool for the parts that compute. geopandas, shapely and GDAL do their work in C, and a single overlay() call blocks the whole event loop for as long as it runs. The design question is not “async or not” but “where is the boundary between the waiting and the computing, and what carries work across it”.

Most spatial pipelines are a mixture: an I/O-heavy fetch phase followed by a CPU-heavy transform phase, repeated per partition. Running the whole thing synchronously wastes most of the fetch phase; running the whole thing asynchronously means the transform phase blocks every concurrent fetch and the pipeline is slower than the synchronous version. Getting real benefit means being explicit about which phase is which.

It is worth naming why this comes up so often in geospatial work specifically. Spatial pipelines sit at an awkward intersection: they are far more I/O-heavy than a typical analytics job, because the data lives in remote tiles, scenes and services rather than in one warehouse; and they are far more CPU-heavy than a typical web backend, because every byte fetched is then resampled, reprojected or intersected. Advice written for either world alone lands badly here, which is how pipelines end up either fetching four hundred tiles one at a time or running every reprojection on the event loop.

Prerequisites & Architecture Baseline#

• Python 3.11+, for asyncio.TaskGroup and asyncio.timeout()
• httpx ≥ 0.27 with AsyncClient, or aioboto3 for object storage
• A measured split of where the time goes: profile one partition and record the I/O and CPU shares before changing anything
• concurrent.futures.ProcessPoolExecutor available, and a worker whose memory can hold one process per pool slot
• Semaphores sized from the endpoint's limits, not from a round number
• Prefect ≥ 2.14 or Dagster ≥ 1.5 if the async flow is orchestrated, since both support async task bodies natively

Core Principles#

1. Async buys concurrency on waiting, never on computing. A hundred concurrent HTTP fetches on one thread is a large win because ninety-nine of them are idle at any moment. A hundred concurrent shapely.union calls on one thread is exactly as slow as doing them one after another, plus scheduling overhead. The event loop interleaves waiting, not work.

2. Any blocking call inside a coroutine blocks everything. This is the failure that produces the most confusion, because nothing errors: the pipeline simply runs at synchronous speed while looking asynchronous. A gpd.read_file() in the middle of an async function stalls every other in-flight request for its whole duration, and the symptom is a throughput graph that flatlines regardless of how high you set the concurrency.

3. Push CPU work to processes, not threads. The GIL means a thread pool gives no parallelism for pure-Python work. GDAL and shapely release the GIL during their C calls, so threads do help there — but unevenly, and a process pool is the predictable choice. run_in_executor with a ProcessPoolExecutor is the standard bridge, and offloading GDAL work to a process pool covers its details.

4. Bound concurrency per resource, not globally. One semaphore for “the pipeline” is meaningless when a task touches three services with three different limits. Give each endpoint its own semaphore sized from its published capacity, and let a task acquire all the ones it needs. This is the same idea that makes exponential backoff work: the limit belongs to the resource.

5. Geometry objects cross process boundaries badly. Pickling a GeoDataFrame to a worker process and back can cost more than the computation. Pass file paths and ranges rather than objects, and let each process read what it needs — the same discipline that makes ETL chains use files between steps.

6. Async complicates cancellation, so decide early. A cancelled coroutine raises CancelledError at its next await point, which is clean — unless it is blocked in a C call, in which case it is not cancellable at all. That interacts directly with timeout budgets and cancellation, and it is another argument for keeping heavy work out of process.

The middle row of that chart also explains a common and misleading benchmark result. Teams frequently test an async rewrite on a small extent, where the tiles are few and the transforms are quick, and see a large speed-up — because at that size the pipeline really is dominated by network round trips. Scaled up to a national run, the transform time grows with the data while the fetch time grows more slowly, the ratio inverts, and the same code delivers a few per cent. Benchmark the rewrite on the largest partition you actually run, not on the one that fits in a test fixture.

Production Implementation#

The flow below fetches concurrently with a per-endpoint semaphore, then hands the CPU-bound work to a process pool. The two phases are visibly separate, which is the point.

from __future__ import annotations

import asyncio
from concurrent.futures import ProcessPoolExecutor
from dataclasses import dataclass
from pathlib import Path
from typing import Sequence

import httpx


@dataclass(frozen=True)
class TileJob:
    tile_id: str
    source_url: str
    scratch: Path


# One semaphore per endpoint, sized from that endpoint's published capacity.
LIMITS = {
    "tiles.example.org": asyncio.Semaphore(16),
    "wfs.example.org": asyncio.Semaphore(2),
}


async def fetch(client: httpx.AsyncClient, job: TileJob) -> Path:
    """I/O only. Nothing here touches shapely, geopandas or GDAL."""
    host = httpx.URL(job.source_url).host
    async with LIMITS[host]:
        response = await client.get(job.source_url, timeout=httpx.Timeout(10.0, read=120.0))
        response.raise_for_status()
    raw = job.scratch / f"{job.tile_id}.tif"
    # Write with a thread, not inline: a large file write blocks the loop too.
    await asyncio.to_thread(raw.write_bytes, response.content)
    return raw


def transform(raw_path: Path, dst_crs: str) -> Path:
    """CPU-bound, and deliberately a plain function: it runs in another process.

    It takes and returns PATHS. Pickling a GeoDataFrame across the process
    boundary routinely costs more than the computation it was meant to speed up.
    """
    import rasterio
    from rasterio.warp import calculate_default_transform, reproject, Resampling

    out_path = raw_path.with_suffix(".proj.tif")
    with rasterio.open(raw_path) as src:
        transform_, width, height = calculate_default_transform(
            src.crs, dst_crs, src.width, src.height, *src.bounds
        )
        profile = src.profile | {
            "crs": dst_crs, "transform": transform_,
            "width": width, "height": height,
            "nodata": src.nodata if src.nodata is not None else -9999,
        }
        with rasterio.open(out_path, "w", **profile) as dst:
            for band in range(1, src.count + 1):
                reproject(
                    source=rasterio.band(src, band),
                    destination=rasterio.band(dst, band),
                    resampling=Resampling.bilinear,
                )
    return out_path


async def run(jobs: Sequence[TileJob], dst_crs: str = "EPSG:3857",
              pool_size: int = 4) -> list[Path]:
    loop = asyncio.get_running_loop()
    results: list[Path] = []

    with ProcessPoolExecutor(max_workers=pool_size) as pool:
        async with httpx.AsyncClient() as client:
            async with asyncio.TaskGroup() as group:
                fetch_tasks = [group.create_task(fetch(client, job)) for job in jobs]

            # Transforms are submitted to the pool; awaiting them yields the loop,
            # so further fetches (if any) keep making progress.
            async with asyncio.TaskGroup() as group:
                transform_tasks = [
                    group.create_task(loop.run_in_executor(pool, transform, task.result(), dst_crs))
                    for task in fetch_tasks
                ]
            results = [task.result() for task in transform_tasks]

    return results

Step-by-Step Walkthrough#

1. Separate the phases before optimising either. The fetch coroutine touches no geospatial library, and transform touches no network. That separation is what makes it possible to reason about which one benefits from which kind of concurrency.
2. Give each host its own semaphore. LIMITS is keyed by hostname because the capacity being protected belongs to the host. A single global semaphore either starves the fast endpoint or overruns the slow one.
3. Write files with asyncio.to_thread. A 200 MB write_bytes on the event loop blocks every other coroutine for its duration. Moving it to a thread is one line and removes a stall that is otherwise almost impossible to see.
4. Make the CPU-bound function a plain, importable function. ProcessPoolExecutor pickles the callable by reference, so a closure or a lambda fails at submission with a confusing error. Module-level functions work; nested ones do not.
5. Pass paths across the process boundary. The transform takes a Path and returns a Path. Nothing large is pickled, so the boundary costs microseconds instead of seconds.
6. Size the pool from memory, not from cores. Each process holds its own GDAL buffers. Four processes each peaking at 3 GB will not fit on a 8 GB worker regardless of how many cores it has.
7. Use TaskGroup rather than gather. On failure, a TaskGroup cancels its siblings and raises an ExceptionGroup that names every failure, which is far more useful than gather’s first-exception-wins behaviour when eight tiles fail for three different reasons.

Edge Cases & Failure Recovery#

A blocking call that nobody notices. The classic symptom is that raising concurrency from 8 to 64 changes nothing. The cause is almost always a synchronous call inside a coroutine — requests instead of httpx, open().read() on a large file, or a geopandas call that slipped into the fetch path. asyncio.run(..., debug=True) logs any callback that occupies the loop for more than 100 ms, which finds these in one run.

Process pool workers that inherit a broken state. GDAL configuration set with gdal.SetConfigOption in the parent is not reliably inherited by pool workers, and neither are environment variables set after the pool was created. Configure inside the worker function, or create the pool after all configuration is done.

Memory multiplied by pool size. A transform that peaks at 3 GB is fine alone and fatal at four concurrent copies. This is the most common way an async rewrite makes a pipeline less reliable: throughput rises, memory rises with it, and the OOM killer arrives. Size the pool by measuring one transform’s peak and dividing available memory by it.

Cancellation that does not reach the pool. Cancelling the coroutine awaiting run_in_executor does not stop the worker process; it only stops waiting for it. A long transform continues to completion. For work that must be cancellable, run it as a subprocess whose process group you control rather than in a pool — the reasoning in cancelling in-flight tile jobs cleanly.

Exceptions that arrive as a group. With TaskGroup, eight failures raise one ExceptionGroup. Code that catches a specific exception type will miss it unless it uses except* or unwraps the group. This is a small syntax change that is easy to forget and produces a confusing “unhandled exception” when the handler was right there.

Fetches that outrun the transforms. Fetching 400 tiles concurrently while the pool processes 4 at a time fills scratch storage with 396 waiting files. Bound the fetch phase by the pool’s appetite — a queue with a maximum size, or simply chunking the job list — so disk usage stays proportional to concurrency rather than to the batch.

A semaphore acquired inside the pool. Semaphores belong to one event loop in one process; a ProcessPoolExecutor worker has neither. Code that tries to rate-limit from inside a pool worker either fails outright or, worse, creates a fresh unshared semaphore per process and silently multiplies the concurrency by the pool size. Keep all rate limiting on the async side of the boundary, where the semaphore actually exists.

Logging that interleaves unreadably. Eight concurrent coroutines writing to one logger produce a stream in which no two consecutive lines belong to the same tile. Attach the tile id to every record — a logging.LoggerAdapter or a contextvar set at the top of each coroutine — or the logs become unusable exactly when concurrency is highest and you most need them. Structured logging for geospatial flows covers the shape.

Configuration Reference#

That last row is the one to act on first. If profiling shows a partition spends 15% of its time waiting and 85% computing, an async rewrite has at most 15% to give and will cost days — the same effort spent on the process pool, or on making the computation cheaper, returns far more. Async is worth it when the ratio runs the other way, which for tile-fetching and API-heavy pipelines it very often does. Measure the split once, write it in the flow’s docstring, and let the number rather than the fashion decide.

Frequently Asked Questions#

Related#

• Running async geopandas tasks safely — the blocking-call trap in detail
• Offloading GDAL work to a process pool — sizing and wiring the pool
• Building ETL chains for vector data — why paths, not objects, cross boundaries
• Timeout budgets and cancellation for geotasks — what can and cannot be cancelled
• Right-sizing workers for raster mosaic jobs — the memory arithmetic behind the pool size

← Spatial Task Design & Dependency Mapping