Running Async GeoPandas Tasks Safely

geopandas has no async API, and calling it inside a coroutine blocks the entire event loop for the duration of the call — every concurrent fetch, every timeout, every heartbeat stops. The safe pattern is to keep GeoPandas out of coroutines entirely: wrap each call in asyncio.to_thread when the operation releases the GIL and is short, or hand it to a ProcessPoolExecutor when it is long or CPU-bound. Then turn on asyncio debug mode in CI, so the next blocking call someone adds is caught by a test rather than by a throughput graph six months later.

When to Use This Pattern#

• An async pipeline needs to touch vector data — reading a GeoPackage, running an overlay, writing a result — which is the normal case once fetching is concurrent.
• Concurrency was raised and throughput did not move, the classic signature of a blocked loop.
• The flow mixes fast I/O with occasional heavy geometry work, so the loop must stay responsive between the heavy parts.
• Timeouts are firing that should not, because the loop was blocked and could not service them.

Complete Working Example#

The wrapper below makes the boundary explicit: anything named *_blocking is synchronous and must never be called directly from a coroutine, and the async helpers are the only sanctioned way across.

from __future__ import annotations

import asyncio
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path
from typing import Any

import geopandas as gpd


# ---- synchronous side: plain functions, no awaits, module level so they pickle ----

def read_layer_blocking(path: Path, layer: str = "features") -> gpd.GeoDataFrame:
    """Reads GIL-releasing C code, but for a big file it still takes seconds."""
    return gpd.read_file(path, layer=layer)


def overlay_blocking(left_path: Path, right_path: Path, out_path: Path,
                     how: str = "intersection") -> Path:
    """CPU-bound. Takes and returns PATHS so nothing large crosses a process boundary."""
    left = gpd.read_file(left_path)
    right = gpd.read_file(right_path)
    if left.crs != right.crs:
        # Never silently reproject inside a worker: it hides a contract violation
        # and doubles the cost of an operation that is already the expensive one.
        raise ValueError(f"CRS mismatch: {left.crs} vs {right.crs}")
    result = gpd.overlay(left, right, how=how, keep_geom_type=True)
    result.to_file(out_path, driver="GPKG", layer="features")
    return out_path


# ---- async side: the only sanctioned crossings ----

async def read_layer(path: Path, layer: str = "features") -> gpd.GeoDataFrame:
    """Short reads: a thread is enough, and it avoids pickling the frame back."""
    return await asyncio.to_thread(read_layer_blocking, path, layer)


async def overlay(pool: ProcessPoolExecutor, left: Path, right: Path, out: Path) -> Path:
    """Long CPU work: a separate process, so the GIL and the loop are both spared."""
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(pool, overlay_blocking, left, right, out)


async def main(jobs: list[tuple[Path, Path, Path]]) -> list[Path]:
    with ProcessPoolExecutor(max_workers=4) as pool:
        async with asyncio.TaskGroup() as group:
            tasks = [group.create_task(overlay(pool, a, b, out)) for a, b, out in jobs]
    return [t.result() for t in tasks]


if __name__ == "__main__":
    # debug=True logs any callback that holds the loop for more than 100 ms —
    # which is exactly the bug this whole file exists to prevent.
    asyncio.run(main(JOBS), debug=True)

The naming convention in that module is not decoration. read_layer_blocking and overlay_blocking are impossible to call accidentally from a coroutine without the suffix appearing in the diff, which turns a subtle runtime problem into something a reviewer notices while reading. The alternative — identical names on both sides of the boundary, distinguished only by whether they are awaited — is exactly the situation in which a blocking call gets added during a hurried fix and nobody sees it for months.

Parameter & Option Reference#

Verification & Testing#

The test that matters asserts the loop stayed responsive — not that the result was correct, which a synchronous test already covers.

import asyncio
import time

import pytest


@pytest.mark.asyncio
async def test_overlay_does_not_block_the_loop(tmp_path, big_layers) -> None:
    """A heartbeat coroutine must keep ticking while the overlay runs."""
    ticks: list[float] = []

    async def heartbeat() -> None:
        while True:
            ticks.append(time.monotonic())
            await asyncio.sleep(0.05)

    beat = asyncio.create_task(heartbeat())
    with ProcessPoolExecutor(max_workers=1) as pool:
        await overlay(pool, big_layers.left, big_layers.right, tmp_path / "out.gpkg")
    beat.cancel()

    gaps = [b - a for a, b in zip(ticks, ticks[1:])]
    # If the loop had been blocked, one gap would be the whole overlay duration.
    assert max(gaps) < 0.5, f"event loop stalled for {max(gaps):.1f}s"


@pytest.mark.asyncio
async def test_crs_mismatch_fails_loudly(tmp_path, mismatched_layers) -> None:
    with ProcessPoolExecutor(max_workers=1) as pool:
        with pytest.raises(ValueError, match="CRS mismatch"):
            await overlay(pool, mismatched_layers.a, mismatched_layers.b, tmp_path / "x.gpkg")

In a running pipeline the equivalent signal is the debug-mode warning, which names the offending callback and its duration:

WARNING asyncio Executing <Task ... coro=<process_tile() running at flows/tiles.py:88>>
        took 8.412 seconds

One line, the file, the line number and the duration — everything needed to find the blocking call. Leaving debug mode on in CI and off in production is the usual arrangement, since it adds measurable overhead but catches the regression before it ships.

It is worth being clear about what the throughput chart does not show, because the missing part is what makes this bug persistent. Latency per individual tile is unchanged between the two lines — each tile still takes the same time from start to finish. What changes is how many are in flight. That means every per-tile metric on a dashboard looks identical whether the loop is blocked or not, and only an aggregate throughput number reveals the problem. Teams that monitor p50 and p99 task duration exclusively can run a fully serialised async pipeline for a year without a single graph looking wrong.

Common Pitfalls#

• gpd.read_file directly in a coroutine. The most common instance, because reading feels lightweight. On a 2 GB GeoPackage it stalls the loop for tens of seconds, and every concurrent request’s timeout clock keeps running while nothing is being read.
• Wrapping in a thread and expecting parallelism for Python-level work. Threads help only where the GIL is released. A loop over geometries in Python gets no parallelism from to_thread, only the loop’s responsiveness back. Use processes for real CPU parallelism.
• Pickling GeoDataFrames to pool workers. The frame goes out and the result comes back, both serialised. For a large frame this is frequently slower than doing the work inline. Pass paths and let each worker read.
• Creating the pool inside the coroutine, per call. Process pools are expensive to start — hundreds of milliseconds, plus the import cost of GDAL in each worker. Create one pool for the flow and pass it down.
• Ignoring keep_geom_type. gpd.overlay can return mixed geometry types, which flows straight into a typed PostGIS column and fails there. Setting it at the operation is one keyword; discovering it at the load is an afternoon.

Frequently Asked Questions#

Related#

• Async execution for heavy GIS tasks — choosing the execution model per step
• Offloading GDAL work to a process pool — sizing and configuring the pool
• Building ETL chains for vector data — why paths cross boundaries and frames do not
• Repairing invalid geometries before load — another GeoPandas operation worth keeping off the loop

← Async Execution for Heavy GIS Tasks