Geospatial Orchestration Hub

Pinning Geospatial Dependencies With conda-lock

The geospatial stack is a graph of C libraries with shared ABI constraints, and Python wheels are a poor place to express it — several bundle their own copy of GDAL, and which copy wins depends on link order. Conda models the native packages as first-class, and conda-lock turns an environment specification into a per-platform lock file listing exact builds and hashes. Resolve once, commit the lock, and every worker, laptop and CI runner installs the same libraries rather than whatever resolved that morning.

When to Use This Pattern

  • The stack includes GDAL, PROJ, GEOS and Python bindings that must agree with each other.
  • Development happens on macOS and production runs on Linux, so a single-platform lock is not enough.
  • A transformation result must be reproducible across machines and across months.
  • Wheels have already caused a problem — two GDAL versions in one process is the usual symptom.

Complete Working Example

The specification is short and states only what the pipeline actually requires:

# environment.yml — intent, not resolution.
name: spatial-worker
channels:
  - conda-forge          # one channel: mixing defaults and conda-forge mixes ABIs
dependencies:
  - python=3.12
  - gdal=3.9.2           # pin the native stack exactly; it decides results
  - libgdal-core=3.9.2
  - proj=9.4.1
  - proj-data            # the transformation grids, as a package
  - geos=3.12.1
  - rasterio=1.3.10
  - fiona=1.9.6
  - shapely=2.0.4
  - pyproj=3.6.1
  - psycopg=3.2.1
  - prefect=3.0.*

conda-lock turns it into one lock file per platform, with builds and hashes:

# Resolve for every platform the team uses. Commit the result.
conda-lock lock \
  --file environment.yml \
  --platform linux-64 \
  --platform osx-arm64 \
  --kind explicit \
  --lockfile conda-lock.yml

# Render a per-platform explicit list, which installs with no solver at all.
conda-lock render --kind explicit --platform linux-64 conda-lock.yml

Installing from the lock is then deterministic and fast, because nothing is resolved:

FROM mambaorg/micromamba:1.5.8 AS deps
COPY --chown=$MAMBA_USER:$MAMBA_USER conda-linux-64.lock /tmp/env.lock
# --file with an explicit lock: no solver runs, so no version can drift.
RUN micromamba install --yes --name base --file /tmp/env.lock \
 && micromamba clean --all --yes

And the versions are asserted rather than assumed:

# tests/test_stack.py — runs in CI inside the built image.
import fiona, pyproj, rasterio, shapely

EXPECTED = {
    "gdal": "3.9.2",
    "proj": "9.4.1",
    "geos": "3.12.1",
}


def test_every_binding_reports_the_pinned_native_version() -> None:
    assert rasterio.__gdal_version__ == EXPECTED["gdal"]
    assert fiona.__gdal_version__ == EXPECTED["gdal"], "fiona links a different GDAL"
    assert pyproj.proj_version_str == EXPECTED["proj"]
    assert shapely.geos_version_string.startswith(EXPECTED["geos"])
Resolve once, install everywhereA single environment specification is resolved into a Linux lock file and a macOS lock file. Each is installed without running a solver, so no version can drift between machines.environment.ymlintentconda-linux-64.lockbuilds + hashesconda-osx-arm64.lockbuilds + hashesworkers and CIno solver runsdeveloper laptopsno solver runsThe solver runs once, in a pull request that can be reviewed, rather than on every machine at install time.
The lock files are artefacts of a decision, which is why they belong in version control rather than being regenerated on demand.

The fiona.__gdal_version__ assertion is not redundant with the rasterio one, and the reason is the whole argument for locking. Both packages link GDAL, and if one comes from conda and the other from a wheel, the process ends up with two copies loaded simultaneously — the same symbol names resolved to different implementations, with behaviour depending on which was imported first. It manifests as a driver available in one library and missing in the other, or a transformation that differs between the vector and raster halves of the same pipeline. Asserting that both report the same version is a two-line test for a class of bug that is otherwise very hard to see.

Parameter & Option Reference

Setting Value Spatial notes
Channels conda-forge only Mixing channels mixes ABI conventions; the geospatial stack is where that hurts first.
gdal pin exact version It decides transformation and resampling results, so it belongs at the same specificity as the code.
proj-data included The grids as a package; without them accuracy silently drops to a fallback method.
--kind explicit yes An explicit lock installs with no solver, which is faster and cannot drift.
Platforms every one used A Linux-only lock leaves developers resolving independently, which is where parity is lost.
pip section last resort Where a package has no conda build. Keep the list short and never put a native package there.
Re-lock cadence monthly, in a pull request A visible diff of what changed is the point; automatic re-locking discards it.
Two native stacks in one processRasterio installed from conda and fiona from a wheel each link their own GDAL. The two disagree about available drivers and about transformation results.one Python process, two GDALsrasterio, from condaGDAL 3.9.2FlatGeobuf: availablegrids: presentfiona, from a wheelGDAL 3.8.4FlatGeobuf: missinggrids: not bundledThe vector and raster halves of the same pipeline now transform coordinates differently.
Neither import fails, and the disagreement is only visible if something compares the two — which is what the version test does.

Verification & Testing

import subprocess
from pathlib import Path


def test_lockfiles_are_current() -> None:
    """A lock that has drifted from the spec is worse than no lock."""
    before = Path("conda-linux-64.lock").read_text()
    subprocess.run(["conda-lock", "lock", "--file", "environment.yml",
                    "--platform", "linux-64", "--check-input-hash"], check=True)
    assert Path("conda-linux-64.lock").read_text() == before, (
        "environment.yml changed without re-locking"
    )


def test_every_platform_has_a_lock() -> None:
    for platform in ("linux-64", "osx-arm64"):
        assert Path(f"conda-{platform}.lock").exists()


def test_no_native_package_installed_by_pip() -> None:
    out = subprocess.run(["pip", "list", "--format=freeze"],
                         capture_output=True, text=True).stdout
    for pkg in ("rasterio", "fiona", "GDAL", "pyproj", "shapely"):
        assert f"{pkg}==" not in out, f"{pkg} came from pip, not conda"


def test_transformation_matches_a_known_answer() -> None:
    import pyproj
    e, n = pyproj.Transformer.from_crs("EPSG:4326", "EPSG:27700").transform(
        55.9533, -3.1883)
    assert abs(e - 325863.108) < 0.01 and abs(n - 673795.836) < 0.01

The --check-input-hash test is what stops the lock file becoming decorative. conda-lock records a hash of the input specification, so re-running it when nothing has changed is a no-op and re-running it after an edit produces a diff — which means a pull request that adds a dependency to environment.yml and forgets to re-lock fails CI rather than shipping an environment where the specification and the installed packages disagree. That divergence is the commonest way a locked project quietly stops being locked.

Three ways to install the stackPip resolution is fast and not reproducible. A conda solve is slow and reproducible only if the channel is unchanged. An explicit lock is fast and fully reproducible.approachinstall timesame bits everywherepip, unpinned40 snoconda solve3 to 6 minonly if unchangedexplicit lock35 syesThe bottom row is faster than the middle one because no solver runs, which is also why it cannot drift.
Reproducibility and install speed usually trade against each other; here the same mechanism delivers both.

It is worth being explicit about the division of labour between this and the image, because they overlap and are not the same control. The lock file answers “which packages”, down to a build string and a hash; the image answers “which environment those packages run in” — the environment variables, the user, the grids if they are synced rather than packaged, the entry point. A locked environment installed into an unconfigured image is still a worker whose driver cache is unbounded and whose reads list a whole prefix. Both artefacts are needed, and confusing them leads to teams that lock meticulously and then wonder why two workers behave differently.

The other thing the lock does not cover is what the code asks the libraries to do. Pinning GDAL to 3.9.2 fixes the resampling implementations available; it does not fix which one a call site chose, and a change from bilinear to cubic in application code will change every output while every version assertion continues to pass. That is the correct division — one artefact for the stack, one for the intent — and it is why the canary comparison in rolling out GDAL upgrades diffs outputs rather than versions.

Common Pitfalls

  • Mixing conda and pip for native packages. Two GDALs in one process, disagreeing quietly.
  • Locking only for Linux. Developers then resolve their own environments and reproduce bugs the fleet does not have.
  • Mixing defaults and conda-forge. The ABI conventions differ and the geospatial stack is where the mismatch surfaces first.
  • Leaving proj-data out. Accuracy drops to a fallback transformation with no error anywhere.
  • Regenerating the lock in CI. It defeats the purpose: the lock is a reviewed decision, not a build artefact.
  • Pinning Python packages and not the native ones. The native versions decide the results; the bindings mostly decide the API.

Frequently Asked Questions

Is conda necessary, or will pip do?

Pip is workable when the wheels happen to agree — and increasingly they do, since rasterio and fiona now ship consistent GDAL builds. Conda is worth the weight when the stack is larger than that, when a specific GDAL version is required, or when grids need to be a managed dependency rather than a build step.

How large does the environment get?

A worker environment with GDAL, PROJ data and the usual bindings lands around 1.5 GiB before cleanup and under a gigabyte after micromamba clean. That is larger than a wheel-based image and smaller than most people expect.

What about `pixi` or `uv`?

Both are good and moving quickly, and the argument here is about locking rather than about the tool. Any mechanism that resolves the native stack once, per platform, with hashes, and installs without a solver satisfies the requirement.

How often should the lock be refreshed?

Monthly, deliberately, in a pull request whose diff shows what moved. Automatic weekly re-locking gives up the one thing the lock is for, which is that a change in the native stack is a visible event somebody approved.

Does the lock cover the transformation grids?

Through proj-data, yes, which is the tidiest arrangement because the grids then version alongside PROJ. The alternative — projsync in the image — gives finer control over which regions are included and is worth it when image size matters. See containerizing GDAL workers with Docker.

← Deployment & CI/CD for Spatial Workers

Related in this section