Spatial Validation & Sync Tasks

In short: validation belongs at the boundary where untrusted data enters, done once, with the result recorded as a guarantee the rest of the pipeline can rely on. Everything downstream then trusts the contract instead of re-checking, which is both faster and — more importantly — makes it obvious where a violated assumption came from.

Spatial data arrives wrong in a small number of well-known ways: a projection that is declared but not true, geometry that violates the simple-features rules, coordinates outside the CRS’s valid area, and attribute encodings that survive every geometric check. None of these are exotic, and all of them are cheap to detect at the door and expensive to detect three joins later. The discipline is not “validate more”; it is “validate once, at the right place, and record what you found”.

There is a second half to this discipline that is easy to skip: the sync side. Validation at the boundary establishes that what entered was trustworthy. Sync tasks establish that what is still there matches what entered — that the copy in the warehouse has the same feature count as the copy in PostGIS, that the published tiles cover the same extent as the source, that a mirror in another region has not silently fallen a week behind. Those are the failures that no amount of entry validation catches, because nothing went wrong at the door; something went wrong afterwards, quietly, in a system that reported success.

Prerequisites & Architecture Baseline#

• pyproj ≥ 3.6 and shapely ≥ 2.0, pinned — validity semantics and CRS lookups both participate in what "valid" means
• PostGIS ≥ 3.3 if validation happens in the database, for ST_IsValidDetail and ST_MakeValid
• A place to record the validation outcome: the pipeline's data contract, a column, or a run artefact
• A quarantine destination for records that fail — a dead-letter queue, not a log line
• The authoritative extent of each CRS you accept, so "outside the valid area" is a computable statement
• A policy, written down, on whether a failed record fails the batch or is quarantined
• Test fixtures containing genuinely broken data — a bow-tie polygon, a missing projection, a Latin-1 attribute file — so the validator's failure paths are exercised rather than assumed

Core Principles#

1. Validate at the boundary, exactly once. The boundary is wherever data crosses from something you do not control into something you do. Inside that line, the contract holds; outside it, nothing is assumed. Re-validating in every task is not defence in depth, it is the absence of a contract, and it costs real time on every large dataset.

2. Distinguish “declared” from “true”. A .prj file saying EPSG:4326 does not make the coordinates geographic. The cheapest and most valuable check in spatial validation is comparing the coordinate ranges against the declared CRS’s valid area — it catches the single most damaging class of error for the price of reading the bounding box.

3. Validity is topological, not semantic. ST_IsValid tells you a polygon does not self-intersect. It says nothing about whether the polygon is in the right place, the right size, or the right units. A dataset can be one hundred per cent valid and entirely wrong, so a validity check alone is never sufficient.

4. Repair is a decision, not a default. ST_MakeValid will produce something for almost any input, including a GEOMETRYCOLLECTION where a polygon was expected or an empty geometry where a sliver was. Repairing silently means those surprises reach the load. Repair deliberately, record what changed, and quarantine what cannot be repaired without changing its meaning.

5. Quarantine beats failing the batch. A delivery of four hundred thousand features containing nine broken ones should load 399 991 features and put nine somewhere a human can look. Failing the whole batch converts a small data-quality problem into an outage, and it trains everyone to re-run without looking.

6. Sync tasks are validation with a comparison. Checking that a downstream copy matches its source — feature counts, extents, checksums per partition — is the same discipline applied across systems rather than at an entry point. It answers the question monitoring cannot: not “did the pipeline run?” but “does what it produced still match what it came from?”

The range check deserves its prominence because of how its failures propagate. A wrong projection does not corrupt one feature, it displaces every feature in the dataset — and every downstream operation continues to work perfectly, because the geometry is internally consistent. Spatial joins return no matches, or the wrong ones. Zonal statistics compute over the wrong areas. Tiles render blank, which is usually the first symptom anybody notices, several steps and often several days later. Comparing the declared CRS against the coordinate ranges takes one bounding-box read and catches the whole class at the door.

Production Implementation#

The validator below runs at the boundary and returns a report rather than raising, so the caller decides what to do with partial failure.

from __future__ import annotations

from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional

import fiona
import shapely
from pyproj import CRS
from shapely.geometry import shape


@dataclass
class ValidationReport:
    """What was checked, what passed, and exactly what did not."""

    declared_crs: Optional[str] = None
    detected_issue: Optional[str] = None
    total: int = 0
    invalid_geometries: list[tuple[str, str]] = field(default_factory=list)
    out_of_range: list[str] = field(default_factory=list)
    empty_geometries: list[str] = field(default_factory=list)

    @property
    def ok(self) -> bool:
        return (
            self.detected_issue is None
            and not self.invalid_geometries
            and not self.out_of_range
        )


def crs_area_bounds(crs: CRS) -> tuple[float, float, float, float]:
    """The CRS's own declared area of use, projected into its own units."""
    area = crs.area_of_use
    if area is None:
        return (-float("inf"), -float("inf"), float("inf"), float("inf"))
    if crs.is_geographic:
        return (area.west, area.south, area.east, area.north)
    from pyproj import Transformer
    to_crs = Transformer.from_crs(CRS.from_epsg(4326), crs, always_xy=True)
    x0, y0 = to_crs.transform(area.west, area.south)
    x1, y1 = to_crs.transform(area.east, area.north)
    return (min(x0, x1), min(y0, y1), max(x0, x1), max(y0, y1))


def validate_layer(path: Path, layer: str = "features",
                   id_field: str = "gml_id") -> ValidationReport:
    report = ValidationReport()

    with fiona.open(path, layer=layer) as src:
        if not src.crs:
            report.detected_issue = "no CRS declared — the coordinates mean nothing yet"
            return report

        crs = CRS.from_user_input(src.crs)
        report.declared_crs = crs.to_string()
        minx, miny, maxx, maxy = crs_area_bounds(crs)

        for feature in src:
            report.total += 1
            fid = str(feature["properties"].get(id_field) or feature["id"])
            if feature["geometry"] is None:
                report.empty_geometries.append(fid)
                continue

            geom = shape(feature["geometry"])
            if geom.is_empty:
                report.empty_geometries.append(fid)
                continue

            gminx, gminy, gmaxx, gmaxy = geom.bounds
            # The most valuable check here: coordinates outside the declared CRS's
            # own area of use mean the declaration is wrong, whatever it says.
            if not (minx <= gminx and gmaxx <= maxx and miny <= gminy and gmaxy <= maxy):
                report.out_of_range.append(fid)
                continue

            if not shapely.is_valid(geom):
                report.invalid_geometries.append((fid, shapely.is_valid_reason(geom)))

    return report

Step-by-Step Walkthrough#

1. Refuse an undeclared CRS immediately. Without a projection the coordinates are just numbers, and every subsequent check is meaningless. This is the one failure that should stop the batch rather than quarantine records.
2. Derive the valid area from the CRS itself. pyproj’s area_of_use is authoritative and free. Hand-written bounding boxes go stale and are wrong for every CRS but the one they were written for.
3. Check ranges before validity. A range failure means the declaration is wrong, which makes the validity result irrelevant — a polygon can be perfectly valid and sitting in the Gulf of Guinea. Ordering the checks this way also means the expensive one runs on fewer features.
4. Record the reason, not just the count. is_valid_reason returns text like Self-intersection[6.12 58.4], which names the offending vertex. Storing that turns triage from an investigation into a query.
5. Treat empty and null geometries separately. They are usually a source-side artefact rather than corruption, and they often should be dropped rather than quarantined — but the decision differs per dataset, so keep them in their own bucket.
6. Return a report; do not raise. The caller knows the policy. A validator that raises forces every consumer into the same all-or-nothing behaviour and makes partial acceptance impossible.
7. Record the report with the run. The next question after “why is this feature missing?” is always “what did validation say that night”, and it is only answerable if the report was stored.

Edge Cases & Failure Recovery#

A CRS with no declared area of use. Custom and legacy CRSs frequently lack one, so the range check silently passes everything. Where a pipeline accepts such a CRS, supply the expected extent explicitly from the dataset’s documentation — an unbounded check that always passes is worse than no check, because it looks like coverage.

Coordinates that are valid but implausible. A parcel dataset for one municipality containing a feature two hundred kilometres away is within the CRS’s area of use and completely wrong. The fix is a second, dataset-specific extent check — the delivery’s own declared bounding box, or the administrative boundary it claims to cover.

Validity that changes with the library version. GEOS has tightened its interpretation of the simple-features rules over time, so a dataset that validated cleanly two years ago may not today. This is not a regression to work around; it is the check getting better. Pin the version, and treat an upgrade as an event that may quarantine records which previously passed.

Repairs that change the geometry type. ST_MakeValid on a bow-tie polygon returns a MULTIPOLYGON; on a degenerate sliver it may return a LINESTRING or an empty geometry. A pipeline that repairs and then loads into a typed column will fail at the load with an error that names the column rather than the repair. Re-describe after repairing, as in repairing invalid geometries before load.

Attribute encoding that no geometry check sees. A .dbf in Latin-1 read as UTF-8 gives mojibake in every place name while the geometry is perfect. Nothing in the spatial validation path catches it. Assert on a known accented string from the dataset in your fixtures, and set the encoding explicitly on read.

Validation that runs after the expensive step. Checking the CRS after reprojecting is checking your own arithmetic, not the source’s declaration — and it has already cost the reprojection. The boundary is before the first expensive operation, not after the last cheap one, and moving a check earlier is usually the highest-value change available to a pipeline that validates in the wrong place.

A sync check that compares the wrong things. Feature counts match while geometries differ, because the copy was made before a repair step. Comparing counts is cheap and weak; comparing a per-partition checksum of geometry WKB is a little more expensive and actually answers the question.

Configuration Reference#

The failure policy is the one row that should be revisited per dataset rather than set once. A basemap can tolerate nine missing parcels; a cadastral register of record cannot, and for that dataset quarantining silently is the wrong behaviour — the batch should complete, the nine should be quarantined, and the run should be marked as requiring review before the output is published. That third state, “completed with quarantine”, is worth having explicitly in your run vocabulary, because collapsing it into either “succeeded” or “failed” loses the distinction that matters to whoever consumes the data.

Frequently Asked Questions#

Related#

• Validating coordinate systems before ETL — the CRS and range checks in detail
• Repairing invalid geometries before load — repairing deliberately rather than by default
• Building ETL chains for vector data — the contract this validation establishes
• Storing failed geometries in a PostGIS dead-letter queue — where quarantined features go
• Monitoring geometry validity rates as an SLO — tracking the outcome over time

← Spatial Task Design & Dependency Mapping