Measuring Tile Generation Latency Percentiles

To measure p50, p95, and p99 tile-generation latency, record each tile’s render time in a Prometheus Histogram whose buckets straddle your real durations, label observations by zoom level, and compute percentiles at query time with histogram_quantile(). Averages hide the slow tail that actually degrades a tile server, and a single global percentile blurs the fact that a 512px zoom-3 tile and a 256px zoom-18 tile live in different time regimes. This page shows the Histogram setup, the bucket reasoning, and the PromQL that turns buckets into percentiles.

When to Use This Pattern#

• You serve or pre-render map tiles and need tail-latency visibility, not just a mean.
• Slow tiles at specific zoom levels are suspected but not proven.
• You compare 256px versus 512px tile rendering cost and need per-size percentiles.
• You are setting an SLO like “p99 tile latency under two seconds” and must measure against it.

Complete Working Example#

The instrumentation records one Histogram observation per rendered tile, labeled by zoom level and tile size — both bounded sets — so histogram_quantile() can later resolve percentiles per group. Zoom is capped to a small label set; tile coordinates are never used as labels.

import time
import mercantile
from pyproj import CRS
from prometheus_client import Histogram

WEB_MERCATOR: CRS = CRS.from_epsg(3857)

TILE_SECONDS = Histogram(
    "tile_generation_seconds",
    "Time to render a single map tile",
    labelnames=("zoom", "tile_px"),
    # Buckets span fast high-zoom 256px tiles up to slow low-zoom 512px tiles.
    buckets=(0.02, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0),
)

def render_tile(x: int, y: int, z: int, tile_px: int = 256) -> bytes:
    """Render one tile and record its latency, labeled by zoom and size."""
    # Bound the label: clamp zoom to the documented pyramid depth.
    zoom_label: str = str(z) if 0 <= z <= 22 else "out_of_range"
    bounds = mercantile.xy_bounds(x, y, z)  # extent in EPSG:3857 metres

    start = time.perf_counter()
    png_bytes: bytes = _rasterize_extent(bounds, tile_px, crs=WEB_MERCATOR)
    TILE_SECONDS.labels(zoom=zoom_label, tile_px=str(tile_px)).observe(
        time.perf_counter() - start
    )
    return png_bytes

def _rasterize_extent(bounds, tile_px: int, crs: CRS) -> bytes:
    # Placeholder for the actual renderer (e.g. rasterio + a color ramp).
    # Real work lives here; the wrapper only measures its duration.
    ...

Labeling on zoom and tile_px keeps the series count to at most 23 zoom levels times two sizes times the bucket count — bounded and cheap — while giving enough resolution to separate the fast and slow regimes.

Parameter & Option Reference#

Verification & Testing#

Scrape the endpoint and confirm the bucket, sum, and count series exist for each zoom and size group:

curl -s localhost:9105/metrics | grep tile_generation_seconds
# tile_generation_seconds_bucket{le="0.5",tile_px="256",zoom="14"} 812
# tile_generation_seconds_sum{tile_px="256",zoom="14"} 190.4
# tile_generation_seconds_count{tile_px="256",zoom="14"} 1000

Compute the p99 per zoom level in the Prometheus expression browser. Aggregating the _bucket series by (le, zoom) before applying the quantile is what yields a correct per-zoom percentile:

histogram_quantile(
  0.99,
  sum(rate(tile_generation_seconds_bucket{tile_px="256"}[5m])) by (le, zoom)
)

A guard test confirms your buckets are not saturated — if the top bucket count equals the total count, the true tail lives above your highest bound and the percentile is unreliable:

def test_buckets_are_not_saturated():
    samples = {s.labels["le"]: s.value
               for s in TILE_SECONDS.collect()[0].samples
               if s.name.endswith("_bucket")}
    total = TILE_SECONDS.collect()[0].samples[-1].value  # _count
    top_bucket_count = samples["+Inf"] - samples["8.0"]
    assert top_bucket_count / total < 0.01, "tail exceeds top bucket; widen buckets"

Common Pitfalls#

• Reporting an average instead of a percentile. rate(_sum) / rate(_count) gives a mean that a handful of multi-second tiles barely move, hiding exactly the tail latency users feel. Always compute p95/p99 from the buckets.
• A single global percentile across all zooms. Mixing fast zoom-18 tiles with slow zoom-3 tiles produces a meaningless blended number. Break the quantile down by (le, zoom) so each regime is visible.
• Buckets that top out too low. If your slowest 512px tiles take six seconds but the top bucket is at two, histogram_quantile clamps at the boundary and reports flat, wrong tails. Set the highest bound above your observed maximum.
• Forgetting by (le, ...) in the quantile query. histogram_quantile() needs the le label preserved through aggregation; dropping it or failing to sum the rate first returns nonsense.

Related: This tiling instrumentation extends the metric family in Prometheus metrics for raster throughput, complements the warp counters in instrumenting gdalwarp with Prometheus counters, and the percentile panels belong in Grafana dashboards for GIS workflows within the wider Observability & Monitoring for Geospatial Pipelines practice. See the Prometheus histogram documentation for the quantile-estimation math.

← Back to Prometheus Metrics for Raster Throughput