StreetZim — offline OpenStreetMap in a ZIM file

StreetZim packages OpenStreetMap data into a single ZIM file that opens in Kiwix (iOS, Android, desktop) or in a browser, with no network at all. A ZIM contains:

• Vector map rendered on the device by MapLibre GL JS (OpenMapTiles schema, z0–14, overzoomed beyond).
• Search over places, administrative areas (countries to wards, from the extract's boundary relations), streets, addresses, POIs, peaks, parks and water, plus Kiwix's own title and full-text search over a detail page for each place, administrative area, park, peak, water feature and airport (POIs too with --kiwix-poi-pages; never streets or addresses: see KIWIX_PAGE_TYPES in streetzim/zim_writer.py and docs/zimfarm.md).
• Find page with category chips (Food & Drink, Bars, Hotels, Museums, Parks, Health, Shops, Gas…) and distance sorting.
• Offline routing (drive / walk / bike) in a Web Worker, with a GPS turn-by-turn HUD.
• Terrain: hillshade and 3D from the Copernicus DEM (on in the streetzim command with --profile full, the default; --terrain for create_osm_zim.py).
• Optional satellite imagery (see the licence note below), Wikidata facts and bundled Wikipedia articles.
• Optional Overture Maps addresses and place details (websites, phones, brands).

Published ZIMs are listed at https://streetzim.web.app. That site also has a PWA at /drive/ that opens a local ZIM, or streams one from archive.org, in the browser.

CHANGELOG.md lists what StreetZim can do and when each feature arrived.

Quick start (build Monaco in about a minute)

CI runs these steps on every push (.github/workflows/ci.yml, job monaco-e2e, which also adds the production routing and chip flags), so they are known to work on Ubuntu 24.04.

1. System tools. You need Python ≥ 3.10, osmium-tool, and tilemaker 3.x. Ubuntu's apt install tilemaker (2.4) is too old for resources/tilemaker/process-openmaptiles.lua, so build it from source:

sudo apt install osmium-tool unzip build-essential cmake libboost-dev \
  libboost-filesystem-dev libboost-iostreams-dev libboost-program-options-dev \
  libboost-system-dev liblua5.1-0-dev libshp-dev libsqlite3-dev rapidjson-dev zlib1g-dev
git clone --depth 1 --branch v3.2.0 https://github.com/systemed/tilemaker.git /tmp/tilemaker
cmake -S /tmp/tilemaker -B /tmp/tilemaker/build -DCMAKE_BUILD_TYPE=Release
cmake --build /tmp/tilemaker/build -j"$(nproc)" && sudo cp /tmp/tilemaker/build/tilemaker /usr/local/bin/

On macOS: brew install osmium-tool, then build tilemaker the same way (brew install boost lua rapidjson shapelib sqlite cmake for its dependencies) or use Homebrew's tilemaker if it is 3.2 or newer (3.0 runs the profile too, but writes some labels into many tiles; see docs/language-ready.md).

2. Python environment.

python3 -m venv venv && . venv/bin/activate
pip install -r requirements.txt

requirements.txt pulls stock libzim (python-libzim), which includes the libzim fixes StreetZim contributed upstream; the old patches in patches/ are kept for history only. It lists only what the builder needs; to run the tests and linters, install requirements-dev.txt instead (MAINTAINING.md).

3. Coastline and Natural Earth shapefiles. tilemaker reads them relative to the directory you build in. Without them the build still succeeds but has no ocean, and create_osm_zim.py prints a warning.

scripts/fetch-shapefiles.sh        # ~900 MB download, once

4. Build, validate, view.

python create_osm_zim.py --area monaco --routing -o osm-monaco.zim
python cloud/validate_zim.py osm-monaco.zim     # the pre-upload gate
kiwix-serve --port 8888 osm-monaco.zim          # from kiwix-tools; or:
python cloud/serve_zim_entries.py osm-monaco.zim --port 8888

StreetZim uses the standard openZIM tools. It writes ZIMs with python-libzim (libzim), checks them with zimcheck, and serves them with kiwix-serve. Nothing else is needed. On every push, CI builds, rewrites and serves a ZIM with exactly this stack.

cloud/validate_zim.py also runs zimcheck if it is on your PATH. Use zim-tools 3.8 or newer from https://download.openzim.org/release/zim-tools/; Ubuntu's 3.2 misreads ZIMs written by current libzim.

Or with Docker

The image contains tilemaker 3, osmium and the Python stack; CI builds it and makes a ZIM inside it on every push.

docker build -t streetzim .
mkdir -p out
docker run --rm -v "$PWD/out:/output" streetzim fetch-shapefiles.sh /output   # once
docker run --rm -v "$PWD/out:/output" streetzim \
    create_osm_zim.py --area monaco --routing -o /output/osm-monaco.zim

Or the openZIM way: streetzim

streetzim is the same builder behind openZIM's scraper conventions: maps2zim's flag names, ZIM metadata flags checked before any download, {name}_{period}.zim in an output folder, and a Zimfarm progress file. Its flags are described for Zimfarm in offliner-definition.json; docs/zimfarm.md covers the profiles (--profile full, the default, adds Wikidata, Wikipedia articles, Overture Maps and terrain like StreetZim's own builds, plus POIs in Kiwix's search; --profile basic fetches nothing but the OSM data), how they compare with the production builds, and what a build costs.

pip install -e .              # or a wheel (python -m build; docs/packaging.md), or the Docker image
streetzim --name osm_en_monaco --title Monaco \
    --description "Offline map of Monaco with search and routing" \
    --area monaco --output out --stats-filename out/task_progress.json

--cpus N sets how many cores the build uses at once. By default it is the usable cores, capped by a container's CPU quota and at one per 2 GiB of its memory limit; in Docker with only a CPU share (as on Zimfarm) pass it, since a share hides no cores and each core costs memory (docs/zimfarm.md).

For Docker to choose workers from RAM, set its memory limit and leave --cpus unset. For example, an 8 GiB Netherlands full build automatically uses at most 4 workers for tilemaker, search and terrain:

docker run --rm --memory 8g --memory-swap 8g \
    -v "$PWD/out:/output" streetzim \
    streetzim --name osm_en_netherlands --title Netherlands \
    --description "Offline map of the Netherlands with search, routing and terrain" \
    --include-poly https://download.geofabrik.de/europe/netherlands.poly \
    --profile full --zim-workers 4 --output /output

--memory-swap 8g makes the combined RAM/swap limit equal to the RAM limit, so this example uses no swap. Compression has a separate, smaller memory cost; --zim-workers 4 sets its worker count explicitly. --cpus N overrides the automatic worker budget. Fixed allocations, including the initial native PBF extraction, still consume RAM even with one worker; scaling workers does not guarantee every region fits the same container limit.

Satellite imagery is off by default in streetzim. --satellite adds EOX's 2016 mosaic (CC BY 4.0); the sharper 2021 mosaic is non-commercial and takes --satellite-source s2cloudless-2021 --satellite-accept-noncommercial, which also labels the ZIM as restricted (docs/zimfarm.md).

Choosing what goes in

Input, one of:

flag	input
--area NAME	a preset: monaco, liechtenstein, dc, manhattan, san-francisco, austin, portland, virginia, colorado, california, iran, united-states
--geofabrik europe/liechtenstein --name …	any Geofabrik extract
--pbf file.osm.pbf --name …	a local PBF; add --bbox minlon,minlat,maxlon,maxlat to cut it (minlon > maxlon for an area across the antimeridian, e.g. Fiji)
--mbtiles tiles.mbtiles	skip tilemaker and reuse existing tiles (with --pbf for search/routing)

Areas across the antimeridian (Fiji, Chukotka, Kiribati) are supported: the extract, tiles, map bounds, search and routing cover both sides of ±180° (docs/zimfarm.md, docs/formats.md).

Main feature flags (all off by default; python create_osm_zim.py --help lists all of them, about 70):

flag	adds	needs
--routing	routing graph (SZRG v4)	—
--spatial-chunk-scale 10	routing split into 0.1° cells loaded on demand (what production ships; needed for large regions on phones)	--routing
--split-find-chips	one file per Find chip instead of the whole POI list	—
--terrain	hillshade / 3D terrain tiles to z12	GDAL (via rasterio), network to AWS S3
--satellite	Sentinel-2 imagery; --satellite-source picks the year: s2cloudless-2021 (default here, CC BY-NC-SA 4.0, non-commercial) or s2cloudless-2016 (CC BY 4.0)	network to EOX
--wikidata	population, descriptions, Wikipedia extracts	network to Wikidata/Wikipedia
--bundle-wiki-articles --wiki-articles-source enwiki.zim	full Wikipedia articles, read from a local Wikipedia ZIM	a Wikipedia ZIM
--overture-addresses/--overture-places PARQUET	Overture data, from download_overture_data.py	DuckDB, network to S3

Size and memory: a city builds in minutes on a laptop. Country and continent builds need tens to ~100 GB of RAM and are run with the production wrappers described in MAINTAINING.md.

How it works

Geofabrik / planet PBF
   │  tilemaker (OpenMapTiles profile, resources/tilemaker/)
   ▼
MBTiles ──► tiles/{z}/{x}/{y}.pbf ──┐
PBF ──► search records, addresses ──┤  (+ Overture, Wikidata, terrain, satellite)
PBF ──► routing graph (pyosmium) ───┤
resources/viewer/ (MapLibre app) ───┤
                                    ▼
                         create_osm_zim.py → osm-<name>-<date>.zim
                                    │
             Kiwix / the /drive/ PWA serve entries to the viewer, which renders
             tiles, searches the sharded JSON and routes in a Web Worker.

• Every byte format and ZIM path is specified in docs/formats.md; search records and category files in docs/search-records.md.
• Tile sources: StreetZim's tilemaker tiles give +31% street names and about 2x named places compared with OpenFreeMap's (docs/tile-sources.md).
• Working with openzim/maps: StreetZim also builds from the OpenFreeMap tiles maps2zim uses (scripts/fetch-openfreemap-mbtiles.py, then --mbtiles); we propose openZIM adopt StreetZim as its maps scraper (docs/adoption-plan.md, docs/zimfarm.md); docs/openzim-integration.md compares the two and sets out the porting route for reference.
• Routing internals: docs/routing.md. Search sharding: docs/search-prefix-locality.md.
• Kiwix reader quirks the design works around (service-worker request drops, cluster size limits, zstd windows): docs/zim-packaging-gotchas.md.
• The in-ZIM apps (places.html, detail pages, the #dest= deep-link protocol): docs/in-zim-apps.md.

Repository map

path	what
create_osm_zim.py	the builder's command line
streetzim/cli.py	streetzim, the openZIM-style command (maps2zim's flags; offliner-definition.json)
resources/viewer/	the viewer shipped inside every ZIM (index.html, built from src/index/ by tools/build_viewer.py; places.html; routing-worker.js)
resources/vendor/, resources/viewer-assets.lock.json	vendored MapLibre GL JS and the sha256 pins for it and the font glyphs (tools/pin_viewer_assets.py, docs/viewer-supply-chain.md)
resources/tilemaker/	tilemaker config and Lua profile
cloud/	Python modules the builder imports (chip_rules, search_shards, viewer_slots, …) and the ZIM tools (validate_zim, repackage_zim, patch_viewer_inplace, …)
streetzim/	the builder's modules: tiles, terrain, satellite, addresses, admin_areas, search_extract, zim_writer (one function per ZIM phase), zim_metadata (openZIM metadata rules), progress (Zimfarm progress file), cpus (the --cpus budget), cli (the streetzim command), area, download, paths, mbtiles, overture, overture_taxonomy, tile_alias, viewer_assets, glyph_fallback, satellite_sources, scraperlib, source_report, common, and routing/ (graph build, formats, reference routers)
tests/	pytest and Node tests
web/	streetzim.web.app catalogue and the /drive/ PWA (operations; moves to ops/ in stage 2)
preview-proxy/	archive.org range proxy for online previews
ops/	StreetZim's own operations (build host, uploads, rollouts); not needed to build a ZIM. The old paths are symlinks into it for now: ops/README.md

MAINTAINING.md explains which parts are core and which are the author's hosting/operations tooling, how releases are built and shipped, and the known technical debt.

Licence

Code: MIT (see LICENSE).

Data in the ZIM files:

source	licence	attribution
OpenStreetMap	ODbL 1.0	© OpenStreetMap contributors
OpenMapTiles schema	CC BY 4.0	© OpenMapTiles
OSM water polygons	ODbL 1.0	© OpenStreetMap contributors
Natural Earth	public domain	—
Copernicus GLO-30 / GLO-90 DEM (--terrain)	Copernicus free & open	© DLR e.V. 2010-2014 and © Airbus Defence and Space GmbH 2014-2018, provided under COPERNICUS by the EU and ESA
EOxCloudless Sentinel-2 cloudless 2016 by EOX (--satellite-source s2cloudless-2016; streetzim --satellite)	CC BY 4.0	EOxCloudless https://cloudless.eox.at by EOX IT Services GmbH (Contains modified Copernicus Sentinel data 2016 & 2017)
EOxCloudless Sentinel-2 cloudless 2021 by EOX (create_osm_zim.py --satellite)	CC BY-NC-SA 4.0	EOxCloudless https://cloudless.eox.at by EOX IT Services GmbH (Contains modified Copernicus Sentinel data 2021)
Overture Maps (--overture-*)	CDLA-Permissive-2.0 and others per source; written to overture-sources.json	per source
Wikidata	CC0	—
Wikipedia text	CC BY-SA 4.0	Wikipedia contributors

Bundled software: MapLibre GL JS (BSD-3-Clause); mapbox-gl-rtl-text 0.3.0 (BSD-2-Clause, with ICU under the Unicode licence; vendored in resources/vendor/mapbox-gl-rtl-text/, shaping Arabic/Hebrew labels); Maki 8.2.0 POI icons (CC0 1.0, inlined in the viewer); label glyphs from Open Sans (Apache 2.0) and, for Arabic, Armenian, Georgian, Hebrew, Lao and Thai labels when a map has them, Noto Sans (SIL Open Font License 1.1, shipped in the ZIM as fonts/NotoSans/OFL.txt; pinned in resources/viewer-assets.lock.json, licence in resources/vendor/noto-sans/).

Satellite imagery: check which year a ZIM has. EOX licenses each year of its Sentinel-2 cloudless mosaic separately. The 2021 layer is CC BY-NC-SA 4.0: non-commercial use only. create_osm_zim.py only includes it with --satellite, but the production wrappers pass that flag, so most published StreetZim ZIMs contain it and may only be used and redistributed non-commercially. Build without --satellite (or use a variant with satellite=no in cloud/region-variants.tsv) for a ZIM with no non-commercial data. The 2016 layer is CC BY 4.0 (attribution only), and is what the openZIM-style streetzim --satellite adds; streetzim takes the 2021 layer only with --satellite-accept-noncommercial, and then labels the ZIM as restricted. A ZIM's License metadata, its viewer credits and the on-map attribution name the year it contains. Details: docs/zimfarm.md.

Optional writer and native search indexer

The author's production host builds whole continents. It uses two optional backends, both switched off by default:

• --zim-builder manifest uses the Python manifest packer with configurable compression, streamed large bodies and raw Xapian/routing clusters. It cannot run libzim's indexer: without --xapian builder (--xapian none, the streetzim default for this backend) the ZIM has no Xapian indexes, so Kiwix has no full-text search and no Xapian title suggestions (only prefix matches on titles).
• --xapian builder uses an external indexer (xapianbuilder) over the same Kiwix search pages a libzim build writes.

Both produce ordinary ZIMs, the same as libzim's. Neither is needed for anything in this README. docs/zim-builder-python.md explains when they help.