mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 21:17:58 +00:00
Closes #2428. Adds free and open map providers (topographic, satellite, gray and vector basemaps), a hillshade overlay, and support for operators' own tile servers. Everything is additive and opt-in: Carto stays the default, every new provider group is off until the operator turns it on, and no existing config key or provider was removed. An upgrade with an untouched config changes nothing in the picker. > **Dependency:** #2426 has merged. The branch is rebased onto current `master` (`6c78a1d2`) with no conflicts. ## What's new - **Provider registry** (`public/map-tile-providers.js`). New groups are **opt-in**: each is off until `map.tiles.providers.<group>.enabled` is `true` (the boolean), so licences and volunteer servers are the operator's call: - **EOX**: Sentinel-2 satellite (2025 and 2016), Terrain Light, and a place-labels overlay. EOX 2025 imagery is CC BY-NC-SA 4.0 (non-commercial); a commercial deployment should leave `eox` off or license it. - **OSM France**: Humanitarian, CyclOSM, OSM Gray and OSM Gray Dark (volunteer-run, fair use). - **OpenFreeMap** and **VersaTiles** vector styles: Liberty, Bright, Positron, Dark, Fiord, Colorful, Muted and the Gray variants. - **Hillshade** (below) is opt-in the same way. **OpenTopoMap**, the OSM community server, Stamen and USGS keep their existing opt-in. - **Custom tile servers**: `map.tiles.custom` adds raster (XYZ), vector (MapLibre style URL) and raster-overlay entries as `custom-<id>`. They can be used in `darkDefault` / `lightDefault`. Invalid entries are skipped with a console warning. - **Hillshade overlay** (`public/map-hillshade.js`): shaded in the browser from Terrarium elevation tiles. `map.tiles.providers.hillshade.url` can point at a mirror. - **Vector basemaps** (`public/map-vector.js`) use a vendored MapLibre GL 5.24.0 (CSP build) and `@maplibre/maplibre-gl-leaflet` 0.1.4. - They are lazy-loaded only when a vector map is picked (about 380 KB gzipped). - The load times out after 15 s. - Without WebGL the map falls back to the theme's default raster map. - **One picker on Map, Live and Repeater Planner**: Auto, Street, Topographic, Satellite, plus an Overlays group. - **A map picked there is saved** (`localStorage` `mc-tile-pinned`, untrusted: honoured only while registered). Every picker reopens on it after a reload or on another page, until the visitor picks **Auto**. - The other maps (home mini-map, node detail, node reach, analytics) follow the visitor's choice through the shared builder: the saved pick when it suits the current theme (satellite always does), otherwise the theme's map. - Those small maps never load MapLibre, so a vector choice shows the theme's default raster map there. - **Security contract**: the `CONTRACT` block at the top of `map-tile-providers.js` (S1 to S7) covers one shared tile-URL validator (`https?://` or a same-origin path, no credentials in the authority, whitelisted placeholders), attribution HTML-escaped once, picker labels set with `textContent`, explicit MapLibre options, pinned vendored versions, CORS-safe hillshade decoding, and treating stored picks as untrusted input. `map.tiles` is served publicly in `/api/config/client`, so keys put there are public (the docs warn about this). ## Fixes included The race stress suite written for this PR found teardown races when leaving map pages mid-load: - a lazy MapLibre load finishing after navigation; - a hillshade tile decoding after `remove()`; - a provider-change event arriving during teardown. It also found a pre-existing document-listener leak in `AreaFilter` / `RegionFilter`. All of these are fixed in `e6709e74`, with regression tests in `tests/e2e/test-map-teardown-races.js`. The first CI run then caught a bug in that fix (`7f8d792c`). `MC_safeRemoveMap` called `map.off()` before `map.remove()`, so the map's `unload` event reached no one. Leaflet relies on that event to detach controls and clean up the zoom-animation proxy and BoxZoom, and `test-issue-1206` watches it to confirm node maps are disposed. `off()` now runs after `remove()`, and a new regression test checks that `unload` fires, controls detach and no listener is left. A later stress run (a new seed) found a real two-tab bug, fixed in `0506fc1d`. When another tab's overlay pick is already in localStorage but its `storage` event hasn't arrived, a layer-picker click here triggered a sync that added that overlay while Leaflet was still applying the click. Leaflet doesn't redraw the list for changes it sees mid-click, so the overlay's checkbox stayed unticked, and the next click on any overlay removed it and wrote the removal back to every tab. The overlay set now defers a sync until the click has finished. New E2E check `4b` in `test-map-providers-e2e.js` plants another tab's pick and clicks twice: one write per click, the planted overlay stays ticked. It fails on the previous code. ## Docs - `docs/deployment.md`, "Map Tile Providers" section, rewritten. It covers every provider and its licence, turning groups on and off, custom servers, vector maps, hillshade, self-hosting, and what is public. - User guide: - `map.md` gets a "Map style and overlays" section; - `live.md` notes the picker; - `configuration.md` gets a "Map tiles" section; - the FAQ gets a new entry on satellite, terrain and gray maps. - `CHANGELOG.md`: an Unreleased entry. ## Test data Two datasets were used, so every number below says which one it came from. **1. Large synthetic dataset** (perf A/B and screenshots). It is generated, contains no real mesh data, and is not checked in. - **Generator:** `synthgen` 1.0.0, seed 1, "full" scale. Built from upstream `be16173e` and retimed so its 18-day window (2026-10-31 to 2026-11-18) ends at the time of the run. - **Database:** SQLite, 3.94 GB (586 MB as zstd), 11.6 M rows in all. | Table | Rows | |---|---| | Nodes | 18,102 (12,606 companions, 3,652 repeaters, 953 rooms, 891 sensors) | | Nodes with a location | 13,452, spread worldwide (latitude −37° to 60°, longitude −123° to 175°) | | Observers | 1,175, in 42 regions on six continents | | Transmissions | 759,133 | | Observations | 10,569,668 | | Neighbour edges | 42,806 | | Advert route evidence | 212,142 | During the perf runs, the server's packet store was pinned to the newest 48 h with a 1.5 GB memory cap. Each server held about 65 k transmissions and 0.9 M observations in memory (baseline 65,647 / 906,467; candidate 64,862 / 896,248), plus all 18,102 nodes (about 17,650 active) and 1,175 observers. The tracked store was about 450 MB. **2. The repo's CI fixture** (`test-fixtures/e2e-fixture.db`, 704 KB). All E2E suites and the race stress gates ran against it, freshened the way CI does: 204 nodes, 31 observers, 500 transmissions, 502 observations. A second copy with the Repeater Planner enabled was used for the planner-on stress runs. **Perf environment:** - Headless Chromium 158 via Playwright 1.58.2, at 1280×800 with device scale factor 1. - 9 measured runs per scenario plus 1 warm-up, on each of `master` and this branch. - Host: Linux, AMD Ryzen 7 3700X (16 threads), 63 GB RAM. - **Tiles were stubbed.** The harness answers every tile, elevation and style request locally with the same stub on both sides (`--mode stub`), so perf timings involve no network. Only the screenshot gallery uses real tiles. ## Screenshots These were taken from the synthetic dataset with real tiles in a real browser (headless Chromium, 1280×800). Click a thumbnail for the full size. The images are hosted on a separate assets branch, so they are not in this diff. #### Street | | | | |---|---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-light.png" width="260" alt="Street: Carto (default) (light)"></a><br>Carto (default) (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-dark.png" width="260" alt="Street: Carto (default) (dark)"></a><br>Carto (default) (dark) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-osmfr-hot-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-osmfr-hot-light.png" width="260" alt="Street: OSM France Humanitarian (light)"></a><br>OSM France Humanitarian (light) | #### Gray | | | | |---|---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-light.png" width="260" alt="Gray: OSM France (light)"></a><br>OSM France (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-dark.png" width="260" alt="Gray: OSM France (dark)"></a><br>OSM France (dark) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-light.png" width="260" alt="Gray: VersaTiles (light)"></a><br>VersaTiles (light) | | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-dark.png" width="260" alt="Gray: VersaTiles (dark)"></a><br>VersaTiles (dark) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-light.png" width="260" alt="Gray: OpenFreeMap Positron (light)"></a><br>OpenFreeMap Positron (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-dark.png" width="260" alt="Gray: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap Dark (dark) | #### Topographic | | | | |---|---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-cyclosm-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-cyclosm-light.png" width="260" alt="Topo: CyclOSM (light)"></a><br>CyclOSM (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-eox-terrain-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-eox-terrain-light.png" width="260" alt="Topo: EOX Terrain Light (light)"></a><br>EOX Terrain Light (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-opentopomap-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-opentopomap-light.png" width="260" alt="Topo: OpenTopoMap (opt-in) (light)"></a><br>OpenTopoMap (opt-in) (light) | #### Satellite | | | | |---|---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-light.png" width="260" alt="Satellite: Sentinel-2 2025 (light)"></a><br>Sentinel-2 2025 (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-light.png" width="260" alt="Satellite: Sentinel-2 2025 with labels (light)"></a><br>Sentinel-2 2025 with labels (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-dark.png" width="260" alt="Satellite: Sentinel-2 2025 (dark)"></a><br>Sentinel-2 2025 (dark) | | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-dark.png" width="260" alt="Satellite: Sentinel-2 2025 with labels (dark)"></a><br>Sentinel-2 2025 with labels (dark) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-light.png" width="260" alt="Satellite: Sentinel-2 2016 (light)"></a><br>Sentinel-2 2016 (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-labels-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-labels-light.png" width="260" alt="Satellite: Sentinel-2 2016 with labels (light)"></a><br>Sentinel-2 2016 with labels (light) | #### Vector (MapLibre) | | | | |---|---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-liberty-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-liberty-light.png" width="260" alt="Vector: OpenFreeMap Liberty (light)"></a><br>OpenFreeMap Liberty (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-dark-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-dark-dark.png" width="260" alt="Vector: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap Dark (dark) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-light.png" width="260" alt="Vector: VersaTiles Colorful (light)"></a><br>VersaTiles Colorful (light) | | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-dark.png" width="260" alt="Vector: VersaTiles Colorful Dark (dark)"></a><br>VersaTiles Colorful Dark (dark) | | | #### Hillshade overlay | | | | |---|---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-light.png" width="260" alt="Overlay: hillshade over Sentinel-2 (light)"></a><br>hillshade over Sentinel-2 (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-light.png" width="260" alt="Overlay: hillshade over Carto (light)"></a><br>hillshade over Carto (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-dark.png" width="260" alt="Overlay: hillshade over Sentinel-2 (dark)"></a><br>hillshade over Sentinel-2 (dark) | | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-dark.png" width="260" alt="Overlay: hillshade over Carto (dark)"></a><br>hillshade over Carto (dark) | | | #### Repeater Planner | | | |---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-light.png" width="260" alt="Planner: Sentinel-2 + hillshade (light)"></a><br>Sentinel-2 + hillshade (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-dark.png" width="260" alt="Planner: Sentinel-2 + hillshade (dark)"></a><br>Sentinel-2 + hillshade (dark) | #### Layer picker | | | |---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-light.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-light.png" width="260" alt="Layer picker open (light)"></a><br>Layer picker open (light) | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-dark.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-dark.png" width="260" alt="Layer picker open (dark)"></a><br>Layer picker open (dark) | #### Custom tile server | | | |---|---| | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-map.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-map.png" width="260" alt="Custom raster provider on the map"></a><br>Custom raster provider on the map | <a href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-picker.png"><img src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-picker.png" width="260" alt="Custom raster provider in the picker"></a><br>Custom raster provider in the picker | ## Testing The full suite ran twice, each time on a fresh clone: | | Pass 1 | Pass 2 | |---|---|---| | Commit (before rebase) | `8a56e841` | `9355169b` | | `test-all.sh` (unit) | pass | pass | | `npm run lint` | 0 errors (87 warnings, all pre-existing) | 0 errors (87 warnings, all pre-existing) | | Go `cmd/server` / `cmd/ingestor` | ok / ok | ok / ok | | E2E, map-related files (CI fixture) | 44/44 | 44/44 + planner-on run | | `test-e2e-playwright.js` | 149 passed, 3 skipped | 149 passed, 3 skipped | | Race stress gate (seeds) | 101, 202: all pass | 303, 404: all pass | | Stress with planner enabled | pass | pass (seed 505) | | Gallery | 32/32 | 32/32 | The rebased equivalents of the pass 1 and pass 2 commits are `fde2552c` and `e4d5c447`. **After the first CI run.** The two passes covered only the map-related E2E files, and CI exposed two problems: 1. `test-issue-1206` failed 7 checks. This was the `unload` bug described under "Fixes included". 2. The JS unit gate runs without `node_modules`, so the two new jsdom suites crashed `test-all.sh`. Locally, jsdom was reachable from a parent checkout, which hid this. The second problem is fixed in `57720bf3`, following the `domIntegration` convention in `scripts/non-unit-tests.json`. `test-map-teardown-races.js` moved to `tests/e2e/`, and the three jsdom zoom-limit tests split out into `tests/e2e/test-map-vector-leaflet-zoom.js`, so 30 + 3 = 33 tests are kept. Both run from E2E shard 2 and take a few seconds. The heavy suites (race stress, perf, gallery) stay local-only and are not run in GitHub Actions. These checks ran on the fixed, rebased tree (`57720bf3`): | Check | Result | |---|---| | `test-all.sh` from a copy with no `node_modules` (as CI runs it) | pass | | `test-test-inventory.js` | 399 suites classified | | `npm run lint` | 0 errors | | Go `cmd/server` / `cmd/ingestor` | ok / ok | | `test-issue-1206-resize-observer-leak-e2e.js` | 13/13 (7 were failing) | | `test-e2e-playwright.js` | 149 passed, 3 skipped | | `test-map-providers-e2e.js` / harness E2E | 17/17, 16/16 | | `test-map-teardown-races.js` / `test-map-vector-leaflet-zoom.js` | 19/19, 3/3 | | Race stress (3 iterations) | all scenarios pass | The commits since pass 2 are `76c67184` (docs only), `7f8d792c` and `57720bf3`. **GitHub Actions on `0506fc1d` (rebased onto `master`, with the two-tab fix): every check passed** (13 passed, 4 skipped as usual for a PR). That covers Go build and test, all three E2E shards, the server race detector and both image builds. The moved suites ran in shard 2 (`test-map-teardown-races.js` 19/19, `test-map-vector-leaflet-zoom.js` 3/3), and the unit gate ran `test-map-vector.js` (30/30). **Race stress suite** (`tests/e2e/test-map-race-stress-e2e.js`). Each gate run does 20 iterations of scenarios a to e and 50 cycles of f. The scenarios cover: - rapid provider switching; - theme flips during a vector load; - navigating away mid-load (including a 15 s MapLibre timeout); - repeated map-page navigation with heap checks; - overlay toggles; - cross-page picker persistence. Results for each run: - Listener counts stay flat: at most 3 `mc-tile-provider-changed` listeners and 5 `storage` listeners. - At most one live WebGL context at a time. - Heap growth in the second half of scenario d was 0.25 to 0.45 MB across the four gate runs. - Upstream `master` grows about 0.39 MB per map navigation loop in the same scenario; this branch levels off. ## Maintainer review (addressed) 1. **Licence default.** All five new groups (`eox`, `osmfr`, `openfreemap`, `versatiles`, `hillshade`) are opt-in: off unless `enabled: true` (`"true"`, `1` or `{}` don't count). With an untouched config the picker lists the same 6 base maps as before. Docs, `config.example.json` (now `enabled: false` for each) and the changelog say so, and each group's terms are listed before you turn it on. 2. **Saved pick.** The on-map picker now saves the pick (option 1 of your two), as the docs promised. Behaviour, and what an other-theme pick does: - Every picker opens on the pinned map, on any page and after a reload, until **Auto** is picked. A pin is shown whatever the theme, as an explicit pick already was. - Maps without a picker show it when it fits the current theme (`type` is the theme or `any`), otherwise the theme's map. So a light pin isn't drawn on a dark node map; satellite always is. - A stored pin whose group is turned off reads as Auto, and comes back if the group is turned on again; only picking Auto clears it. - On a map with a picker, Auto always means the theme's map: the pin is only the explicit layer, never loaded into the hidden Auto layer. A choice in the customizer, and Reset All, clear the pin. A failed write (quota full) never brings back an older stored pin. - E2E `7a` to `7c` in `test-map-providers-e2e.js`: pick, reload, the radio and tiles are the pick, the node detail map draws it, and a light pin in dark is kept but not drawn. Auto clears the key; junk in it gives Auto; an untouched config lists none of the new groups. 7a and 7b fail with the save line removed. Unit tests are in `test-map-tile-pin.js` (21). 3. **CI.** `test-map-providers-e2e.js` (24 checks, including `4b`, the two-tab regression check) and `test-map-providers-harness-e2e.js` (16) now run in E2E shard 3 against the fixture server: every tile host is stubbed. Non-blocking: - `getTileUrl()` now returns the raster fallback for a vector pick (`MC_rasterTileUrl`). There's a test in `test-tile-resolver.js` that fails on the old code. - The `map-vector.js` header and the binding's vendor README list the Leaflet and binding internals to recheck on an upgrade. - `docs/deployment.md` says SRI makes a JS-rewriting proxy drop vector maps to raster, so serve `public/vendor/` unmodified. - `scripts/map-perf/README.md` says where `SEED_DB` and `SHIFT_SH` come from. - The maps with `attributionControl: false` are left as they are. That predates this PR, and with every new group opt-in, an upgrade adds no uncredited imagery. Follow-up for all of them: #2441. - The OpenFreeMap credit line is `OpenFreeMap © OpenMapTiles Data from OpenStreetMap` (`2424f3c3`), the line OpenFreeMap gives for clients that don't add it automatically. Verified after these changes: - `test-all.sh` and lint pass, with 0 errors. - Both provider E2E suites pass: 24/24 and 16/16. - Race stress, seed 979: scenarios a, b, d and e pass. c (15/15) and f (37/37) pass after updating their model of the pick: they expected Auto after a return, which was the old unsaved behaviour. - The stress suite now enables the five groups itself, so it keeps exercising them. ## Performance A/B test: `master` against this branch on the large synthetic dataset, with 9 runs per scenario and stubbed tiles (see Test data; `scripts/map-perf/`). The full HTML report is not attached; ask if you want it. - **No budget failures** in either pass. The `raster-no-maplibre` budget holds: no raster scenario loads MapLibre. - **DOMContentLoaded is unchanged on every page**, even with the two new scripts (`map-hillshade.js` and `map-vector.js`, about 635 lines in all; MapLibre itself loads only on demand). - **Switching providers 50 times (raster only), median ms per switch:** - dark: 146 → 108.5; - light: 208 → 208.5. - Heap growth during the run: 0.3 to 0.4% on both sides. - **Flagged and checked:** - Live and Analytics `loadEventMs` are flagged in some runs. This metric has two modes on **both** branches: `load` fires either before Leaflet adds its first tile `<img>`s (about 150 ms) or after they finish (about 400 ms). A direct probe of `master` alone gave 428, 404 and 124 ms. On these scenarios, map creation, first tile, all tiles, first marker and wall time are within a few ms on both sides. - Carto and Positron `allTilesSinceMapMs` are 4 to 6 ms slower, just above `master`'s p95. Tiles are stubbed locally, so this is not network noise. The cause is not isolated. Both providers now go through the shared layer builder, and on the same scenarios first tile, all tiles and wall time are within 10 ms of `master` (Carto Voyager +2 to +3 ms; Positron Dark +6 to +9 ms). - The Analytics map shows one long task of about 50 ms against a `master` median of 0, but `master`'s p95 is 52 to 84 ms. This is disclosed in case it shows up for you. - Other desktop apps were running during the perf runs, so expect some noise. ## Commits 1. `c6bf12ed` feat(map): tile provider registry with custom servers, gray, topo and satellite maps, and overlays 2. `9237230b` feat(map): client-side hillshade overlay 3. `781e41b4` feat(map): vector basemaps via vendored MapLibre GL 5.24.0 4. `dae36d2e` refactor(map): every map surface uses the shared basemap builder 5. `3d4afcdc` fix(map): teardown races and listener leaks when leaving map pages 6. `b798b2fe` test(server): pin map.tiles passthrough in /api/config/client 7. `171b1bc4` test(map): E2E, race stress suite, perf harness and screenshot gallery 8. `61f73875` docs(map): providers, custom tile servers, overlays and self-hosting 9. `135a9b12` test(map): gallery shows opt-in OpenTopoMap and retries slow tile servers 10. `3833572b` test(map): harness waits for vector basemaps; stress suite fixes from test pass 1 11. `3e543cad` test(map-perf): redact tile-URL credentials; compare switch50 like with like 12. `d5006111` docs(map): user guide for map styles, overlays and map.tiles 13. `90ee1d5d` fix(map): MC_safeRemoveMap lets unload listeners run before dropping them 14. `5fc57b5c` test(map): run the jsdom map suites in the E2E job, not the unit gate 15. `bd36b634` fix(map): another tab's overlay pick that lands mid-click is no longer removed 16. `38df65d2` fix(map): new tile groups are opt-in; a map picked in the picker is saved 17. `eafbcac9` ci(e2e): run the map provider E2E suites in shard 3 18. `65b223ec` fix(map): getTileUrl() never returns a vector style url; review notes 19. `f3402c5d` test(map): race stress enables the opt-in groups and expects the saved pick 20. `e92b8631` fix(map): Auto on a picker map ignores the saved pick; customizer clears it 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
138 lines
7.2 KiB
JavaScript
138 lines
7.2 KiB
JavaScript
/**
|
|
* E2E self-test of the map harness (tests/e2e/lib/map-harness.js, #2428):
|
|
* in stub mode every tile on #/map is answered locally in light and dark,
|
|
* nothing leaks to an external host, waitForTiles sees no failures, the
|
|
* screenshot is a real PNG, and a runtime provider switch is drawn by the
|
|
* picked provider's stub.
|
|
*
|
|
* Usage: BASE_URL=http://localhost:13581 node tests/e2e/test-map-providers-harness-e2e.js
|
|
*/
|
|
'use strict';
|
|
const fs = require('fs');
|
|
const os = require('os');
|
|
const path = require('path');
|
|
const { chromium } = require('playwright');
|
|
const MH = require('./lib/map-harness');
|
|
|
|
const BASE = process.env.BASE_URL || 'http://localhost:13581';
|
|
const OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'map-harness-e2e-'));
|
|
|
|
let passed = 0, failed = 0;
|
|
async function test(name, fn) {
|
|
try { await fn(); passed++; console.log(` ✅ ${name}`); }
|
|
catch (e) { failed++; console.log(` ❌ ${name}: ${e.message}`); }
|
|
}
|
|
function assert(cond, msg) { if (!cond) throw new Error(msg || 'Assertion failed'); }
|
|
|
|
const EXPECTED_KEY = { light: 'carto-light', dark: 'carto-dark' };
|
|
|
|
async function openMap(browser, theme) {
|
|
const context = await browser.newContext({ viewport: { width: 1280, height: 800 }, serviceWorkers: 'block' });
|
|
const page = await context.newPage();
|
|
page.setDefaultTimeout(15000);
|
|
await MH.installCollectors(page);
|
|
await MH.installTileStub(page, { mode: 'stub', baseUrl: BASE });
|
|
await MH.presetStorage(page, { theme: theme, provider: 'auto' });
|
|
await page.goto(`${BASE}/#/map`, { waitUntil: 'domcontentloaded' });
|
|
await page.waitForSelector('#leaflet-map .leaflet-tile', { timeout: 15000 });
|
|
const tiles = await MH.waitForTiles(page, { timeout: 15000 });
|
|
return { context, page, tiles };
|
|
}
|
|
|
|
async function tileSrcs(page) {
|
|
return page.$$eval('#leaflet-map img.leaflet-tile', els => els.map(e => e.src));
|
|
}
|
|
|
|
async function run() {
|
|
const browser = await chromium.launch({
|
|
headless: true,
|
|
executablePath: process.env.CHROMIUM_PATH || undefined,
|
|
args: ['--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage'],
|
|
});
|
|
console.log(`\nMap harness self-test against ${BASE}\n`);
|
|
try {
|
|
for (const theme of ['light', 'dark']) {
|
|
console.log(`--- ${theme} ---`);
|
|
const { context, page, tiles } = await openMap(browser, theme);
|
|
try {
|
|
await test(`${theme}: data-theme applied from preset`, async () => {
|
|
const t = await page.evaluate(() => document.documentElement.getAttribute('data-theme'));
|
|
assert(t === theme, `data-theme=${t}`);
|
|
});
|
|
await test(`${theme}: waitForTiles reports visible tiles loaded, 0 failed`, async () => {
|
|
assert(!tiles.timedOut, `timed out: ${JSON.stringify(tiles)}`);
|
|
assert(tiles.loaded > 0, `loaded=${tiles.loaded}`);
|
|
assert(tiles.failed === 0, `failed=${tiles.failed}`);
|
|
assert(tiles.pending === 0, `pending=${tiles.pending}`);
|
|
});
|
|
await test(`${theme}: every tile on #/map was intercepted by the stub`, async () => {
|
|
const log = MH.getRequestLog(page);
|
|
const seen = new Set(log.intercepted.map(r => r.url));
|
|
const srcs = await tileSrcs(page);
|
|
assert(srcs.length > 0, 'no tile <img> elements');
|
|
const missing = srcs.filter(s => !seen.has(s));
|
|
assert(missing.length === 0, `not intercepted: ${missing.slice(0, 3).join(', ')}`);
|
|
const keys = Object.keys(log.counts);
|
|
assert(keys.length === 1 && keys[0] === EXPECTED_KEY[theme], `provider keys ${JSON.stringify(log.counts)}`);
|
|
});
|
|
await test(`${theme}: stub tiles decode as 256px images`, async () => {
|
|
const dims = await page.$$eval('#leaflet-map img.leaflet-tile-loaded', els => els.map(e => e.naturalWidth));
|
|
assert(dims.length > 0 && dims.every(w => w === 256), `naturalWidth ${dims.slice(0, 5)}`);
|
|
});
|
|
await test(`${theme}: zero external requests leaked, none failed, no MapLibre`, async () => {
|
|
const log = MH.getRequestLog(page);
|
|
assert(log.external.length === 0, `external: ${log.external.map(r => r.url).slice(0, 5).join(', ')}`);
|
|
assert(log.failed.length === 0, `failed: ${JSON.stringify(log.failed.slice(0, 3))}`);
|
|
assert(log.maplibre.length === 0, `maplibre requested: ${log.maplibre[0]}`);
|
|
});
|
|
await test(`${theme}: screenshot is a non-empty PNG`, async () => {
|
|
const p = await MH.screenshot(page, OUT, `harness-map-${theme}`);
|
|
const buf = fs.readFileSync(p);
|
|
assert(buf.length > 5000, `size ${buf.length}`);
|
|
assert(buf.subarray(1, 4).toString('ascii') === 'PNG', 'not a PNG');
|
|
});
|
|
await test(`${theme}: collectors report first tile and long-task stats`, async () => {
|
|
const m = await MH.collectMetrics(page);
|
|
assert(typeof m.firstTileMs === 'number' && m.firstTileMs > 0, `firstTileMs ${m.firstTileMs}`);
|
|
assert(typeof m.allTilesMs === 'number' && m.allTilesMs >= m.firstTileMs, `allTilesMs ${m.allTilesMs}`);
|
|
assert(typeof m.longTaskCount === 'number', 'longTaskCount');
|
|
assert(m.layers && m.layers.maps >= 1 && m.layers.layers > 0, `layers ${JSON.stringify(m.layers)}`);
|
|
});
|
|
if (theme === 'dark') {
|
|
await test('dark: setTileProvider switch is drawn by the picked provider stub', async () => {
|
|
const provs = await MH.listProviders(page);
|
|
const target = provs.find(p => p.type === 'dark' && p.id === 'esri-darkgray-labels') || provs.find(p => p.type === 'dark' && p.id !== 'carto-dark');
|
|
assert(target, `no alternative dark provider in ${JSON.stringify(provs)}`);
|
|
const ok = await MH.setTileProvider(page, { theme: 'dark', id: target.id });
|
|
assert(ok, 'MC_setDarkTileProvider returned false');
|
|
await page.waitForFunction((id) => localStorage.getItem('mc-dark-tile-provider') === id, target.id);
|
|
await page.waitForFunction(() => Array.from(document.querySelectorAll('#leaflet-map img.leaflet-tile')).some(i => !/cartocdn/.test(i.src)));
|
|
const t = await MH.waitForTiles(page, { timeout: 15000 });
|
|
assert(t.failed === 0 && !t.timedOut, JSON.stringify(t));
|
|
const log = MH.getRequestLog(page);
|
|
assert(Object.keys(log.counts).some(k => k !== 'carto-dark'), `keys ${JSON.stringify(log.counts)}`);
|
|
assert(log.external.length === 0, 'external leak after switch');
|
|
});
|
|
await test('dark: setTheme flips the app to light via its own toggle', async () => {
|
|
await MH.setTheme(page, 'light');
|
|
const stored = await page.evaluate(() => localStorage.getItem('meshcore-theme'));
|
|
assert(stored === 'light', `meshcore-theme=${stored}`);
|
|
const t = await MH.waitForTiles(page, { timeout: 15000 });
|
|
assert(t.failed === 0 && !t.timedOut, JSON.stringify(t));
|
|
assert((MH.getRequestLog(page).counts['carto-light'] || 0) > 0, 'light tiles not requested after theme flip');
|
|
});
|
|
}
|
|
} finally {
|
|
await context.close();
|
|
}
|
|
}
|
|
} finally {
|
|
await browser.close();
|
|
fs.rmSync(OUT, { recursive: true, force: true });
|
|
}
|
|
console.log(`\n${passed} passed, ${failed} failed`);
|
|
if (failed) process.exit(1);
|
|
}
|
|
|
|
run().catch((e) => { console.error(e); process.exit(1); });
|