Commit Graph
159 Commits
Author SHA1 Message Date
gadgethd 1ba7674782 style(privacy): match site-prose/prose-section layout of sibling pages 2026-08-12 20:49:42 +00:00
gadgethd 3400d3c1fc fix(telemetry): correct telemetry=off gate — key is telemetry, value off (probe caught it) 2026-08-12 20:36:19 +00:00
gadgethd 20ecb5f427 fix(telemetry): lazy telemetry=off check so module imports safely in node tests 2026-08-12 20:23:40 +00:00
gadgethd b8628a78bf feat(telemetry+gdpr): shared client error capture, console.error hook, privacy page, retention jobs, packet_paths store
- clientErrors.ts: window.onerror/unhandledrejection for BOTH domains via main.tsx
  (was app-only in App.tsx); resource-failure classification; telemetry=off gate;
  console.error hook (kind warning) with terrain-tiles noise filter; page field
  now pathname-only (no query strings)
- PrivacyPage at /privacy linked from footer + cookie banner; docs/privacy.md LIA
- retention: packet_decryptions added to row-table retention targets; packets
  180d->30d in lifecycle policies; migration 050 schedules timescaledb retention
  jobs (packets 30d, node_status 180d, node_neighbor 7d) that never existed
- migration 050: packet_paths hypertable (content-stripped path store, no
  retention) + backfill of 2.56M path-bearing packets + ingest dual-write
2026-08-12 20:23:13 +00:00
gadgethd 18cb600912 fix: mobile stats page — scrollable tabs, local-time peak hour, chart header wrap
- Tabs now stay on one line (horizontal scroll) with 40px tap targets ≤480px
- Peak hour card renders a compact local date+time (fmtPeakHour, unit tested)
  instead of the raw ISO timestamp that broke mid-string
- Chart header wraps cleanly on narrow screens; stat cards get tighter
  mobile padding + overflow-wrap on values; refresh row wraps aligned
2026-08-12 14:41:19 +00:00
gadgethd 89ea473cfa fix: mobile shell grid — floating controls no longer own a grid row
Removing .mobile-controls from flow auto-placed map-layer into the empty
auto row (height 0), blanking the map on ≤640px. Shell is now a 2-row
grid (48px topbar + 1fr map); legend and activity-replay chips sit below
the floating Layers/Live bar.
2026-08-12 13:33:10 +00:00
gadgethd 199d7b3d33 feat: mobile Live chip ensures feed layer on when opened; e2e coverage for floating menu
- App: opening the Live feed chip also enables filters.livePackets if off.
- mobile.spec.ts: LOS width assertion updated for the menu grid layout; new
  test covers floating-bar default, feed chip toggle, layers menu toggles
  and Escape close.
2026-08-12 13:17:50 +00:00
gadgethd efa746d7bf feat: mobile map de-cramp — collapsible Layers menu, feed chip, coverage details expander
- MobileControls: stacked top panel replaced with a floating bar (Layers menu
  toggle + Live feed chip) and a collapsible menu sheet: layers grid, view
  modes, LOS tools, node search, watchlist. Closes on Esc/outside tap.
- App: mobileFeedOpen state drives data-feed-open so the packet feed stays
  hidden on mobile until the Live chip is pressed (desktop unchanged).
- RfCoverageStatus: details dl + note wrap behind a Show/Hide toggle on
  mobile only; gradient + tier buttons stay visible.
- map-app.css: new ≤640px block (floating bar, menu sheet, feed gating,
  coverage details); removed superseded .mobile-controls/__primary rules.
2026-08-12 13:16:21 +00:00
gadgethd 7eb531101d fix: call maplibre v6 transform shim before MapboxOverlay addControl 2026-08-12 12:55:45 +00:00
gadgethd fa949aeee0 fix: shim maplibre v6 transform surface for deck.gl MapboxOverlay (visgl/deck.gl#10501) 2026-08-12 12:51:08 +00:00
gadgethd bae0ddeb88 fix: emit all maplibre-gl dist .mjs (worker imports shared module) 2026-08-12 11:36:00 +00:00
gadgethd fb787d3f9d fix: guard packet-privacy rewrite to privacy transitions only (049); emit maplibre-gl worker asset; serve .mjs as JS 2026-08-12 11:25:42 +00:00
hermes-gadgetandhermes-gadget e069f6b102 fix: close Wave 3 medium findings (owner auth, deploy, newuser, alerts, gitleaks, viewshed) (#64)
* fix(owner): Wave 3 mediums — password handling, deploy rollback, newuser races, session revocation

- BUG-017: stop trimming MQTT passwords (frontend + backend). A valid
  broker password may begin/end with whitespace; trimming made it
  permanently unauthenticatable. Username trimming unchanged.
- BUG-013: deploy-website.sh now set -Eeuo pipefail with an EXIT rollback
  trap armed only after the pin mutates; container-down and bundle-mismatch
  paths fail loudly and restore the old pin + verify the restored service.
- BUG-012: newuser.sh installs an EXIT trap (ERR does not fire on explicit
  exit 1 from die()) so every post-mutation failure rolls back; rollback is
  idempotent and a no-op before any mutation flag. Disarm points clear EXIT.
- BUG-011: newuser.sh re-reads + re-validates OWNER_MQTT_USERNAME_MAP under
  the re-acquired lock after the long unlocked discovery window, then merges
  the new grant into the current map — concurrent provisioning runs can no
  longer be overwritten by a stale snapshot.
- BUG-010: owner sessions are now v3 cookies carrying a credential
  generation (Redis-backed). When the broker rejects a previously-valid
  password (revocation detected), the generation bumps and every older
  session is rejected on the next request. TTL shortened 30d -> 7d.

Verified: tsc clean, 310/310 backend tests pass, bash -n on both scripts.

* fix(alert-receiver): durable alert delivery with bounded retry + dead-letter (BUG-014)

Forwarding was fire-and-forget: failures logged asynchronously after HTTP 202,
/healthz stayed green, and alerts could be archived to the local JSONL while
operators never saw them. Now:
- Every receipt is enqueued for delivery with bounded exponential backoff
  (ALERT_FORWARD_MAX_ATTEMPTS=5, base 1s, cap 60s) and dead-lettered to
  receipts.jsonl.dead after exhausting attempts.
- /healthz keeps returning 200 (compose wget healthcheck must not restart
  the container) but the body reports degraded status + detail; new /readyz
  returns 503 when ALERT_FORWARD_URL is unset (archive-only), no successful
  forward since startup, or alerts stuck undelivered >5min.

Verified: tsc clean, 310/310 tests pass.

* fix(ci): classify reviewed public channel keys + fixtures in gitleaks (BUG-009)

The nightly full-history secret scan flagged 42 findings spanning the
documented community channel keys (VALIDATED_CHANNELS — intentionally
public, each verified to decrypt real UK Mesh group text), fake test
fixtures, and a deployed-commit SHA in the website live manifests. That
made a genuine credential easy to dismiss among expected hits.

- Rule-scoped allowlists with exact fingerprints (regexTarget: secret,
  anchored full-value matches) for the Public channel key, test fixtures
  (0123456789... / abcdef0123...), and the manifest commit SHA.
- Structure-exact line allowlist for channelRegistry.ts VALIDATED_CHANNELS.
- Verified locally with gitleaks 8.24.3 full-history scan: 42 -> 0 findings.

* fix(viewshed): side-effect completion markers prevent skipped link jobs (BUG-006)

store_coverage() commits on an autocommit connection, then the worker queues
physical-link jobs and publishes Redis notifications. A Redis failure after
the DB commit NACKed the job; on retry already_calculated() saw the coverage
row and returned early — link work and frontend notifications were skipped
forever.

- Record a Redis completion marker (viewshed:side-effects:<node>) only after
  EVERY side effect succeeds.
- On the already_calculated early return, a missing marker triggers an
  idempotent replay: link jobs are re-enqueued from the stored node position
  (admission is idempotent) and coverage_update/node_upsert notifications are
  re-published from the stored coverage row.
- Redis read failures are treated as incomplete (replay attempt re-raises and
  NACKs rather than silently skipping).
- Added tests/test_side_effect_markers.py (5 tests) with a conftest that stubs
  osgeo/psycopg2 so pure-logic worker tests run without GDAL (CI keeps real
  GDAL via setdefault). Verified: 5/5 new tests pass; full suite 34 passed,
  2 GDAL-required terrain tests fail only in stub env (pass in CI image).

---------

Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>
2026-08-11 22:50:16 +01:00
d6b66424f2 fix: migrate frontend to maplibre-gl 6 / recharts 3 (dependabot #61) (#62)
* fix: migrate frontend dependencies for #61

* fix(frontend): maplibre 6 migration — react-is peer dep, worker resolution, e2e timing

- Add react-is ^19.2.8 as a direct dependency: recharts 3 declares it as
  a peer dep (recharts 2 bundled it), so vite's dep optimizer 500'd on
  'Failed to resolve import "react-is"' and the whole dashboard failed
  to mount (error boundary).
- Exclude maplibre-gl from vite optimizeDeps: v6 loads its worker via
  new URL(..., import.meta.url); pre-bundling rewrote that to
  node_modules/.vite/deps/maplibre-gl-worker.mjs (404) killing the
  worker and all custom raster protocols (hopreach-rf tiles never
  fetched) in dev.
- Harden two dashboard e2e assertions from the default 5s to 15s:
  maplibre 6's larger ESM bundle makes initial map mount slower under
  4-worker parallel CI load; the app behavior is unchanged (verified
  vs main's pass).

Verified locally: 84/84 unit tests, tsc + vite build, full e2e
(public-desktop + dashboard-desktop + dashboard-mobile) 27 passed.

---------

Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>
2026-08-11 22:18:38 +01:00
hermes-gadgetandgadgethd ab6372b8a9 fix(ci): pin gitleaks-action to valid commit after upstream force-push (#63)
* fix(ci): pin gitleaks-action to valid commit after upstream force-push

Upstream deleted dcedce43 (force-push), so every ci.yml run fails at
workflow-parse with 0 jobs. The dependabot actions-group bump (#30)
contains this fix but cannot merge whole: its docker/* actions require
node24, which GitHub runners do not support yet. Pin gitleaks alone
(ff98106e is node20).

* fix(ci): downgrade v7 actions to node20-compatible versions

actions/checkout@v7, setup-node@v7, setup-python@v7 require node24,
which GitHub-hosted runners do not support yet — every ci.yml run has
failed at workflow-parse (0 jobs) since the Aug 3 v7 bump. Pin to the
latest node20 versions (checkout v4, setup-node v4, setup-python v5).
Also pins gitleaks-action to a valid commit (upstream force-pushed away
dcedce43).

* ci: noop retrigger

* fix(ci): escape literal ${{ in bash string so GitHub parser accepts workflow

The 'Validate tracked build and Compose inventory' step matched literal
${{ inside run-block strings, which GitHub's expression parser reads as
the start of an expression (${{'* is invalid) — the workflow fails at
parse time with 0 jobs. Split the literal as '$''{{' (bash concatenates
adjacent quoted strings at runtime; the source no longer contains a
contiguous ${{).

* fix(ci): resolve all three pre-existing ukmesh CI failures

Backend (packetBatchWriteCoalesce):
- 045: restore nodes_public_visibility_generation trigger — 015 is
  superseded by 044 on fresh DBs so the trigger was never created,
  leaving generation/visibility_generation out of sync and public
  packet reads returning empty.
- 046: restore packet privacy classification — the 042 fence design
  assumes sync_private_node_prefixes rewrites packets on privacy
  change, but 026's version only maintains prefixes. Node flips to
  private left old packets visible; direct SQL inserts were never
  classified. Restore the packet rewrite in the sync trigger and add
  a BEFORE INSERT trigger mirroring the batch path's is_private /
  visibility_ok computation.
- packetBatch.integration.test.ts: exclude the prefix-cache refresh
  query from statementCount (matches unit-test convention).

Frontend e2e:
- public.spec.ts: assert the TopologyMap component's actual labels
  ('Geographic repeater topology map', '2 mapped repeaters · 2
  observed relationships') instead of the pre-map SVG graph labels.

Workers (pip install):
- viewshed-worker: bump numpy 1.26.4→2.3.5, scipy 1.13.1→1.16.3,
  psycopg2-binary 2.9.10→2.9.11 (cp314 wheels); the base gdal image
  ships Python 3.14 so the old pins had no wheels. Inherit shapely
  from the image's python3-shapely apt package (no cp314 wheel
  exists; source builds are GEOS 3.14-incompatible).

* fix(ci): restore data-plane services for compose validation and smoke test

Commit 8c5e1c8 split the long-lived data-plane services (timescaledb,
mosquitto, redis, mosquitto-reloader) into the external meshcore-infra
project on the live host, but the Workers-and-Compose CI job still
asserted their presence in the app compose project (max_worker_processes
check, inventory check, smoke-test execs). CI had been broken since
Aug 3 so the mismatch was never caught.

Add docker-compose.ci.yml — a CI-only overlay restoring the four
data-plane services from their pre-split definitions — and point every
compose invocation in the Workers-and-Compose job at
'-f docker-compose.yml -f docker-compose.ci.yml'.

Validated locally: merged config parses, the jq assertions and the full
inventory (db-migrate backend app-ukmesh website-ukmesh website-dev
mesh-health-check mosquitto-reloader link-worker link-backfill-worker
hopreach timescaledb mosquitto) all pass.

* fix(ci): include dev profile in compose inventory check

website-dev is a dev-profile service; the inventory gate compared the
base-config service list, so the merged stack never matched. This gate
has been red since the workflow was first written (CI parse-broken from
Aug 3 until the gitleaks pin landed).

* fix(ci): db-migrate must wait for timescaledb health

The Aug 9 data-plane split removed db-migrate's depends_on along with
the timescaledb service definition, so the migration runner raced the
fresh database init in the empty-volume smoke test and failed on the
not-yet-created base schema. Restore the pre-split condition:
timescaledb service_healthy.

* fix(ci): restore internal TCP MQTT listener for smoke test + exporter

Commit 40e843f dropped the 1883 TCP listener, breaking the CI smoke
(mosquitto_pub targets 1883) and the Prometheus mosquitto exporter.
The listener is internal-only: compose never publishes 1883 on the
host, and the password/ACL policy still applies.

---------

Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
2026-08-11 19:55:29 +01:00
hermes-gadgetandgadgethd 2fa099ff8b fix: unblock fresh-DB migrations + retire removed CI components + restore frontend CI (#59)
* fix: unblock fresh database migrations (#44)

* fix: remove retired CI components (#48)

* fix: restore frontend CI and RF control (#49)

---------

Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
2026-08-11 03:38:57 +01:00
gadgethd feb582be57 chore: bump ip-address + nanoid to clear npm audit findings 2026-08-09 23:03:00 +00:00
gadgethd 4d9872768a perf: reduce monitoring load and retire path history 2026-08-09 19:37:59 +00:00
gadgethd 47811106d2 feat(site): add /contact page with operator Discord + email, footer link
Contact page (site footer -> Contact) with Discord sudogadget and
ukmesh@proton.me, plus an observer-onboarding blurb. Wired into
publicRoutes (sitemap: true), lazy-loaded ContactPage chunk, and the
shared SiteLayout footer (app + website surfaces). Re-baselined
website-live-manifest.json for the new website image (e5db6790).
2026-08-09 02:45:59 +00:00
gadgethd 40e843ff72 chore: save pending state — pathing experiment reports, stats & mosquitto updates (pre-migration sweep) 2026-08-08 16:46:49 +01:00
gadgethd 049b063a8c seo: static JSON-LD, sitemap lastmod, og-image for public website (crawler-only, no UI change) 2026-08-08 00:16:20 +00:00
gadgethd 5b4da86e05 fix(mqtt,stats): accept UK neighbours topic spelling; render chart times in viewer local tz
- topic.ts/brokerLog.ts/client.ts: accept the neighbours suffix (official
  MeshCore firmware publishes the UK spelling; our MQTT fork uses neighbors)
- aclManager.ts: render + verify BOTH spellings in owner ACL grants (ukmesh
  and test scopes); inventoryOwnerAuthorization regex updated to match
- statsService.ts: chart bucket labels are now machine-readable ISO instead
  of server-timezone HH:MM/en-GB strings (backend container runs UTC;
  viewers saw UTC hours)
- StatsPage.tsx: axis ticks + tooltips + summaries formatted via
  statsTimeFormat.ts (Intl local timezone, raw fallback for legacy cached
  snapshots) + unit tests
2026-08-07 11:13:12 +00:00
gadgethd 33bfdc1c6d Expose owner node telemetry and heard neighbors 2026-08-07 00:55:51 +00:00
gadgethd 991b99fef7 fix(frontend): keep mobile map controls in flow 2026-08-06 21:47:20 +00:00
gadgethd eb4201b9dd fix(frontend): resolve mobile map and owner layout defects 2026-08-06 21:34:07 +00:00
gadgethd 7fc2fb03ef revert(owner): drop card-click hint line 2026-08-06 21:11:56 +00:00
gadgethd 14cdb96189 feat(owner): clickable identity cards replace node dropdown selector 2026-08-06 21:08:49 +00:00
gadgethd 7b6e860b57 fix(topology): keep map instance stable after load 2026-08-06 13:57:38 +00:00
gadgethd be0a68995a feat(topology): render live repeater links on UK map 2026-08-06 13:37:09 +00:00
gadgethd 1c3841cc52 chore(website): spring-clean public pages 2026-08-06 13:04:39 +00:00
gadgethd 52820d004b fix(map): keep live feed viewport visible 2026-08-06 12:44:54 +00:00
gadgethd 45698318e5 feat(map): spring-clean live feed and repeater rendering 2026-08-06 11:47:46 +00:00
gadgethd 1e5bc3b299 feat(owner): clean up spring dashboard 2026-08-06 11:25:07 +00:00
gadgethd f5a2a2b93a feat(feed): load historical channel messages 2026-08-06 05:13:22 +00:00
gadgethd 61fdcd9d11 feat: first-class MeshCore channel decryption
- channelRegistry: 39 validated channel secrets committed as the service
  default (Public + 38 community/hashtag channels recovered via the
  derivation audit); MESHCORE_CHANNEL_SECRETS env appends on top
- shared buildSummary/identifyChannel (single implementation for ingest
  and offline tools)
- tools/backfillDecrypt: keyset-paginated retroactive decryption of the
  stored raw_hex corpus (idempotent, resumable)
- feed: channel scope sidebar now lists every decrypted channel
2026-08-06 01:00:05 +00:00
gadgethd f6f8290c70 Color live paths by packet identity and extend TTL 2026-08-04 11:38:49 +01:00
gadgethd 5dd15d94c1 feat(pathing): add physics gates and held-path refinement 2026-08-04 11:04:04 +01:00
gadgethd 469c84ed13 feat(pathing): slow-mode packet resolution — wait out propagation window
Backend: schedule ONE final multi-observer resolution per path-bearing
packet after the propagation window closes (default 60s, calibrated on
real first->last observer spread p50 11s / p95 34s / p99 50s; median 17
observers per packet). The final runs with the COMPLETE observer set and
pushes into the resolve cache after invalidation, so the next API read
serves the full-propagation answer. mode=slow on /path-beta/resolve-multi
returns 202 {status:pending, remainingMs} while inside the window; new
/path-beta/slow-mode status endpoint; observability table
path_slow_resolutions (migration 034). Env knobs:
PATH_SLOW_MODE_ENABLED / PATH_SLOW_MODE_WINDOW_MS / PATH_SLOW_MODE_MIN_HOPS
/ PATH_SLOW_MODE_PENDING_MAX. Best-effort: restart loss is harmless.

Frontend: pinned packets resolve in slow mode (bounded 202 retry), local
path renders meanwhile, upgraded when the final lands. Live feed unchanged.

Tests: 4 new slowMode unit tests (dedupe, disable, bound, window timing).
Backend build clean, 257/258 (1 pre-existing workerPool timing flake on
base too); frontend build clean, 74/74.
2026-08-04 10:41:09 +01:00
gadgethd 7bcabcc426 fix(map): keep live path arcs above terrain 2026-08-04 03:07:37 +01:00
gadgethd 262a8a2515 fix(map): bind live paths to resolved packets 2026-08-04 02:12:16 +01:00
gadgethd d7e5b0ff92 fix(map): stream hop animation along stable arcs 2026-08-04 01:50:14 +01:00
gadgethd bcaa35fd27 fix(map): remove inferred nodes and restore historic hops 2026-08-03 22:47:50 +01:00
gadgethd 209989832e fix(map): reduce live-path line TTL to 5s 2026-08-03 22:07:33 +01:00
gadgethd a76ab1e16b fix(map): terrain-aware live paths + airborne hop marker 2026-08-03 22:00:38 +01:00
gadgethd f103029af7 fix(map): never shift the map on node selection 2026-08-03 21:39:43 +01:00
gadgethd 884e2cfa40 test(map): update TTL assertion to 15s 2026-08-03 21:13:03 +01:00
gadgethd 143bc97100 fix(map): reduce live-path line TTL to 15s 2026-08-03 21:10:06 +01:00
gadgethd 5ba192856e fix(map): retain branched live paths 2026-08-03 20:42:50 +01:00
gadgethd 2d740ec07c fix(map): one-click node info — remove center-snap, enlarge click hitbox 2026-08-03 20:14:22 +01:00
gadgethd 1464e94075 feat(frontend): render canonical blue paths 2026-08-03 15:12:56 +01:00