Commit Graph
27 Commits
Author SHA1 Message Date
gadgethd 1ebc496965 Map UI redesign, live-path visibility, feed latency, and security hardening (#19)
* Fix map node freshness consistency

* Harden output, ingest, caches, and WebSocket limits

* Enforce public visibility across derived data

* Harden proxy and operator deployment boundary

* Make owner grants authoritative and reconcile ACLs safely

* Bound path, spam, and statistics analysis

* Make link and coverage jobs crash-safe

* Implement strategic security remediation

* Fix production cutover configuration

* Fix disabled viewshed worker health signal

* Serve stale stats during background refresh

* Retain stale stats through refresh windows

* Bound analytics work to protect ingestion

* Prioritize summary warmup over chart scans

* Throttle path history rebuilds

* Bound path history result memory

* Stream path history aggregation

* Give bounded path rebuild one CPU

* Serve stale charts during bounded refresh

* Prioritize startup stats before chart scans

* Bound path history segment cardinality

* Pin path rebuild context to privacy generation

* Self-host original frontend fonts

* Allow bounded path rebuild to complete

* Improve live map UI and low-latency group feed

- Dock node details on the right with selection highlight and collapsible layers
- Add node legend, 24h activity sparkline, copy-link, and layout/overlap fixes
- Keep all repeaters visible during Live Path focus
- Send GroupText feed packets immediately over WebSocket (no batch delay)
- Cache expensive stats/observer activity more aggressively to protect ingest
- Remove stale local planning/audit markdown from the tree

* fix(ci): supply OPERATOR_SITE_TOKEN for compose validation

Workers/Compose CI failed because docker-compose requires
OPERATOR_SITE_TOKEN. Add CI placeholders for that and MQTT_PASSWORD.
2026-07-27 02:39:12 +01:00
gadgethd 93cc3e1b38 Expand network intelligence and harden operations (#9)
Consolidates the stacked backend, privacy, network-intelligence, frontend, operations, mobile, and owner-cache changes after resolving main conflicts and passing the full CI suite.
2026-07-23 20:50:47 +01:00
BenandClaude Sonnet 4.6 edc8206a22 Add ML path learner, Anubis bot protection, planned coverage, companion page, and misc improvements
- ml-path-learner: new Python worker that trains a prefix→node ML model from gold paths and writes scores to ml_path_prefix_scores
- lazyResolver: integrate ML scores and edge priors into tiered candidate ranking; bidirectional pass-2 propagation; global cross-group direct anchors
- Anubis: add sidecar bot-protection containers for all public HTTP services; botPolicy.yaml
- Planned coverage: new API route + frontend map layers for placing hypothetical repeaters and computing coverage
- Map: light/dark theme toggle; map-tools button group (LOS, Repeater, theme); plan-repeater mode with click-to-place/remove and polling
- Frontend: remove inferred nodes from map display (backend inference kept for internal use)
- Stats page: observer region summary, channel traffic, companion activity endpoints and UI
- UKCompanionPage: new companion activity page
- Teesside site: extracted into standalone build context (teesside-site/)
- healthcheck overrides: share.html, sw.js, region filter support, SVG score ring updates
- DB: dedicated analytics pool; DATABASE_SKIP_SCHEMA_INIT flag; path hash prefix indexes; lateral join fixes for node_link_radio_reports
- docker-compose: ml-path-learner service; anubis sidecars; DATABASE_SKIP_SCHEMA_INIT on all workers; OBSERVER_RETENTION_SECONDS; REGIONS_FILE
- backend-site: internal operator dashboard routes
- docker/mesh-health-check-entrypoint.sh: extract channel secret from MESHCORE_CHANNEL_SECRETS at startup
- scripts: observer key generation, observer registration, healthcheck tunnel check
- vacuum-compressed-chunks.sh: maintenance script

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-03 01:33:48 +00:00
Ben 4bf515b735 Switch mesh-health-check build to gadgethd fork, track main branch 2026-04-13 22:32:57 +00:00
BenandClaude Sonnet 4.6 a0622c9861 Replace hardcoded terrain tiles path with env var default
Use ${TERRAIN_TILES_PATH:-./terrain-tiles} so the path is configurable
per-host without leaking machine-specific paths in the repo.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 01:22:34 +00:00
BenandClaude Sonnet 4.6 587093d48a Add multi-node LOS overlay with terrain profile and custom LOS tool
- Multi-node LOS: multiple repeaters can have LOS active simultaneously,
  each auto-expiring after 15 seconds; LOS persists after popup closes
- Custom LOS tool: desktop-only map button to sample terrain between two
  points (click map or repeater); renders per-segment green/clear or
  red/obstructed based on terrain height; prohibited nodes are excluded
- Terrain exaggeration: deck.gl altitudes multiplied by exaggeration factor
  (2×) to match MapLibre's visual terrain mesh
- LOS line rendering: bloom+core PathLayer tube effect with ScatterplotLayer
  endpoint markers; popup z-index fix so it renders above deck.gl canvas
- Terrain sampler: client-side Terrarium PNG tile decoder for elevation sampling
- DB schema: terrain_profile_json JSONB column added to node_links
- Nginx: terrain tile serving block added
- Docker: terrain tiles volume mount added

Excluded: PacketDetailPanel, UKFeedPage, styles/globals.css (unrelated local
changes), backfill_profiles.py (server-side utility, not intended for repo)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 01:15:39 +00:00
BenandClaude Sonnet 4.6 f8e8717454 Add lazy path resolver, ACL watcher, feed map, and observability stack
- Lazy path resolver: geographic hop reconstruction from path_hashes without
  pre-training data; fixes observer self-hash trimming bug
- ACL watcher worker: auto-provisions Mosquitto ACL entries when PUBLISH is
  denied for a known user; eliminates the recurring gnomemoe ACL removal issue
- connectionMonitor: increase historical scan to 50 MB; fix OOM by streaming
  lines instead of buffering; fixes missed backfill on restart
- Feed page: replace path-beta with lazy paths; add clickable node markers
  showing name and public key; observer column nowrap; summary text unclipped
- PacketFeed: GRP-only message store protected from ADV eviction; scroll to
  bottom on new message; 24-hour message backfill on initial load
- getRecentMessages: separate DB query for last 50 GRP messages last 24h
- getPacketDetail: new endpoint for full packet detail including observations
- Grafana/Loki/Promtail observability stack with Docker log collection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-22 23:28:57 +00:00
BenandClaude Sonnet 4.6 490171c16b Auto-provision Mosquitto ACL and refresh owner session from DB
- New aclManager.ts rewrites a user's ACL block (all linked nodes) and
  sends SIGHUP to the Mosquitto container whenever autoLinkOwnerNodeIds
  runs at owner login, eliminating the need to manually edit the ACL
  file when a user connects a second node.
- requireOwnerSession now resolves node IDs from owner_account_nodes
  on every request instead of using the frozen session cookie list,
  so manually- or auto-linked nodes are visible without re-login.
- docker-compose: mount mosquitto config dir (rw) and Docker socket
  into backend so it can update the ACL file and signal Mosquitto.
- Add dockerode dependency for container SIGHUP signalling.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-22 19:18:55 +00:00
Ben 7d57943b60 Improve owner dashboard telemetry and bare-domain support
- narrow owner last-hop telemetry to the selected MQTT node and cache it per node with hourly background refresh
- tighten repeater-only last-hop classification and clean up the owner chart selector/filtering
- fix owner direct-sender map basemap, static behavior, and viewport fitting/centering
- allow bare ukmesh.com and teessidemesh.com host handling in backend/frontend config
2026-03-21 00:16:21 +00:00
Ben 812aa95837 Tune RF link model and fix map stale link nodes 2026-03-20 00:41:02 +00:00
Ben 8db9a71e43 Rewrite map stack and tighten path topology 2026-03-19 22:42:35 +00:00
BenandClaude Sonnet 4.6 acd71b0913 Fix owner portal login; add tile worker and cross-network improvements
Owner portal fixes:
- Fix session node ID case mismatch: decryptOwnerSession was lowercasing
  node IDs but /owner/live uppercased the ?nodeId= param, so the find()
  never matched — users always got "Node is not owned by this session"
- Revert sameSite strict → lax (broke sessions after March 18 deploy)
- Remove LOWER() wrappers from all owner live SQL (node IDs are uppercase)
- Add Fireball's second node (TS17 Yagi North) to owner_account_nodes

Tile worker:
- Server-side PNG tile rendering via @napi-rs/canvas
- Tile snapshot cache, Redis-backed tile cache, tile worker service
- /api/tiles/nodes/:z/:x/:y.png endpoint with rate limiter
- UK tile TTL logic

Cross-network connectivity:
- Add 7-day hourly history series to /cross-network-connectivity response
- Cache result for 60s to avoid repeated join-heavy query
- Fix query to use literal interval instead of interpolated variable

Other:
- Remove LOWER() from packet/node SQL throughout (IDs stored uppercase)
- Various performance and cleanup improvements

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 15:55:47 +00:00
BenandClaude Sonnet 4.6 caaa45c5d6 Remove private radio bot URL from public config defaults
RADIO_BOT_URL is a private integration — strip the default service name
from docker-compose and remove it from the README env table.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 02:09:09 +00:00
BenandClaude Sonnet 4.6 0c5c97df39 Add Teesside Mesh dashboard and path-beta worker improvements
- New www.teessidemesh.com single-page dashboard (TeessieDashboard.tsx)
  with amber/charcoal theme, connectivity status, repeater cards with
  7-day battery sparklines, MQTT observer grid, and decoded packet feed
- Backend: cross-network connectivity endpoint (hop-count packet matching),
  radio-stats and radio-history proxy endpoints, observer-activity endpoint,
  nodesDay stat field
- docker-compose: website-teesside service on port 3002
- Path-beta: worker pool, resolve cache, and pool refactor for performance
- Connection monitor, WS server and MQTT client improvements
- Mosquitto config, viewshed worker, and frontend UI polish

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 02:01:54 +00:00
Ben ebb8fe55b4 Ship path history, site consolidation, and telemetry fixes 2026-03-12 23:12:33 +00:00
Ben f294b96592 Remove path simulation workers 2026-03-08 12:53:44 +00:00
BenandClaude Opus 4.6 b9a231ef36 Dedup relay copies per observer and increase DB connection limit
Drop duplicate MQTT packets when the same packet hash arrives at the
same observer with the same hop count within 120s. Increase TimescaleDB
max_connections to 300 to prevent connection exhaustion on startup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 12:26:09 +00:00
Ben a7503ef3f2 Refine coverage, owner links, and UK feed 2026-03-08 01:16:58 +00:00
Ben 41c01bb70b Add isolated test feed and fix live public stats 2026-03-07 21:27:59 +00:00
Ben 05514fdc18 Add owner auth DB, website refresh, and coverage rollback 2026-03-07 03:23:52 +00:00
Ben 729a966e6a Add server-side beta path resolver and mobile layer controls
- move beta path resolution onto the backend with packet-level diagnostics and live API access
- add path simulation worker plumbing, health/schema updates, and compose wiring
- update app overlays to fetch server-computed beta paths with animated rendering and request caching
- refine mobile controls so map layers collapse behind a toggle and compact the links legend
2026-03-07 00:42:31 +00:00
Ben 0d77a5027c feat: update owner auth and refresh stats visualizations 2026-03-05 23:05:29 +00:00
Ben 0e365479a6 Split workers, fix frontend proxy resilience, expand health UX/docs 2026-03-05 00:39:44 +00:00
BenandClaude Sonnet 4.6 8865a6444d Add ukmesh.com frontend and MQTT infrastructure
www.ukmesh.com (website-ukmesh container, port 3004):
- UKLayout, UKHomePage, UKInstallPage, UKMqttPage components
- VITE_SITE=ukmesh build arg routes to UK pages in main.tsx
- Stats fetched via relative /api path (proxied by Nginx)

app.ukmesh.com (app-ukmesh container, port 3003):
- Same dashboard build with VITE_APP_HOSTNAME=app.ukmesh.com
- VITE_NETWORK=ukmesh for future API filtering

MQTT:
- node-ukmesh Mosquitto user for ukmesh observer nodes
- ACL updated: backend reads meshcore/# and ukmesh/#
- ukmesh nodes publish to ukmesh/{IATA}/{PUBKEY}/packets

Cloudflare routes to add in Zero Trust:
  www.ukmesh.com  -> http://website-ukmesh:80
  app.ukmesh.com  -> http://app-ukmesh:80
  mqtt.ukmesh.com -> http://mosquitto:9001

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-04 11:16:26 +00:00
BenandClaude Sonnet 4.6 a946a07488 Separate frontend and backend into distinct containers
Splits the monolithic app container into:
- backend: Node.js handling MQTT, WebSocket, API, DB (no static files)
- app: Nginx serving the React SPA, proxying /api and /ws to backend

This means future dashboards (app.ukmesh.com etc.) can be added as
additional Nginx containers pointing at the same backend with no
duplicate calculations or DB connections.

Also updates nginx.website.conf to proxy to backend instead of app,
and bumps website host port from 3001 to 3002 to avoid conflict.

NOTE: Cloudflare Zero Trust tunnel route for app.teessidemesh.com
must be updated from http://app:3000 to http://app:80.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-04 10:58:23 +00:00
Ben 521731bd78 Fix website container missing VITE_APP_HOSTNAME build arg 2026-03-03 19:19:25 +00:00
Ben cb713d6c1a Initial commit 2026-03-03 19:11:05 +00:00