- clientErrors.ts: window.onerror/unhandledrejection for BOTH domains via main.tsx
(was app-only in App.tsx); resource-failure classification; telemetry=off gate;
console.error hook (kind warning) with terrain-tiles noise filter; page field
now pathname-only (no query strings)
- PrivacyPage at /privacy linked from footer + cookie banner; docs/privacy.md LIA
- retention: packet_decryptions added to row-table retention targets; packets
180d->30d in lifecycle policies; migration 050 schedules timescaledb retention
jobs (packets 30d, node_status 180d, node_neighbor 7d) that never existed
- migration 050: packet_paths hypertable (content-stripped path store, no
retention) + backfill of 2.56M path-bearing packets + ingest dual-write
* fix(owner): Wave 3 mediums — password handling, deploy rollback, newuser races, session revocation
- BUG-017: stop trimming MQTT passwords (frontend + backend). A valid
broker password may begin/end with whitespace; trimming made it
permanently unauthenticatable. Username trimming unchanged.
- BUG-013: deploy-website.sh now set -Eeuo pipefail with an EXIT rollback
trap armed only after the pin mutates; container-down and bundle-mismatch
paths fail loudly and restore the old pin + verify the restored service.
- BUG-012: newuser.sh installs an EXIT trap (ERR does not fire on explicit
exit 1 from die()) so every post-mutation failure rolls back; rollback is
idempotent and a no-op before any mutation flag. Disarm points clear EXIT.
- BUG-011: newuser.sh re-reads + re-validates OWNER_MQTT_USERNAME_MAP under
the re-acquired lock after the long unlocked discovery window, then merges
the new grant into the current map — concurrent provisioning runs can no
longer be overwritten by a stale snapshot.
- BUG-010: owner sessions are now v3 cookies carrying a credential
generation (Redis-backed). When the broker rejects a previously-valid
password (revocation detected), the generation bumps and every older
session is rejected on the next request. TTL shortened 30d -> 7d.
Verified: tsc clean, 310/310 backend tests pass, bash -n on both scripts.
* fix(alert-receiver): durable alert delivery with bounded retry + dead-letter (BUG-014)
Forwarding was fire-and-forget: failures logged asynchronously after HTTP 202,
/healthz stayed green, and alerts could be archived to the local JSONL while
operators never saw them. Now:
- Every receipt is enqueued for delivery with bounded exponential backoff
(ALERT_FORWARD_MAX_ATTEMPTS=5, base 1s, cap 60s) and dead-lettered to
receipts.jsonl.dead after exhausting attempts.
- /healthz keeps returning 200 (compose wget healthcheck must not restart
the container) but the body reports degraded status + detail; new /readyz
returns 503 when ALERT_FORWARD_URL is unset (archive-only), no successful
forward since startup, or alerts stuck undelivered >5min.
Verified: tsc clean, 310/310 tests pass.
* fix(ci): classify reviewed public channel keys + fixtures in gitleaks (BUG-009)
The nightly full-history secret scan flagged 42 findings spanning the
documented community channel keys (VALIDATED_CHANNELS — intentionally
public, each verified to decrypt real UK Mesh group text), fake test
fixtures, and a deployed-commit SHA in the website live manifests. That
made a genuine credential easy to dismiss among expected hits.
- Rule-scoped allowlists with exact fingerprints (regexTarget: secret,
anchored full-value matches) for the Public channel key, test fixtures
(0123456789... / abcdef0123...), and the manifest commit SHA.
- Structure-exact line allowlist for channelRegistry.ts VALIDATED_CHANNELS.
- Verified locally with gitleaks 8.24.3 full-history scan: 42 -> 0 findings.
* fix(viewshed): side-effect completion markers prevent skipped link jobs (BUG-006)
store_coverage() commits on an autocommit connection, then the worker queues
physical-link jobs and publishes Redis notifications. A Redis failure after
the DB commit NACKed the job; on retry already_calculated() saw the coverage
row and returned early — link work and frontend notifications were skipped
forever.
- Record a Redis completion marker (viewshed:side-effects:<node>) only after
EVERY side effect succeeds.
- On the already_calculated early return, a missing marker triggers an
idempotent replay: link jobs are re-enqueued from the stored node position
(admission is idempotent) and coverage_update/node_upsert notifications are
re-published from the stored coverage row.
- Redis read failures are treated as incomplete (replay attempt re-raises and
NACKs rather than silently skipping).
- Added tests/test_side_effect_markers.py (5 tests) with a conftest that stubs
osgeo/psycopg2 so pure-logic worker tests run without GDAL (CI keeps real
GDAL via setdefault). Verified: 5/5 new tests pass; full suite 34 passed,
2 GDAL-required terrain tests fail only in stub env (pass in CI image).
---------
Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>
* fix(ci): pin gitleaks-action to valid commit after upstream force-push
Upstream deleted dcedce43 (force-push), so every ci.yml run fails at
workflow-parse with 0 jobs. The dependabot actions-group bump (#30)
contains this fix but cannot merge whole: its docker/* actions require
node24, which GitHub runners do not support yet. Pin gitleaks alone
(ff98106e is node20).
* fix(ci): downgrade v7 actions to node20-compatible versions
actions/checkout@v7, setup-node@v7, setup-python@v7 require node24,
which GitHub-hosted runners do not support yet — every ci.yml run has
failed at workflow-parse (0 jobs) since the Aug 3 v7 bump. Pin to the
latest node20 versions (checkout v4, setup-node v4, setup-python v5).
Also pins gitleaks-action to a valid commit (upstream force-pushed away
dcedce43).
* ci: noop retrigger
* fix(ci): escape literal ${{ in bash string so GitHub parser accepts workflow
The 'Validate tracked build and Compose inventory' step matched literal
${{ inside run-block strings, which GitHub's expression parser reads as
the start of an expression (${{'* is invalid) — the workflow fails at
parse time with 0 jobs. Split the literal as '$''{{' (bash concatenates
adjacent quoted strings at runtime; the source no longer contains a
contiguous ${{).
* fix(ci): resolve all three pre-existing ukmesh CI failures
Backend (packetBatchWriteCoalesce):
- 045: restore nodes_public_visibility_generation trigger — 015 is
superseded by 044 on fresh DBs so the trigger was never created,
leaving generation/visibility_generation out of sync and public
packet reads returning empty.
- 046: restore packet privacy classification — the 042 fence design
assumes sync_private_node_prefixes rewrites packets on privacy
change, but 026's version only maintains prefixes. Node flips to
private left old packets visible; direct SQL inserts were never
classified. Restore the packet rewrite in the sync trigger and add
a BEFORE INSERT trigger mirroring the batch path's is_private /
visibility_ok computation.
- packetBatch.integration.test.ts: exclude the prefix-cache refresh
query from statementCount (matches unit-test convention).
Frontend e2e:
- public.spec.ts: assert the TopologyMap component's actual labels
('Geographic repeater topology map', '2 mapped repeaters · 2
observed relationships') instead of the pre-map SVG graph labels.
Workers (pip install):
- viewshed-worker: bump numpy 1.26.4→2.3.5, scipy 1.13.1→1.16.3,
psycopg2-binary 2.9.10→2.9.11 (cp314 wheels); the base gdal image
ships Python 3.14 so the old pins had no wheels. Inherit shapely
from the image's python3-shapely apt package (no cp314 wheel
exists; source builds are GEOS 3.14-incompatible).
* fix(ci): restore data-plane services for compose validation and smoke test
Commit 8c5e1c8 split the long-lived data-plane services (timescaledb,
mosquitto, redis, mosquitto-reloader) into the external meshcore-infra
project on the live host, but the Workers-and-Compose CI job still
asserted their presence in the app compose project (max_worker_processes
check, inventory check, smoke-test execs). CI had been broken since
Aug 3 so the mismatch was never caught.
Add docker-compose.ci.yml — a CI-only overlay restoring the four
data-plane services from their pre-split definitions — and point every
compose invocation in the Workers-and-Compose job at
'-f docker-compose.yml -f docker-compose.ci.yml'.
Validated locally: merged config parses, the jq assertions and the full
inventory (db-migrate backend app-ukmesh website-ukmesh website-dev
mesh-health-check mosquitto-reloader link-worker link-backfill-worker
hopreach timescaledb mosquitto) all pass.
* fix(ci): include dev profile in compose inventory check
website-dev is a dev-profile service; the inventory gate compared the
base-config service list, so the merged stack never matched. This gate
has been red since the workflow was first written (CI parse-broken from
Aug 3 until the gitleaks pin landed).
* fix(ci): db-migrate must wait for timescaledb health
The Aug 9 data-plane split removed db-migrate's depends_on along with
the timescaledb service definition, so the migration runner raced the
fresh database init in the empty-volume smoke test and failed on the
not-yet-created base schema. Restore the pre-split condition:
timescaledb service_healthy.
* fix(ci): restore internal TCP MQTT listener for smoke test + exporter
Commit 40e843f dropped the 1883 TCP listener, breaking the CI smoke
(mosquitto_pub targets 1883) and the Prometheus mosquitto exporter.
The listener is internal-only: compose never publishes 1883 on the
host, and the password/ACL policy still applies.
---------
Co-authored-by: gadgethd <111318106+gadgethd@users.noreply.github.com>
The cold WS initial-state fetch (fetchInitialState -> getRecentPackets +
getRecentMessages) called meshcore_canonical_node_id() ~100k times per
fetch (per-row subquery against node_identity_aliases). 6.97s cold on the
2-vCPU box; under load it exceeded the 8s WS_INITIAL_STATE_DB_TIMEOUT_MS
abort, so the map never received an initial snapshot (empty map +
synthetic_check_failed). Replace the per-row function calls with
node_identity_aliases LEFT JOINs (COALESCE(alias.canonical_node_id,
upper(btrim(node_id))) - exact function semantics, table ~20 rows).
Measured: full getRecentMessages 6968ms -> 1772ms (4x); getRecentPackets
same pattern. Also raise the WS_INITIAL_STATE_DB_TIMEOUT_MS env cap
9s -> 60s (default stays 8s; deploy sets 30s) so slow-but-working fetches
complete and warm the cache instead of aborting forever.
281/281 backend tests pass; tsc clean. Live: initial_state 502ms warm,
all synthetic checks green.
Root cause: node_identity_nodes was estimated at one scoped row, so the combined two-endpoint filter formed a nested-loop Cartesian search and issued roughly 174 million composite link-index probes. The 14-row radio-report table was not the bottleneck.
Split the scoped-node and viable-link reads, intersect endpoints in memory, and preserve the existing all-time neighbor SUM/MAX fields. Thread an 8s AbortSignal through every initial-state database read so a pathological cold fetch releases its gate slot before the 10s synthetic deadline.
Evidence:
- before EXPLAIN (plan only): Nested Loop with node_identity_links_pkey and net_nodes rows=1
- fixed EXPLAIN ANALYZE (ARRAY['ukmesh']): 258.325ms + 61.944ms = 320.269ms server time; 0.52s psql wall
- production getViableLinks('ukmesh'): 20,448 rows in 312.1ms
- semantics check: 20,448 links; neighbor rows=11, reports=15, best SNR=12 dB
- npm test: 280 passed, 0 failed
- npx tsc --noEmit: PASS
The hourly rollup rebuild was filtering source rows with
PUBLIC_PACKET_PRIVACY_SQL, so a backfill reproduced the public-only
rollup (~26k/24h) even after the incremental writer (packetBatch.ts)
was changed to count all rows (~101k/24h). The tool's hourly apply
slice + dry-run + catchUpCurrentHour now have NO privacy filter,
matching the runtime writer. The daily max-hop and observer-region
phases KEEP the privacy filter - their runtime writers still filter.
The hourly/daily chart series come from packet_hourly_stats, whose
incremental writer (packetBatch.ts) filtered WHERE path_is_valid AND
NOT is_private - so the charts only ever showed public-valid rows
(~26k/24h) while the totals now count all observations (~101k/24h).
Per Ben's spec (every packet received by every observer counts), the
rollup must include private rows too.
Changes:
- packetBatch.ts: rollup INSERT drops the path_is_valid/is_private
filter (keeps the meshcore-test topic exclusion) - all rows counted
- statsRepository.ts: aggregate raw-slice CTEs (fresh hour + rollup
gap) use totalFilters; legacy/observer chart series (perHour,
perDay, types, hops, routes, transports) use totalFilters; region
summary counts all rows
- rollup history will be rebuilt via stats:backfill-rollups (the
backfill tool already counts all rows)
Verified: totals ~101k/24h; charts will sum to the same after backfill.
Per Ben's spec the "Observed packets" totals must count EVERY packet
received by EVERY observer - duplicate receptions of the same packet by
multiple stations each count. Reverts the COUNT(DISTINCT packet_hash)
semantics from 8dff7d7 and additionally drops the privacy visibility
filter from the totals so they match the chart rollup path (which has
never been privacy-filtered - aggregateScope filters by network only).
Changes:
- packetsDay, totalPackets24h, totalPackets7d: COUNT(*) over all rows
(network/test-topic scoped, rx_node_id present), no visibility filter
- channelTraffic: COUNT(*) all rows (percentages stay consistent with
the all-rows total denominator)
- networkFilters(): new includePrivacy option (default true) so total
queries can opt out of visibility conditions
- legacy/observer-scoped chart series back to COUNT(*) (row semantics
everywhere; transportCodes alias fix from 40be809 retained)
- tests: includePrivacy coverage added
Verified live: 24h total ~99k rows (vs 25k privacy-filtered, vs 5.8k
distinct) - consistent with the hourly chart series which already
counted all rows.
The packets table stores one row per observer per packet, so COUNT(*)
overcounted by the average observation multiplicity (~4.3x public, ~9.5x
overall). A packet heard by 5 stations counted 5 times.
Converted to COUNT(DISTINCT packet_hash) on every surface that presents a
"packets" number:
- /api/stats summary packetsDay (app + home dashboard card)
- charts summary totalPackets24h / totalPackets7d (StatsPage cards)
- legacy/observer-scoped chart series: packetsPerHour, packetsPerDay,
packetTypes, hopDistribution, routeTypes, transportCodes
- channelTraffic channel counts (keeps allPct consistent with the
distinct denominator)
The canonical rollup path (packet_hourly_stats, incrementally maintained)
remains row-based - it cannot maintain distinct counts incrementally and
stays as the trend-shape source for the public charts.
Verified: public distinct 24h = 5,835 vs 25,236 public rows vs 99,079
total rows (74% of traffic is from privacy-marked nodes).
df7cfbe migrated listNodeLinks to node_identity_nodes but kept the
un-aliased filters.nodes fragment, whose EXISTS subquery references
nodes.node_id with no nodes table in scope, so every GET /nodes/:id/links
500'd (missing FROM-clause entry for table nodes) and the repeater page
showed no simulated neighbours for any node. Use filters.nodesAlias like
the sibling identity queries do.
- topic.ts/brokerLog.ts/client.ts: accept the neighbours suffix (official
MeshCore firmware publishes the UK spelling; our MQTT fork uses neighbors)
- aclManager.ts: render + verify BOTH spellings in owner ACL grants (ukmesh
and test scopes); inventoryOwnerAuthorization regex updated to match
- statsService.ts: chart bucket labels are now machine-readable ISO instead
of server-timezone HH:MM/en-GB strings (backend container runs UTC;
viewers saw UTC hours)
- StatsPage.tsx: axis ticks + tooltips + summaries formatted via
statsTimeFormat.ts (Intl local timezone, raw fallback for legacy cached
snapshots) + unit tests
gadgethd