The cold WS initial-state fetch (fetchInitialState -> getRecentPackets +
getRecentMessages) called meshcore_canonical_node_id() ~100k times per
fetch (per-row subquery against node_identity_aliases). 6.97s cold on the
2-vCPU box; under load it exceeded the 8s WS_INITIAL_STATE_DB_TIMEOUT_MS
abort, so the map never received an initial snapshot (empty map +
synthetic_check_failed). Replace the per-row function calls with
node_identity_aliases LEFT JOINs (COALESCE(alias.canonical_node_id,
upper(btrim(node_id))) - exact function semantics, table ~20 rows).
Measured: full getRecentMessages 6968ms -> 1772ms (4x); getRecentPackets
same pattern. Also raise the WS_INITIAL_STATE_DB_TIMEOUT_MS env cap
9s -> 60s (default stays 8s; deploy sets 30s) so slow-but-working fetches
complete and warm the cache instead of aborting forever.
281/281 backend tests pass; tsc clean. Live: initial_state 502ms warm,
all synthetic checks green.
Root cause: node_identity_nodes was estimated at one scoped row, so the combined two-endpoint filter formed a nested-loop Cartesian search and issued roughly 174 million composite link-index probes. The 14-row radio-report table was not the bottleneck.
Split the scoped-node and viable-link reads, intersect endpoints in memory, and preserve the existing all-time neighbor SUM/MAX fields. Thread an 8s AbortSignal through every initial-state database read so a pathological cold fetch releases its gate slot before the 10s synthetic deadline.
Evidence:
- before EXPLAIN (plan only): Nested Loop with node_identity_links_pkey and net_nodes rows=1
- fixed EXPLAIN ANALYZE (ARRAY['ukmesh']): 258.325ms + 61.944ms = 320.269ms server time; 0.52s psql wall
- production getViableLinks('ukmesh'): 20,448 rows in 312.1ms
- semantics check: 20,448 links; neighbor rows=11, reports=15, best SNR=12 dB
- npm test: 280 passed, 0 failed
- npx tsc --noEmit: PASS
The hourly rollup rebuild was filtering source rows with
PUBLIC_PACKET_PRIVACY_SQL, so a backfill reproduced the public-only
rollup (~26k/24h) even after the incremental writer (packetBatch.ts)
was changed to count all rows (~101k/24h). The tool's hourly apply
slice + dry-run + catchUpCurrentHour now have NO privacy filter,
matching the runtime writer. The daily max-hop and observer-region
phases KEEP the privacy filter - their runtime writers still filter.
The hourly/daily chart series come from packet_hourly_stats, whose
incremental writer (packetBatch.ts) filtered WHERE path_is_valid AND
NOT is_private - so the charts only ever showed public-valid rows
(~26k/24h) while the totals now count all observations (~101k/24h).
Per Ben's spec (every packet received by every observer counts), the
rollup must include private rows too.
Changes:
- packetBatch.ts: rollup INSERT drops the path_is_valid/is_private
filter (keeps the meshcore-test topic exclusion) - all rows counted
- statsRepository.ts: aggregate raw-slice CTEs (fresh hour + rollup
gap) use totalFilters; legacy/observer chart series (perHour,
perDay, types, hops, routes, transports) use totalFilters; region
summary counts all rows
- rollup history will be rebuilt via stats:backfill-rollups (the
backfill tool already counts all rows)
Verified: totals ~101k/24h; charts will sum to the same after backfill.
Per Ben's spec the "Observed packets" totals must count EVERY packet
received by EVERY observer - duplicate receptions of the same packet by
multiple stations each count. Reverts the COUNT(DISTINCT packet_hash)
semantics from 8dff7d7 and additionally drops the privacy visibility
filter from the totals so they match the chart rollup path (which has
never been privacy-filtered - aggregateScope filters by network only).
Changes:
- packetsDay, totalPackets24h, totalPackets7d: COUNT(*) over all rows
(network/test-topic scoped, rx_node_id present), no visibility filter
- channelTraffic: COUNT(*) all rows (percentages stay consistent with
the all-rows total denominator)
- networkFilters(): new includePrivacy option (default true) so total
queries can opt out of visibility conditions
- legacy/observer-scoped chart series back to COUNT(*) (row semantics
everywhere; transportCodes alias fix from 40be809 retained)
- tests: includePrivacy coverage added
Verified live: 24h total ~99k rows (vs 25k privacy-filtered, vs 5.8k
distinct) - consistent with the hourly chart series which already
counted all rows.
The packets table stores one row per observer per packet, so COUNT(*)
overcounted by the average observation multiplicity (~4.3x public, ~9.5x
overall). A packet heard by 5 stations counted 5 times.
Converted to COUNT(DISTINCT packet_hash) on every surface that presents a
"packets" number:
- /api/stats summary packetsDay (app + home dashboard card)
- charts summary totalPackets24h / totalPackets7d (StatsPage cards)
- legacy/observer-scoped chart series: packetsPerHour, packetsPerDay,
packetTypes, hopDistribution, routeTypes, transportCodes
- channelTraffic channel counts (keeps allPct consistent with the
distinct denominator)
The canonical rollup path (packet_hourly_stats, incrementally maintained)
remains row-based - it cannot maintain distinct counts incrementally and
stays as the trend-shape source for the public charts.
Verified: public distinct 24h = 5,835 vs 25,236 public rows vs 99,079
total rows (74% of traffic is from privacy-marked nodes).
df7cfbe migrated listNodeLinks to node_identity_nodes but kept the
un-aliased filters.nodes fragment, whose EXISTS subquery references
nodes.node_id with no nodes table in scope, so every GET /nodes/:id/links
500'd (missing FROM-clause entry for table nodes) and the repeater page
showed no simulated neighbours for any node. Use filters.nodesAlias like
the sibling identity queries do.
- topic.ts/brokerLog.ts/client.ts: accept the neighbours suffix (official
MeshCore firmware publishes the UK spelling; our MQTT fork uses neighbors)
- aclManager.ts: render + verify BOTH spellings in owner ACL grants (ukmesh
and test scopes); inventoryOwnerAuthorization regex updated to match
- statsService.ts: chart bucket labels are now machine-readable ISO instead
of server-timezone HH:MM/en-GB strings (backend container runs UTC;
viewers saw UTC hours)
- StatsPage.tsx: axis ticks + tooltips + summaries formatted via
statsTimeFormat.ts (Intl local timezone, raw fallback for legacy cached
snapshots) + unit tests
The planner inlined the view CTEs into consumers and re-ran the links
aggregation once per candidate row (loops=13070, 24s+ for getViableLinks).
Wrapping each view body in WITH v AS MATERIALIZED forces one evaluation
per query: viable-links query 24.3s -> 3.1s, WS initial state <1s.
Also drops the dead terrain_profile_json column from node_identity_links
(no readers; frontend terrain comes from DEM tiles).
- channelRegistry: 39 validated channel secrets committed as the service
default (Public + 38 community/hashtag channels recovered via the
derivation audit); MESHCORE_CHANNEL_SECRETS env appends on top
- shared buildSummary/identifyChannel (single implementation for ingest
and offline tools)
- tools/backfillDecrypt: keyset-paginated retroactive decryption of the
stored raw_hex corpus (idempotent, resumable)
- feed: channel scope sidebar now lists every decrypted channel
gadgethd