The hourly rollup rebuild was filtering source rows with
PUBLIC_PACKET_PRIVACY_SQL, so a backfill reproduced the public-only
rollup (~26k/24h) even after the incremental writer (packetBatch.ts)
was changed to count all rows (~101k/24h). The tool's hourly apply
slice + dry-run + catchUpCurrentHour now have NO privacy filter,
matching the runtime writer. The daily max-hop and observer-region
phases KEEP the privacy filter - their runtime writers still filter.
The hourly/daily chart series come from packet_hourly_stats, whose
incremental writer (packetBatch.ts) filtered WHERE path_is_valid AND
NOT is_private - so the charts only ever showed public-valid rows
(~26k/24h) while the totals now count all observations (~101k/24h).
Per Ben's spec (every packet received by every observer counts), the
rollup must include private rows too.
Changes:
- packetBatch.ts: rollup INSERT drops the path_is_valid/is_private
filter (keeps the meshcore-test topic exclusion) - all rows counted
- statsRepository.ts: aggregate raw-slice CTEs (fresh hour + rollup
gap) use totalFilters; legacy/observer chart series (perHour,
perDay, types, hops, routes, transports) use totalFilters; region
summary counts all rows
- rollup history will be rebuilt via stats:backfill-rollups (the
backfill tool already counts all rows)
Verified: totals ~101k/24h; charts will sum to the same after backfill.
Per Ben's spec the "Observed packets" totals must count EVERY packet
received by EVERY observer - duplicate receptions of the same packet by
multiple stations each count. Reverts the COUNT(DISTINCT packet_hash)
semantics from 8dff7d7 and additionally drops the privacy visibility
filter from the totals so they match the chart rollup path (which has
never been privacy-filtered - aggregateScope filters by network only).
Changes:
- packetsDay, totalPackets24h, totalPackets7d: COUNT(*) over all rows
(network/test-topic scoped, rx_node_id present), no visibility filter
- channelTraffic: COUNT(*) all rows (percentages stay consistent with
the all-rows total denominator)
- networkFilters(): new includePrivacy option (default true) so total
queries can opt out of visibility conditions
- legacy/observer-scoped chart series back to COUNT(*) (row semantics
everywhere; transportCodes alias fix from 40be809 retained)
- tests: includePrivacy coverage added
Verified live: 24h total ~99k rows (vs 25k privacy-filtered, vs 5.8k
distinct) - consistent with the hourly chart series which already
counted all rows.
The packets table stores one row per observer per packet, so COUNT(*)
overcounted by the average observation multiplicity (~4.3x public, ~9.5x
overall). A packet heard by 5 stations counted 5 times.
Converted to COUNT(DISTINCT packet_hash) on every surface that presents a
"packets" number:
- /api/stats summary packetsDay (app + home dashboard card)
- charts summary totalPackets24h / totalPackets7d (StatsPage cards)
- legacy/observer-scoped chart series: packetsPerHour, packetsPerDay,
packetTypes, hopDistribution, routeTypes, transportCodes
- channelTraffic channel counts (keeps allPct consistent with the
distinct denominator)
The canonical rollup path (packet_hourly_stats, incrementally maintained)
remains row-based - it cannot maintain distinct counts incrementally and
stays as the trend-shape source for the public charts.
Verified: public distinct 24h = 5,835 vs 25,236 public rows vs 99,079
total rows (74% of traffic is from privacy-marked nodes).
df7cfbe migrated listNodeLinks to node_identity_nodes but kept the
un-aliased filters.nodes fragment, whose EXISTS subquery references
nodes.node_id with no nodes table in scope, so every GET /nodes/:id/links
500'd (missing FROM-clause entry for table nodes) and the repeater page
showed no simulated neighbours for any node. Use filters.nodesAlias like
the sibling identity queries do.
The overlay still carried pre-migration 172.18.30.x addresses; the VPS
meshcore-analytics_default bridge is 172.30.0.0/24 (containers already run
on 172.30.0.x). Stale IPs made compose unable to (re)create app-ukmesh
(no subnet contains 172.18.30.10).
- topic.ts/brokerLog.ts/client.ts: accept the neighbours suffix (official
MeshCore firmware publishes the UK spelling; our MQTT fork uses neighbors)
- aclManager.ts: render + verify BOTH spellings in owner ACL grants (ukmesh
and test scopes); inventoryOwnerAuthorization regex updated to match
- statsService.ts: chart bucket labels are now machine-readable ISO instead
of server-timezone HH:MM/en-GB strings (backend container runs UTC;
viewers saw UTC hours)
- StatsPage.tsx: axis ticks + tooltips + summaries formatted via
statsTimeFormat.ts (Intl local timezone, raw fallback for legacy cached
snapshots) + unit tests
The planner inlined the view CTEs into consumers and re-ran the links
aggregation once per candidate row (loops=13070, 24s+ for getViableLinks).
Wrapping each view body in WITH v AS MATERIALIZED forces one evaluation
per query: viable-links query 24.3s -> 3.1s, WS initial state <1s.
Also drops the dead terrain_profile_json column from node_identity_links
(no readers; frontend terrain comes from DEM tiles).
- channelRegistry: 39 validated channel secrets committed as the service
default (Public + 38 community/hashtag channels recovered via the
derivation audit); MESHCORE_CHANNEL_SECRETS env appends on top
- shared buildSummary/identifyChannel (single implementation for ingest
and offline tools)
- tools/backfillDecrypt: keyset-paginated retroactive decryption of the
stored raw_hex corpus (idempotent, resumable)
- feed: channel scope sidebar now lists every decrypted channel
Backend: schedule ONE final multi-observer resolution per path-bearing
packet after the propagation window closes (default 60s, calibrated on
real first->last observer spread p50 11s / p95 34s / p99 50s; median 17
observers per packet). The final runs with the COMPLETE observer set and
pushes into the resolve cache after invalidation, so the next API read
serves the full-propagation answer. mode=slow on /path-beta/resolve-multi
returns 202 {status:pending, remainingMs} while inside the window; new
/path-beta/slow-mode status endpoint; observability table
path_slow_resolutions (migration 034). Env knobs:
PATH_SLOW_MODE_ENABLED / PATH_SLOW_MODE_WINDOW_MS / PATH_SLOW_MODE_MIN_HOPS
/ PATH_SLOW_MODE_PENDING_MAX. Best-effort: restart loss is harmless.
Frontend: pinned packets resolve in slow mode (bounded 202 retry), local
path renders meanwhile, upgraded when the final lands. Live feed unchanged.
Tests: 4 new slowMode unit tests (dedupe, disable, bound, window timing).
Backend build clean, 257/258 (1 pre-existing workerPool timing flake on
base too); frontend build clean, 74/74.
gadgethd