2 Commits

Include renames

Author	SHA1	Message	Date
efitenandClaude Opus 5.5	927a8eec8a	fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211) ... Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: | Range | Availability | Packets/day | |---|---|---| | 7d | 88.7% | 91 | | 30d | 42.6% | 265 | | All | 3.5% | 22 | Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - **UTC fix.** The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: | Range | Before | After | |---|---|---| | 7d | 82.7% | 83.3%, not partial | | 30d | 19.3% | 83.3%, partial from 2026-10-07 | | All | 1.6% | 83.3%, partial from 2026-10-07 | In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 09:27:21 +02:00
efitenandClaude Opus 5	a059299588	feat(node-analytics): hop-count statistics per node (#1812) (#2021) ... ## Summary Adds per-node hop-count statistics so repeater operators can choose `flood.max`, `flood.max.unscoped` and `flood.max.advert` from what their node actually sees. - New endpoint `GET /api/nodes/{pubkey}/hop_analytics?days=N` (`cmd/server/routes.go:299`, `cmd/server/node_hop_analytics.go:312`), separate from `/analytics` as requested in the issue. - New card "Hop Count at This Node" on the node analytics page (`public/node-hop-analytics.js`, wired at `public/node-analytics.js:130,174`): histogram of hop counts with a box plot on the same x axis, filters `flood.max` (default), `flood.max.advert`, `flood.max.unscoped`, driven by the existing range picker. - The existing "Hop Distribution" chart is unchanged: it shows path length at the observer, a different quantity. - No `direct` tag, although the issue lists one: for DIRECT packets the path is the remaining route and no flood limit applies, so there is no hop count to report. ## Hop count definition (firmware 0679dbef) - `src/helpers/RoutingPolicy.h:15-21`: limits compare `getPathHashCount()`; `.unscoped` applies to route type FLOOD, `.advert` to adverts. - `src/Mesh.cpp:344-350`: `routeRecvPacket` checks with n hashes in the path, then writes its own hash at index n. So hops = the node's zero-based index in the path, no +1. - `src/Mesh.cpp:265-285`: a node forwards a flood once; `src/Mesh.cpp:651,680`: an originator never forwards its own flood. - DIRECT packets are excluded: their path is the remaining route (`src/Mesh.cpp:78-103,334-341`). Response: `{timeRange, packets: [{hash, timestamp, hops, tags}], ambiguous}`. Tags: `flood`, `scoped` or `unscoped`, `advert`. Documented in `docs/api-spec.md:679` and `cmd/server/openapi.go:90`. ## Attribution `cmd/server/node_hop_analytics.go:198-309`. The result depends only on the observed paths, the prefix map and the neighbor graph, so it is the same after a restart as after live ingest. - Every observation of every flood packet in the window is read. `byNode` holds the server resolver's pick at ingest and other picks after a cold load; `byPathHop` indexes only each packet's longest path, which for a busy relay often runs through another branch of the flood. - A packet counts when the node's prefix sits at exactly one index across its observations, and either the node is the only relay candidate for that prefix (`prefixMap.relayCandidates`, `cmd/server/store.go:6795`), or the hop resolves to the node under the ingestor's strict rule (`cmd/ingestor/path_resolver.go:143-214`) in at least one observation and to another node in none. Strict rule: earlier hops identified without a tiebreak, exactly one candidate adjacent in `neighbor_edges` to the previous hop (the originator for hop 0 of an advert), nodes already on the path excluded. - The server resolver's tiebreaks (affinity, GPS distance, advert count, pubkey order) are not used. - Everything else with the node's prefix goes to `ambiguous`. In practice that is most packets with a colliding 1-byte path hash. On a read-only 7-day dump of a 1,669-node mesh DB, for one busy repeater: 23,081 packets attributed, 11,437 ambiguous. Taking candidates from `byPathHop` instead gave 9,995 attributed, with the histogram mode moved from 2 to 3-5 hops. ## Performance Scans `s.packets` under the read lock, no SQL per packet. Per observation: one substring test for the node's first prefix byte; the hop scan only for observations containing it; the strict walk only for colliding prefixes, with per-request caches for candidates and adjacency. `BenchmarkNodeHopPackets` models one 7-day request at that scale (73,782 flood packets, 1,430,280 observations): 44-87 ms/op, 13.4 MB, 40 allocs on a throttling laptop. Response size for that repeater over 7 days: about 23k entries, 2.3 MB JSON, 375 KB gzipped. `hash` and `timestamp` are 61% of the raw and 91% of the gzipped bytes; they stay because the issue asks for them so a client can join entries to packets and bin by time. ## Tests - Go: `cmd/server/node_hop_analytics_test.go`: 12 unit tests, a live-ingest test through `IngestNewFromDB` (a colliding prefix without independent attribution goes to `ambiguous`, not to the node the resolver picked), live ingest versus cold load of the same DB, route test, benchmark. 15 mutations of the attribution logic each fail a test. - JS: `test-node-hop-analytics.js` (filters, histogram, quartiles and whiskers with a fixture that separates 1.5 IQR from 3 IQR, render), registered in `test-all.sh` and `.github/workflows/deploy.yml`. - `gofmt`, `go vet ./...`, `go test ./...` in `cmd/server`, `scripts/check-css-vars.js` pass. ## Staging validation Build `c646310f` (this PR's review follow-up together with the other open follow-ups), after a container restart and full load, on a busy Belgian repeater: - `hop_analytics?days=7`: 23,302 packets, 11,548 ambiguous, median 4, adverts never above hop 7 (matching the firmware default `flood_max_advert = 8`, `examples/simple_repeater/MyMesh.cpp:922`), 1.2 s. The first version reported 23,035 packets and 86 ambiguous in 534 ms, because it trusted the resolver's pick for colliding prefixes. - The card rendered on the first version with no console errors; the rework does not touch the frontend beyond a test fixture. ## Not verified - Response time and lock hold for 30 days on the busiest node on a 14-day store. - Server relay candidates exclude companions and listeners while the ingestor's prefix index does not, so a few strict attributions can differ from the ingestor's persisted `resolved_path`. - Identical numbers across a second container restart were shown in a Go test, not repeated on staging. - Dark theme, phone width, and switching the range picker in the browser. - Filter state is not reflected in the URL hash (the range picker is not either). Fixes #1812 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 19:59:56 +02:00