Commit Graph
1 Commits
Author SHA1 Message Date
efitenandClaude Opus 5.5 927a8eec8a fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211)
Refs #748 (M3a: correct numbers within the in-memory window; the SQL
history in M3b is a separate plan, see the end).

## Problem

The node analytics range picker offers 24h, 7d, 30d and All (365d), but
the store only holds its in-memory window (`retentionHours`, then
eviction). `GetNodeAnalytics` still divided availability and packets/day
by the requested range. Measured on our live instance (14-day window),
repeater BE-BRE-ON8AR:

| Range | Availability | Packets/day |
|---|---|---|
| 7d | 88.7% | 91 |
| 30d | 42.6% | 265 |
| All | 3.5% | 22 |

Nothing on the page said that the data stopped at the edge of the
window.

## What changes

- `coverageStart` returns the edge of the in-memory window: the latest
of `oldestLoaded` (background-load progress), `now - retentionHours`,
and a new `evictedThrough` that eviction sets to the newest `FirstSeen`
it removed, so memory-pressure eviction inside the retention window
moves the edge too.
- `s.packets[0]` is not that edge, and the first version of this PR used
it. Chunks are selected by `last_seen`, so a transmission first heard
weeks ago and heard again recently sits at the head with its old
`FirstSeen`. On staging (7-day window) it returned 21/07.
- When the edge lies more than a minute after the requested start, node
analytics counts only transmissions from the edge, computes availability
and packets/day over that span, and returns `timeRange.dataFrom` and
`timeRange.partial`.
- The minute of slack keeps a range equal to the retention window from
reading as partial. Without it every 7d request on staging (168h) was
partial, because the two edges came from separate clock reads.
- The analytics page shows "Data from 2026-10-07: the server keeps about
7 days of packets in memory. Availability and packets/day cover that
period, not the full 365 days." when partial.
- **UTC fix.** The window bounds of node analytics and node hop
analytics are formatted in UTC. They are compared as strings with UTC
`FirstSeen` values, so on a non-UTC host the 7d window lost its oldest
hours (2 at +02:00). Docker images run in UTC, which is why this did not
show in production.
- `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the
edge is kept.

## Tests

- Go, `node_analytics_coverage_test.go`:
  - full coverage within memory;
  - 30d over 3 days in memory;
  - empty store;
- a re-heard 85-day-old transmission at the head of `s.packets` (edge
stays at retention, packet not counted, no bucket before `dataFrom`);
  - memory-eviction edge;
  - eviction records `evictedThrough`;
  - retention 30 s shorter than the range is not partial, while 30d is;
  - window bounds under a +02:00 `time.Local`.
- Two tests were checked to fail on the old code: the local-zone test
(166 of 168 packets, a `+02:00` bound) and the retention test without
the slack (`dataFrom 09:57:08` vs `from 09:56:38`).
- Full `cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master). One run of five reported `FAIL` and four reruns were green; I
did not capture which test failed.
- Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js`
(5 assertions on the real `node-analytics.js` in `vm`), in
`test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js`
(registered in `deploy.yml` and `scripts/non-unit-tests.json`), against
the real fixture response with no interception: 7d is not partial and
shows no notice; All is partial and the notice names the real start
date. 2/2 locally.

## Validation

Staging (retentionHours 168), same repeater, after the background load
completed:

| Range | Before | After |
|---|---|---|
| 7d | 82.7% | 83.3%, not partial |
| 30d | 19.3% | 83.3%, partial from 2026-10-07 |
| All | 1.6% | 83.3%, partial from 2026-10-07 |

In the browser the notice appears under All and not under 7d.

## Not done

- M3b, older history from SQLite. In the database only the sender is
indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay
hops only in `observations.resolved_path` (filled for about 27% of
observations on our live instance, not indexed). A fallback today would
show only a node's own packets for older days, a different meaning than
the in-memory part. Doing it properly needs a per-node index written by
the ingestor (pubkey, transmission, role); that is a separate plan.
- The other analytics endpoints with a range (observer analytics, RF)
were not checked for the same dilution.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:27:21 +02:00