mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 14:38:28 +00:00
Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: | Range | Availability | Packets/day | |---|---|---| | 7d | 88.7% | 91 | | 30d | 42.6% | 265 | | All | 3.5% | 22 | Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - **UTC fix.** The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: | Range | Before | After | |---|---|---| | 7d | 82.7% | 83.3%, not partial | | 30d | 19.3% | 83.3%, partial from 2026-10-07 | | All | 1.6% | 83.3%, partial from 2026-10-07 | In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
81 lines
3.5 KiB
JavaScript
81 lines
3.5 KiB
JavaScript
/**
|
|
* #748 — node analytics coverage notice: shown when the server's in-memory
|
|
* window starts after the requested range. Loads the real
|
|
* public/node-analytics.js in a vm sandbox.
|
|
*/
|
|
'use strict';
|
|
const fs = require('fs');
|
|
const path = require('path');
|
|
const vm = require('vm');
|
|
const assert = require('assert');
|
|
|
|
const REPO_ROOT = path.resolve(__dirname, '..', '..');
|
|
let passed = 0, failed = 0;
|
|
function t(name, fn) {
|
|
try { fn(); passed++; console.log(' ✓ ' + name); }
|
|
catch (e) { failed++; console.error(' ✗ ' + name + ': ' + e.message); }
|
|
}
|
|
|
|
const ctx = { window: {}, registerPage: () => {}, Date, Math, Number, String, isNaN, console };
|
|
vm.createContext(ctx);
|
|
vm.runInContext(fs.readFileSync(path.join(REPO_ROOT, 'public', 'node-analytics.js'), 'utf8'), ctx);
|
|
const notice = ctx.window.NodeAnalyticsCoverage && ctx.window.NodeAnalyticsCoverage.notice;
|
|
|
|
console.log('node-analytics.js coverage notice');
|
|
|
|
t('exposes NodeAnalyticsCoverage.notice', () => {
|
|
assert.strictEqual(typeof notice, 'function');
|
|
});
|
|
|
|
t('memory limit: names the start date, the window and the requested days', () => {
|
|
const html = notice({ from: '2025-10-14T12:00:00Z', to: '2026-10-14T12:00:00Z', days: 365, dataFrom: '2026-09-30T09:12:00Z', dataFromReason: 'memory', partial: true });
|
|
assert.ok(html.includes('data-testid="analytics-coverage-notice"'), html);
|
|
assert.ok(html.includes('Data from 2026-09-30'), html);
|
|
assert.ok(html.includes('about 14 days'), html);
|
|
assert.ok(html.includes('not the full 365 days'), html);
|
|
});
|
|
|
|
t('database younger than the range: says so, no memory claim', () => {
|
|
const html = notice({ to: '2026-10-14T12:00:00Z', days: 365, dataFrom: '2026-07-01T00:00:00Z', dataFromReason: 'database', partial: true });
|
|
assert.ok(html.includes('Data from 2026-07-01: the database holds no older packets'), html);
|
|
assert.ok(!html.includes('in memory'), html);
|
|
assert.ok(html.includes('not the full 365 days'), html);
|
|
});
|
|
|
|
t('background load still running: says so', () => {
|
|
const html = notice({ to: '2026-10-14T12:00:00Z', days: 30, dataFrom: '2026-10-13T12:00:00Z', dataFromReason: 'loading', partial: true });
|
|
assert.ok(html.includes('the server is still loading older packets'), html);
|
|
});
|
|
|
|
t('window too short for rates: says so instead of the rates claim', () => {
|
|
const html = notice({ to: '2026-10-14T12:00:00Z', days: 7, dataFrom: '2026-10-14T06:00:00Z', dataFromReason: 'memory', partial: true },
|
|
{ availabilityPct: null, avgPacketsPerDay: null });
|
|
assert.ok(html.includes('too short for availability and packets/day'), html);
|
|
assert.ok(!html.includes('not the full'), html);
|
|
});
|
|
|
|
t('rateText: null reads as a dash, numbers keep their suffix', () => {
|
|
const rateText = ctx.window.NodeAnalyticsCoverage.rateText;
|
|
assert.strictEqual(rateText(null, '%'), '—');
|
|
assert.strictEqual(rateText(undefined), '—');
|
|
assert.strictEqual(rateText(0, '%'), '0%');
|
|
assert.strictEqual(rateText(12.5), '12.5');
|
|
});
|
|
|
|
t('full coverage: no notice', () => {
|
|
assert.strictEqual(notice({ from: '2026-10-07T12:00:00Z', to: '2026-10-14T12:00:00Z', days: 7 }), '');
|
|
});
|
|
|
|
t('missing or malformed time range: no notice', () => {
|
|
assert.strictEqual(notice(null), '');
|
|
assert.strictEqual(notice({ partial: true, dataFrom: 'garbage', to: '2026-10-14T12:00:00Z', days: 30 }), '');
|
|
});
|
|
|
|
t('one day of data reads singular', () => {
|
|
const html = notice({ to: '2026-10-14T12:00:00Z', days: 7, dataFrom: '2026-10-13T13:00:00Z', partial: true });
|
|
assert.ok(html.includes('about 1 day of packets'), html);
|
|
});
|
|
|
|
console.log(`\n${passed} passed, ${failed} failed`);
|
|
process.exit(failed ? 1 : 0);
|