mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 12:37:33 +00:00
Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: | Range | Availability | Packets/day | |---|---|---| | 7d | 88.7% | 91 | | 30d | 42.6% | 265 | | All | 3.5% | 22 | Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - **UTC fix.** The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: | Range | Before | After | |---|---|---| | 7d | 82.7% | 83.3%, not partial | | 30d | 19.3% | 83.3%, partial from 2026-10-07 | | All | 1.6% | 83.3%, partial from 2026-10-07 | In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
72 lines
3.2 KiB
JavaScript
72 lines
3.2 KiB
JavaScript
/**
|
|
* E2E test (#748): the node analytics page says from when its data runs when
|
|
* the requested range reaches past the server's in-memory window.
|
|
*
|
|
* The CI server runs without a retention window or memory cap and the
|
|
* fixture's packets span a few months: 7d is fully covered, and "All"
|
|
* (365d) reaches past the oldest packet in the database. That is the
|
|
* "database" case, so the notice must say the database holds no older
|
|
* packets, not that the server keeps them in memory. The memory and
|
|
* loading wordings are covered by the unit tests.
|
|
*
|
|
* Usage: BASE_URL=http://localhost:13581 node test-issue-748-node-analytics-coverage-e2e.js
|
|
*/
|
|
const { chromium } = require('playwright');
|
|
|
|
const BASE = process.env.BASE_URL || 'http://localhost:3000';
|
|
|
|
async function test(name, fn) {
|
|
try {
|
|
await fn();
|
|
console.log(` ✅ ${name}`);
|
|
} catch (err) {
|
|
console.log(` ❌ ${name}: ${err.message}`);
|
|
process.exit(1);
|
|
}
|
|
}
|
|
|
|
function assert(cond, msg) {
|
|
if (!cond) throw new Error(msg || 'Assertion failed');
|
|
}
|
|
|
|
async function run() {
|
|
const browser = await chromium.launch({
|
|
headless: true,
|
|
executablePath: process.env.CHROMIUM_PATH || undefined,
|
|
args: ['--no-sandbox', '--disable-gpu', '--disable-dev-shm-usage']
|
|
});
|
|
const page = await (await browser.newContext({ viewport: { width: 1400, height: 900 } })).newPage();
|
|
page.setDefaultTimeout(15000);
|
|
|
|
console.log(`\nRunning #748 node analytics coverage E2E tests against ${BASE}\n`);
|
|
|
|
const nodes = (await (await page.request.get(`${BASE}/api/nodes?limit=50`)).json()).nodes || [];
|
|
assert(nodes.length > 0, 'fixture has no nodes');
|
|
const pk = nodes[0].public_key;
|
|
|
|
await test('7d inside the in-memory window: not partial, no notice', async () => {
|
|
const r = await (await page.request.get(`${BASE}/api/nodes/${pk}/analytics?days=7`)).json();
|
|
assert(!r.timeRange.partial && !r.timeRange.dataFrom, `7d timeRange: ${JSON.stringify(r.timeRange)}`);
|
|
await page.goto(`${BASE}/#/nodes/${encodeURIComponent(pk)}/analytics`, { waitUntil: 'domcontentloaded' });
|
|
await page.waitForSelector('#timeRangeBtns');
|
|
assert(!(await page.$('[data-testid="analytics-coverage-notice"]')), 'notice shown for a fully covered range');
|
|
});
|
|
|
|
await test('All reaches past the oldest packet: partial, notice names the start date and the database', async () => {
|
|
const r = await (await page.request.get(`${BASE}/api/nodes/${pk}/analytics?days=365`)).json();
|
|
assert(r.timeRange.partial && r.timeRange.dataFrom, `365d timeRange: ${JSON.stringify(r.timeRange)}`);
|
|
assert(r.timeRange.dataFromReason === 'database', `365d reason: ${r.timeRange.dataFromReason}`);
|
|
await page.click('#timeRangeBtns button[data-days="365"]');
|
|
await page.waitForSelector('[data-testid="analytics-coverage-notice"]');
|
|
const text = await page.$eval('[data-testid="analytics-coverage-notice"]', (el) => el.textContent);
|
|
const day = r.timeRange.dataFrom.slice(0, 10);
|
|
assert(text.includes('Data from ' + day + ': the database holds no older packets'), `notice: ${text}`);
|
|
assert(!text.includes('in memory'), `notice claims a memory limit: ${text}`);
|
|
});
|
|
|
|
await browser.close();
|
|
console.log('\nAll #748 node analytics coverage E2E tests passed.');
|
|
}
|
|
|
|
run().catch((e) => { console.error(e); process.exit(1); });
|