Files
meshcore-analyzer/public/node-analytics.js
T
efitenandClaude Opus 5.5 927a8eec8a fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211)
Refs #748 (M3a: correct numbers within the in-memory window; the SQL
history in M3b is a separate plan, see the end).

## Problem

The node analytics range picker offers 24h, 7d, 30d and All (365d), but
the store only holds its in-memory window (`retentionHours`, then
eviction). `GetNodeAnalytics` still divided availability and packets/day
by the requested range. Measured on our live instance (14-day window),
repeater BE-BRE-ON8AR:

| Range | Availability | Packets/day |
|---|---|---|
| 7d | 88.7% | 91 |
| 30d | 42.6% | 265 |
| All | 3.5% | 22 |

Nothing on the page said that the data stopped at the edge of the
window.

## What changes

- `coverageStart` returns the edge of the in-memory window: the latest
of `oldestLoaded` (background-load progress), `now - retentionHours`,
and a new `evictedThrough` that eviction sets to the newest `FirstSeen`
it removed, so memory-pressure eviction inside the retention window
moves the edge too.
- `s.packets[0]` is not that edge, and the first version of this PR used
it. Chunks are selected by `last_seen`, so a transmission first heard
weeks ago and heard again recently sits at the head with its old
`FirstSeen`. On staging (7-day window) it returned 21/07.
- When the edge lies more than a minute after the requested start, node
analytics counts only transmissions from the edge, computes availability
and packets/day over that span, and returns `timeRange.dataFrom` and
`timeRange.partial`.
- The minute of slack keeps a range equal to the retention window from
reading as partial. Without it every 7d request on staging (168h) was
partial, because the two edges came from separate clock reads.
- The analytics page shows "Data from 2026-10-07: the server keeps about
7 days of packets in memory. Availability and packets/day cover that
period, not the full 365 days." when partial.
- **UTC fix.** The window bounds of node analytics and node hop
analytics are formatted in UTC. They are compared as strings with UTC
`FirstSeen` values, so on a non-UTC host the 7d window lost its oldest
hours (2 at +02:00). Docker images run in UTC, which is why this did not
show in production.
- `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the
edge is kept.

## Tests

- Go, `node_analytics_coverage_test.go`:
  - full coverage within memory;
  - 30d over 3 days in memory;
  - empty store;
- a re-heard 85-day-old transmission at the head of `s.packets` (edge
stays at retention, packet not counted, no bucket before `dataFrom`);
  - memory-eviction edge;
  - eviction records `evictedThrough`;
  - retention 30 s shorter than the range is not partial, while 30d is;
  - window bounds under a +02:00 `time.Local`.
- Two tests were checked to fail on the old code: the local-zone test
(166 of 168 packets, a `+02:00` bound) and the retention test without
the slack (`dataFrom 09:57:08` vs `from 09:56:38`).
- Full `cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master). One run of five reported `FAIL` and four reruns were green; I
did not capture which test failed.
- Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js`
(5 assertions on the real `node-analytics.js` in `vm`), in
`test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js`
(registered in `deploy.yml` and `scripts/non-unit-tests.json`), against
the real fixture response with no interception: 7d is not partial and
shows no notice; All is partial and the notice names the real start
date. 2/2 locally.

## Validation

Staging (retentionHours 168), same repeater, after the background load
completed:

| Range | Before | After |
|---|---|---|
| 7d | 82.7% | 83.3%, not partial |
| 30d | 19.3% | 83.3%, partial from 2026-10-07 |
| All | 1.6% | 83.3%, partial from 2026-10-07 |

In the browser the notice appears under All and not under 7d.

## Not done

- M3b, older history from SQLite. In the database only the sender is
indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay
hops only in `observations.resolved_path` (filled for about 27% of
observations on our live instance, not indexed). A fallback today would
show only a node's own packets for older days, a different meaning than
the in-memory part. Doing it properly needs a per-node index written by
the ingestor (pubkey, transmission, role); that is a separate plan.
- The other analytics endpoints with a range (observer analytics, RF)
were not checked for the same dilution.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:27:21 +02:00

413 lines
20 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/* === CoreScope — node-analytics.js === */
'use strict';
(function () {
const PAYLOAD_LABELS = { 0: 'Request', 1: 'Response', 2: 'Direct Msg', 3: 'ACK', 4: 'Advert', 5: 'Channel Msg', 7: 'Anon Req', 8: 'Path', 9: 'Trace', 11: 'Control' };
const CHART_COLORS = ['#4a9eff', '#ff6b6b', '#51cf66', '#fcc419', '#cc5de8', '#20c997', '#ff922b', '#845ef7', '#f06595', '#339af0'];
const GRADE_COLORS = { A: '#51cf66', 'A-': '#51cf66', 'B+': '#339af0', B: '#339af0', C: '#fcc419', D: '#ff6b6b' };
const DAY_NAMES = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'];
let charts = [];
let currentDays = 7;
let currentPubkey = null;
function destroyCharts() {
charts.forEach(c => { try { c.destroy(); } catch {} });
charts = [];
}
function chartDefaults() {
const style = getComputedStyle(document.documentElement);
Chart.defaults.color = style.getPropertyValue('--text-muted').trim() || '#6b7280';
Chart.defaults.borderColor = style.getPropertyValue('--border').trim() || '#e2e5ea';
}
function formatSilence(ms) {
if (!ms) return '—';
const h = Math.floor(ms / 3600000);
const m = Math.floor((ms % 3600000) / 60000);
if (h > 24) return Math.floor(h / 24) + 'd ' + (h % 24) + 'h';
if (h > 0) return h + 'h ' + m + 'm';
return m + 'm';
}
// #748: the server only holds its in-memory window. When that starts
// after the requested range, say from when the data runs, why, and what
// the rates cover; '' otherwise. stats.availabilityPct is null when the
// window is too short for rates.
function coverageNotice(tr, stats) {
if (!tr || !tr.partial || !tr.dataFrom || !tr.to) return '';
const from = Date.parse(tr.dataFrom);
const to = Date.parse(tr.to);
if (isNaN(from) || isNaN(to)) return '';
const spanDays = Math.max(1, Math.round((to - from) / 86400000));
let why;
if (tr.dataFromReason === 'database') why = 'the database holds no older packets';
else if (tr.dataFromReason === 'loading') why = 'the server is still loading older packets';
else why = 'the server keeps about ' + spanDays + ' day' + (spanDays === 1 ? '' : 's') + ' of packets in memory';
const rates = stats && stats.availabilityPct == null
? 'That is under a day, too short for availability and packets/day.'
: 'Availability and packets/day cover that period, not the full ' + tr.days + ' days.';
return '<div class="analytics-coverage-notice" data-testid="analytics-coverage-notice" role="note">' +
'Data from ' + new Date(from).toISOString().slice(0, 10) + ': ' + why + '. ' + rates + '</div>';
}
// A rate the server left out (window too short, #748) reads as a dash.
function rateText(v, suffix) {
return v == null ? '—' : v + (suffix || '');
}
window.NodeAnalyticsCoverage = { notice: coverageNotice, rateText: rateText };
async function loadAnalytics(container, pubkey, days) {
currentPubkey = pubkey;
currentDays = days;
destroyCharts();
chartDefaults();
container.innerHTML = '<div style="padding:40px;text-align:center;color:var(--text-muted)">Loading analytics…</div>';
let data;
try {
data = await api('/nodes/' + encodeURIComponent(pubkey) + '/analytics?days=' + days, { ttl: CLIENT_TTL.nodeAnalytics });
} catch (e) {
container.innerHTML = '<div style="padding:40px;text-align:center;color:#ff6b6b">Failed to load analytics: ' + escapeHtml(e.message) + '</div>';
return;
}
const n = data.node;
const s = data.computedStats;
const nodeName = escapeHtml(n.name || n.public_key.slice(0, 12));
container.innerHTML = `
<div style="max-width:1000px;margin:0 auto;padding:12px 16px">
<div style="margin-bottom:12px">
<a href="#/nodes/${encodeURIComponent(n.public_key)}" style="color:var(--link-color);text-decoration:none;font-size:12px">← Back to ${nodeName}</a>
<h2 style="margin:4px 0 2px;font-size:18px"><svg class="ph-icon" aria-hidden="true"><use href="/icons/phosphor-sprite.svg#ph-chart-bar"/></svg> ${nodeName} — Analytics</h2>
<div style="color:var(--text-muted);font-size:11px">${n.role || 'Unknown role'} · ${s.totalTransmissions || s.totalPackets} packets in ${days}d window</div>
</div>
<div class="analytics-time-range" id="timeRangeBtns">
<button data-days="1" ${days===1?'class="active"':''}>24h</button>
<button data-days="7" ${days===7?'class="active"':''}>7d</button>
<button data-days="30" ${days===30?'class="active"':''}>30d</button>
<button data-days="365" ${days===365?'class="active"':''}>All</button>
</div>
${coverageNotice(data.timeRange, s)}
<div class="analytics-stats">
<div class="analytics-stat-card">
<div class="analytics-stat-label">Availability</div>
<div class="analytics-stat-value">${rateText(s.availabilityPct, '%')}</div>
<div class="analytics-stat-desc">% of time windows with at least one packet</div>
</div>
<div class="analytics-stat-card">
<div class="analytics-stat-label">Signal Grade</div>
<div class="analytics-stat-value" style="color:${GRADE_COLORS[s.signalGrade]||'var(--text)'}">${s.signalGrade}</div>
<div class="analytics-stat-desc">A–F based on average SNR across all observers</div>
</div>
<div class="analytics-stat-card">
<div class="analytics-stat-label">Packets / Day</div>
<div class="analytics-stat-value">${rateText(s.avgPacketsPerDay)}</div>
<div class="analytics-stat-desc">Average daily packet volume in this window</div>
</div>
<div class="analytics-stat-card">
<div class="analytics-stat-label">Observers</div>
<div class="analytics-stat-value">${s.uniqueObservers}</div>
<div class="analytics-stat-desc">Distinct stations that heard this node</div>
</div>
<div class="analytics-stat-card">
<div class="analytics-stat-label">Relay %</div>
<div class="analytics-stat-value">${s.relayPct}%</div>
<div class="analytics-stat-desc">Packets forwarded through repeaters vs direct</div>
</div>
<div class="analytics-stat-card">
<div class="analytics-stat-label">Longest Silence</div>
<div class="analytics-stat-value" style="font-size:18px">${formatSilence(s.longestSilenceMs)}</div>
<div class="analytics-stat-desc">Longest gap between consecutive packets</div>
</div>
</div>
<div class="analytics-charts">
<div class="analytics-chart-card full">
<h4>Activity Timeline</h4>
<div class="analytics-chart-desc">Packet count per time bucket — shows when this node is most active</div>
<canvas id="activityChart" role="img" aria-label="Activity timeline chart"></canvas>
</div>
<div class="analytics-chart-card">
<h4>SNR Trend</h4>
<div class="analytics-chart-desc">Signal-to-noise ratio over time — higher is better reception</div>
<canvas id="snrChart" role="img" aria-label="SNR trend chart"></canvas>
</div>
<div class="analytics-chart-card">
<h4>Packet Types</h4>
<div class="analytics-chart-desc">Breakdown of advert, position, text, and other packet types</div>
<canvas id="packetTypeChart" role="img" aria-label="Packet types chart"></canvas>
</div>
<div class="analytics-chart-card">
<h4>Observer Coverage</h4>
<div class="analytics-chart-desc">Which stations hear this node and how often</div>
<canvas id="observerChart" role="img" aria-label="Observer coverage chart"></canvas>
</div>
<div class="analytics-chart-card">
<h4>Hop Distribution</h4>
<div class="analytics-chart-desc">How many repeater hops packets take — 0 means direct</div>
<canvas id="hopChart" role="img" aria-label="Hop distribution chart"></canvas>
</div>
<div class="analytics-chart-card full">
<h4>Hop Count at This Node</h4>
<div class="analytics-chart-desc">For each flood packet this node forwarded: how many hops it already had, the number the flood.max, flood.max.unscoped and flood.max.advert limits are checked against. Bars count packets per hop count; the box above spans the middle half of the packets with a line at the median, and the whiskers reach the furthest packet within 1.5 box widths of the box.</div>
<div id="hopCountSection"><div style="padding:20px;text-align:center;color:var(--text-muted);font-size:12px">Loading hop counts...</div></div>
</div>
<div class="analytics-chart-card full">
<h4>Battery Voltage <span id="batteryStatusBadge" style="font-size:11px;font-weight:normal;margin-left:8px"></span></h4>
<div class="analytics-chart-desc">Battery voltage over time from observer status reports — flat line means full, downward slope means draining</div>
<canvas id="batteryChart" role="img" aria-label="Battery voltage trend chart"></canvas>
<div id="batteryEmpty" style="display:none;padding:20px;text-align:center;color:var(--text-muted);font-size:12px">No battery telemetry recorded for this node in this window.</div>
</div>
<div class="analytics-chart-card full">
<h4>Uptime Heatmap</h4>
<div class="analytics-chart-desc">Hour-by-hour activity grid — darker = more packets in that slot</div>
<div id="heatmapGrid" class="analytics-heatmap"></div>
</div>
${data.peerInteractions.length ? `<div class="analytics-chart-card full">
<h4>Peer Interactions</h4>
<div class="analytics-chart-desc">Nodes this device has exchanged messages with</div>
<div class="analytics-table-scroll"><table class="analytics-peer-table">
<thead><tr><th scope="col">Peer</th><th scope="col">Messages</th><th scope="col">Last Contact</th></tr></thead>
<tbody>${data.peerInteractions.map(p => `<tr>
<td><a href="#/nodes/${encodeURIComponent(p.peer_key)}" style="color:var(--link-color)">${escapeHtml(p.peer_name)}</a></td>
<td>${p.messageCount}</td>
<td>${timeAgo(p.lastContact)}</td>
</tr>`).join('')}</tbody>
</table></div>
</div>` : ''}
</div>
</div>`;
// Time range buttons
container.querySelectorAll('#timeRangeBtns button').forEach(btn => {
btn.addEventListener('click', () => {
const d = Number(btn.dataset.days);
loadAnalytics(container, pubkey, d);
});
});
// Build charts
buildActivityChart(data);
buildSnrChart(data);
buildPacketTypeChart(data);
buildObserverChart(data);
buildHopChart(data);
buildHeatmap(data);
loadBatteryChart(pubkey, currentDays);
NodeHopAnalytics.load(document.getElementById('hopCountSection'), pubkey, currentDays);
}
function buildActivityChart(data) {
const ctx = document.getElementById('activityChart');
if (!ctx) return;
const tl = data.activityTimeline;
const c = new Chart(ctx, {
type: 'bar',
data: {
labels: tl.map(b => {
const d = new Date(b.bucket);
return (typeof formatChartAxisLabel === 'function') ? formatChartAxisLabel(d, currentDays <= 3) : (currentDays <= 3 ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' }) : d.toLocaleDateString([], { month: 'short', day: 'numeric' }));
}),
datasets: [{ label: 'Packets', data: tl.map(b => b.count), backgroundColor: 'rgba(74,158,255,0.5)', borderColor: '#4a9eff', borderWidth: 1 }]
},
options: { responsive: true, plugins: { legend: { display: false } }, scales: { x: { ticks: { maxTicksAutoSkip: true, maxRotation: 45 } }, y: { beginAtZero: true } } }
});
charts.push(c);
}
function buildSnrChart(data) {
const ctx = document.getElementById('snrChart');
if (!ctx) return;
// Group by observer
const byObs = {};
data.snrTrend.forEach(p => {
const key = p.observer_id || 'unknown';
if (!byObs[key]) byObs[key] = { name: p.observer_name || key, points: [] };
byObs[key].points.push({ x: new Date(p.timestamp), y: p.snr });
});
const datasets = Object.values(byObs).map((obs, i) => ({
label: obs.name, data: obs.points.map(p => p.y), borderColor: CHART_COLORS[i % CHART_COLORS.length],
backgroundColor: 'transparent', pointRadius: 1, borderWidth: 1.5, tension: 0.3
}));
// Use labels from the observer with most points
const longestObs = Object.values(byObs).sort((a, b) => b.points.length - a.points.length)[0];
const labels = longestObs ? longestObs.points.map(p => {
const d = p.x;
return (typeof formatChartAxisLabel === 'function') ? formatChartAxisLabel(d, false) : d.toLocaleDateString([], { month: 'short', day: 'numeric' }) + ' ' + d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
}) : [];
const c = new Chart(ctx, {
type: 'line',
data: { labels, datasets },
options: {
responsive: true,
scales: { x: { display: false }, y: { title: { display: true, text: 'SNR (dB)' } } },
plugins: { legend: { position: 'bottom', labels: { boxWidth: 12, font: { size: 10 } } } }
}
});
charts.push(c);
}
function buildPacketTypeChart(data) {
const ctx = document.getElementById('packetTypeChart');
if (!ctx) return;
const items = data.packetTypeBreakdown;
const c = new Chart(ctx, {
type: 'doughnut',
data: {
labels: items.map(i => PAYLOAD_LABELS[i.payload_type] || 'Type ' + i.payload_type),
datasets: [{ data: items.map(i => i.count), backgroundColor: items.map((_, i) => CHART_COLORS[i % CHART_COLORS.length]) }]
},
options: { responsive: true, plugins: { legend: { position: 'bottom', labels: { boxWidth: 12, font: { size: 10 } } } } }
});
charts.push(c);
}
function buildObserverChart(data) {
const ctx = document.getElementById('observerChart');
if (!ctx) return;
const obs = data.observerCoverage;
const c = new Chart(ctx, {
type: 'bar',
data: {
labels: obs.map(o => (o.observer_name || o.observer_id || '?').slice(0, 20)),
datasets: [{ label: 'Packets', data: obs.map(o => o.packetCount), backgroundColor: obs.map(o => {
const snr = o.avgSnr || 0;
const alpha = Math.min(1, Math.max(0.3, snr / 20));
return `rgba(74,158,255,${alpha})`;
}) }]
},
options: { indexAxis: 'y', responsive: true, plugins: { legend: { display: false } }, scales: { x: { beginAtZero: true } } }
});
charts.push(c);
}
function buildHopChart(data) {
const ctx = document.getElementById('hopChart');
if (!ctx) return;
const hops = data.hopDistribution;
const c = new Chart(ctx, {
type: 'bar',
data: {
labels: hops.map(h => h.hops + ' hop' + (h.hops !== '1' ? 's' : '')),
datasets: [{ label: 'Packets', data: hops.map(h => h.count), backgroundColor: 'rgba(81,207,102,0.6)', borderColor: '#51cf66', borderWidth: 1 }]
},
options: { responsive: true, plugins: { legend: { display: false } }, scales: { y: { beginAtZero: true } } }
});
charts.push(c);
}
function buildHeatmap(data) {
const grid = document.getElementById('heatmapGrid');
if (!grid) return;
// Build lookup
const lookup = {};
let maxCount = 1;
data.uptimeHeatmap.forEach(h => {
const key = h.dayOfWeek + '-' + h.hour;
lookup[key] = h.count;
if (h.count > maxCount) maxCount = h.count;
});
// Header row
grid.innerHTML = '<div class="analytics-heatmap-label"></div>';
for (let h = 0; h < 24; h++) {
grid.innerHTML += `<div class="analytics-heatmap-label" style="justify-content:center;font-size:9px">${h}</div>`;
}
// Day rows
for (let d = 0; d < 7; d++) {
grid.innerHTML += `<div class="analytics-heatmap-label">${DAY_NAMES[d]}</div>`;
for (let h = 0; h < 24; h++) {
const count = lookup[d + '-' + h] || 0;
const intensity = count / maxCount;
const bg = count === 0 ? 'var(--card-bg)' : `rgba(74,158,255,${0.15 + intensity * 0.85})`;
grid.innerHTML += `<div class="analytics-heatmap-cell" style="background:${bg}" title="${DAY_NAMES[d]} ${h}:00 — ${count} packets"></div>`;
}
}
}
async function loadBatteryChart(pubkey, days) {
let data;
try {
data = await api('/nodes/' + encodeURIComponent(pubkey) + '/battery?days=' + days);
} catch (e) {
const empty = document.getElementById('batteryEmpty');
if (empty) { empty.style.display = 'block'; empty.textContent = 'Battery data unavailable: ' + e.message; }
return;
}
const ctx = document.getElementById('batteryChart');
const empty = document.getElementById('batteryEmpty');
const badge = document.getElementById('batteryStatusBadge');
const samples = (data && data.samples) || [];
const thr = (data && data.thresholds) || { low_mv: 3300, critical_mv: 3000 };
if (badge) {
const STATUS_COLOR = { ok: '#51cf66', low: '#fcc419', critical: '#ff6b6b', unknown: 'var(--text-muted)' };
const label = data && data.status === 'ok' ? '<svg class="ph-icon" aria-hidden="true"><use href="/icons/phosphor-sprite.svg#ph-battery-high"/></svg> OK'
: data && data.status === 'low' ? '<svg class="ph-icon" aria-hidden="true"><use href="/icons/phosphor-sprite.svg#ph-warning"/></svg> Low'
: data && data.status === 'critical' ? '<svg class="ph-icon" aria-hidden="true"><use href="/icons/phosphor-sprite.svg#ph-battery-low"/></svg> Critical'
: 'No data';
const mv = data && data.latest_mv ? ' · ' + data.latest_mv + ' mV' : '';
badge.textContent = label + mv;
badge.style.color = STATUS_COLOR[(data && data.status) || 'unknown'];
}
if (!ctx || samples.length === 0) {
if (ctx) ctx.style.display = 'none';
if (empty) empty.style.display = 'block';
return;
}
if (empty) empty.style.display = 'none';
ctx.style.display = '';
const labels = samples.map(p => {
const d = new Date(p.timestamp);
return (typeof formatChartAxisLabel === 'function')
? formatChartAxisLabel(d, days <= 3)
: (days <= 3 ? d.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
: d.toLocaleDateString([], { month: 'short', day: 'numeric' }));
});
const values = samples.map(p => p.battery_mv);
const c = new Chart(ctx, {
type: 'line',
data: {
labels: labels,
datasets: [
{ label: 'Battery (mV)', data: values, borderColor: '#4a9eff', backgroundColor: 'rgba(74,158,255,0.15)', tension: 0.25, pointRadius: 2, fill: true },
{ label: 'Low threshold', data: values.map(() => thr.low_mv), borderColor: '#fcc419', borderDash: [6, 4], pointRadius: 0, fill: false },
{ label: 'Critical', data: values.map(() => thr.critical_mv), borderColor: '#ff6b6b', borderDash: [6, 4], pointRadius: 0, fill: false }
]
},
options: {
responsive: true,
plugins: { legend: { display: true, position: 'bottom' } },
scales: { x: { ticks: { maxTicksAutoSkip: true, maxRotation: 45 } }, y: { title: { display: true, text: 'mV' } } }
}
});
charts.push(c);
}
function init(container, routeParam) {
// routeParam is "PUBKEY/analytics"
if (!routeParam || !routeParam.endsWith('/analytics')) {
container.innerHTML = '<div style="padding:40px;text-align:center">Invalid analytics URL</div>';
return;
}
const pubkey = routeParam.slice(0, -'/analytics'.length);
loadAnalytics(container, pubkey, 7);
}
function destroy() {
destroyCharts();
NodeHopAnalytics.destroy();
currentPubkey = null;
}
registerPage('node-analytics', { init, destroy });
})();