mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-03-30 13:35:42 +00:00
Compare commits
42 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7d23ee2ba0 | ||
|
|
a058b8aa8b | ||
|
|
21eeb593d8 | ||
|
|
0eb6b90e32 | ||
|
|
1e6a3b41a3 | ||
|
|
1e34ab18a7 | ||
|
|
60688993c6 | ||
|
|
aab67353d6 | ||
|
|
6bb9235992 | ||
|
|
0a4586f2a9 | ||
|
|
b876fd8a60 | ||
|
|
6a4d2a0844 | ||
|
|
abea3a81e7 | ||
|
|
8104d8bac7 | ||
|
|
881956ffdb | ||
|
|
4dc179e0b9 | ||
|
|
1fe40cf85a | ||
|
|
67ea437429 | ||
|
|
d47c7d18b1 | ||
|
|
d0c5384883 | ||
|
|
3e89fdac67 | ||
|
|
4779193edf | ||
|
|
fba9af5955 | ||
|
|
e09f902a17 | ||
|
|
c261c7b322 | ||
|
|
c36abe2b24 | ||
|
|
536459ee53 | ||
|
|
162782209c | ||
|
|
6e30ba1539 | ||
|
|
971c58b369 | ||
|
|
1f88232f94 | ||
|
|
f410b7d49b | ||
|
|
32ffcb6be4 | ||
|
|
f2d7dabbb2 | ||
|
|
2146ec64ec | ||
|
|
441cd52608 | ||
|
|
832421f581 | ||
|
|
215003f295 | ||
|
|
3049ec3e1c | ||
|
|
936609f729 | ||
|
|
915484f54a | ||
|
|
e6911a3915 |
67
PERFORMANCE.md
Normal file
67
PERFORMANCE.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Performance — v2.1.0
|
||||
|
||||
**Dataset:** 28,014 packets, ~650 nodes, 2 observers
|
||||
**Hardware:** ARM64 (MikroTik CCR2116), single-core Node.js
|
||||
|
||||
## A/B Benchmark: v2.0.1 (before) vs v2.1.0 (after)
|
||||
|
||||
All times are averages over 3 runs. "Cached" = warm TTL cache hit.
|
||||
|
||||
| Endpoint | v2.0.1 | v2.1.0 (cold) | v2.1.0 (cached) | Speedup |
|
||||
|---|---|---|---|---|
|
||||
| **Bulk Health** | 7,059 ms | 3 ms | 1 ms | **7,059×** |
|
||||
| **Node Analytics** | 381 ms | 2 ms | 1 ms | **381×** |
|
||||
| **Hash Sizes** | 353 ms | 193 ms | 1 ms | **353×** |
|
||||
| **Topology** | 685 ms | 579 ms | 2 ms | **342×** |
|
||||
| **RF Analytics** | 253 ms | 235 ms | 1 ms | **253×** |
|
||||
| **Channels** | 206 ms | 77 ms | 1 ms | **206×** |
|
||||
| **Node Health** | 195 ms | 1 ms | 1 ms | **195×** |
|
||||
| **Node Detail** | 133 ms | 1 ms | 1 ms | **133×** |
|
||||
| **Channel Analytics** | 95 ms | 73 ms | 2 ms | **47×** |
|
||||
| **Packets (grouped)** | 76 ms | 33 ms | 28 ms | **2×** |
|
||||
| **Stats** | 2 ms | 1 ms | 1 ms | 2× |
|
||||
| **Nodes List** | 3 ms | 2 ms | 2 ms | 1× |
|
||||
| **Observers** | 1 ms | 8 ms | 1 ms | 1× |
|
||||
|
||||
## Architecture
|
||||
|
||||
### Two-Layer Performance Stack
|
||||
|
||||
1. **In-Memory Packet Store** (`packet-store.js`)
|
||||
- All packets loaded from SQLite into RAM on startup (~28K packets = ~12MB)
|
||||
- Indexed by `id`, `hash`, `observer`, and `node` (Map-based O(1) lookup)
|
||||
- Ring buffer with configurable max memory (default 1GB, ~2.3M packets)
|
||||
- SQLite becomes **write-only** for packets — reads never touch disk
|
||||
- New packets from MQTT written to both RAM + SQLite
|
||||
|
||||
2. **TTL Cache** (`server.js`)
|
||||
- Computed API responses cached with configurable TTLs (via `config.json`)
|
||||
- Smart invalidation: packet bursts only invalidate channels/observers; analytics expire by TTL only
|
||||
- Pre-warmed on startup: subpaths, RF, topology, channels, hash-sizes, bulk-health
|
||||
- Result: most API responses served in **1-2ms** from cache
|
||||
|
||||
### Key Optimizations
|
||||
|
||||
- **Eliminated all `LIKE '%pubkey%'` queries**: Every node-specific endpoint was doing full-table scans on the packets table via `decoded_json LIKE '%pubkey%'`. Replaced with O(1) `pktStore.byNode` Map lookups.
|
||||
- **Single-pass computations**: Channels, analytics, and subpaths computed in one loop instead of multiple SQL queries.
|
||||
- **Client-side WebSocket prepend**: New packets appended to the table without re-fetching the API.
|
||||
- **RF response compression**: Server-side histograms + scatter downsampling (1MB → 15KB).
|
||||
- **Configurable everything**: All TTLs, packet store limits, and thresholds in `config.json`.
|
||||
|
||||
### What Didn't Work
|
||||
|
||||
- **Background refresh (`setInterval`)**: Attempted to re-warm caches at 80% TTL. Blocked the event loop — Node.js is single-threaded. Response times went from 3ms to 1,200ms. Reverted immediately.
|
||||
- **Worker threads**: `structuredClone` overhead of 416ms for 28K packets negated the compute savings. Only viable at 10× data growth or with `SharedArrayBuffer` (zero-copy).
|
||||
|
||||
## Running the Benchmark
|
||||
|
||||
```bash
|
||||
# Stop the production server first
|
||||
supervisorctl stop meshcore-analyzer
|
||||
|
||||
# Run A/B benchmark (launches two servers: old v2.0.1 vs current)
|
||||
./benchmark-ab.sh
|
||||
|
||||
# Restart production
|
||||
supervisorctl start meshcore-analyzer
|
||||
```
|
||||
14
README.md
14
README.md
@@ -53,6 +53,20 @@ Full experience on your phone — proper touch controls, iOS safe area support,
|
||||
- **Mobile Responsive** — proper two-row VCR bar, iOS safe area support, touch-friendly
|
||||
- **Accessible** — ARIA patterns, keyboard navigation, screen reader support, distinct marker shapes
|
||||
|
||||
### ⚡ Performance (v2.1.0)
|
||||
|
||||
Two-layer caching architecture: in-memory packet store + TTL response cache. All packet reads served from RAM — SQLite is write-only. Heavy endpoints pre-warmed on startup.
|
||||
|
||||
| Endpoint | Before | After | Speedup |
|
||||
|---|---|---|---|
|
||||
| Bulk Health | 7,059 ms | 1 ms | **7,059×** |
|
||||
| Node Analytics | 381 ms | 1 ms | **381×** |
|
||||
| Topology | 685 ms | 2 ms | **342×** |
|
||||
| Node Health | 195 ms | 1 ms | **195×** |
|
||||
| Node Detail | 133 ms | 1 ms | **133×** |
|
||||
|
||||
See [PERFORMANCE.md](PERFORMANCE.md) for the full benchmark.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
131
benchmark-ab.sh
Executable file
131
benchmark-ab.sh
Executable file
@@ -0,0 +1,131 @@
|
||||
#!/bin/bash
|
||||
# A/B benchmark: old (pre-perf) vs new (current)
|
||||
# Usage: ./benchmark-ab.sh
|
||||
set -e
|
||||
|
||||
PORT_OLD=13003
|
||||
PORT_NEW=13004
|
||||
RUNS=3
|
||||
DB_PATH="$(pwd)/data/meshcore.db"
|
||||
|
||||
OLD_COMMIT="23caae4"
|
||||
NEW_COMMIT="$(git rev-parse HEAD)"
|
||||
|
||||
echo "═══════════════════════════════════════════════════════"
|
||||
echo " A/B Benchmark: Pre-optimization vs Current"
|
||||
echo "═══════════════════════════════════════════════════════"
|
||||
echo "OLD: $OLD_COMMIT (v2.0.1 — before any perf work)"
|
||||
echo "NEW: $NEW_COMMIT (current)"
|
||||
echo "Runs per endpoint: $RUNS"
|
||||
echo ""
|
||||
|
||||
# Get a real node pubkey for testing
|
||||
ORIG_DIR="$(pwd)"
|
||||
PUBKEY=$(sqlite3 "$DB_PATH" "SELECT public_key FROM nodes ORDER BY last_seen DESC LIMIT 1")
|
||||
echo "Test node: ${PUBKEY:0:16}..."
|
||||
echo ""
|
||||
|
||||
# Setup old version in temp dir
|
||||
OLD_DIR=$(mktemp -d)
|
||||
echo "Cloning old version to $OLD_DIR..."
|
||||
git worktree add "$OLD_DIR" "$OLD_COMMIT" --quiet 2>/dev/null || {
|
||||
git worktree add "$OLD_DIR" "$OLD_COMMIT" --detach --quiet
|
||||
}
|
||||
# Copy config + db symlink
|
||||
# Copy config + db + share node_modules
|
||||
cp config.json "$OLD_DIR/"
|
||||
mkdir -p "$OLD_DIR/data"
|
||||
cp "$ORIG_DIR/data/meshcore.db" "$OLD_DIR/data/meshcore.db"
|
||||
ln -sf "$ORIG_DIR/node_modules" "$OLD_DIR/node_modules"
|
||||
|
||||
ENDPOINTS=(
|
||||
"Stats|/api/stats"
|
||||
"Packets(50)|/api/packets?limit=50"
|
||||
"PacketsGrouped|/api/packets?limit=50&groupByHash=true"
|
||||
"NodesList|/api/nodes?limit=50"
|
||||
"NodeDetail|/api/nodes/$PUBKEY"
|
||||
"NodeHealth|/api/nodes/$PUBKEY/health"
|
||||
"NodeAnalytics|/api/nodes/$PUBKEY/analytics?days=7"
|
||||
"BulkHealth|/api/nodes/bulk-health?limit=50"
|
||||
"NetworkStatus|/api/nodes/network-status"
|
||||
"Channels|/api/channels"
|
||||
"Observers|/api/observers"
|
||||
"RF|/api/analytics/rf"
|
||||
"Topology|/api/analytics/topology"
|
||||
"ChannelAnalytics|/api/analytics/channels"
|
||||
"HashSizes|/api/analytics/hash-sizes"
|
||||
)
|
||||
|
||||
bench_endpoint() {
|
||||
local port=$1 path=$2 runs=$3 nocache=$4
|
||||
local total=0
|
||||
for i in $(seq 1 $runs); do
|
||||
local url="http://127.0.0.1:$port$path"
|
||||
if [ "$nocache" = "1" ]; then
|
||||
if echo "$path" | grep -q '?'; then
|
||||
url="${url}&nocache=1"
|
||||
else
|
||||
url="${url}?nocache=1"
|
||||
fi
|
||||
fi
|
||||
local ms=$(curl -s -o /dev/null -w "%{time_total}" "$url" 2>/dev/null)
|
||||
local ms_int=$(echo "$ms * 1000" | bc | cut -d. -f1)
|
||||
total=$((total + ms_int))
|
||||
done
|
||||
echo $((total / runs))
|
||||
}
|
||||
|
||||
# Launch old server
|
||||
echo "Starting OLD server (port $PORT_OLD)..."
|
||||
cd "$OLD_DIR"
|
||||
PORT=$PORT_OLD node server.js &>/dev/null &
|
||||
OLD_PID=$!
|
||||
cd - >/dev/null
|
||||
|
||||
# Launch new server
|
||||
echo "Starting NEW server (port $PORT_NEW)..."
|
||||
PORT=$PORT_NEW node server.js &>/dev/null &
|
||||
NEW_PID=$!
|
||||
|
||||
# Wait for both
|
||||
sleep 12 # old server has no memory store; new needs prewarm
|
||||
|
||||
# Verify
|
||||
curl -s "http://127.0.0.1:$PORT_OLD/api/stats" >/dev/null 2>&1 || { echo "OLD server failed to start"; kill $OLD_PID $NEW_PID 2>/dev/null; exit 1; }
|
||||
curl -s "http://127.0.0.1:$PORT_NEW/api/stats" >/dev/null 2>&1 || { echo "NEW server failed to start"; kill $OLD_PID $NEW_PID 2>/dev/null; exit 1; }
|
||||
|
||||
echo ""
|
||||
echo "Warming up caches on new server..."
|
||||
for ep in "${ENDPOINTS[@]}"; do
|
||||
path="${ep#*|}"
|
||||
curl -s -o /dev/null "http://127.0.0.1:$PORT_NEW$path" 2>/dev/null
|
||||
done
|
||||
sleep 2
|
||||
|
||||
printf "\n%-22s %9s %9s %9s %9s\n" "Endpoint" "Old(ms)" "New-cold" "New-cache" "Speedup"
|
||||
printf "%-22s %9s %9s %9s %9s\n" "──────────────────────" "─────────" "─────────" "─────────" "─────────"
|
||||
|
||||
for ep in "${ENDPOINTS[@]}"; do
|
||||
name="${ep%%|*}"
|
||||
path="${ep#*|}"
|
||||
|
||||
old_ms=$(bench_endpoint $PORT_OLD "$path" $RUNS 0)
|
||||
new_cold=$(bench_endpoint $PORT_NEW "$path" $RUNS 1)
|
||||
new_cached=$(bench_endpoint $PORT_NEW "$path" $RUNS 0)
|
||||
|
||||
if [ "$old_ms" -gt 0 ] && [ "$new_cached" -gt 0 ]; then
|
||||
speedup="${old_ms}/${new_cached}"
|
||||
speedup_x=$(echo "scale=0; $old_ms / $new_cached" | bc 2>/dev/null || echo "?")
|
||||
printf "%-22s %7dms %7dms %7dms %7d×\n" "$name" "$old_ms" "$new_cold" "$new_cached" "$speedup_x"
|
||||
else
|
||||
printf "%-22s %7dms %7dms %7dms %9s\n" "$name" "$old_ms" "$new_cold" "$new_cached" "∞"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "═══════════════════════════════════════════════════════"
|
||||
|
||||
# Cleanup
|
||||
kill $OLD_PID $NEW_PID 2>/dev/null
|
||||
git worktree remove "$OLD_DIR" --force 2>/dev/null
|
||||
echo "Done."
|
||||
246
benchmark.js
Normal file
246
benchmark.js
Normal file
@@ -0,0 +1,246 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Benchmark suite for meshcore-analyzer.
|
||||
* Launches two server instances — one with in-memory store, one with pure SQLite —
|
||||
* and compares performance side by side.
|
||||
*
|
||||
* Usage: node benchmark.js [--runs 5] [--json]
|
||||
*/
|
||||
|
||||
const http = require('http');
|
||||
const { spawn } = require('child_process');
|
||||
const path = require('path');
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const RUNS = Number(args.find((a, i) => args[i - 1] === '--runs') || 5);
|
||||
const JSON_OUT = args.includes('--json');
|
||||
|
||||
const PORT_MEM = 13001; // In-memory store
|
||||
const PORT_SQL = 13002; // SQLite-only
|
||||
|
||||
const ENDPOINTS = [
|
||||
{ name: 'Stats', path: '/api/stats' },
|
||||
{ name: 'Packets (50)', path: '/api/packets?limit=50' },
|
||||
{ name: 'Packets (100)', path: '/api/packets?limit=100' },
|
||||
{ name: 'Packets grouped', path: '/api/packets?limit=100&groupByHash=true' },
|
||||
{ name: 'Packets filtered', path: '/api/packets?limit=50&type=5' },
|
||||
{ name: 'Packets timestamps', path: '/api/packets/timestamps?since=2020-01-01' },
|
||||
{ name: 'Nodes list', path: '/api/nodes?limit=50' },
|
||||
{ name: 'Node detail', path: '/api/nodes/__FIRST_NODE__' },
|
||||
{ name: 'Node health', path: '/api/nodes/__FIRST_NODE__/health' },
|
||||
{ name: 'Bulk health', path: '/api/nodes/bulk-health?limit=50' },
|
||||
{ name: 'Network status', path: '/api/nodes/network-status' },
|
||||
{ name: 'Observers', path: '/api/observers' },
|
||||
{ name: 'Channels', path: '/api/channels' },
|
||||
{ name: 'RF Analytics', path: '/api/analytics/rf' },
|
||||
{ name: 'Topology', path: '/api/analytics/topology' },
|
||||
{ name: 'Channel Analytics', path: '/api/analytics/channels' },
|
||||
{ name: 'Hash Sizes', path: '/api/analytics/hash-sizes' },
|
||||
{ name: 'Subpaths 2-hop', path: '/api/analytics/subpaths?minLen=2&maxLen=2&limit=50' },
|
||||
{ name: 'Subpaths 3-hop', path: '/api/analytics/subpaths?minLen=3&maxLen=3&limit=30' },
|
||||
{ name: 'Subpaths 4-hop', path: '/api/analytics/subpaths?minLen=4&maxLen=4&limit=20' },
|
||||
{ name: 'Subpaths 5-8 hop', path: '/api/analytics/subpaths?minLen=5&maxLen=8&limit=15' },
|
||||
];
|
||||
|
||||
function fetch(url) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const t0 = process.hrtime.bigint();
|
||||
const req = http.get(url, (res) => {
|
||||
let body = '';
|
||||
res.on('data', c => body += c);
|
||||
res.on('end', () => {
|
||||
const ms = Number(process.hrtime.bigint() - t0) / 1e6;
|
||||
resolve({ ms, bytes: Buffer.byteLength(body), status: res.statusCode, body });
|
||||
});
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.setTimeout(60000, () => { req.destroy(); reject(new Error('timeout')); });
|
||||
});
|
||||
}
|
||||
|
||||
function median(arr) { const s = [...arr].sort((a,b)=>a-b); return s[Math.floor(s.length/2)]; }
|
||||
function p95(arr) { const s = [...arr].sort((a,b)=>a-b); return s[Math.floor(s.length*0.95)]; }
|
||||
function avg(arr) { return arr.reduce((a,b)=>a+b,0)/arr.length; }
|
||||
function fmt(ms) { return ms >= 1000 ? (ms/1000).toFixed(1)+'s' : ms.toFixed(1)+'ms'; }
|
||||
function fmtSize(b) { return b >= 1048576 ? (b/1048576).toFixed(1)+'MB' : b >= 1024 ? (b/1024).toFixed(0)+'KB' : b+'B'; }
|
||||
|
||||
function launchServer(port, env = {}) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const child = spawn('node', ['server.js'], {
|
||||
cwd: __dirname,
|
||||
env: { ...process.env, PORT: String(port), ...env },
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
let started = false;
|
||||
const timeout = setTimeout(() => { if (!started) { child.kill(); reject(new Error('Server start timeout')); } }, 30000);
|
||||
|
||||
child.stdout.on('data', (d) => {
|
||||
if (!started && (d.toString().includes('listening') || d.toString().includes('running'))) {
|
||||
started = true; clearTimeout(timeout); resolve(child);
|
||||
}
|
||||
});
|
||||
child.stderr.on('data', (d) => {
|
||||
if (!started && (d.toString().includes('listening') || d.toString().includes('running'))) {
|
||||
started = true; clearTimeout(timeout); resolve(child);
|
||||
}
|
||||
});
|
||||
child.on('exit', (code) => { if (!started) { clearTimeout(timeout); reject(new Error(`Server exited with ${code}`)); } });
|
||||
|
||||
// Fallback: wait longer (SQLite-only mode pre-warms subpaths ~6s)
|
||||
setTimeout(() => {
|
||||
if (!started) {
|
||||
started = true; clearTimeout(timeout);
|
||||
resolve(child);
|
||||
}
|
||||
}, 15000);
|
||||
});
|
||||
}
|
||||
|
||||
async function waitForServer(port, maxMs = 20000) {
|
||||
const t0 = Date.now();
|
||||
while (Date.now() - t0 < maxMs) {
|
||||
try {
|
||||
const r = await fetch(`http://127.0.0.1:${port}/api/stats`);
|
||||
if (r.status === 200) return true;
|
||||
} catch {}
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
}
|
||||
throw new Error(`Server on port ${port} didn't start`);
|
||||
}
|
||||
|
||||
async function benchmarkEndpoints(port, endpoints, nocache = false) {
|
||||
const results = [];
|
||||
for (const ep of endpoints) {
|
||||
const suffix = nocache ? (ep.path.includes('?') ? '&nocache=1' : '?nocache=1') : '';
|
||||
const url = `http://127.0.0.1:${port}${ep.path}${suffix}`;
|
||||
|
||||
// Warm-up
|
||||
try { await fetch(url); } catch {}
|
||||
|
||||
const times = [];
|
||||
let bytes = 0;
|
||||
let failed = false;
|
||||
|
||||
for (let i = 0; i < RUNS; i++) {
|
||||
try {
|
||||
const r = await fetch(url);
|
||||
if (r.status !== 200) { failed = true; break; }
|
||||
times.push(r.ms);
|
||||
bytes = r.bytes;
|
||||
} catch { failed = true; break; }
|
||||
}
|
||||
|
||||
if (failed || !times.length) {
|
||||
results.push({ name: ep.name, failed: true });
|
||||
} else {
|
||||
results.push({
|
||||
name: ep.name,
|
||||
avg: Math.round(avg(times) * 10) / 10,
|
||||
p50: Math.round(median(times) * 10) / 10,
|
||||
p95: Math.round(p95(times) * 10) / 10,
|
||||
bytes
|
||||
});
|
||||
}
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
async function run() {
|
||||
console.log(`\nMeshCore Analyzer Benchmark — ${RUNS} runs per endpoint`);
|
||||
console.log('Launching servers...\n');
|
||||
|
||||
// Launch both servers
|
||||
let memServer, sqlServer;
|
||||
try {
|
||||
console.log(' Starting in-memory server (port ' + PORT_MEM + ')...');
|
||||
memServer = await launchServer(PORT_MEM, {});
|
||||
await waitForServer(PORT_MEM);
|
||||
console.log(' ✅ In-memory server ready');
|
||||
|
||||
console.log(' Starting SQLite-only server (port ' + PORT_SQL + ')...');
|
||||
sqlServer = await launchServer(PORT_SQL, { NO_MEMORY_STORE: '1' });
|
||||
await waitForServer(PORT_SQL);
|
||||
console.log(' ✅ SQLite-only server ready\n');
|
||||
} catch (e) {
|
||||
console.error('Failed to start servers:', e.message);
|
||||
if (memServer) memServer.kill();
|
||||
if (sqlServer) sqlServer.kill();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Get first node pubkey
|
||||
let firstNode = '';
|
||||
try {
|
||||
const r = await fetch(`http://127.0.0.1:${PORT_MEM}/api/nodes?limit=1`);
|
||||
const data = JSON.parse(r.body);
|
||||
firstNode = data.nodes?.[0]?.public_key || '';
|
||||
} catch {}
|
||||
|
||||
const endpoints = ENDPOINTS.map(e => ({
|
||||
...e,
|
||||
path: e.path.replace('__FIRST_NODE__', firstNode),
|
||||
}));
|
||||
|
||||
// Get packet count
|
||||
try {
|
||||
const r = await fetch(`http://127.0.0.1:${PORT_MEM}/api/stats`);
|
||||
const stats = JSON.parse(r.body);
|
||||
console.log(`Dataset: ${(stats.totalPackets || '?').toLocaleString()} packets\n`);
|
||||
} catch {}
|
||||
|
||||
// Run benchmarks
|
||||
console.log('Benchmarking in-memory store (nocache for true compute cost)...');
|
||||
const memResults = await benchmarkEndpoints(PORT_MEM, endpoints, true);
|
||||
|
||||
console.log('Benchmarking SQLite-only (nocache)...');
|
||||
const sqlResults = await benchmarkEndpoints(PORT_SQL, endpoints, true);
|
||||
|
||||
// Also test cached in-memory for the full picture
|
||||
console.log('Benchmarking in-memory store (cached)...');
|
||||
const memCachedResults = await benchmarkEndpoints(PORT_MEM, endpoints, false);
|
||||
|
||||
// Kill servers
|
||||
memServer.kill();
|
||||
sqlServer.kill();
|
||||
|
||||
if (JSON_OUT) {
|
||||
console.log(JSON.stringify({ memoryNocache: memResults, sqliteNocache: sqlResults, memoryCached: memCachedResults }, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
// Print results
|
||||
const W = 94;
|
||||
console.log(`\n${'═'.repeat(W)}`);
|
||||
console.log(' 🏁 BENCHMARK RESULTS: SQLite vs In-Memory Store');
|
||||
console.log(`${'═'.repeat(W)}`);
|
||||
console.log(`${'Endpoint'.padEnd(24)} ${'SQLite'.padStart(9)} ${'Memory'.padStart(9)} ${'Cached'.padStart(9)} ${'Speedup'.padStart(9)} ${'Size (SQL)'.padStart(10)} ${'Size (Mem)'.padStart(10)}`);
|
||||
console.log(`${'─'.repeat(24)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(10)} ${'─'.repeat(10)}`);
|
||||
|
||||
for (let i = 0; i < endpoints.length; i++) {
|
||||
const sql = sqlResults[i];
|
||||
const mem = memResults[i];
|
||||
const cached = memCachedResults[i];
|
||||
if (!sql || sql.failed || !mem || mem.failed) {
|
||||
console.log(`${endpoints[i].name.padEnd(24)} ${'FAILED'.padStart(9)}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const speedup = sql.avg > 0 && mem.avg > 0 ? Math.round(sql.avg / mem.avg) + '×' : '—';
|
||||
const cachedStr = cached && !cached.failed ? fmt(cached.avg) : '—';
|
||||
|
||||
console.log(
|
||||
`${sql.name.padEnd(24)} ${fmt(sql.avg).padStart(9)} ${fmt(mem.avg).padStart(9)} ${cachedStr.padStart(9)} ${speedup.padStart(9)} ${fmtSize(sql.bytes).padStart(10)} ${fmtSize(mem.bytes).padStart(10)}`
|
||||
);
|
||||
}
|
||||
|
||||
// Summary
|
||||
const sqlTotal = sqlResults.filter(r => !r.failed).reduce((s, r) => s + r.avg, 0);
|
||||
const memTotal = memResults.filter(r => !r.failed).reduce((s, r) => s + r.avg, 0);
|
||||
console.log(`${'─'.repeat(24)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)}`);
|
||||
console.log(`${'TOTAL'.padEnd(24)} ${fmt(sqlTotal).padStart(9)} ${fmt(memTotal).padStart(9)} ${''.padStart(9)} ${(Math.round(sqlTotal/memTotal)+'×').padStart(9)}`);
|
||||
console.log(`\n${'═'.repeat(W)}\n`);
|
||||
}
|
||||
|
||||
run().catch(e => { console.error(e); process.exit(1); });
|
||||
@@ -22,5 +22,31 @@
|
||||
"OAK": "Oakland, US",
|
||||
"MRY": "Monterey, US",
|
||||
"LAR": "Los Angeles, US"
|
||||
},
|
||||
"cacheTTL": {
|
||||
"stats": 10,
|
||||
"nodeDetail": 300,
|
||||
"nodeHealth": 300,
|
||||
"nodeList": 90,
|
||||
"bulkHealth": 600,
|
||||
"networkStatus": 600,
|
||||
"observers": 300,
|
||||
"channels": 15,
|
||||
"channelMessages": 10,
|
||||
"analyticsRF": 1800,
|
||||
"analyticsTopology": 1800,
|
||||
"analyticsChannels": 1800,
|
||||
"analyticsHashSizes": 3600,
|
||||
"analyticsSubpaths": 3600,
|
||||
"analyticsSubpathDetail": 3600,
|
||||
"nodeAnalytics": 60,
|
||||
"nodeSearch": 10,
|
||||
"invalidationDebounce": 30,
|
||||
"_comment": "All values in seconds. Server uses these directly. Client fetches via /api/config/cache."
|
||||
},
|
||||
"packetStore": {
|
||||
"maxMemoryMB": 1024,
|
||||
"estimatedPacketBytes": 450,
|
||||
"_comment": "In-memory packet store. maxMemoryMB caps RAM usage. All packets loaded on startup, served from RAM."
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "meshcore-analyzer",
|
||||
"version": "2.0.1",
|
||||
"version": "2.1.0",
|
||||
"description": "Community-run alternative to the closed-source `analyzer.letsmesh.net`. MQTT packet collection + open-source web analyzer for the Bay Area MeshCore mesh.",
|
||||
"main": "index.js",
|
||||
"scripts": {
|
||||
|
||||
342
packet-store.js
Normal file
342
packet-store.js
Normal file
@@ -0,0 +1,342 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* In-memory packet store — loads all packets from SQLite on startup,
|
||||
* serves reads from RAM, writes to both RAM + SQLite.
|
||||
* Caps memory at configurable limit (default 1GB).
|
||||
*/
|
||||
class PacketStore {
|
||||
constructor(dbModule, config = {}) {
|
||||
this.dbModule = dbModule; // The full db module (has .db, .insertPacket, .getPacket)
|
||||
this.db = dbModule.db; // Raw better-sqlite3 instance for queries
|
||||
this.maxBytes = (config.maxMemoryMB || 1024) * 1024 * 1024;
|
||||
this.estPacketBytes = config.estimatedPacketBytes || 450;
|
||||
this.maxPackets = Math.floor(this.maxBytes / this.estPacketBytes);
|
||||
|
||||
// SQLite-only mode: skip RAM loading, all reads go to DB
|
||||
this.sqliteOnly = process.env.NO_MEMORY_STORE === '1';
|
||||
|
||||
// Core storage: array sorted by timestamp DESC (newest first)
|
||||
this.packets = [];
|
||||
// Indexes
|
||||
this.byId = new Map();
|
||||
this.byHash = new Map(); // hash → [packet, ...]
|
||||
this.byObserver = new Map(); // observer_id → [packet, ...]
|
||||
this.byNode = new Map(); // pubkey → [packet, ...]
|
||||
|
||||
this.loaded = false;
|
||||
this.stats = { totalLoaded: 0, evicted: 0, inserts: 0, queries: 0 };
|
||||
}
|
||||
|
||||
/** Load all packets from SQLite into memory */
|
||||
load() {
|
||||
if (this.sqliteOnly) {
|
||||
console.log('[PacketStore] SQLite-only mode (NO_MEMORY_STORE=1) — all reads go to database');
|
||||
this.loaded = true;
|
||||
return this;
|
||||
}
|
||||
const t0 = Date.now();
|
||||
const rows = this.db.prepare(
|
||||
'SELECT * FROM packets ORDER BY timestamp DESC'
|
||||
).all();
|
||||
|
||||
for (const row of rows) {
|
||||
if (this.packets.length >= this.maxPackets) break;
|
||||
this._index(row);
|
||||
this.packets.push(row);
|
||||
}
|
||||
|
||||
this.stats.totalLoaded = this.packets.length;
|
||||
this.loaded = true;
|
||||
const elapsed = Date.now() - t0;
|
||||
console.log(`[PacketStore] Loaded ${this.packets.length} packets in ${elapsed}ms (${Math.round(this.packets.length * this.estPacketBytes / 1024 / 1024)}MB est)`);
|
||||
return this;
|
||||
}
|
||||
|
||||
/** Index a packet into all lookup maps */
|
||||
_index(pkt) {
|
||||
this.byId.set(pkt.id, pkt);
|
||||
|
||||
if (pkt.hash) {
|
||||
if (!this.byHash.has(pkt.hash)) this.byHash.set(pkt.hash, []);
|
||||
this.byHash.get(pkt.hash).push(pkt);
|
||||
}
|
||||
|
||||
if (pkt.observer_id) {
|
||||
if (!this.byObserver.has(pkt.observer_id)) this.byObserver.set(pkt.observer_id, []);
|
||||
this.byObserver.get(pkt.observer_id).push(pkt);
|
||||
}
|
||||
|
||||
// Index by node pubkeys mentioned in decoded_json
|
||||
this._indexByNode(pkt);
|
||||
}
|
||||
|
||||
/** Extract node pubkeys/names from decoded_json and index */
|
||||
_indexByNode(pkt) {
|
||||
if (!pkt.decoded_json) return;
|
||||
try {
|
||||
const decoded = JSON.parse(pkt.decoded_json);
|
||||
const keys = new Set();
|
||||
if (decoded.pubKey) keys.add(decoded.pubKey);
|
||||
if (decoded.destPubKey) keys.add(decoded.destPubKey);
|
||||
if (decoded.srcPubKey) keys.add(decoded.srcPubKey);
|
||||
for (const k of keys) {
|
||||
if (!this.byNode.has(k)) this.byNode.set(k, []);
|
||||
this.byNode.get(k).push(pkt);
|
||||
}
|
||||
} catch {}
|
||||
}
|
||||
|
||||
/** Remove oldest packets when over memory limit */
|
||||
_evict() {
|
||||
while (this.packets.length > this.maxPackets) {
|
||||
const old = this.packets.pop();
|
||||
this.byId.delete(old.id);
|
||||
// Remove from hash index
|
||||
if (old.hash && this.byHash.has(old.hash)) {
|
||||
const arr = this.byHash.get(old.hash).filter(p => p.id !== old.id);
|
||||
if (arr.length) this.byHash.set(old.hash, arr); else this.byHash.delete(old.hash);
|
||||
}
|
||||
// Remove from observer index
|
||||
if (old.observer_id && this.byObserver.has(old.observer_id)) {
|
||||
const arr = this.byObserver.get(old.observer_id).filter(p => p.id !== old.id);
|
||||
if (arr.length) this.byObserver.set(old.observer_id, arr); else this.byObserver.delete(old.observer_id);
|
||||
}
|
||||
// Skip node index cleanup for eviction (expensive, low value)
|
||||
this.stats.evicted++;
|
||||
}
|
||||
}
|
||||
|
||||
/** Insert a new packet (to both memory and SQLite) */
|
||||
insert(packetData) {
|
||||
const id = this.dbModule.insertPacket(packetData);
|
||||
const row = this.dbModule.getPacket(id);
|
||||
if (row) {
|
||||
this.packets.unshift(row); // newest first
|
||||
this._index(row);
|
||||
this._evict();
|
||||
this.stats.inserts++;
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
/** Query packets with filters — all from memory (or SQLite in fallback mode) */
|
||||
query({ limit = 50, offset = 0, type, route, region, observer, hash, since, until, node, order = 'DESC' } = {}) {
|
||||
this.stats.queries++;
|
||||
|
||||
if (this.sqliteOnly) return this._querySQLite({ limit, offset, type, route, region, observer, hash, since, until, node, order });
|
||||
|
||||
let results = this.packets;
|
||||
|
||||
// Use indexes for single-key filters when possible
|
||||
if (hash && !type && !route && !region && !observer && !since && !until && !node) {
|
||||
results = this.byHash.get(hash) || [];
|
||||
} else if (observer && !type && !route && !region && !hash && !since && !until && !node) {
|
||||
results = this.byObserver.get(observer) || [];
|
||||
} else if (node && !type && !route && !region && !observer && !hash && !since && !until) {
|
||||
results = this.byNode.get(node) || [];
|
||||
} else {
|
||||
// Apply filters sequentially
|
||||
if (type !== undefined) {
|
||||
const t = Number(type);
|
||||
results = results.filter(p => p.payload_type === t);
|
||||
}
|
||||
if (route !== undefined) {
|
||||
const r = Number(route);
|
||||
results = results.filter(p => p.route_type === r);
|
||||
}
|
||||
if (observer) results = results.filter(p => p.observer_id === observer);
|
||||
if (hash) results = results.filter(p => p.hash === hash);
|
||||
if (since) results = results.filter(p => p.timestamp > since);
|
||||
if (until) results = results.filter(p => p.timestamp < until);
|
||||
if (region) {
|
||||
// Need to look up observers for this region
|
||||
const regionObservers = new Set();
|
||||
try {
|
||||
const obs = this.db.prepare('SELECT id FROM observers WHERE iata = ?').all(region);
|
||||
obs.forEach(o => regionObservers.add(o.id));
|
||||
} catch {}
|
||||
results = results.filter(p => regionObservers.has(p.observer_id));
|
||||
}
|
||||
if (node) {
|
||||
// Check indexed results first, fall back to text search
|
||||
const indexed = this.byNode.get(node);
|
||||
if (indexed) {
|
||||
const idSet = new Set(indexed.map(p => p.id));
|
||||
results = results.filter(p => idSet.has(p.id));
|
||||
} else {
|
||||
// Text search fallback (node name)
|
||||
results = results.filter(p =>
|
||||
p.decoded_json && p.decoded_json.includes(node)
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const total = results.length;
|
||||
|
||||
// Sort
|
||||
if (order === 'ASC') {
|
||||
results = results.slice().sort((a, b) => {
|
||||
if (a.timestamp < b.timestamp) return -1;
|
||||
if (a.timestamp > b.timestamp) return 1;
|
||||
return 0;
|
||||
});
|
||||
}
|
||||
// Default DESC — packets array is already sorted newest-first
|
||||
|
||||
// Paginate
|
||||
const paginated = results.slice(Number(offset), Number(offset) + Number(limit));
|
||||
return { packets: paginated, total };
|
||||
}
|
||||
|
||||
/** Query with groupByHash — aggregate packets by content hash */
|
||||
queryGrouped({ limit = 50, offset = 0, type, route, region, observer, hash, since, until, node } = {}) {
|
||||
this.stats.queries++;
|
||||
|
||||
if (this.sqliteOnly) return this._queryGroupedSQLite({ limit, offset, type, route, region, observer, hash, since, until, node });
|
||||
|
||||
// Get filtered results first
|
||||
const { packets: filtered, total: filteredTotal } = this.query({
|
||||
limit: 999999, offset: 0, type, route, region, observer, hash, since, until, node
|
||||
});
|
||||
|
||||
// Group by hash
|
||||
const groups = new Map();
|
||||
for (const p of filtered) {
|
||||
const h = p.hash || p.id.toString();
|
||||
if (!groups.has(h)) {
|
||||
groups.set(h, {
|
||||
hash: p.hash,
|
||||
observer_count: new Set(),
|
||||
count: 0,
|
||||
latest: p.timestamp,
|
||||
observer_id: p.observer_id,
|
||||
observer_name: p.observer_name,
|
||||
path_json: p.path_json,
|
||||
payload_type: p.payload_type,
|
||||
raw_hex: p.raw_hex,
|
||||
decoded_json: p.decoded_json,
|
||||
});
|
||||
}
|
||||
const g = groups.get(h);
|
||||
g.count++;
|
||||
if (p.observer_id) g.observer_count.add(p.observer_id);
|
||||
if (p.timestamp > g.latest) {
|
||||
g.latest = p.timestamp;
|
||||
}
|
||||
// Keep longest path
|
||||
if (p.path_json && (!g.path_json || p.path_json.length > g.path_json.length)) {
|
||||
g.path_json = p.path_json;
|
||||
g.raw_hex = p.raw_hex;
|
||||
}
|
||||
}
|
||||
|
||||
// Sort by latest DESC, paginate
|
||||
const sorted = [...groups.values()]
|
||||
.map(g => ({ ...g, observer_count: g.observer_count.size }))
|
||||
.sort((a, b) => b.latest.localeCompare(a.latest));
|
||||
|
||||
const total = sorted.length;
|
||||
const paginated = sorted.slice(Number(offset), Number(offset) + Number(limit));
|
||||
return { packets: paginated, total };
|
||||
}
|
||||
|
||||
/** Get timestamps for sparkline */
|
||||
getTimestamps(since) {
|
||||
if (this.sqliteOnly) {
|
||||
return this.db.prepare('SELECT timestamp FROM packets WHERE timestamp > ? ORDER BY timestamp ASC').all(since).map(r => r.timestamp);
|
||||
}
|
||||
const results = [];
|
||||
for (const p of this.packets) {
|
||||
if (p.timestamp <= since) break;
|
||||
results.push(p.timestamp);
|
||||
}
|
||||
return results.reverse();
|
||||
}
|
||||
|
||||
/** Get a single packet by ID */
|
||||
getById(id) {
|
||||
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets WHERE id = ?').get(id) || null;
|
||||
return this.byId.get(id) || null;
|
||||
}
|
||||
|
||||
/** Get all siblings of a packet (same hash) */
|
||||
getSiblings(hash) {
|
||||
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets WHERE hash = ? ORDER BY timestamp DESC').all(hash);
|
||||
return this.byHash.get(hash) || [];
|
||||
}
|
||||
|
||||
/** Get all packets (raw array reference — do not mutate) */
|
||||
all() {
|
||||
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets ORDER BY timestamp DESC').all();
|
||||
return this.packets;
|
||||
}
|
||||
|
||||
/** Get all packets matching a filter function */
|
||||
filter(fn) {
|
||||
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets ORDER BY timestamp DESC').all().filter(fn);
|
||||
return this.packets.filter(fn);
|
||||
}
|
||||
|
||||
/** Memory stats */
|
||||
getStats() {
|
||||
return {
|
||||
...this.stats,
|
||||
inMemory: this.sqliteOnly ? 0 : this.packets.length,
|
||||
sqliteOnly: this.sqliteOnly,
|
||||
maxPackets: this.maxPackets,
|
||||
estimatedMB: this.sqliteOnly ? 0 : Math.round(this.packets.length * this.estPacketBytes / 1024 / 1024),
|
||||
maxMB: Math.round(this.maxBytes / 1024 / 1024),
|
||||
indexes: {
|
||||
byHash: this.byHash.size,
|
||||
byObserver: this.byObserver.size,
|
||||
byNode: this.byNode.size,
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/** SQLite fallback: query with filters */
|
||||
_querySQLite({ limit, offset, type, route, region, observer, hash, since, until, node, order }) {
|
||||
const where = []; const params = [];
|
||||
if (type !== undefined) { where.push('payload_type = ?'); params.push(Number(type)); }
|
||||
if (route !== undefined) { where.push('route_type = ?'); params.push(Number(route)); }
|
||||
if (observer) { where.push('observer_id = ?'); params.push(observer); }
|
||||
if (hash) { where.push('hash = ?'); params.push(hash); }
|
||||
if (since) { where.push('timestamp > ?'); params.push(since); }
|
||||
if (until) { where.push('timestamp < ?'); params.push(until); }
|
||||
if (region) { where.push('observer_id IN (SELECT id FROM observers WHERE iata = ?)'); params.push(region); }
|
||||
if (node) { where.push('decoded_json LIKE ?'); params.push(`%${node}%`); }
|
||||
const w = where.length ? 'WHERE ' + where.join(' AND ') : '';
|
||||
const total = this.db.prepare(`SELECT COUNT(*) as c FROM packets ${w}`).get(...params).c;
|
||||
const packets = this.db.prepare(`SELECT * FROM packets ${w} ORDER BY timestamp ${order === 'ASC' ? 'ASC' : 'DESC'} LIMIT ? OFFSET ?`).all(...params, limit, offset);
|
||||
return { packets, total };
|
||||
}
|
||||
|
||||
/** SQLite fallback: grouped query */
|
||||
_queryGroupedSQLite({ limit, offset, type, route, region, observer, hash, since, until, node }) {
|
||||
const where = []; const params = [];
|
||||
if (type !== undefined) { where.push('payload_type = ?'); params.push(Number(type)); }
|
||||
if (route !== undefined) { where.push('route_type = ?'); params.push(Number(route)); }
|
||||
if (observer) { where.push('observer_id = ?'); params.push(observer); }
|
||||
if (hash) { where.push('hash = ?'); params.push(hash); }
|
||||
if (since) { where.push('timestamp > ?'); params.push(since); }
|
||||
if (until) { where.push('timestamp < ?'); params.push(until); }
|
||||
if (region) { where.push('observer_id IN (SELECT id FROM observers WHERE iata = ?)'); params.push(region); }
|
||||
if (node) { where.push('decoded_json LIKE ?'); params.push(`%${node}%`); }
|
||||
const w = where.length ? 'WHERE ' + where.join(' AND ') : '';
|
||||
|
||||
const sql = `SELECT hash, COUNT(*) as count, COUNT(DISTINCT observer_id) as observer_count,
|
||||
MAX(timestamp) as latest, MIN(observer_id) as observer_id, MIN(observer_name) as observer_name,
|
||||
MIN(path_json) as path_json, MIN(payload_type) as payload_type, MIN(raw_hex) as raw_hex,
|
||||
MIN(decoded_json) as decoded_json
|
||||
FROM packets ${w} GROUP BY hash ORDER BY latest DESC LIMIT ? OFFSET ?`;
|
||||
const packets = this.db.prepare(sql).all(...params, limit, offset);
|
||||
|
||||
const countSql = `SELECT COUNT(DISTINCT hash) as c FROM packets ${w}`;
|
||||
const total = this.db.prepare(countSql).get(...params).c;
|
||||
return { packets, total };
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = PacketStore;
|
||||
@@ -40,7 +40,15 @@
|
||||
return svg;
|
||||
}
|
||||
|
||||
function histogram(values, bins, color, w = 800, h = 180) {
|
||||
function histogram(data, bins, color, w = 800, h = 180) {
|
||||
// Support pre-computed histogram from server { bins: [{x, w, count}], min, max }
|
||||
if (data && data.bins && Array.isArray(data.bins)) {
|
||||
const buckets = data.bins.map(b => b.count);
|
||||
const labels = data.bins.map(b => b.x.toFixed(1));
|
||||
return { svg: barChart(buckets, labels, color, w, h), buckets, labels };
|
||||
}
|
||||
// Legacy: raw values array
|
||||
const values = data;
|
||||
const min = Math.min(...values), max = Math.max(...values);
|
||||
const step = (max - min) / bins;
|
||||
const buckets = Array(bins).fill(0);
|
||||
@@ -101,10 +109,10 @@
|
||||
try {
|
||||
_analyticsData = {};
|
||||
const [hashData, rfData, topoData, chanData] = await Promise.all([
|
||||
api('/analytics/hash-sizes'),
|
||||
api('/analytics/rf'),
|
||||
api('/analytics/topology'),
|
||||
api('/analytics/channels'),
|
||||
api('/analytics/hash-sizes', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
api('/analytics/rf', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
api('/analytics/topology', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
api('/analytics/channels', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
]);
|
||||
_analyticsData = { hashData, rfData, topoData, chanData };
|
||||
renderTab('overview');
|
||||
@@ -747,7 +755,7 @@
|
||||
</div>
|
||||
`;
|
||||
let allNodes = [];
|
||||
try { const nd = await api('/nodes?limit=2000'); allNodes = nd.nodes || []; } catch {}
|
||||
try { const nd = await api('/nodes?limit=2000', { ttl: CLIENT_TTL.nodeList }); allNodes = nd.nodes || []; } catch {}
|
||||
renderHashMatrix(data.topHops, allNodes);
|
||||
renderCollisions(data.topHops, allNodes);
|
||||
}
|
||||
@@ -938,10 +946,10 @@
|
||||
el.innerHTML = '<div class="text-center text-muted" style="padding:40px">Analyzing route patterns…</div>';
|
||||
try {
|
||||
const [d2, d3, d4, d5] = await Promise.all([
|
||||
api('/analytics/subpaths?minLen=2&maxLen=2&limit=50'),
|
||||
api('/analytics/subpaths?minLen=3&maxLen=3&limit=30'),
|
||||
api('/analytics/subpaths?minLen=4&maxLen=4&limit=20'),
|
||||
api('/analytics/subpaths?minLen=5&maxLen=8&limit=15')
|
||||
api('/analytics/subpaths?minLen=2&maxLen=2&limit=50', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
api('/analytics/subpaths?minLen=3&maxLen=3&limit=30', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
api('/analytics/subpaths?minLen=4&maxLen=4&limit=20', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
api('/analytics/subpaths?minLen=5&maxLen=8&limit=15', { ttl: CLIENT_TTL.analyticsRF })
|
||||
]);
|
||||
|
||||
function renderTable(data, title) {
|
||||
@@ -1032,7 +1040,7 @@
|
||||
panel.classList.remove('collapsed');
|
||||
panel.innerHTML = '<div class="text-center text-muted" style="padding:40px">Loading…</div>';
|
||||
try {
|
||||
const data = await api('/analytics/subpath-detail?hops=' + encodeURIComponent(hopsStr));
|
||||
const data = await api('/analytics/subpath-detail?hops=' + encodeURIComponent(hopsStr), { ttl: CLIENT_TTL.analyticsRF });
|
||||
renderSubpathDetail(panel, data);
|
||||
} catch (e) {
|
||||
panel.innerHTML = `<div class="text-muted">Error: ${e.message}</div>`;
|
||||
@@ -1141,9 +1149,9 @@
|
||||
el.innerHTML = '<div style="padding:40px;text-align:center;color:var(--text-muted)">Loading node analytics…</div>';
|
||||
try {
|
||||
const [nodesResp, bulkHealth, netStatus] = await Promise.all([
|
||||
api('/nodes?limit=200&sortBy=lastSeen'),
|
||||
api('/nodes/bulk-health?limit=50'),
|
||||
api('/nodes/network-status')
|
||||
api('/nodes?limit=200&sortBy=lastSeen', { ttl: CLIENT_TTL.nodeList }),
|
||||
api('/nodes/bulk-health?limit=50', { ttl: CLIENT_TTL.analyticsRF }),
|
||||
api('/nodes/network-status', { ttl: CLIENT_TTL.analyticsRF })
|
||||
]);
|
||||
const nodes = nodesResp.nodes || nodesResp;
|
||||
const myNodes = JSON.parse(localStorage.getItem('meshcore-my-nodes') || '[]');
|
||||
|
||||
@@ -11,12 +11,81 @@ function payloadTypeName(n) { return PAYLOAD_TYPES[n] || 'UNKNOWN'; }
|
||||
function payloadTypeColor(n) { return PAYLOAD_COLORS[n] || 'unknown'; }
|
||||
|
||||
// --- Utilities ---
|
||||
async function api(path) {
|
||||
const res = await fetch('/api' + path);
|
||||
if (!res.ok) throw new Error(`API ${res.status}: ${path}`);
|
||||
return res.json();
|
||||
const _apiPerf = { calls: 0, totalMs: 0, log: [], cacheHits: 0 };
|
||||
const _apiCache = new Map();
|
||||
const _inflight = new Map();
|
||||
// Client-side TTLs (ms) — loaded from server config, with defaults
|
||||
const CLIENT_TTL = {
|
||||
stats: 10000, nodeDetail: 240000, nodeHealth: 240000, nodeList: 90000,
|
||||
bulkHealth: 300000, networkStatus: 300000, observers: 120000,
|
||||
channels: 15000, channelMessages: 10000, analyticsRF: 300000,
|
||||
analyticsTopology: 300000, analyticsChannels: 300000, analyticsHashSizes: 300000,
|
||||
analyticsSubpaths: 300000, analyticsSubpathDetail: 300000,
|
||||
nodeAnalytics: 60000, nodeSearch: 10000
|
||||
};
|
||||
// Fetch server cache config and use as client TTLs (server values are in seconds)
|
||||
fetch('/api/config/cache').then(r => r.json()).then(cfg => {
|
||||
for (const [k, v] of Object.entries(cfg)) {
|
||||
if (k in CLIENT_TTL && typeof v === 'number') CLIENT_TTL[k] = v * 1000;
|
||||
}
|
||||
}).catch(() => {});
|
||||
async function api(path, { ttl = 0, bust = false } = {}) {
|
||||
const t0 = performance.now();
|
||||
if (!bust && ttl > 0) {
|
||||
const cached = _apiCache.get(path);
|
||||
if (cached && Date.now() < cached.expires) {
|
||||
_apiPerf.calls++;
|
||||
_apiPerf.cacheHits++;
|
||||
_apiPerf.log.push({ path, ms: 0, time: Date.now(), cached: true });
|
||||
if (_apiPerf.log.length > 200) _apiPerf.log.shift();
|
||||
return cached.data;
|
||||
}
|
||||
}
|
||||
// Deduplicate in-flight requests
|
||||
if (_inflight.has(path)) return _inflight.get(path);
|
||||
const promise = (async () => {
|
||||
const res = await fetch('/api' + path);
|
||||
if (!res.ok) throw new Error(`API ${res.status}: ${path}`);
|
||||
const data = await res.json();
|
||||
const ms = performance.now() - t0;
|
||||
_apiPerf.calls++;
|
||||
_apiPerf.totalMs += ms;
|
||||
_apiPerf.log.push({ path, ms: Math.round(ms), time: Date.now() });
|
||||
if (_apiPerf.log.length > 200) _apiPerf.log.shift();
|
||||
if (ms > 500) console.warn(`[SLOW API] ${path} took ${Math.round(ms)}ms`);
|
||||
if (ttl > 0) _apiCache.set(path, { data, expires: Date.now() + ttl });
|
||||
return data;
|
||||
})();
|
||||
_inflight.set(path, promise);
|
||||
promise.finally(() => _inflight.delete(path));
|
||||
return promise;
|
||||
}
|
||||
|
||||
function invalidateApiCache(prefix) {
|
||||
for (const key of _apiCache.keys()) {
|
||||
if (key.startsWith(prefix || '')) _apiCache.delete(key);
|
||||
}
|
||||
}
|
||||
// Expose for console debugging: apiPerf()
|
||||
window.apiPerf = function() {
|
||||
const byPath = {};
|
||||
_apiPerf.log.forEach(e => {
|
||||
if (!byPath[e.path]) byPath[e.path] = { count: 0, totalMs: 0, maxMs: 0 };
|
||||
byPath[e.path].count++;
|
||||
byPath[e.path].totalMs += e.ms;
|
||||
if (e.ms > byPath[e.path].maxMs) byPath[e.path].maxMs = e.ms;
|
||||
});
|
||||
const rows = Object.entries(byPath).map(([p, s]) => ({
|
||||
path: p, count: s.count, avgMs: Math.round(s.totalMs / s.count), maxMs: s.maxMs,
|
||||
totalMs: Math.round(s.totalMs)
|
||||
})).sort((a, b) => b.totalMs - a.totalMs);
|
||||
console.table(rows);
|
||||
const hitRate = _apiPerf.calls ? Math.round(_apiPerf.cacheHits / _apiPerf.calls * 100) : 0;
|
||||
const misses = _apiPerf.calls - _apiPerf.cacheHits;
|
||||
console.log(`Cache: ${_apiPerf.cacheHits} hits / ${misses} misses (${hitRate}% hit rate)`);
|
||||
return { calls: _apiPerf.calls, avgMs: Math.round(_apiPerf.totalMs / (misses || 1)), cacheHits: _apiPerf.cacheHits, cacheMisses: misses, cacheHitRate: hitRate, endpoints: rows };
|
||||
};
|
||||
|
||||
function timeAgo(iso) {
|
||||
if (!iso) return '—';
|
||||
const s = Math.floor((Date.now() - new Date(iso).getTime()) / 1000);
|
||||
@@ -140,6 +209,15 @@ function connectWS() {
|
||||
ws.onmessage = (e) => {
|
||||
try {
|
||||
const msg = JSON.parse(e.data);
|
||||
// Debounce cache invalidation — don't nuke on every packet
|
||||
if (!api._invalidateTimer) {
|
||||
api._invalidateTimer = setTimeout(() => {
|
||||
api._invalidateTimer = null;
|
||||
invalidateApiCache('/stats');
|
||||
invalidateApiCache('/nodes');
|
||||
invalidateApiCache('/channels');
|
||||
}, 5000);
|
||||
}
|
||||
wsListeners.forEach(fn => fn(msg));
|
||||
} catch {}
|
||||
};
|
||||
@@ -217,7 +295,10 @@ function navigate() {
|
||||
|
||||
const app = document.getElementById('app');
|
||||
if (pages[basePage]?.init) {
|
||||
const t0 = performance.now();
|
||||
pages[basePage].init(app, routeParam);
|
||||
const ms = performance.now() - t0;
|
||||
if (ms > 100) console.warn(`[SLOW PAGE] ${basePage} init took ${Math.round(ms)}ms`);
|
||||
app.classList.remove('page-enter'); void app.offsetWidth; app.classList.add('page-enter');
|
||||
} else {
|
||||
app.innerHTML = `<div style="padding:40px;text-align:center;color:#6b7280"><h2>${route}</h2><p>Page not yet implemented.</p></div>`;
|
||||
@@ -290,7 +371,7 @@ window.addEventListener('DOMContentLoaded', () => {
|
||||
favDropdown.innerHTML = '<div class="fav-dd-loading">Loading...</div>';
|
||||
const items = await Promise.all(favs.map(async (pk) => {
|
||||
try {
|
||||
const h = await api('/nodes/' + pk + '/health');
|
||||
const h = await api('/nodes/' + pk + '/health', { ttl: CLIENT_TTL.nodeHealth });
|
||||
const age = h.stats.lastHeard ? Date.now() - new Date(h.stats.lastHeard).getTime() : null;
|
||||
const status = age === null ? '🔴' : age < 3600000 ? '🟢' : age < 86400000 ? '🟡' : '🔴';
|
||||
return '<a href="#/nodes/' + pk + '" class="fav-dd-item" data-key="' + pk + '">'
|
||||
@@ -394,7 +475,7 @@ window.addEventListener('DOMContentLoaded', () => {
|
||||
// --- Nav Stats ---
|
||||
async function updateNavStats() {
|
||||
try {
|
||||
const stats = await api('/stats');
|
||||
const stats = await api('/stats', { ttl: CLIENT_TTL.stats });
|
||||
const el = document.getElementById('navStats');
|
||||
if (el) {
|
||||
el.innerHTML = `<span class="stat-val">${stats.totalPackets}</span> pkts · <span class="stat-val">${stats.totalNodes}</span> nodes · <span class="stat-val">${stats.totalObservers}</span> obs`;
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
if (cached && !cached.fetchedAt) return cached; // legacy null entries
|
||||
}
|
||||
try {
|
||||
const data = await api('/nodes/search?q=' + encodeURIComponent(name));
|
||||
const data = await api('/nodes/search?q=' + encodeURIComponent(name), { ttl: CLIENT_TTL.channelMessages });
|
||||
// Try exact match first, then case-insensitive, then contains
|
||||
const nodes = data.nodes || [];
|
||||
const match = nodes.find(n => n.name === name)
|
||||
@@ -110,7 +110,7 @@
|
||||
}
|
||||
|
||||
try {
|
||||
const detail = await api('/nodes/' + encodeURIComponent(node.public_key));
|
||||
const detail = await api('/nodes/' + encodeURIComponent(node.public_key), { ttl: CLIENT_TTL.nodeDetail });
|
||||
const n = detail.node;
|
||||
const adverts = detail.recentAdverts || [];
|
||||
const role = n.is_repeater ? '📡 Repeater' : n.is_room ? '🏠 Room' : n.is_sensor ? '🌡 Sensor' : '📻 Companion';
|
||||
@@ -389,7 +389,7 @@
|
||||
|
||||
async function loadChannels(silent) {
|
||||
try {
|
||||
const data = await api('/channels');
|
||||
const data = await api('/channels', { ttl: CLIENT_TTL.channels });
|
||||
channels = (data.channels || []).sort((a, b) => (b.lastActivity || '').localeCompare(a.lastActivity || ''));
|
||||
renderChannelList();
|
||||
} catch (e) {
|
||||
@@ -451,7 +451,7 @@
|
||||
msgEl.innerHTML = '<div class="ch-loading">Loading messages…</div>';
|
||||
|
||||
try {
|
||||
const data = await api(`/channels/${hash}/messages?limit=200`);
|
||||
const data = await api(`/channels/${hash}/messages?limit=200`, { ttl: CLIENT_TTL.channelMessages });
|
||||
messages = data.messages || [];
|
||||
renderMessages();
|
||||
scrollToBottom();
|
||||
@@ -466,7 +466,7 @@
|
||||
if (!msgEl) return;
|
||||
const wasAtBottom = msgEl.scrollHeight - msgEl.scrollTop - msgEl.clientHeight < 60;
|
||||
try {
|
||||
const data = await api(`/channels/${selectedHash}/messages?limit=200`);
|
||||
const data = await api(`/channels/${selectedHash}/messages?limit=200`, { ttl: CLIENT_TTL.channelMessages });
|
||||
const newMsgs = data.messages || [];
|
||||
// #92: Use message ID/hash for change detection instead of count + timestamp
|
||||
var _getLastId = function (arr) { var m = arr.length ? arr[arr.length - 1] : null; return m ? (m.id || m.packetId || m.timestamp || '') : ''; };
|
||||
|
||||
BIN
public/favicon.ico
Normal file
BIN
public/favicon.ico
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
13
public/favicon.svg
Normal file
13
public/favicon.svg
Normal file
@@ -0,0 +1,13 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
|
||||
<rect width="32" height="32" rx="6" fill="#1a1a2e"/>
|
||||
<circle cx="16" cy="8" r="3" fill="#00d4ff"/>
|
||||
<circle cx="7" cy="22" r="3" fill="#00d4ff"/>
|
||||
<circle cx="25" cy="22" r="3" fill="#00d4ff"/>
|
||||
<circle cx="16" cy="18" r="2" fill="#00ff88"/>
|
||||
<line x1="16" y1="8" x2="7" y2="22" stroke="#00d4ff" stroke-width="1" opacity="0.6"/>
|
||||
<line x1="16" y1="8" x2="25" y2="22" stroke="#00d4ff" stroke-width="1" opacity="0.6"/>
|
||||
<line x1="7" y1="22" x2="25" y2="22" stroke="#00d4ff" stroke-width="1" opacity="0.6"/>
|
||||
<line x1="16" y1="8" x2="16" y2="18" stroke="#00ff88" stroke-width="1" opacity="0.5"/>
|
||||
<line x1="7" y1="22" x2="16" y2="18" stroke="#00ff88" stroke-width="1" opacity="0.5"/>
|
||||
<line x1="25" y1="22" x2="16" y2="18" stroke="#00ff88" stroke-width="1" opacity="0.5"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 851 B |
@@ -146,7 +146,7 @@
|
||||
if (!q) { suggest.classList.remove('open'); input.setAttribute('aria-expanded', 'false'); input.setAttribute('aria-activedescendant', ''); return; }
|
||||
searchTimeout = setTimeout(async () => {
|
||||
try {
|
||||
const data = await api('/nodes/search?q=' + encodeURIComponent(q));
|
||||
const data = await api('/nodes/search?q=' + encodeURIComponent(q), { ttl: CLIENT_TTL.nodeSearch });
|
||||
const nodes = data.nodes || [];
|
||||
if (!nodes.length) {
|
||||
suggest.innerHTML = '<div class="suggest-empty">No nodes found</div>';
|
||||
@@ -247,7 +247,7 @@
|
||||
|
||||
const cards = await Promise.all(myNodes.map(async (mn) => {
|
||||
try {
|
||||
const h = await api('/nodes/' + encodeURIComponent(mn.pubkey) + '/health');
|
||||
const h = await api('/nodes/' + encodeURIComponent(mn.pubkey) + '/health', { ttl: CLIENT_TTL.nodeHealth });
|
||||
const node = h.node || {};
|
||||
const stats = h.stats || {};
|
||||
const obs = h.observers || [];
|
||||
@@ -369,7 +369,7 @@
|
||||
// ==================== STATS ====================
|
||||
async function loadStats() {
|
||||
try {
|
||||
const s = await api('/stats');
|
||||
const s = await api('/stats', { ttl: CLIENT_TTL.nodeSearch });
|
||||
const el = document.getElementById('homeStats');
|
||||
if (!el) return;
|
||||
el.innerHTML = `
|
||||
@@ -391,7 +391,7 @@
|
||||
if (journey) journey.classList.remove('visible');
|
||||
|
||||
try {
|
||||
const h = await api('/nodes/' + encodeURIComponent(pubkey) + '/health');
|
||||
const h = await api('/nodes/' + encodeURIComponent(pubkey) + '/health', { ttl: CLIENT_TTL.nodeHealth });
|
||||
const node = h.node || {};
|
||||
const stats = h.stats || {};
|
||||
const packets = h.recentPackets || [];
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<link rel="icon" href="favicon.ico" type="image/x-icon">
|
||||
<link rel="icon" href="favicon.svg" type="image/svg+xml">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
|
||||
<title>MeshCore Analyzer</title>
|
||||
|
||||
@@ -20,7 +22,7 @@
|
||||
<meta name="twitter:title" content="MeshCore Analyzer">
|
||||
<meta name="twitter:description" content="Real-time MeshCore LoRa mesh network analyzer — live packet visualization, node tracking, channel decryption, and route analysis.">
|
||||
<meta name="twitter:image" content="https://raw.githubusercontent.com/Kpa-clawbot/meshcore-analyzer/master/public/og-image.png">
|
||||
<link rel="stylesheet" href="style.css?v=1773969261">
|
||||
<link rel="stylesheet" href="style.css?v=1773976827">
|
||||
<link rel="stylesheet" href="home.css">
|
||||
<link rel="stylesheet" href="live.css?v=1773966856">
|
||||
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css"
|
||||
@@ -51,6 +53,7 @@
|
||||
<a href="#/traces" class="nav-link" data-route="traces">Traces</a>
|
||||
<a href="#/observers" class="nav-link" data-route="observers">Observers</a>
|
||||
<a href="#/analytics" class="nav-link" data-route="analytics">Analytics</a>
|
||||
<a href="#/perf" class="nav-link" data-route="perf">⚡ Perf</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="nav-right">
|
||||
@@ -76,16 +79,17 @@
|
||||
<main id="app" role="main"></main>
|
||||
|
||||
<script src="vendor/qrcode.js"></script>
|
||||
<script src="app.js?v=1774079160"></script>
|
||||
<script src="home.js?v=1774079160"></script>
|
||||
<script src="packets.js?v=1773969349"></script>
|
||||
<script src="map.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="channels.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="nodes.js?v=1773961950" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="traces.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="analytics.js?v=1773961035" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="app.js?v=1773977027"></script>
|
||||
<script src="home.js?v=1773977027"></script>
|
||||
<script src="packets.js?v=1773977027"></script>
|
||||
<script src="map.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="channels.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="nodes.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="traces.js?v=1773972187" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="analytics.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="live.js?v=1773964458" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="observers.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="node-analytics.js?v=1773961276" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="observers.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="node-analytics.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
<script src="perf.js?v=1773985649" onerror="console.error('Failed to load:', this.src)"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@@ -980,7 +980,7 @@
|
||||
addNodeMarker(n);
|
||||
}
|
||||
});
|
||||
document.getElementById('liveNodeCount').textContent = Object.keys(nodeMarkers).length;
|
||||
const _el2 = document.getElementById('liveNodeCount'); if (_el2) _el2.textContent = Object.keys(nodeMarkers).length;
|
||||
} catch (e) { console.error('Failed to load nodes:', e); }
|
||||
}
|
||||
|
||||
@@ -1066,7 +1066,7 @@
|
||||
function animatePacket(pkt) {
|
||||
packetCount++;
|
||||
pktTimestamps.push(Date.now());
|
||||
document.getElementById('livePktCount').textContent = packetCount;
|
||||
const _el = document.getElementById('livePktCount'); if (_el) _el.textContent = packetCount;
|
||||
|
||||
const decoded = pkt.decoded || {};
|
||||
const header = decoded.header || {};
|
||||
@@ -1086,7 +1086,7 @@
|
||||
const n = { public_key: key, name: payload.name || key.slice(0,8), role: payload.role || 'unknown', lat: payload.lat, lon: payload.lon };
|
||||
nodeData[key] = n;
|
||||
addNodeMarker(n);
|
||||
document.getElementById('liveNodeCount').textContent = Object.keys(nodeMarkers).length;
|
||||
const _el2 = document.getElementById('liveNodeCount'); if (_el2) _el2.textContent = Object.keys(nodeMarkers).length;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -245,12 +245,12 @@
|
||||
|
||||
async function loadNodes() {
|
||||
try {
|
||||
const data = await api(`/nodes?limit=10000&lastHeard=${filters.lastHeard}`);
|
||||
const data = await api(`/nodes?limit=10000&lastHeard=${filters.lastHeard}`, { ttl: CLIENT_TTL.nodeList });
|
||||
nodes = data.nodes || [];
|
||||
buildRoleChecks(data.counts || {});
|
||||
|
||||
// Load observers for jump buttons
|
||||
const obsData = await api('/observers');
|
||||
const obsData = await api('/observers', { ttl: CLIENT_TTL.observers });
|
||||
observers = obsData.observers || [];
|
||||
buildJumpButtons();
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = await api('/nodes/' + encodeURIComponent(pubkey) + '/analytics?days=' + days);
|
||||
data = await api('/nodes/' + encodeURIComponent(pubkey) + '/analytics?days=' + days, { ttl: CLIENT_TTL.nodeAnalytics });
|
||||
} catch (e) {
|
||||
container.innerHTML = '<div style="padding:40px;text-align:center;color:#ff6b6b">Failed to load analytics: ' + escapeHtml(e.message) + '</div>';
|
||||
return;
|
||||
|
||||
@@ -85,8 +85,8 @@
|
||||
const body = document.getElementById('nodeFullBody');
|
||||
try {
|
||||
const [nodeData, healthData] = await Promise.all([
|
||||
api('/nodes/' + encodeURIComponent(pubkey)),
|
||||
api('/nodes/' + encodeURIComponent(pubkey) + '/health').catch(() => null)
|
||||
api('/nodes/' + encodeURIComponent(pubkey), { ttl: CLIENT_TTL.nodeDetail }),
|
||||
api('/nodes/' + encodeURIComponent(pubkey) + '/health', { ttl: CLIENT_TTL.nodeDetail }).catch(() => null)
|
||||
]);
|
||||
const n = nodeData.node;
|
||||
const adverts = nodeData.recentAdverts || [];
|
||||
@@ -228,7 +228,7 @@
|
||||
if (activeTab !== 'all') params.set('role', activeTab);
|
||||
if (search) params.set('search', search);
|
||||
if (lastHeard) params.set('lastHeard', lastHeard);
|
||||
const data = await api('/nodes?' + params);
|
||||
const data = await api('/nodes?' + params, { ttl: CLIENT_TTL.nodeList });
|
||||
nodes = data.nodes || [];
|
||||
counts = data.counts || {};
|
||||
|
||||
@@ -238,7 +238,7 @@
|
||||
const missing = myNodes.filter(mn => !existingKeys.has(mn.pubkey));
|
||||
if (missing.length) {
|
||||
const fetched = await Promise.allSettled(
|
||||
missing.map(mn => api('/nodes/' + encodeURIComponent(mn.pubkey)))
|
||||
missing.map(mn => api('/nodes/' + encodeURIComponent(mn.pubkey), { ttl: CLIENT_TTL.nodeDetail }))
|
||||
);
|
||||
fetched.forEach(r => {
|
||||
if (r.status === 'fulfilled' && r.value && r.value.public_key) nodes.push(r.value);
|
||||
@@ -401,8 +401,8 @@
|
||||
|
||||
try {
|
||||
const [data, healthData] = await Promise.all([
|
||||
api('/nodes/' + encodeURIComponent(pubkey)),
|
||||
api('/nodes/' + encodeURIComponent(pubkey) + '/health').catch(() => null)
|
||||
api('/nodes/' + encodeURIComponent(pubkey), { ttl: CLIENT_TTL.nodeDetail }),
|
||||
api('/nodes/' + encodeURIComponent(pubkey) + '/health', { ttl: CLIENT_TTL.nodeDetail }).catch(() => null)
|
||||
]);
|
||||
data.healthData = healthData;
|
||||
renderDetail(panel, data);
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
|
||||
async function loadObservers() {
|
||||
try {
|
||||
const data = await api('/observers');
|
||||
const data = await api('/observers', { ttl: CLIENT_TTL.observers });
|
||||
observers = data.observers || [];
|
||||
render();
|
||||
} catch (e) {
|
||||
|
||||
@@ -182,9 +182,80 @@
|
||||
} catch {}
|
||||
}
|
||||
wsHandler = debouncedOnWS(function (msgs) {
|
||||
if (msgs.some(function (m) { return m.type === 'packet'; })) {
|
||||
loadPackets();
|
||||
const newPkts = msgs
|
||||
.filter(m => m.type === 'packet' && m.data?.packet)
|
||||
.map(m => m.data.packet);
|
||||
if (!newPkts.length) return;
|
||||
|
||||
// Check if new packets pass current filters
|
||||
const filtered = newPkts.filter(p => {
|
||||
if (filters.type !== undefined && filters.type !== '' && p.payload_type !== Number(filters.type)) return false;
|
||||
if (filters.observer && p.observer_id !== filters.observer) return false;
|
||||
if (filters.hash && p.hash !== filters.hash) return false;
|
||||
if (filters.region) {
|
||||
const obs = observers.find(o => o.id === p.observer_id);
|
||||
if (!obs || obs.iata !== filters.region) return false;
|
||||
}
|
||||
if (filters.node && !(p.decoded_json || '').includes(filters.node)) return false;
|
||||
return true;
|
||||
});
|
||||
if (!filtered.length) return;
|
||||
|
||||
// Resolve any new hops, then update and re-render
|
||||
const newHops = new Set();
|
||||
for (const p of filtered) {
|
||||
try { JSON.parse(p.path_json || '[]').forEach(h => { if (!(h in hopNameCache)) newHops.add(h); }); } catch {}
|
||||
}
|
||||
(newHops.size ? resolveHops([...newHops]) : Promise.resolve()).then(() => {
|
||||
if (groupByHash) {
|
||||
// Update existing groups or create new ones
|
||||
for (const p of filtered) {
|
||||
const h = p.hash;
|
||||
const existing = packets.find(g => g.hash === h);
|
||||
if (existing) {
|
||||
existing.count = (existing.count || 1) + 1;
|
||||
existing.latest = p.timestamp > existing.latest ? p.timestamp : existing.latest;
|
||||
// Track unique observers
|
||||
if (p.observer_id && p.observer_id !== existing.observer_id) {
|
||||
existing.observer_count = (existing.observer_count || 1) + 1;
|
||||
}
|
||||
// Keep longest path
|
||||
if (p.path_json && (!existing.path_json || p.path_json.length > existing.path_json.length)) {
|
||||
existing.path_json = p.path_json;
|
||||
existing.raw_hex = p.raw_hex;
|
||||
}
|
||||
// Update decoded_json to latest
|
||||
if (p.decoded_json) existing.decoded_json = p.decoded_json;
|
||||
// Update expanded children if this group is expanded
|
||||
if (expandedHashes.has(h) && existing._children) {
|
||||
existing._children.unshift(p);
|
||||
}
|
||||
} else {
|
||||
// New group
|
||||
packets.unshift({
|
||||
hash: h,
|
||||
count: 1,
|
||||
observer_count: 1,
|
||||
latest: p.timestamp,
|
||||
observer_id: p.observer_id,
|
||||
observer_name: p.observer_name,
|
||||
path_json: p.path_json,
|
||||
payload_type: p.payload_type,
|
||||
raw_hex: p.raw_hex,
|
||||
decoded_json: p.decoded_json,
|
||||
});
|
||||
}
|
||||
}
|
||||
// Re-sort by latest DESC, cap size
|
||||
packets.sort((a, b) => (b.latest || '').localeCompare(a.latest || ''));
|
||||
packets = packets.slice(0, 200);
|
||||
} else {
|
||||
// Flat mode: prepend
|
||||
packets = filtered.concat(packets).slice(0, 200);
|
||||
}
|
||||
totalCount += filtered.length;
|
||||
renderTableRows();
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
@@ -207,7 +278,7 @@
|
||||
|
||||
async function loadObservers() {
|
||||
try {
|
||||
const data = await api('/observers');
|
||||
const data = await api('/observers', { ttl: CLIENT_TTL.observers });
|
||||
observers = data.observers || [];
|
||||
} catch {}
|
||||
}
|
||||
|
||||
144
public/perf.js
Normal file
144
public/perf.js
Normal file
@@ -0,0 +1,144 @@
|
||||
/* === MeshCore Analyzer — perf.js === */
|
||||
'use strict';
|
||||
|
||||
(function () {
|
||||
let interval = null;
|
||||
|
||||
async function render(app) {
|
||||
app.innerHTML = '<div style="height:100%;overflow-y:auto;padding:16px 24px;"><h2>⚡ Performance Dashboard</h2><div id="perfContent">Loading...</div></div>';
|
||||
await refresh();
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
const el = document.getElementById('perfContent');
|
||||
if (!el) return;
|
||||
try {
|
||||
const [server, client] = await Promise.all([
|
||||
fetch('/api/perf').then(r => r.json()),
|
||||
Promise.resolve(window.apiPerf ? window.apiPerf() : null)
|
||||
]);
|
||||
|
||||
// Also fetch health telemetry
|
||||
const health = await fetch('/api/health').then(r => r.json()).catch(() => null);
|
||||
|
||||
let html = '';
|
||||
|
||||
// Server overview
|
||||
html += `<div style="display:flex;gap:16px;flex-wrap:wrap;margin:16px 0;">
|
||||
<div class="perf-card"><div class="perf-num">${server.totalRequests}</div><div class="perf-label">Total Requests</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${server.avgMs}ms</div><div class="perf-label">Avg Response</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${health ? health.uptimeHuman : Math.round(server.uptime / 60) + 'm'}</div><div class="perf-label">Uptime</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${server.slowQueries.length}</div><div class="perf-label">Slow (>100ms)</div></div>
|
||||
</div>`;
|
||||
|
||||
// System health (memory, event loop, WS)
|
||||
if (health) {
|
||||
const m = health.memory, el = health.eventLoop;
|
||||
const elColor = el.p95Ms > 500 ? '#ef4444' : el.p95Ms > 100 ? '#f59e0b' : '#22c55e';
|
||||
const memColor = m.heapUsed > m.heapTotal * 0.85 ? '#ef4444' : m.heapUsed > m.heapTotal * 0.7 ? '#f59e0b' : '#22c55e';
|
||||
html += `<h3>System Health</h3><div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
|
||||
<div class="perf-card"><div class="perf-num" style="color:${memColor}">${m.heapUsed}MB</div><div class="perf-label">Heap Used / ${m.heapTotal}MB</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${m.rss}MB</div><div class="perf-label">RSS</div></div>
|
||||
<div class="perf-card"><div class="perf-num" style="color:${elColor}">${el.p95Ms}ms</div><div class="perf-label">Event Loop p95</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${el.maxLagMs}ms</div><div class="perf-label">EL Max Lag</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${el.currentLagMs}ms</div><div class="perf-label">EL Current</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${health.websocket.clients}</div><div class="perf-label">WS Clients</div></div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// Cache stats
|
||||
if (server.cache) {
|
||||
const c = server.cache;
|
||||
const clientCache = _apiCache ? _apiCache.size : 0;
|
||||
html += `<h3>Cache</h3><div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
|
||||
<div class="perf-card"><div class="perf-num">${c.size}</div><div class="perf-label">Server Entries</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${c.hits}</div><div class="perf-label">Server Hits</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${c.misses}</div><div class="perf-label">Server Misses</div></div>
|
||||
<div class="perf-card"><div class="perf-num" style="color:${c.hitRate > 50 ? '#22c55e' : c.hitRate > 20 ? '#f59e0b' : '#ef4444'}">${c.hitRate}%</div><div class="perf-label">Server Hit Rate</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${c.staleHits || 0}</div><div class="perf-label">Stale Hits (SWR)</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${c.recomputes || 0}</div><div class="perf-label">Recomputes</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${clientCache}</div><div class="perf-label">Client Entries</div></div>
|
||||
</div>`;
|
||||
if (client) {
|
||||
html += `<div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
|
||||
<div class="perf-card"><div class="perf-num">${client.cacheHits || 0}</div><div class="perf-label">Client Hits</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${client.cacheMisses || 0}</div><div class="perf-label">Client Misses</div></div>
|
||||
<div class="perf-card"><div class="perf-num" style="color:${(client.cacheHitRate||0) > 50 ? '#22c55e' : '#f59e0b'}">${client.cacheHitRate || 0}%</div><div class="perf-label">Client Hit Rate</div></div>
|
||||
</div>`;
|
||||
}
|
||||
}
|
||||
|
||||
// Packet Store stats
|
||||
if (server.packetStore) {
|
||||
const ps = server.packetStore;
|
||||
html += `<h3>In-Memory Packet Store</h3><div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
|
||||
<div class="perf-card"><div class="perf-num">${ps.inMemory.toLocaleString()}</div><div class="perf-label">Packets in RAM</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.estimatedMB}MB</div><div class="perf-label">Memory Used</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.maxMB}MB</div><div class="perf-label">Memory Limit</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.queries.toLocaleString()}</div><div class="perf-label">Queries Served</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.inserts.toLocaleString()}</div><div class="perf-label">Live Inserts</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.evicted.toLocaleString()}</div><div class="perf-label">Evicted</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.indexes.byHash.toLocaleString()}</div><div class="perf-label">Unique Hashes</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.indexes.byObserver}</div><div class="perf-label">Observers</div></div>
|
||||
<div class="perf-card"><div class="perf-num">${ps.indexes.byNode.toLocaleString()}</div><div class="perf-label">Indexed Nodes</div></div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// Server endpoints table
|
||||
const eps = Object.entries(server.endpoints);
|
||||
if (eps.length) {
|
||||
html += '<h3>Server Endpoints (sorted by total time)</h3>';
|
||||
html += '<div style="overflow-x:auto"><table class="perf-table"><thead><tr><th>Endpoint</th><th>Count</th><th>Avg</th><th>P50</th><th>P95</th><th>Max</th><th>Total</th></tr></thead><tbody>';
|
||||
for (const [path, s] of eps) {
|
||||
const total = Math.round(s.count * s.avgMs);
|
||||
const cls = s.p95Ms > 200 ? ' class="perf-slow"' : s.p95Ms > 50 ? ' class="perf-warn"' : '';
|
||||
html += `<tr${cls}><td><code>${path}</code></td><td>${s.count}</td><td>${s.avgMs}ms</td><td>${s.p50Ms}ms</td><td>${s.p95Ms}ms</td><td>${s.maxMs}ms</td><td>${total}ms</td></tr>`;
|
||||
}
|
||||
html += '</tbody></table></div>';
|
||||
}
|
||||
|
||||
// Client API calls
|
||||
if (client && client.endpoints.length) {
|
||||
html += '<h3>Client API Calls (this session)</h3>';
|
||||
html += '<div style="overflow-x:auto"><table class="perf-table"><thead><tr><th>Endpoint</th><th>Count</th><th>Avg</th><th>Max</th><th>Total</th></tr></thead><tbody>';
|
||||
for (const s of client.endpoints) {
|
||||
const cls = s.maxMs > 500 ? ' class="perf-slow"' : s.avgMs > 200 ? ' class="perf-warn"' : '';
|
||||
html += `<tr${cls}><td><code>${s.path}</code></td><td>${s.count}</td><td>${s.avgMs}ms</td><td>${s.maxMs}ms</td><td>${s.totalMs}ms</td></tr>`;
|
||||
}
|
||||
html += '</tbody></table></div>';
|
||||
}
|
||||
|
||||
// Slow queries
|
||||
if (server.slowQueries.length) {
|
||||
html += '<h3>Recent Slow Queries (>100ms)</h3>';
|
||||
html += '<div style="overflow-x:auto"><table class="perf-table"><thead><tr><th>Time</th><th>Path</th><th>Duration</th><th>Status</th></tr></thead><tbody>';
|
||||
for (const q of server.slowQueries.slice().reverse()) {
|
||||
html += `<tr class="perf-slow"><td>${new Date(q.time).toLocaleTimeString()}</td><td><code>${q.path}</code></td><td>${q.ms}ms</td><td>${q.status}</td></tr>`;
|
||||
}
|
||||
html += '</tbody></table></div>';
|
||||
}
|
||||
|
||||
html += `<div style="margin-top:16px"><button id="perfReset" style="padding:8px 16px;cursor:pointer">Reset Stats</button> <button id="perfRefresh" style="padding:8px 16px;cursor:pointer">Refresh</button></div>`;
|
||||
el.innerHTML = html;
|
||||
|
||||
document.getElementById('perfReset')?.addEventListener('click', async () => {
|
||||
await fetch('/api/perf/reset', { method: 'POST' });
|
||||
if (window._apiPerf) { window._apiPerf = { calls: 0, totalMs: 0, log: [] }; }
|
||||
refresh();
|
||||
});
|
||||
document.getElementById('perfRefresh')?.addEventListener('click', refresh);
|
||||
} catch (err) {
|
||||
el.innerHTML = `<p style="color:red">Error: ${err.message}</p>`;
|
||||
}
|
||||
}
|
||||
|
||||
registerPage('perf', {
|
||||
init(app) {
|
||||
render(app);
|
||||
interval = setInterval(refresh, 5000);
|
||||
},
|
||||
destroy() {
|
||||
if (interval) { clearInterval(interval); interval = null; }
|
||||
}
|
||||
});
|
||||
})();
|
||||
@@ -1413,3 +1413,16 @@ tr[data-hops]:hover { background: rgba(59,130,246,0.1); }
|
||||
}
|
||||
.mobile-sheet-close:hover { color: var(--text); }
|
||||
.mobile-sheet-content { padding-top: 4px; }
|
||||
|
||||
/* Perf dashboard */
|
||||
.perf-card { background: var(--surface-1); border: 1px solid var(--border); border-radius: 8px; padding: 12px 20px; min-width: 120px; text-align: center; }
|
||||
.perf-num { font-size: 24px; font-weight: 800; color: var(--text); font-variant-numeric: tabular-nums; }
|
||||
.perf-label { font-size: 11px; color: var(--text-muted); text-transform: uppercase; letter-spacing: 0.5px; margin-top: 4px; }
|
||||
.perf-table { width: 100%; border-collapse: collapse; font-size: 13px; }
|
||||
.perf-table th { text-align: left; padding: 6px 10px; border-bottom: 2px solid var(--border); color: var(--text-muted); font-size: 11px; text-transform: uppercase; }
|
||||
.perf-table td { padding: 5px 10px; border-bottom: 1px solid var(--border); font-variant-numeric: tabular-nums; }
|
||||
.perf-table code { font-size: 12px; color: var(--text); }
|
||||
.perf-table .perf-slow { background: rgba(239, 68, 68, 0.08); }
|
||||
.perf-table .perf-slow td { color: #ef4444; }
|
||||
.perf-table .perf-warn { background: rgba(251, 191, 36, 0.06); }
|
||||
.perf-table .perf-warn td { color: #f59e0b; }
|
||||
|
||||
Reference in New Issue
Block a user