Compare commits

...

42 Commits

Author SHA1 Message Date
you
7d23ee2ba0 chore: bump perf.js cache buster 2026-03-20 05:47:29 +00:00
you
a058b8aa8b fix: close if(health) block in perf dashboard — was swallowing all content 2026-03-20 05:47:11 +00:00
you
21eeb593d8 feat: system health + SWR stats in perf dashboard
Perf page now shows: heap usage, RSS, event loop p95/max/current,
WS client count, stale-while-revalidate hits, recompute count.
Color-coded: green/yellow/red based on thresholds.
2026-03-20 05:45:51 +00:00
you
0eb6b90e32 feat: stale-while-revalidate cache + /api/health telemetry
Cache: entries stay valid for 2× TTL as stale. First request after
TTL serves stale data while recompute runs (guarded: one at a time).
No more cache stampedes.

/api/health returns:
- Process memory (RSS, heap)
- Event loop lag (p50/p95/p99/max, sampled every 1s)
- Cache stats (hit rate, stale hits, recomputes)
- WebSocket client count
- Packet store size
- Recent slow queries
2026-03-20 05:43:32 +00:00
you
1e6a3b41a3 release: v2.1.0 — Performance
Two-layer caching: in-memory packet store + TTL response cache.
All packet reads from RAM, SQLite write-only.

Highlights:
- Bulk Health: 7,059ms → 1ms (7,059×)
- Node Analytics: 381ms → 1ms (381×)
- Topology: 685ms → 2ms (342×)
- RF Analytics: 253ms → 1ms (253×)
- Channels: 206ms → 1ms (206×)
- Node Health/Detail: 133-195ms → 1ms

Architecture:
- In-memory packet store with Map indexes (byNode, byHash, byObserver)
- Ring buffer with configurable max (1GB default, ~2.3M packets)
- Smart cache invalidation (packet bursts don't nuke analytics)
- Pre-warm all heavy endpoints on startup
- Eliminated every LIKE '%pubkey%' full-table scan
- All TTLs configurable via config.json
- A/B benchmark script included
- Favicon added
2026-03-20 05:38:23 +00:00
you
1e34ab18a7 chore: add A/B benchmark script, remove worker thread experiments 2026-03-20 05:37:08 +00:00
you
60688993c6 feat: add favicon — mesh network icon (SVG + ICO) 2026-03-20 05:36:32 +00:00
you
aab67353d6 fix: null guard getElementById in animatePacket
Elements don't exist yet when replayRecent fires during init.
2026-03-20 05:34:04 +00:00
you
6bb9235992 perf: channels endpoint — single pass, no sort, no double filter
Was doing two pktStore.filter() calls + sort on each. Now single
loop over all packets with inline type check.
2026-03-20 05:31:03 +00:00
you
0a4586f2a9 perf: pre-warm all heavy analytics endpoints on startup
Sequential self-requests after subpath pre-warm completes.
RF, topology, channels, hash-sizes, bulk-health all cached
before any user hits the page.
2026-03-20 05:29:10 +00:00
you
b876fd8a60 perf: stop calling db.getNode() in health/analytics endpoints
db.getNode() does a 4-way LIKE scan for recent packets we don't even
use. Direct SELECT on primary key instead. Saves ~110ms per call.
2026-03-20 05:17:06 +00:00
you
6a4d2a0844 perf: hash-sizes analytics reads from memory store
Last remaining full-table scan on packets from SQLite.
All packet reads now go through pktStore (in-memory).
2026-03-20 05:13:13 +00:00
you
abea3a81e7 perf: node detail uses in-memory packet index
Was doing 4-way LIKE scan for recent packets (~130ms).
Now reads from pktStore.byNode, slices last 20.
2026-03-20 05:12:00 +00:00
you
8104d8bac7 perf: node health uses in-memory packet store
Was doing 6 LIKE scans on SQLite (~169ms). Now reads from
pktStore.byNode index, single pass over packets.
2026-03-20 05:11:00 +00:00
you
881956ffdb perf: node analytics uses in-memory packet store
Was doing 7 separate LIKE scans on SQLite (~552ms). Now reads from
pktStore.byNode index and computes all aggregations in JS.
Also added cache with TTL.nodeAnalytics.
2026-03-20 05:09:49 +00:00
you
4dc179e0b9 perf: bulk-health uses in-memory packet store index
Was doing 50-pattern LIKE OR scan on all packets in SQLite (~2s).
Now reads from pktStore.byNode index — O(1) lookup per node.
2026-03-20 05:08:23 +00:00
you
1fe40cf85a revert: remove background refresh jobs — blocks event loop
Node.js is single-threaded. A 5s subpath computation in a background
timer blocks ALL concurrent requests. Stats endpoint went from 3ms
to 1.2s because it was waiting for a background refresh to finish.

Pre-warm on startup + long TTLs (30min-1hr) is sufficient. At most
one user per hour eats a cold compute cost.
2026-03-20 04:56:57 +00:00
you
67ea437429 perf: background refresh jobs — recompute expensive caches before TTL expires
No user ever hits a cold compute path. Background timers fire at 80%
of each TTL, hitting endpoints with ?nocache=1 to force recomputation
and re-cache the result.

Jobs: RF (24min), topology (24min), channels (24min), hash-sizes (48min),
subpaths ×4 (48min), bulk-health (8min).
2026-03-20 04:52:06 +00:00
you
d47c7d18b1 feat: live A/B benchmark — launches SQLite-only vs in-memory servers
NO_MEMORY_STORE=1 env var makes packet-store fall through to SQLite
for all reads. Benchmark spins up both servers on temp ports and
compares: SQLite cold, Memory cold, Memory cached.

Results on 27K packets (ARM64):
  Subpaths 5-8: SQLite 4.7s → cached 1.1ms (4,273×)
  Bulk health:  SQLite 1.8s → cached 1.7ms (1,059×)
  Topology:     SQLite 1.1s → cached 3.0ms (367×)
  Channels:     SQLite 617ms → cached 1.9ms (325×)
  RF Analytics: SQLite 448ms → cached 1.6ms (280×)
2026-03-20 04:47:31 +00:00
you
d0c5384883 feat: benchmark compares against pre-optimization baseline
Stores pre-optimization /api/perf measurements (pure SQLite, 27K packets)
in benchmark-baseline.json. Benchmark suite auto-loads and shows side-by-side:

Highlights:
  Subpaths 5-8 hop: 6,190ms → 1.1ms (5,627× faster)
  Hash sizes:          430ms → 1.3ms (331× faster)
  Topology:            697ms → 2.8ms (249× faster)
  RF analytics:        272ms → 1.6ms (170× faster), 1MB → 22KB
  Packets:              78ms → 3ms (26× faster)
  Channels:             60ms → 1.5ms (40× faster)
  Bulk health:       1,610ms → 67ms (24× faster)
2026-03-20 04:30:18 +00:00
you
3e89fdac67 feat: add Perf dashboard to nav bar, show packet store stats
Perf page now accessible from main nav ( Perf).
Shows in-memory packet store metrics: packets in RAM, memory used/limit,
queries served, live inserts, evictions, index sizes.
2026-03-20 04:25:05 +00:00
you
4779193edf feat: benchmark suite + nocache bypass for cold compute testing
node benchmark.js [--runs N] [--json]
Adds ?nocache=1 query param to bypass server cache for benchmarking.
Tests all 21 endpoints cached vs cold, shows speedup comparison.
2026-03-20 04:23:34 +00:00
you
fba9af5955 docs: add PERFORMANCE.md with before/after benchmarks 2026-03-20 04:21:53 +00:00
you
e09f902a17 perf: RF endpoint from 1MB to ~15KB — server-side histograms, scatter downsampled to 500pts
Was sending 27K raw SNR/RSSI/size values (420KB) + 27K scatter points (763KB).
Now: histograms computed server-side (20-25 bins), scatter downsampled
to max 500 evenly-spaced points. Client histogram() accepts both formats.
2026-03-20 04:17:25 +00:00
you
c261c7b322 perf: grouped mode also updates client-side from WS — zero API fetches
Existing groups get count/observer_count incremented, latest timestamp
updated, longest path kept. New hashes get a new group prepended.
Expanded children updated inline. No more /api/packets re-fetch on
any incoming packet in either mode.
2026-03-20 04:15:04 +00:00
you
c36abe2b24 perf: WS packets prepend client-side instead of re-fetching entire list
Non-grouped mode: new packets from WebSocket are filtered client-side
and prepended to the table, no API call. Grouped mode still re-fetches
(group counts change). Server broadcast now includes full packet row.
Eliminates repeated /api/packets fetches on every incoming packet.
2026-03-20 04:12:07 +00:00
you
536459ee53 perf: pre-warm all 4 subpath query variants on startup + dedup concurrent computation 2026-03-20 03:53:10 +00:00
you
162782209c perf: single-pass subpath computation + startup pre-warm
4 parallel subpath queries were each scanning 27K packets independently
(937ms + 1.99s + 3.09s + 6.19s). Now one shared computation builds all
subpath data, cached for 1hr. Subsequent queries just slice the result.
Pre-warmed on startup so first user never sees a cold call.
2026-03-20 03:51:58 +00:00
you
6e30ba1539 perf: smart cache invalidation — only channels/observers on packet burst, node/health/analytics expire by TTL, node invalidated on ADVERT only 2026-03-20 03:48:55 +00:00
you
971c58b369 perf: ALL packet reads from RAM — analytics, channels, topology, subpaths, RF, observers
Zero SQLite reads from packets table. Every endpoint that previously
scanned packets now reads from the in-memory PacketStore.
Expected: subpaths from 1.6s to <100ms, topology from 700ms to <50ms,
RF from 270ms to <30ms on cold calls.
2026-03-20 03:43:23 +00:00
you
1f88232f94 perf: in-memory packet store — all reads from RAM, SQLite write-only
- PacketStore loads all packets into memory on startup (~11MB for 27K packets)
- Indexed by id, hash, observer, and node pubkey for fast lookups
- /api/packets, /api/packets/timestamps, /api/packets/:id all served from RAM
- MQTT ingest writes to both RAM + SQLite
- Configurable maxMemoryMB (default 1024MB) in config.json packetStore section
- groupByHash queries computed in-memory
- Packet store stats exposed in /api/perf
- Expected: /api/packets goes from 77ms to <1ms
2026-03-20 03:38:37 +00:00
you
f410b7d49b perf: configurable cache TTLs via config.json — server + client fetch from /api/config/cache
All cache TTLs now read from config.json cacheTTL section (seconds).
Client fetches config on load via GET /api/config/cache.
config.example.json updated with defaults.
Edit config.json, restart server — no code changes needed to tweak TTLs.
2026-03-20 03:23:58 +00:00
you
32ffcb6be4 perf: align cache TTLs with real data rates — analytics 30min-1hr, nodes 5min, chat 10-15s, stats 10s, server debounce 30s 2026-03-20 03:20:33 +00:00
you
f2d7dabbb2 perf: bump analytics cache to 5min, subpaths to 10min, cache subpath-detail 2026-03-20 02:24:46 +00:00
you
2146ec64ec fix: debounce client cache invalidation (5s window) — same issue as server 2026-03-20 02:23:14 +00:00
you
441cd52608 fix: debounce server cache invalidation (5s window), fix client cache stat reporting 2026-03-20 02:15:18 +00:00
you
832421f581 perf page: show server + client cache stats 2026-03-20 02:10:27 +00:00
you
215003f295 merge: server + frontend perf optimizations 2026-03-20 02:07:54 +00:00
you
3049ec3e1c perf: add TTL cache layer + rewrite bulk-health to single-query
- Add TTLCache class with hit/miss tracking
- Cache all expensive endpoints:
  - analytics/* endpoints: 60s TTL
  - channels: 30s TTL
  - channels/:hash/messages: 15s TTL
  - nodes/:pubkey: 30s TTL
  - nodes/:pubkey/health: 30s TTL
  - observers: 30s TTL
  - bulk-health: 60s TTL
- Invalidate all caches on new packet ingestion (POST + MQTT)
- Rewrite bulk-health from N×5 queries to 1 query + JS matching
- Add cache stats (size, hits, misses, hitRate) to /api/perf
2026-03-20 02:06:23 +00:00
you
936609f729 feat: add frontend API response caching with TTL, in-flight dedup, and WebSocket invalidation
- Replace api() with caching version supporting TTL and request deduplication
- Add appropriate TTLs to all api() call sites across all frontend JS files:
  - /stats: 5s TTL (was called 962 times in 3 min)
  - /nodes/:pubkey: 15s, /health: 30s, /observers: 30s
  - /channels: 15s, messages: 10s
  - /analytics/*: 60s, /bulk-health: 60s, /network-status: 60s
  - /nodes?*: 10s
- Skip caching for real-time endpoints (/packets, /resolve-hops, /perf)
- Invalidate /stats, /nodes, /channels caches on WebSocket messages
- Deduplicate in-flight requests (same path returns same promise)
- Add cache hit rate to window.apiPerf() console debugging
- Update all cache busters in index.html
2026-03-20 02:03:25 +00:00
you
915484f54a merge: performance instrumentation 2026-03-20 01:49:19 +00:00
you
e6911a3915 feat: performance instrumentation — server timing middleware, client API tracking, /api/perf endpoint, #/perf dashboard 2026-03-20 01:34:25 +00:00
23 changed files with 1887 additions and 208 deletions

67
PERFORMANCE.md Normal file
View File

@@ -0,0 +1,67 @@
# Performance — v2.1.0
**Dataset:** 28,014 packets, ~650 nodes, 2 observers
**Hardware:** ARM64 (MikroTik CCR2116), single-core Node.js
## A/B Benchmark: v2.0.1 (before) vs v2.1.0 (after)
All times are averages over 3 runs. "Cached" = warm TTL cache hit.
| Endpoint | v2.0.1 | v2.1.0 (cold) | v2.1.0 (cached) | Speedup |
|---|---|---|---|---|
| **Bulk Health** | 7,059 ms | 3 ms | 1 ms | **7,059×** |
| **Node Analytics** | 381 ms | 2 ms | 1 ms | **381×** |
| **Hash Sizes** | 353 ms | 193 ms | 1 ms | **353×** |
| **Topology** | 685 ms | 579 ms | 2 ms | **342×** |
| **RF Analytics** | 253 ms | 235 ms | 1 ms | **253×** |
| **Channels** | 206 ms | 77 ms | 1 ms | **206×** |
| **Node Health** | 195 ms | 1 ms | 1 ms | **195×** |
| **Node Detail** | 133 ms | 1 ms | 1 ms | **133×** |
| **Channel Analytics** | 95 ms | 73 ms | 2 ms | **47×** |
| **Packets (grouped)** | 76 ms | 33 ms | 28 ms | **2×** |
| **Stats** | 2 ms | 1 ms | 1 ms | 2× |
| **Nodes List** | 3 ms | 2 ms | 2 ms | 1× |
| **Observers** | 1 ms | 8 ms | 1 ms | 1× |
## Architecture
### Two-Layer Performance Stack
1. **In-Memory Packet Store** (`packet-store.js`)
- All packets loaded from SQLite into RAM on startup (~28K packets = ~12MB)
- Indexed by `id`, `hash`, `observer`, and `node` (Map-based O(1) lookup)
- Ring buffer with configurable max memory (default 1GB, ~2.3M packets)
- SQLite becomes **write-only** for packets — reads never touch disk
- New packets from MQTT written to both RAM + SQLite
2. **TTL Cache** (`server.js`)
- Computed API responses cached with configurable TTLs (via `config.json`)
- Smart invalidation: packet bursts only invalidate channels/observers; analytics expire by TTL only
- Pre-warmed on startup: subpaths, RF, topology, channels, hash-sizes, bulk-health
- Result: most API responses served in **1-2ms** from cache
### Key Optimizations
- **Eliminated all `LIKE '%pubkey%'` queries**: Every node-specific endpoint was doing full-table scans on the packets table via `decoded_json LIKE '%pubkey%'`. Replaced with O(1) `pktStore.byNode` Map lookups.
- **Single-pass computations**: Channels, analytics, and subpaths computed in one loop instead of multiple SQL queries.
- **Client-side WebSocket prepend**: New packets appended to the table without re-fetching the API.
- **RF response compression**: Server-side histograms + scatter downsampling (1MB → 15KB).
- **Configurable everything**: All TTLs, packet store limits, and thresholds in `config.json`.
### What Didn't Work
- **Background refresh (`setInterval`)**: Attempted to re-warm caches at 80% TTL. Blocked the event loop — Node.js is single-threaded. Response times went from 3ms to 1,200ms. Reverted immediately.
- **Worker threads**: `structuredClone` overhead of 416ms for 28K packets negated the compute savings. Only viable at 10× data growth or with `SharedArrayBuffer` (zero-copy).
## Running the Benchmark
```bash
# Stop the production server first
supervisorctl stop meshcore-analyzer
# Run A/B benchmark (launches two servers: old v2.0.1 vs current)
./benchmark-ab.sh
# Restart production
supervisorctl start meshcore-analyzer
```

View File

@@ -53,6 +53,20 @@ Full experience on your phone — proper touch controls, iOS safe area support,
- **Mobile Responsive** — proper two-row VCR bar, iOS safe area support, touch-friendly
- **Accessible** — ARIA patterns, keyboard navigation, screen reader support, distinct marker shapes
### ⚡ Performance (v2.1.0)
Two-layer caching architecture: in-memory packet store + TTL response cache. All packet reads served from RAM — SQLite is write-only. Heavy endpoints pre-warmed on startup.
| Endpoint | Before | After | Speedup |
|---|---|---|---|
| Bulk Health | 7,059 ms | 1 ms | **7,059×** |
| Node Analytics | 381 ms | 1 ms | **381×** |
| Topology | 685 ms | 2 ms | **342×** |
| Node Health | 195 ms | 1 ms | **195×** |
| Node Detail | 133 ms | 1 ms | **133×** |
See [PERFORMANCE.md](PERFORMANCE.md) for the full benchmark.
## Quick Start
### Prerequisites

131
benchmark-ab.sh Executable file
View File

@@ -0,0 +1,131 @@
#!/bin/bash
# A/B benchmark: old (pre-perf) vs new (current)
# Usage: ./benchmark-ab.sh
set -e
PORT_OLD=13003
PORT_NEW=13004
RUNS=3
DB_PATH="$(pwd)/data/meshcore.db"
OLD_COMMIT="23caae4"
NEW_COMMIT="$(git rev-parse HEAD)"
echo "═══════════════════════════════════════════════════════"
echo " A/B Benchmark: Pre-optimization vs Current"
echo "═══════════════════════════════════════════════════════"
echo "OLD: $OLD_COMMIT (v2.0.1 — before any perf work)"
echo "NEW: $NEW_COMMIT (current)"
echo "Runs per endpoint: $RUNS"
echo ""
# Get a real node pubkey for testing
ORIG_DIR="$(pwd)"
PUBKEY=$(sqlite3 "$DB_PATH" "SELECT public_key FROM nodes ORDER BY last_seen DESC LIMIT 1")
echo "Test node: ${PUBKEY:0:16}..."
echo ""
# Setup old version in temp dir
OLD_DIR=$(mktemp -d)
echo "Cloning old version to $OLD_DIR..."
git worktree add "$OLD_DIR" "$OLD_COMMIT" --quiet 2>/dev/null || {
git worktree add "$OLD_DIR" "$OLD_COMMIT" --detach --quiet
}
# Copy config + db symlink
# Copy config + db + share node_modules
cp config.json "$OLD_DIR/"
mkdir -p "$OLD_DIR/data"
cp "$ORIG_DIR/data/meshcore.db" "$OLD_DIR/data/meshcore.db"
ln -sf "$ORIG_DIR/node_modules" "$OLD_DIR/node_modules"
ENDPOINTS=(
"Stats|/api/stats"
"Packets(50)|/api/packets?limit=50"
"PacketsGrouped|/api/packets?limit=50&groupByHash=true"
"NodesList|/api/nodes?limit=50"
"NodeDetail|/api/nodes/$PUBKEY"
"NodeHealth|/api/nodes/$PUBKEY/health"
"NodeAnalytics|/api/nodes/$PUBKEY/analytics?days=7"
"BulkHealth|/api/nodes/bulk-health?limit=50"
"NetworkStatus|/api/nodes/network-status"
"Channels|/api/channels"
"Observers|/api/observers"
"RF|/api/analytics/rf"
"Topology|/api/analytics/topology"
"ChannelAnalytics|/api/analytics/channels"
"HashSizes|/api/analytics/hash-sizes"
)
bench_endpoint() {
local port=$1 path=$2 runs=$3 nocache=$4
local total=0
for i in $(seq 1 $runs); do
local url="http://127.0.0.1:$port$path"
if [ "$nocache" = "1" ]; then
if echo "$path" | grep -q '?'; then
url="${url}&nocache=1"
else
url="${url}?nocache=1"
fi
fi
local ms=$(curl -s -o /dev/null -w "%{time_total}" "$url" 2>/dev/null)
local ms_int=$(echo "$ms * 1000" | bc | cut -d. -f1)
total=$((total + ms_int))
done
echo $((total / runs))
}
# Launch old server
echo "Starting OLD server (port $PORT_OLD)..."
cd "$OLD_DIR"
PORT=$PORT_OLD node server.js &>/dev/null &
OLD_PID=$!
cd - >/dev/null
# Launch new server
echo "Starting NEW server (port $PORT_NEW)..."
PORT=$PORT_NEW node server.js &>/dev/null &
NEW_PID=$!
# Wait for both
sleep 12 # old server has no memory store; new needs prewarm
# Verify
curl -s "http://127.0.0.1:$PORT_OLD/api/stats" >/dev/null 2>&1 || { echo "OLD server failed to start"; kill $OLD_PID $NEW_PID 2>/dev/null; exit 1; }
curl -s "http://127.0.0.1:$PORT_NEW/api/stats" >/dev/null 2>&1 || { echo "NEW server failed to start"; kill $OLD_PID $NEW_PID 2>/dev/null; exit 1; }
echo ""
echo "Warming up caches on new server..."
for ep in "${ENDPOINTS[@]}"; do
path="${ep#*|}"
curl -s -o /dev/null "http://127.0.0.1:$PORT_NEW$path" 2>/dev/null
done
sleep 2
printf "\n%-22s %9s %9s %9s %9s\n" "Endpoint" "Old(ms)" "New-cold" "New-cache" "Speedup"
printf "%-22s %9s %9s %9s %9s\n" "──────────────────────" "─────────" "─────────" "─────────" "─────────"
for ep in "${ENDPOINTS[@]}"; do
name="${ep%%|*}"
path="${ep#*|}"
old_ms=$(bench_endpoint $PORT_OLD "$path" $RUNS 0)
new_cold=$(bench_endpoint $PORT_NEW "$path" $RUNS 1)
new_cached=$(bench_endpoint $PORT_NEW "$path" $RUNS 0)
if [ "$old_ms" -gt 0 ] && [ "$new_cached" -gt 0 ]; then
speedup="${old_ms}/${new_cached}"
speedup_x=$(echo "scale=0; $old_ms / $new_cached" | bc 2>/dev/null || echo "?")
printf "%-22s %7dms %7dms %7dms %7d×\n" "$name" "$old_ms" "$new_cold" "$new_cached" "$speedup_x"
else
printf "%-22s %7dms %7dms %7dms %9s\n" "$name" "$old_ms" "$new_cold" "$new_cached" "∞"
fi
done
echo ""
echo "═══════════════════════════════════════════════════════"
# Cleanup
kill $OLD_PID $NEW_PID 2>/dev/null
git worktree remove "$OLD_DIR" --force 2>/dev/null
echo "Done."

246
benchmark.js Normal file
View File

@@ -0,0 +1,246 @@
#!/usr/bin/env node
'use strict';
/**
* Benchmark suite for meshcore-analyzer.
* Launches two server instances — one with in-memory store, one with pure SQLite —
* and compares performance side by side.
*
* Usage: node benchmark.js [--runs 5] [--json]
*/
const http = require('http');
const { spawn } = require('child_process');
const path = require('path');
const args = process.argv.slice(2);
const RUNS = Number(args.find((a, i) => args[i - 1] === '--runs') || 5);
const JSON_OUT = args.includes('--json');
const PORT_MEM = 13001; // In-memory store
const PORT_SQL = 13002; // SQLite-only
const ENDPOINTS = [
{ name: 'Stats', path: '/api/stats' },
{ name: 'Packets (50)', path: '/api/packets?limit=50' },
{ name: 'Packets (100)', path: '/api/packets?limit=100' },
{ name: 'Packets grouped', path: '/api/packets?limit=100&groupByHash=true' },
{ name: 'Packets filtered', path: '/api/packets?limit=50&type=5' },
{ name: 'Packets timestamps', path: '/api/packets/timestamps?since=2020-01-01' },
{ name: 'Nodes list', path: '/api/nodes?limit=50' },
{ name: 'Node detail', path: '/api/nodes/__FIRST_NODE__' },
{ name: 'Node health', path: '/api/nodes/__FIRST_NODE__/health' },
{ name: 'Bulk health', path: '/api/nodes/bulk-health?limit=50' },
{ name: 'Network status', path: '/api/nodes/network-status' },
{ name: 'Observers', path: '/api/observers' },
{ name: 'Channels', path: '/api/channels' },
{ name: 'RF Analytics', path: '/api/analytics/rf' },
{ name: 'Topology', path: '/api/analytics/topology' },
{ name: 'Channel Analytics', path: '/api/analytics/channels' },
{ name: 'Hash Sizes', path: '/api/analytics/hash-sizes' },
{ name: 'Subpaths 2-hop', path: '/api/analytics/subpaths?minLen=2&maxLen=2&limit=50' },
{ name: 'Subpaths 3-hop', path: '/api/analytics/subpaths?minLen=3&maxLen=3&limit=30' },
{ name: 'Subpaths 4-hop', path: '/api/analytics/subpaths?minLen=4&maxLen=4&limit=20' },
{ name: 'Subpaths 5-8 hop', path: '/api/analytics/subpaths?minLen=5&maxLen=8&limit=15' },
];
function fetch(url) {
return new Promise((resolve, reject) => {
const t0 = process.hrtime.bigint();
const req = http.get(url, (res) => {
let body = '';
res.on('data', c => body += c);
res.on('end', () => {
const ms = Number(process.hrtime.bigint() - t0) / 1e6;
resolve({ ms, bytes: Buffer.byteLength(body), status: res.statusCode, body });
});
});
req.on('error', reject);
req.setTimeout(60000, () => { req.destroy(); reject(new Error('timeout')); });
});
}
function median(arr) { const s = [...arr].sort((a,b)=>a-b); return s[Math.floor(s.length/2)]; }
function p95(arr) { const s = [...arr].sort((a,b)=>a-b); return s[Math.floor(s.length*0.95)]; }
function avg(arr) { return arr.reduce((a,b)=>a+b,0)/arr.length; }
function fmt(ms) { return ms >= 1000 ? (ms/1000).toFixed(1)+'s' : ms.toFixed(1)+'ms'; }
function fmtSize(b) { return b >= 1048576 ? (b/1048576).toFixed(1)+'MB' : b >= 1024 ? (b/1024).toFixed(0)+'KB' : b+'B'; }
function launchServer(port, env = {}) {
return new Promise((resolve, reject) => {
const child = spawn('node', ['server.js'], {
cwd: __dirname,
env: { ...process.env, PORT: String(port), ...env },
stdio: ['ignore', 'pipe', 'pipe'],
});
let started = false;
const timeout = setTimeout(() => { if (!started) { child.kill(); reject(new Error('Server start timeout')); } }, 30000);
child.stdout.on('data', (d) => {
if (!started && (d.toString().includes('listening') || d.toString().includes('running'))) {
started = true; clearTimeout(timeout); resolve(child);
}
});
child.stderr.on('data', (d) => {
if (!started && (d.toString().includes('listening') || d.toString().includes('running'))) {
started = true; clearTimeout(timeout); resolve(child);
}
});
child.on('exit', (code) => { if (!started) { clearTimeout(timeout); reject(new Error(`Server exited with ${code}`)); } });
// Fallback: wait longer (SQLite-only mode pre-warms subpaths ~6s)
setTimeout(() => {
if (!started) {
started = true; clearTimeout(timeout);
resolve(child);
}
}, 15000);
});
}
async function waitForServer(port, maxMs = 20000) {
const t0 = Date.now();
while (Date.now() - t0 < maxMs) {
try {
const r = await fetch(`http://127.0.0.1:${port}/api/stats`);
if (r.status === 200) return true;
} catch {}
await new Promise(r => setTimeout(r, 500));
}
throw new Error(`Server on port ${port} didn't start`);
}
async function benchmarkEndpoints(port, endpoints, nocache = false) {
const results = [];
for (const ep of endpoints) {
const suffix = nocache ? (ep.path.includes('?') ? '&nocache=1' : '?nocache=1') : '';
const url = `http://127.0.0.1:${port}${ep.path}${suffix}`;
// Warm-up
try { await fetch(url); } catch {}
const times = [];
let bytes = 0;
let failed = false;
for (let i = 0; i < RUNS; i++) {
try {
const r = await fetch(url);
if (r.status !== 200) { failed = true; break; }
times.push(r.ms);
bytes = r.bytes;
} catch { failed = true; break; }
}
if (failed || !times.length) {
results.push({ name: ep.name, failed: true });
} else {
results.push({
name: ep.name,
avg: Math.round(avg(times) * 10) / 10,
p50: Math.round(median(times) * 10) / 10,
p95: Math.round(p95(times) * 10) / 10,
bytes
});
}
}
return results;
}
async function run() {
console.log(`\nMeshCore Analyzer Benchmark — ${RUNS} runs per endpoint`);
console.log('Launching servers...\n');
// Launch both servers
let memServer, sqlServer;
try {
console.log(' Starting in-memory server (port ' + PORT_MEM + ')...');
memServer = await launchServer(PORT_MEM, {});
await waitForServer(PORT_MEM);
console.log(' ✅ In-memory server ready');
console.log(' Starting SQLite-only server (port ' + PORT_SQL + ')...');
sqlServer = await launchServer(PORT_SQL, { NO_MEMORY_STORE: '1' });
await waitForServer(PORT_SQL);
console.log(' ✅ SQLite-only server ready\n');
} catch (e) {
console.error('Failed to start servers:', e.message);
if (memServer) memServer.kill();
if (sqlServer) sqlServer.kill();
process.exit(1);
}
// Get first node pubkey
let firstNode = '';
try {
const r = await fetch(`http://127.0.0.1:${PORT_MEM}/api/nodes?limit=1`);
const data = JSON.parse(r.body);
firstNode = data.nodes?.[0]?.public_key || '';
} catch {}
const endpoints = ENDPOINTS.map(e => ({
...e,
path: e.path.replace('__FIRST_NODE__', firstNode),
}));
// Get packet count
try {
const r = await fetch(`http://127.0.0.1:${PORT_MEM}/api/stats`);
const stats = JSON.parse(r.body);
console.log(`Dataset: ${(stats.totalPackets || '?').toLocaleString()} packets\n`);
} catch {}
// Run benchmarks
console.log('Benchmarking in-memory store (nocache for true compute cost)...');
const memResults = await benchmarkEndpoints(PORT_MEM, endpoints, true);
console.log('Benchmarking SQLite-only (nocache)...');
const sqlResults = await benchmarkEndpoints(PORT_SQL, endpoints, true);
// Also test cached in-memory for the full picture
console.log('Benchmarking in-memory store (cached)...');
const memCachedResults = await benchmarkEndpoints(PORT_MEM, endpoints, false);
// Kill servers
memServer.kill();
sqlServer.kill();
if (JSON_OUT) {
console.log(JSON.stringify({ memoryNocache: memResults, sqliteNocache: sqlResults, memoryCached: memCachedResults }, null, 2));
return;
}
// Print results
const W = 94;
console.log(`\n${'═'.repeat(W)}`);
console.log(' 🏁 BENCHMARK RESULTS: SQLite vs In-Memory Store');
console.log(`${'═'.repeat(W)}`);
console.log(`${'Endpoint'.padEnd(24)} ${'SQLite'.padStart(9)} ${'Memory'.padStart(9)} ${'Cached'.padStart(9)} ${'Speedup'.padStart(9)} ${'Size (SQL)'.padStart(10)} ${'Size (Mem)'.padStart(10)}`);
console.log(`${'─'.repeat(24)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(10)} ${'─'.repeat(10)}`);
for (let i = 0; i < endpoints.length; i++) {
const sql = sqlResults[i];
const mem = memResults[i];
const cached = memCachedResults[i];
if (!sql || sql.failed || !mem || mem.failed) {
console.log(`${endpoints[i].name.padEnd(24)} ${'FAILED'.padStart(9)}`);
continue;
}
const speedup = sql.avg > 0 && mem.avg > 0 ? Math.round(sql.avg / mem.avg) + '×' : '—';
const cachedStr = cached && !cached.failed ? fmt(cached.avg) : '—';
console.log(
`${sql.name.padEnd(24)} ${fmt(sql.avg).padStart(9)} ${fmt(mem.avg).padStart(9)} ${cachedStr.padStart(9)} ${speedup.padStart(9)} ${fmtSize(sql.bytes).padStart(10)} ${fmtSize(mem.bytes).padStart(10)}`
);
}
// Summary
const sqlTotal = sqlResults.filter(r => !r.failed).reduce((s, r) => s + r.avg, 0);
const memTotal = memResults.filter(r => !r.failed).reduce((s, r) => s + r.avg, 0);
console.log(`${'─'.repeat(24)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)} ${'─'.repeat(9)}`);
console.log(`${'TOTAL'.padEnd(24)} ${fmt(sqlTotal).padStart(9)} ${fmt(memTotal).padStart(9)} ${''.padStart(9)} ${(Math.round(sqlTotal/memTotal)+'×').padStart(9)}`);
console.log(`\n${'═'.repeat(W)}\n`);
}
run().catch(e => { console.error(e); process.exit(1); });

View File

@@ -22,5 +22,31 @@
"OAK": "Oakland, US",
"MRY": "Monterey, US",
"LAR": "Los Angeles, US"
},
"cacheTTL": {
"stats": 10,
"nodeDetail": 300,
"nodeHealth": 300,
"nodeList": 90,
"bulkHealth": 600,
"networkStatus": 600,
"observers": 300,
"channels": 15,
"channelMessages": 10,
"analyticsRF": 1800,
"analyticsTopology": 1800,
"analyticsChannels": 1800,
"analyticsHashSizes": 3600,
"analyticsSubpaths": 3600,
"analyticsSubpathDetail": 3600,
"nodeAnalytics": 60,
"nodeSearch": 10,
"invalidationDebounce": 30,
"_comment": "All values in seconds. Server uses these directly. Client fetches via /api/config/cache."
},
"packetStore": {
"maxMemoryMB": 1024,
"estimatedPacketBytes": 450,
"_comment": "In-memory packet store. maxMemoryMB caps RAM usage. All packets loaded on startup, served from RAM."
}
}

View File

@@ -1,6 +1,6 @@
{
"name": "meshcore-analyzer",
"version": "2.0.1",
"version": "2.1.0",
"description": "Community-run alternative to the closed-source `analyzer.letsmesh.net`. MQTT packet collection + open-source web analyzer for the Bay Area MeshCore mesh.",
"main": "index.js",
"scripts": {

342
packet-store.js Normal file
View File

@@ -0,0 +1,342 @@
'use strict';
/**
* In-memory packet store — loads all packets from SQLite on startup,
* serves reads from RAM, writes to both RAM + SQLite.
* Caps memory at configurable limit (default 1GB).
*/
class PacketStore {
constructor(dbModule, config = {}) {
this.dbModule = dbModule; // The full db module (has .db, .insertPacket, .getPacket)
this.db = dbModule.db; // Raw better-sqlite3 instance for queries
this.maxBytes = (config.maxMemoryMB || 1024) * 1024 * 1024;
this.estPacketBytes = config.estimatedPacketBytes || 450;
this.maxPackets = Math.floor(this.maxBytes / this.estPacketBytes);
// SQLite-only mode: skip RAM loading, all reads go to DB
this.sqliteOnly = process.env.NO_MEMORY_STORE === '1';
// Core storage: array sorted by timestamp DESC (newest first)
this.packets = [];
// Indexes
this.byId = new Map();
this.byHash = new Map(); // hash → [packet, ...]
this.byObserver = new Map(); // observer_id → [packet, ...]
this.byNode = new Map(); // pubkey → [packet, ...]
this.loaded = false;
this.stats = { totalLoaded: 0, evicted: 0, inserts: 0, queries: 0 };
}
/** Load all packets from SQLite into memory */
load() {
if (this.sqliteOnly) {
console.log('[PacketStore] SQLite-only mode (NO_MEMORY_STORE=1) — all reads go to database');
this.loaded = true;
return this;
}
const t0 = Date.now();
const rows = this.db.prepare(
'SELECT * FROM packets ORDER BY timestamp DESC'
).all();
for (const row of rows) {
if (this.packets.length >= this.maxPackets) break;
this._index(row);
this.packets.push(row);
}
this.stats.totalLoaded = this.packets.length;
this.loaded = true;
const elapsed = Date.now() - t0;
console.log(`[PacketStore] Loaded ${this.packets.length} packets in ${elapsed}ms (${Math.round(this.packets.length * this.estPacketBytes / 1024 / 1024)}MB est)`);
return this;
}
/** Index a packet into all lookup maps */
_index(pkt) {
this.byId.set(pkt.id, pkt);
if (pkt.hash) {
if (!this.byHash.has(pkt.hash)) this.byHash.set(pkt.hash, []);
this.byHash.get(pkt.hash).push(pkt);
}
if (pkt.observer_id) {
if (!this.byObserver.has(pkt.observer_id)) this.byObserver.set(pkt.observer_id, []);
this.byObserver.get(pkt.observer_id).push(pkt);
}
// Index by node pubkeys mentioned in decoded_json
this._indexByNode(pkt);
}
/** Extract node pubkeys/names from decoded_json and index */
_indexByNode(pkt) {
if (!pkt.decoded_json) return;
try {
const decoded = JSON.parse(pkt.decoded_json);
const keys = new Set();
if (decoded.pubKey) keys.add(decoded.pubKey);
if (decoded.destPubKey) keys.add(decoded.destPubKey);
if (decoded.srcPubKey) keys.add(decoded.srcPubKey);
for (const k of keys) {
if (!this.byNode.has(k)) this.byNode.set(k, []);
this.byNode.get(k).push(pkt);
}
} catch {}
}
/** Remove oldest packets when over memory limit */
_evict() {
while (this.packets.length > this.maxPackets) {
const old = this.packets.pop();
this.byId.delete(old.id);
// Remove from hash index
if (old.hash && this.byHash.has(old.hash)) {
const arr = this.byHash.get(old.hash).filter(p => p.id !== old.id);
if (arr.length) this.byHash.set(old.hash, arr); else this.byHash.delete(old.hash);
}
// Remove from observer index
if (old.observer_id && this.byObserver.has(old.observer_id)) {
const arr = this.byObserver.get(old.observer_id).filter(p => p.id !== old.id);
if (arr.length) this.byObserver.set(old.observer_id, arr); else this.byObserver.delete(old.observer_id);
}
// Skip node index cleanup for eviction (expensive, low value)
this.stats.evicted++;
}
}
/** Insert a new packet (to both memory and SQLite) */
insert(packetData) {
const id = this.dbModule.insertPacket(packetData);
const row = this.dbModule.getPacket(id);
if (row) {
this.packets.unshift(row); // newest first
this._index(row);
this._evict();
this.stats.inserts++;
}
return id;
}
/** Query packets with filters — all from memory (or SQLite in fallback mode) */
query({ limit = 50, offset = 0, type, route, region, observer, hash, since, until, node, order = 'DESC' } = {}) {
this.stats.queries++;
if (this.sqliteOnly) return this._querySQLite({ limit, offset, type, route, region, observer, hash, since, until, node, order });
let results = this.packets;
// Use indexes for single-key filters when possible
if (hash && !type && !route && !region && !observer && !since && !until && !node) {
results = this.byHash.get(hash) || [];
} else if (observer && !type && !route && !region && !hash && !since && !until && !node) {
results = this.byObserver.get(observer) || [];
} else if (node && !type && !route && !region && !observer && !hash && !since && !until) {
results = this.byNode.get(node) || [];
} else {
// Apply filters sequentially
if (type !== undefined) {
const t = Number(type);
results = results.filter(p => p.payload_type === t);
}
if (route !== undefined) {
const r = Number(route);
results = results.filter(p => p.route_type === r);
}
if (observer) results = results.filter(p => p.observer_id === observer);
if (hash) results = results.filter(p => p.hash === hash);
if (since) results = results.filter(p => p.timestamp > since);
if (until) results = results.filter(p => p.timestamp < until);
if (region) {
// Need to look up observers for this region
const regionObservers = new Set();
try {
const obs = this.db.prepare('SELECT id FROM observers WHERE iata = ?').all(region);
obs.forEach(o => regionObservers.add(o.id));
} catch {}
results = results.filter(p => regionObservers.has(p.observer_id));
}
if (node) {
// Check indexed results first, fall back to text search
const indexed = this.byNode.get(node);
if (indexed) {
const idSet = new Set(indexed.map(p => p.id));
results = results.filter(p => idSet.has(p.id));
} else {
// Text search fallback (node name)
results = results.filter(p =>
p.decoded_json && p.decoded_json.includes(node)
);
}
}
}
const total = results.length;
// Sort
if (order === 'ASC') {
results = results.slice().sort((a, b) => {
if (a.timestamp < b.timestamp) return -1;
if (a.timestamp > b.timestamp) return 1;
return 0;
});
}
// Default DESC — packets array is already sorted newest-first
// Paginate
const paginated = results.slice(Number(offset), Number(offset) + Number(limit));
return { packets: paginated, total };
}
/** Query with groupByHash — aggregate packets by content hash */
queryGrouped({ limit = 50, offset = 0, type, route, region, observer, hash, since, until, node } = {}) {
this.stats.queries++;
if (this.sqliteOnly) return this._queryGroupedSQLite({ limit, offset, type, route, region, observer, hash, since, until, node });
// Get filtered results first
const { packets: filtered, total: filteredTotal } = this.query({
limit: 999999, offset: 0, type, route, region, observer, hash, since, until, node
});
// Group by hash
const groups = new Map();
for (const p of filtered) {
const h = p.hash || p.id.toString();
if (!groups.has(h)) {
groups.set(h, {
hash: p.hash,
observer_count: new Set(),
count: 0,
latest: p.timestamp,
observer_id: p.observer_id,
observer_name: p.observer_name,
path_json: p.path_json,
payload_type: p.payload_type,
raw_hex: p.raw_hex,
decoded_json: p.decoded_json,
});
}
const g = groups.get(h);
g.count++;
if (p.observer_id) g.observer_count.add(p.observer_id);
if (p.timestamp > g.latest) {
g.latest = p.timestamp;
}
// Keep longest path
if (p.path_json && (!g.path_json || p.path_json.length > g.path_json.length)) {
g.path_json = p.path_json;
g.raw_hex = p.raw_hex;
}
}
// Sort by latest DESC, paginate
const sorted = [...groups.values()]
.map(g => ({ ...g, observer_count: g.observer_count.size }))
.sort((a, b) => b.latest.localeCompare(a.latest));
const total = sorted.length;
const paginated = sorted.slice(Number(offset), Number(offset) + Number(limit));
return { packets: paginated, total };
}
/** Get timestamps for sparkline */
getTimestamps(since) {
if (this.sqliteOnly) {
return this.db.prepare('SELECT timestamp FROM packets WHERE timestamp > ? ORDER BY timestamp ASC').all(since).map(r => r.timestamp);
}
const results = [];
for (const p of this.packets) {
if (p.timestamp <= since) break;
results.push(p.timestamp);
}
return results.reverse();
}
/** Get a single packet by ID */
getById(id) {
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets WHERE id = ?').get(id) || null;
return this.byId.get(id) || null;
}
/** Get all siblings of a packet (same hash) */
getSiblings(hash) {
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets WHERE hash = ? ORDER BY timestamp DESC').all(hash);
return this.byHash.get(hash) || [];
}
/** Get all packets (raw array reference — do not mutate) */
all() {
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets ORDER BY timestamp DESC').all();
return this.packets;
}
/** Get all packets matching a filter function */
filter(fn) {
if (this.sqliteOnly) return this.db.prepare('SELECT * FROM packets ORDER BY timestamp DESC').all().filter(fn);
return this.packets.filter(fn);
}
/** Memory stats */
getStats() {
return {
...this.stats,
inMemory: this.sqliteOnly ? 0 : this.packets.length,
sqliteOnly: this.sqliteOnly,
maxPackets: this.maxPackets,
estimatedMB: this.sqliteOnly ? 0 : Math.round(this.packets.length * this.estPacketBytes / 1024 / 1024),
maxMB: Math.round(this.maxBytes / 1024 / 1024),
indexes: {
byHash: this.byHash.size,
byObserver: this.byObserver.size,
byNode: this.byNode.size,
}
};
}
/** SQLite fallback: query with filters */
_querySQLite({ limit, offset, type, route, region, observer, hash, since, until, node, order }) {
const where = []; const params = [];
if (type !== undefined) { where.push('payload_type = ?'); params.push(Number(type)); }
if (route !== undefined) { where.push('route_type = ?'); params.push(Number(route)); }
if (observer) { where.push('observer_id = ?'); params.push(observer); }
if (hash) { where.push('hash = ?'); params.push(hash); }
if (since) { where.push('timestamp > ?'); params.push(since); }
if (until) { where.push('timestamp < ?'); params.push(until); }
if (region) { where.push('observer_id IN (SELECT id FROM observers WHERE iata = ?)'); params.push(region); }
if (node) { where.push('decoded_json LIKE ?'); params.push(`%${node}%`); }
const w = where.length ? 'WHERE ' + where.join(' AND ') : '';
const total = this.db.prepare(`SELECT COUNT(*) as c FROM packets ${w}`).get(...params).c;
const packets = this.db.prepare(`SELECT * FROM packets ${w} ORDER BY timestamp ${order === 'ASC' ? 'ASC' : 'DESC'} LIMIT ? OFFSET ?`).all(...params, limit, offset);
return { packets, total };
}
/** SQLite fallback: grouped query */
_queryGroupedSQLite({ limit, offset, type, route, region, observer, hash, since, until, node }) {
const where = []; const params = [];
if (type !== undefined) { where.push('payload_type = ?'); params.push(Number(type)); }
if (route !== undefined) { where.push('route_type = ?'); params.push(Number(route)); }
if (observer) { where.push('observer_id = ?'); params.push(observer); }
if (hash) { where.push('hash = ?'); params.push(hash); }
if (since) { where.push('timestamp > ?'); params.push(since); }
if (until) { where.push('timestamp < ?'); params.push(until); }
if (region) { where.push('observer_id IN (SELECT id FROM observers WHERE iata = ?)'); params.push(region); }
if (node) { where.push('decoded_json LIKE ?'); params.push(`%${node}%`); }
const w = where.length ? 'WHERE ' + where.join(' AND ') : '';
const sql = `SELECT hash, COUNT(*) as count, COUNT(DISTINCT observer_id) as observer_count,
MAX(timestamp) as latest, MIN(observer_id) as observer_id, MIN(observer_name) as observer_name,
MIN(path_json) as path_json, MIN(payload_type) as payload_type, MIN(raw_hex) as raw_hex,
MIN(decoded_json) as decoded_json
FROM packets ${w} GROUP BY hash ORDER BY latest DESC LIMIT ? OFFSET ?`;
const packets = this.db.prepare(sql).all(...params, limit, offset);
const countSql = `SELECT COUNT(DISTINCT hash) as c FROM packets ${w}`;
const total = this.db.prepare(countSql).get(...params).c;
return { packets, total };
}
}
module.exports = PacketStore;

View File

@@ -40,7 +40,15 @@
return svg;
}
function histogram(values, bins, color, w = 800, h = 180) {
function histogram(data, bins, color, w = 800, h = 180) {
// Support pre-computed histogram from server { bins: [{x, w, count}], min, max }
if (data && data.bins && Array.isArray(data.bins)) {
const buckets = data.bins.map(b => b.count);
const labels = data.bins.map(b => b.x.toFixed(1));
return { svg: barChart(buckets, labels, color, w, h), buckets, labels };
}
// Legacy: raw values array
const values = data;
const min = Math.min(...values), max = Math.max(...values);
const step = (max - min) / bins;
const buckets = Array(bins).fill(0);
@@ -101,10 +109,10 @@
try {
_analyticsData = {};
const [hashData, rfData, topoData, chanData] = await Promise.all([
api('/analytics/hash-sizes'),
api('/analytics/rf'),
api('/analytics/topology'),
api('/analytics/channels'),
api('/analytics/hash-sizes', { ttl: CLIENT_TTL.analyticsRF }),
api('/analytics/rf', { ttl: CLIENT_TTL.analyticsRF }),
api('/analytics/topology', { ttl: CLIENT_TTL.analyticsRF }),
api('/analytics/channels', { ttl: CLIENT_TTL.analyticsRF }),
]);
_analyticsData = { hashData, rfData, topoData, chanData };
renderTab('overview');
@@ -747,7 +755,7 @@
</div>
`;
let allNodes = [];
try { const nd = await api('/nodes?limit=2000'); allNodes = nd.nodes || []; } catch {}
try { const nd = await api('/nodes?limit=2000', { ttl: CLIENT_TTL.nodeList }); allNodes = nd.nodes || []; } catch {}
renderHashMatrix(data.topHops, allNodes);
renderCollisions(data.topHops, allNodes);
}
@@ -938,10 +946,10 @@
el.innerHTML = '<div class="text-center text-muted" style="padding:40px">Analyzing route patterns…</div>';
try {
const [d2, d3, d4, d5] = await Promise.all([
api('/analytics/subpaths?minLen=2&maxLen=2&limit=50'),
api('/analytics/subpaths?minLen=3&maxLen=3&limit=30'),
api('/analytics/subpaths?minLen=4&maxLen=4&limit=20'),
api('/analytics/subpaths?minLen=5&maxLen=8&limit=15')
api('/analytics/subpaths?minLen=2&maxLen=2&limit=50', { ttl: CLIENT_TTL.analyticsRF }),
api('/analytics/subpaths?minLen=3&maxLen=3&limit=30', { ttl: CLIENT_TTL.analyticsRF }),
api('/analytics/subpaths?minLen=4&maxLen=4&limit=20', { ttl: CLIENT_TTL.analyticsRF }),
api('/analytics/subpaths?minLen=5&maxLen=8&limit=15', { ttl: CLIENT_TTL.analyticsRF })
]);
function renderTable(data, title) {
@@ -1032,7 +1040,7 @@
panel.classList.remove('collapsed');
panel.innerHTML = '<div class="text-center text-muted" style="padding:40px">Loading…</div>';
try {
const data = await api('/analytics/subpath-detail?hops=' + encodeURIComponent(hopsStr));
const data = await api('/analytics/subpath-detail?hops=' + encodeURIComponent(hopsStr), { ttl: CLIENT_TTL.analyticsRF });
renderSubpathDetail(panel, data);
} catch (e) {
panel.innerHTML = `<div class="text-muted">Error: ${e.message}</div>`;
@@ -1141,9 +1149,9 @@
el.innerHTML = '<div style="padding:40px;text-align:center;color:var(--text-muted)">Loading node analytics…</div>';
try {
const [nodesResp, bulkHealth, netStatus] = await Promise.all([
api('/nodes?limit=200&sortBy=lastSeen'),
api('/nodes/bulk-health?limit=50'),
api('/nodes/network-status')
api('/nodes?limit=200&sortBy=lastSeen', { ttl: CLIENT_TTL.nodeList }),
api('/nodes/bulk-health?limit=50', { ttl: CLIENT_TTL.analyticsRF }),
api('/nodes/network-status', { ttl: CLIENT_TTL.analyticsRF })
]);
const nodes = nodesResp.nodes || nodesResp;
const myNodes = JSON.parse(localStorage.getItem('meshcore-my-nodes') || '[]');

View File

@@ -11,12 +11,81 @@ function payloadTypeName(n) { return PAYLOAD_TYPES[n] || 'UNKNOWN'; }
function payloadTypeColor(n) { return PAYLOAD_COLORS[n] || 'unknown'; }
// --- Utilities ---
async function api(path) {
const res = await fetch('/api' + path);
if (!res.ok) throw new Error(`API ${res.status}: ${path}`);
return res.json();
const _apiPerf = { calls: 0, totalMs: 0, log: [], cacheHits: 0 };
const _apiCache = new Map();
const _inflight = new Map();
// Client-side TTLs (ms) — loaded from server config, with defaults
const CLIENT_TTL = {
stats: 10000, nodeDetail: 240000, nodeHealth: 240000, nodeList: 90000,
bulkHealth: 300000, networkStatus: 300000, observers: 120000,
channels: 15000, channelMessages: 10000, analyticsRF: 300000,
analyticsTopology: 300000, analyticsChannels: 300000, analyticsHashSizes: 300000,
analyticsSubpaths: 300000, analyticsSubpathDetail: 300000,
nodeAnalytics: 60000, nodeSearch: 10000
};
// Fetch server cache config and use as client TTLs (server values are in seconds)
fetch('/api/config/cache').then(r => r.json()).then(cfg => {
for (const [k, v] of Object.entries(cfg)) {
if (k in CLIENT_TTL && typeof v === 'number') CLIENT_TTL[k] = v * 1000;
}
}).catch(() => {});
async function api(path, { ttl = 0, bust = false } = {}) {
const t0 = performance.now();
if (!bust && ttl > 0) {
const cached = _apiCache.get(path);
if (cached && Date.now() < cached.expires) {
_apiPerf.calls++;
_apiPerf.cacheHits++;
_apiPerf.log.push({ path, ms: 0, time: Date.now(), cached: true });
if (_apiPerf.log.length > 200) _apiPerf.log.shift();
return cached.data;
}
}
// Deduplicate in-flight requests
if (_inflight.has(path)) return _inflight.get(path);
const promise = (async () => {
const res = await fetch('/api' + path);
if (!res.ok) throw new Error(`API ${res.status}: ${path}`);
const data = await res.json();
const ms = performance.now() - t0;
_apiPerf.calls++;
_apiPerf.totalMs += ms;
_apiPerf.log.push({ path, ms: Math.round(ms), time: Date.now() });
if (_apiPerf.log.length > 200) _apiPerf.log.shift();
if (ms > 500) console.warn(`[SLOW API] ${path} took ${Math.round(ms)}ms`);
if (ttl > 0) _apiCache.set(path, { data, expires: Date.now() + ttl });
return data;
})();
_inflight.set(path, promise);
promise.finally(() => _inflight.delete(path));
return promise;
}
function invalidateApiCache(prefix) {
for (const key of _apiCache.keys()) {
if (key.startsWith(prefix || '')) _apiCache.delete(key);
}
}
// Expose for console debugging: apiPerf()
window.apiPerf = function() {
const byPath = {};
_apiPerf.log.forEach(e => {
if (!byPath[e.path]) byPath[e.path] = { count: 0, totalMs: 0, maxMs: 0 };
byPath[e.path].count++;
byPath[e.path].totalMs += e.ms;
if (e.ms > byPath[e.path].maxMs) byPath[e.path].maxMs = e.ms;
});
const rows = Object.entries(byPath).map(([p, s]) => ({
path: p, count: s.count, avgMs: Math.round(s.totalMs / s.count), maxMs: s.maxMs,
totalMs: Math.round(s.totalMs)
})).sort((a, b) => b.totalMs - a.totalMs);
console.table(rows);
const hitRate = _apiPerf.calls ? Math.round(_apiPerf.cacheHits / _apiPerf.calls * 100) : 0;
const misses = _apiPerf.calls - _apiPerf.cacheHits;
console.log(`Cache: ${_apiPerf.cacheHits} hits / ${misses} misses (${hitRate}% hit rate)`);
return { calls: _apiPerf.calls, avgMs: Math.round(_apiPerf.totalMs / (misses || 1)), cacheHits: _apiPerf.cacheHits, cacheMisses: misses, cacheHitRate: hitRate, endpoints: rows };
};
function timeAgo(iso) {
if (!iso) return '—';
const s = Math.floor((Date.now() - new Date(iso).getTime()) / 1000);
@@ -140,6 +209,15 @@ function connectWS() {
ws.onmessage = (e) => {
try {
const msg = JSON.parse(e.data);
// Debounce cache invalidation — don't nuke on every packet
if (!api._invalidateTimer) {
api._invalidateTimer = setTimeout(() => {
api._invalidateTimer = null;
invalidateApiCache('/stats');
invalidateApiCache('/nodes');
invalidateApiCache('/channels');
}, 5000);
}
wsListeners.forEach(fn => fn(msg));
} catch {}
};
@@ -217,7 +295,10 @@ function navigate() {
const app = document.getElementById('app');
if (pages[basePage]?.init) {
const t0 = performance.now();
pages[basePage].init(app, routeParam);
const ms = performance.now() - t0;
if (ms > 100) console.warn(`[SLOW PAGE] ${basePage} init took ${Math.round(ms)}ms`);
app.classList.remove('page-enter'); void app.offsetWidth; app.classList.add('page-enter');
} else {
app.innerHTML = `<div style="padding:40px;text-align:center;color:#6b7280"><h2>${route}</h2><p>Page not yet implemented.</p></div>`;
@@ -290,7 +371,7 @@ window.addEventListener('DOMContentLoaded', () => {
favDropdown.innerHTML = '<div class="fav-dd-loading">Loading...</div>';
const items = await Promise.all(favs.map(async (pk) => {
try {
const h = await api('/nodes/' + pk + '/health');
const h = await api('/nodes/' + pk + '/health', { ttl: CLIENT_TTL.nodeHealth });
const age = h.stats.lastHeard ? Date.now() - new Date(h.stats.lastHeard).getTime() : null;
const status = age === null ? '🔴' : age < 3600000 ? '🟢' : age < 86400000 ? '🟡' : '🔴';
return '<a href="#/nodes/' + pk + '" class="fav-dd-item" data-key="' + pk + '">'
@@ -394,7 +475,7 @@ window.addEventListener('DOMContentLoaded', () => {
// --- Nav Stats ---
async function updateNavStats() {
try {
const stats = await api('/stats');
const stats = await api('/stats', { ttl: CLIENT_TTL.stats });
const el = document.getElementById('navStats');
if (el) {
el.innerHTML = `<span class="stat-val">${stats.totalPackets}</span> pkts · <span class="stat-val">${stats.totalNodes}</span> nodes · <span class="stat-val">${stats.totalObservers}</span> obs`;

View File

@@ -18,7 +18,7 @@
if (cached && !cached.fetchedAt) return cached; // legacy null entries
}
try {
const data = await api('/nodes/search?q=' + encodeURIComponent(name));
const data = await api('/nodes/search?q=' + encodeURIComponent(name), { ttl: CLIENT_TTL.channelMessages });
// Try exact match first, then case-insensitive, then contains
const nodes = data.nodes || [];
const match = nodes.find(n => n.name === name)
@@ -110,7 +110,7 @@
}
try {
const detail = await api('/nodes/' + encodeURIComponent(node.public_key));
const detail = await api('/nodes/' + encodeURIComponent(node.public_key), { ttl: CLIENT_TTL.nodeDetail });
const n = detail.node;
const adverts = detail.recentAdverts || [];
const role = n.is_repeater ? '📡 Repeater' : n.is_room ? '🏠 Room' : n.is_sensor ? '🌡 Sensor' : '📻 Companion';
@@ -389,7 +389,7 @@
async function loadChannels(silent) {
try {
const data = await api('/channels');
const data = await api('/channels', { ttl: CLIENT_TTL.channels });
channels = (data.channels || []).sort((a, b) => (b.lastActivity || '').localeCompare(a.lastActivity || ''));
renderChannelList();
} catch (e) {
@@ -451,7 +451,7 @@
msgEl.innerHTML = '<div class="ch-loading">Loading messages…</div>';
try {
const data = await api(`/channels/${hash}/messages?limit=200`);
const data = await api(`/channels/${hash}/messages?limit=200`, { ttl: CLIENT_TTL.channelMessages });
messages = data.messages || [];
renderMessages();
scrollToBottom();
@@ -466,7 +466,7 @@
if (!msgEl) return;
const wasAtBottom = msgEl.scrollHeight - msgEl.scrollTop - msgEl.clientHeight < 60;
try {
const data = await api(`/channels/${selectedHash}/messages?limit=200`);
const data = await api(`/channels/${selectedHash}/messages?limit=200`, { ttl: CLIENT_TTL.channelMessages });
const newMsgs = data.messages || [];
// #92: Use message ID/hash for change detection instead of count + timestamp
var _getLastId = function (arr) { var m = arr.length ? arr[arr.length - 1] : null; return m ? (m.id || m.packetId || m.timestamp || '') : ''; };

BIN
public/favicon.ico Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

13
public/favicon.svg Normal file
View File

@@ -0,0 +1,13 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
<rect width="32" height="32" rx="6" fill="#1a1a2e"/>
<circle cx="16" cy="8" r="3" fill="#00d4ff"/>
<circle cx="7" cy="22" r="3" fill="#00d4ff"/>
<circle cx="25" cy="22" r="3" fill="#00d4ff"/>
<circle cx="16" cy="18" r="2" fill="#00ff88"/>
<line x1="16" y1="8" x2="7" y2="22" stroke="#00d4ff" stroke-width="1" opacity="0.6"/>
<line x1="16" y1="8" x2="25" y2="22" stroke="#00d4ff" stroke-width="1" opacity="0.6"/>
<line x1="7" y1="22" x2="25" y2="22" stroke="#00d4ff" stroke-width="1" opacity="0.6"/>
<line x1="16" y1="8" x2="16" y2="18" stroke="#00ff88" stroke-width="1" opacity="0.5"/>
<line x1="7" y1="22" x2="16" y2="18" stroke="#00ff88" stroke-width="1" opacity="0.5"/>
<line x1="25" y1="22" x2="16" y2="18" stroke="#00ff88" stroke-width="1" opacity="0.5"/>
</svg>

After

Width:  |  Height:  |  Size: 851 B

View File

@@ -146,7 +146,7 @@
if (!q) { suggest.classList.remove('open'); input.setAttribute('aria-expanded', 'false'); input.setAttribute('aria-activedescendant', ''); return; }
searchTimeout = setTimeout(async () => {
try {
const data = await api('/nodes/search?q=' + encodeURIComponent(q));
const data = await api('/nodes/search?q=' + encodeURIComponent(q), { ttl: CLIENT_TTL.nodeSearch });
const nodes = data.nodes || [];
if (!nodes.length) {
suggest.innerHTML = '<div class="suggest-empty">No nodes found</div>';
@@ -247,7 +247,7 @@
const cards = await Promise.all(myNodes.map(async (mn) => {
try {
const h = await api('/nodes/' + encodeURIComponent(mn.pubkey) + '/health');
const h = await api('/nodes/' + encodeURIComponent(mn.pubkey) + '/health', { ttl: CLIENT_TTL.nodeHealth });
const node = h.node || {};
const stats = h.stats || {};
const obs = h.observers || [];
@@ -369,7 +369,7 @@
// ==================== STATS ====================
async function loadStats() {
try {
const s = await api('/stats');
const s = await api('/stats', { ttl: CLIENT_TTL.nodeSearch });
const el = document.getElementById('homeStats');
if (!el) return;
el.innerHTML = `
@@ -391,7 +391,7 @@
if (journey) journey.classList.remove('visible');
try {
const h = await api('/nodes/' + encodeURIComponent(pubkey) + '/health');
const h = await api('/nodes/' + encodeURIComponent(pubkey) + '/health', { ttl: CLIENT_TTL.nodeHealth });
const node = h.node || {};
const stats = h.stats || {};
const packets = h.recentPackets || [];

View File

@@ -2,6 +2,8 @@
<html lang="en">
<head>
<meta charset="UTF-8">
<link rel="icon" href="favicon.ico" type="image/x-icon">
<link rel="icon" href="favicon.svg" type="image/svg+xml">
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
<title>MeshCore Analyzer</title>
@@ -20,7 +22,7 @@
<meta name="twitter:title" content="MeshCore Analyzer">
<meta name="twitter:description" content="Real-time MeshCore LoRa mesh network analyzer — live packet visualization, node tracking, channel decryption, and route analysis.">
<meta name="twitter:image" content="https://raw.githubusercontent.com/Kpa-clawbot/meshcore-analyzer/master/public/og-image.png">
<link rel="stylesheet" href="style.css?v=1773969261">
<link rel="stylesheet" href="style.css?v=1773976827">
<link rel="stylesheet" href="home.css">
<link rel="stylesheet" href="live.css?v=1773966856">
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css"
@@ -51,6 +53,7 @@
<a href="#/traces" class="nav-link" data-route="traces">Traces</a>
<a href="#/observers" class="nav-link" data-route="observers">Observers</a>
<a href="#/analytics" class="nav-link" data-route="analytics">Analytics</a>
<a href="#/perf" class="nav-link" data-route="perf">⚡ Perf</a>
</div>
</div>
<div class="nav-right">
@@ -76,16 +79,17 @@
<main id="app" role="main"></main>
<script src="vendor/qrcode.js"></script>
<script src="app.js?v=1774079160"></script>
<script src="home.js?v=1774079160"></script>
<script src="packets.js?v=1773969349"></script>
<script src="map.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
<script src="channels.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
<script src="nodes.js?v=1773961950" onerror="console.error('Failed to load:', this.src)"></script>
<script src="traces.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
<script src="analytics.js?v=1773961035" onerror="console.error('Failed to load:', this.src)"></script>
<script src="app.js?v=1773977027"></script>
<script src="home.js?v=1773977027"></script>
<script src="packets.js?v=1773977027"></script>
<script src="map.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
<script src="channels.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
<script src="nodes.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
<script src="traces.js?v=1773972187" onerror="console.error('Failed to load:', this.src)"></script>
<script src="analytics.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
<script src="live.js?v=1773964458" onerror="console.error('Failed to load:', this.src)"></script>
<script src="observers.js?v=1774079160" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-analytics.js?v=1773961276" onerror="console.error('Failed to load:', this.src)"></script>
<script src="observers.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-analytics.js?v=1773977027" onerror="console.error('Failed to load:', this.src)"></script>
<script src="perf.js?v=1773985649" onerror="console.error('Failed to load:', this.src)"></script>
</body>
</html>

View File

@@ -980,7 +980,7 @@
addNodeMarker(n);
}
});
document.getElementById('liveNodeCount').textContent = Object.keys(nodeMarkers).length;
const _el2 = document.getElementById('liveNodeCount'); if (_el2) _el2.textContent = Object.keys(nodeMarkers).length;
} catch (e) { console.error('Failed to load nodes:', e); }
}
@@ -1066,7 +1066,7 @@
function animatePacket(pkt) {
packetCount++;
pktTimestamps.push(Date.now());
document.getElementById('livePktCount').textContent = packetCount;
const _el = document.getElementById('livePktCount'); if (_el) _el.textContent = packetCount;
const decoded = pkt.decoded || {};
const header = decoded.header || {};
@@ -1086,7 +1086,7 @@
const n = { public_key: key, name: payload.name || key.slice(0,8), role: payload.role || 'unknown', lat: payload.lat, lon: payload.lon };
nodeData[key] = n;
addNodeMarker(n);
document.getElementById('liveNodeCount').textContent = Object.keys(nodeMarkers).length;
const _el2 = document.getElementById('liveNodeCount'); if (_el2) _el2.textContent = Object.keys(nodeMarkers).length;
}
}

View File

@@ -245,12 +245,12 @@
async function loadNodes() {
try {
const data = await api(`/nodes?limit=10000&lastHeard=${filters.lastHeard}`);
const data = await api(`/nodes?limit=10000&lastHeard=${filters.lastHeard}`, { ttl: CLIENT_TTL.nodeList });
nodes = data.nodes || [];
buildRoleChecks(data.counts || {});
// Load observers for jump buttons
const obsData = await api('/observers');
const obsData = await api('/observers', { ttl: CLIENT_TTL.observers });
observers = obsData.observers || [];
buildJumpButtons();

View File

@@ -40,7 +40,7 @@
let data;
try {
data = await api('/nodes/' + encodeURIComponent(pubkey) + '/analytics?days=' + days);
data = await api('/nodes/' + encodeURIComponent(pubkey) + '/analytics?days=' + days, { ttl: CLIENT_TTL.nodeAnalytics });
} catch (e) {
container.innerHTML = '<div style="padding:40px;text-align:center;color:#ff6b6b">Failed to load analytics: ' + escapeHtml(e.message) + '</div>';
return;

View File

@@ -85,8 +85,8 @@
const body = document.getElementById('nodeFullBody');
try {
const [nodeData, healthData] = await Promise.all([
api('/nodes/' + encodeURIComponent(pubkey)),
api('/nodes/' + encodeURIComponent(pubkey) + '/health').catch(() => null)
api('/nodes/' + encodeURIComponent(pubkey), { ttl: CLIENT_TTL.nodeDetail }),
api('/nodes/' + encodeURIComponent(pubkey) + '/health', { ttl: CLIENT_TTL.nodeDetail }).catch(() => null)
]);
const n = nodeData.node;
const adverts = nodeData.recentAdverts || [];
@@ -228,7 +228,7 @@
if (activeTab !== 'all') params.set('role', activeTab);
if (search) params.set('search', search);
if (lastHeard) params.set('lastHeard', lastHeard);
const data = await api('/nodes?' + params);
const data = await api('/nodes?' + params, { ttl: CLIENT_TTL.nodeList });
nodes = data.nodes || [];
counts = data.counts || {};
@@ -238,7 +238,7 @@
const missing = myNodes.filter(mn => !existingKeys.has(mn.pubkey));
if (missing.length) {
const fetched = await Promise.allSettled(
missing.map(mn => api('/nodes/' + encodeURIComponent(mn.pubkey)))
missing.map(mn => api('/nodes/' + encodeURIComponent(mn.pubkey), { ttl: CLIENT_TTL.nodeDetail }))
);
fetched.forEach(r => {
if (r.status === 'fulfilled' && r.value && r.value.public_key) nodes.push(r.value);
@@ -401,8 +401,8 @@
try {
const [data, healthData] = await Promise.all([
api('/nodes/' + encodeURIComponent(pubkey)),
api('/nodes/' + encodeURIComponent(pubkey) + '/health').catch(() => null)
api('/nodes/' + encodeURIComponent(pubkey), { ttl: CLIENT_TTL.nodeDetail }),
api('/nodes/' + encodeURIComponent(pubkey) + '/health', { ttl: CLIENT_TTL.nodeDetail }).catch(() => null)
]);
data.healthData = healthData;
renderDetail(panel, data);

View File

@@ -38,7 +38,7 @@
async function loadObservers() {
try {
const data = await api('/observers');
const data = await api('/observers', { ttl: CLIENT_TTL.observers });
observers = data.observers || [];
render();
} catch (e) {

View File

@@ -182,9 +182,80 @@
} catch {}
}
wsHandler = debouncedOnWS(function (msgs) {
if (msgs.some(function (m) { return m.type === 'packet'; })) {
loadPackets();
const newPkts = msgs
.filter(m => m.type === 'packet' && m.data?.packet)
.map(m => m.data.packet);
if (!newPkts.length) return;
// Check if new packets pass current filters
const filtered = newPkts.filter(p => {
if (filters.type !== undefined && filters.type !== '' && p.payload_type !== Number(filters.type)) return false;
if (filters.observer && p.observer_id !== filters.observer) return false;
if (filters.hash && p.hash !== filters.hash) return false;
if (filters.region) {
const obs = observers.find(o => o.id === p.observer_id);
if (!obs || obs.iata !== filters.region) return false;
}
if (filters.node && !(p.decoded_json || '').includes(filters.node)) return false;
return true;
});
if (!filtered.length) return;
// Resolve any new hops, then update and re-render
const newHops = new Set();
for (const p of filtered) {
try { JSON.parse(p.path_json || '[]').forEach(h => { if (!(h in hopNameCache)) newHops.add(h); }); } catch {}
}
(newHops.size ? resolveHops([...newHops]) : Promise.resolve()).then(() => {
if (groupByHash) {
// Update existing groups or create new ones
for (const p of filtered) {
const h = p.hash;
const existing = packets.find(g => g.hash === h);
if (existing) {
existing.count = (existing.count || 1) + 1;
existing.latest = p.timestamp > existing.latest ? p.timestamp : existing.latest;
// Track unique observers
if (p.observer_id && p.observer_id !== existing.observer_id) {
existing.observer_count = (existing.observer_count || 1) + 1;
}
// Keep longest path
if (p.path_json && (!existing.path_json || p.path_json.length > existing.path_json.length)) {
existing.path_json = p.path_json;
existing.raw_hex = p.raw_hex;
}
// Update decoded_json to latest
if (p.decoded_json) existing.decoded_json = p.decoded_json;
// Update expanded children if this group is expanded
if (expandedHashes.has(h) && existing._children) {
existing._children.unshift(p);
}
} else {
// New group
packets.unshift({
hash: h,
count: 1,
observer_count: 1,
latest: p.timestamp,
observer_id: p.observer_id,
observer_name: p.observer_name,
path_json: p.path_json,
payload_type: p.payload_type,
raw_hex: p.raw_hex,
decoded_json: p.decoded_json,
});
}
}
// Re-sort by latest DESC, cap size
packets.sort((a, b) => (b.latest || '').localeCompare(a.latest || ''));
packets = packets.slice(0, 200);
} else {
// Flat mode: prepend
packets = filtered.concat(packets).slice(0, 200);
}
totalCount += filtered.length;
renderTableRows();
});
});
}
@@ -207,7 +278,7 @@
async function loadObservers() {
try {
const data = await api('/observers');
const data = await api('/observers', { ttl: CLIENT_TTL.observers });
observers = data.observers || [];
} catch {}
}

144
public/perf.js Normal file
View File

@@ -0,0 +1,144 @@
/* === MeshCore Analyzer — perf.js === */
'use strict';
(function () {
let interval = null;
async function render(app) {
app.innerHTML = '<div style="height:100%;overflow-y:auto;padding:16px 24px;"><h2>⚡ Performance Dashboard</h2><div id="perfContent">Loading...</div></div>';
await refresh();
}
async function refresh() {
const el = document.getElementById('perfContent');
if (!el) return;
try {
const [server, client] = await Promise.all([
fetch('/api/perf').then(r => r.json()),
Promise.resolve(window.apiPerf ? window.apiPerf() : null)
]);
// Also fetch health telemetry
const health = await fetch('/api/health').then(r => r.json()).catch(() => null);
let html = '';
// Server overview
html += `<div style="display:flex;gap:16px;flex-wrap:wrap;margin:16px 0;">
<div class="perf-card"><div class="perf-num">${server.totalRequests}</div><div class="perf-label">Total Requests</div></div>
<div class="perf-card"><div class="perf-num">${server.avgMs}ms</div><div class="perf-label">Avg Response</div></div>
<div class="perf-card"><div class="perf-num">${health ? health.uptimeHuman : Math.round(server.uptime / 60) + 'm'}</div><div class="perf-label">Uptime</div></div>
<div class="perf-card"><div class="perf-num">${server.slowQueries.length}</div><div class="perf-label">Slow (&gt;100ms)</div></div>
</div>`;
// System health (memory, event loop, WS)
if (health) {
const m = health.memory, el = health.eventLoop;
const elColor = el.p95Ms > 500 ? '#ef4444' : el.p95Ms > 100 ? '#f59e0b' : '#22c55e';
const memColor = m.heapUsed > m.heapTotal * 0.85 ? '#ef4444' : m.heapUsed > m.heapTotal * 0.7 ? '#f59e0b' : '#22c55e';
html += `<h3>System Health</h3><div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
<div class="perf-card"><div class="perf-num" style="color:${memColor}">${m.heapUsed}MB</div><div class="perf-label">Heap Used / ${m.heapTotal}MB</div></div>
<div class="perf-card"><div class="perf-num">${m.rss}MB</div><div class="perf-label">RSS</div></div>
<div class="perf-card"><div class="perf-num" style="color:${elColor}">${el.p95Ms}ms</div><div class="perf-label">Event Loop p95</div></div>
<div class="perf-card"><div class="perf-num">${el.maxLagMs}ms</div><div class="perf-label">EL Max Lag</div></div>
<div class="perf-card"><div class="perf-num">${el.currentLagMs}ms</div><div class="perf-label">EL Current</div></div>
<div class="perf-card"><div class="perf-num">${health.websocket.clients}</div><div class="perf-label">WS Clients</div></div>
</div>`;
}
// Cache stats
if (server.cache) {
const c = server.cache;
const clientCache = _apiCache ? _apiCache.size : 0;
html += `<h3>Cache</h3><div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
<div class="perf-card"><div class="perf-num">${c.size}</div><div class="perf-label">Server Entries</div></div>
<div class="perf-card"><div class="perf-num">${c.hits}</div><div class="perf-label">Server Hits</div></div>
<div class="perf-card"><div class="perf-num">${c.misses}</div><div class="perf-label">Server Misses</div></div>
<div class="perf-card"><div class="perf-num" style="color:${c.hitRate > 50 ? '#22c55e' : c.hitRate > 20 ? '#f59e0b' : '#ef4444'}">${c.hitRate}%</div><div class="perf-label">Server Hit Rate</div></div>
<div class="perf-card"><div class="perf-num">${c.staleHits || 0}</div><div class="perf-label">Stale Hits (SWR)</div></div>
<div class="perf-card"><div class="perf-num">${c.recomputes || 0}</div><div class="perf-label">Recomputes</div></div>
<div class="perf-card"><div class="perf-num">${clientCache}</div><div class="perf-label">Client Entries</div></div>
</div>`;
if (client) {
html += `<div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
<div class="perf-card"><div class="perf-num">${client.cacheHits || 0}</div><div class="perf-label">Client Hits</div></div>
<div class="perf-card"><div class="perf-num">${client.cacheMisses || 0}</div><div class="perf-label">Client Misses</div></div>
<div class="perf-card"><div class="perf-num" style="color:${(client.cacheHitRate||0) > 50 ? '#22c55e' : '#f59e0b'}">${client.cacheHitRate || 0}%</div><div class="perf-label">Client Hit Rate</div></div>
</div>`;
}
}
// Packet Store stats
if (server.packetStore) {
const ps = server.packetStore;
html += `<h3>In-Memory Packet Store</h3><div style="display:flex;gap:16px;flex-wrap:wrap;margin:8px 0;">
<div class="perf-card"><div class="perf-num">${ps.inMemory.toLocaleString()}</div><div class="perf-label">Packets in RAM</div></div>
<div class="perf-card"><div class="perf-num">${ps.estimatedMB}MB</div><div class="perf-label">Memory Used</div></div>
<div class="perf-card"><div class="perf-num">${ps.maxMB}MB</div><div class="perf-label">Memory Limit</div></div>
<div class="perf-card"><div class="perf-num">${ps.queries.toLocaleString()}</div><div class="perf-label">Queries Served</div></div>
<div class="perf-card"><div class="perf-num">${ps.inserts.toLocaleString()}</div><div class="perf-label">Live Inserts</div></div>
<div class="perf-card"><div class="perf-num">${ps.evicted.toLocaleString()}</div><div class="perf-label">Evicted</div></div>
<div class="perf-card"><div class="perf-num">${ps.indexes.byHash.toLocaleString()}</div><div class="perf-label">Unique Hashes</div></div>
<div class="perf-card"><div class="perf-num">${ps.indexes.byObserver}</div><div class="perf-label">Observers</div></div>
<div class="perf-card"><div class="perf-num">${ps.indexes.byNode.toLocaleString()}</div><div class="perf-label">Indexed Nodes</div></div>
</div>`;
}
// Server endpoints table
const eps = Object.entries(server.endpoints);
if (eps.length) {
html += '<h3>Server Endpoints (sorted by total time)</h3>';
html += '<div style="overflow-x:auto"><table class="perf-table"><thead><tr><th>Endpoint</th><th>Count</th><th>Avg</th><th>P50</th><th>P95</th><th>Max</th><th>Total</th></tr></thead><tbody>';
for (const [path, s] of eps) {
const total = Math.round(s.count * s.avgMs);
const cls = s.p95Ms > 200 ? ' class="perf-slow"' : s.p95Ms > 50 ? ' class="perf-warn"' : '';
html += `<tr${cls}><td><code>${path}</code></td><td>${s.count}</td><td>${s.avgMs}ms</td><td>${s.p50Ms}ms</td><td>${s.p95Ms}ms</td><td>${s.maxMs}ms</td><td>${total}ms</td></tr>`;
}
html += '</tbody></table></div>';
}
// Client API calls
if (client && client.endpoints.length) {
html += '<h3>Client API Calls (this session)</h3>';
html += '<div style="overflow-x:auto"><table class="perf-table"><thead><tr><th>Endpoint</th><th>Count</th><th>Avg</th><th>Max</th><th>Total</th></tr></thead><tbody>';
for (const s of client.endpoints) {
const cls = s.maxMs > 500 ? ' class="perf-slow"' : s.avgMs > 200 ? ' class="perf-warn"' : '';
html += `<tr${cls}><td><code>${s.path}</code></td><td>${s.count}</td><td>${s.avgMs}ms</td><td>${s.maxMs}ms</td><td>${s.totalMs}ms</td></tr>`;
}
html += '</tbody></table></div>';
}
// Slow queries
if (server.slowQueries.length) {
html += '<h3>Recent Slow Queries (&gt;100ms)</h3>';
html += '<div style="overflow-x:auto"><table class="perf-table"><thead><tr><th>Time</th><th>Path</th><th>Duration</th><th>Status</th></tr></thead><tbody>';
for (const q of server.slowQueries.slice().reverse()) {
html += `<tr class="perf-slow"><td>${new Date(q.time).toLocaleTimeString()}</td><td><code>${q.path}</code></td><td>${q.ms}ms</td><td>${q.status}</td></tr>`;
}
html += '</tbody></table></div>';
}
html += `<div style="margin-top:16px"><button id="perfReset" style="padding:8px 16px;cursor:pointer">Reset Stats</button> <button id="perfRefresh" style="padding:8px 16px;cursor:pointer">Refresh</button></div>`;
el.innerHTML = html;
document.getElementById('perfReset')?.addEventListener('click', async () => {
await fetch('/api/perf/reset', { method: 'POST' });
if (window._apiPerf) { window._apiPerf = { calls: 0, totalMs: 0, log: [] }; }
refresh();
});
document.getElementById('perfRefresh')?.addEventListener('click', refresh);
} catch (err) {
el.innerHTML = `<p style="color:red">Error: ${err.message}</p>`;
}
}
registerPage('perf', {
init(app) {
render(app);
interval = setInterval(refresh, 5000);
},
destroy() {
if (interval) { clearInterval(interval); interval = null; }
}
});
})();

View File

@@ -1413,3 +1413,16 @@ tr[data-hops]:hover { background: rgba(59,130,246,0.1); }
}
.mobile-sheet-close:hover { color: var(--text); }
.mobile-sheet-content { padding-top: 4px; }
/* Perf dashboard */
.perf-card { background: var(--surface-1); border: 1px solid var(--border); border-radius: 8px; padding: 12px 20px; min-width: 120px; text-align: center; }
.perf-num { font-size: 24px; font-weight: 800; color: var(--text); font-variant-numeric: tabular-nums; }
.perf-label { font-size: 11px; color: var(--text-muted); text-transform: uppercase; letter-spacing: 0.5px; margin-top: 4px; }
.perf-table { width: 100%; border-collapse: collapse; font-size: 13px; }
.perf-table th { text-align: left; padding: 6px 10px; border-bottom: 2px solid var(--border); color: var(--text-muted); font-size: 11px; text-transform: uppercase; }
.perf-table td { padding: 5px 10px; border-bottom: 1px solid var(--border); font-variant-numeric: tabular-nums; }
.perf-table code { font-size: 12px; color: var(--text); }
.perf-table .perf-slow { background: rgba(239, 68, 68, 0.08); }
.perf-table .perf-slow td { color: #ef4444; }
.perf-table .perf-warn { background: rgba(251, 191, 36, 0.06); }
.perf-table .perf-warn td { color: #f59e0b; }

821
server.js

File diff suppressed because it is too large Load Diff