perf(analytics): opt-in pauseWhenIdle skips recomputes nobody reads (#2164)

## Summary The analytics recomputers (#1240) recompute eleven snapshots (topology, RF, distance, channels, hash collisions, hash sizes, roles, two clock-skew views, retransmissions, direct-heard) on a fixed interval, 300 s by default, whether or not anyone reads them. That is the right trade for a busy instance: reads always hit a warm cache. On a quiet one it means most of the server's allocation goes on results nobody looks at. Measured on one production instance (2 vCPU / 2 GB): - topology, RF and hash-sizes were each **read 23 times in 45 hours** while being recomputed every 5 minutes; - `analyticsRecomputer.runOnce` accounted for **30% of all bytes allocated** (121 GB of 391 GB in 45 h); - garbage collection was 87% of the process's CPU (30 s CPU profile: `gcDrain` 39.9 s of 45.9 s sampled), with an 805 MB live heap against a 1 GiB `GOMEMLIMIT`, so allocation rate translates directly into collector time. This PR adds an **opt-in** `analytics.pauseWhenIdle`: - a recomputer's tick is skipped when nothing has read its snapshot since the previous compute, and the recomputer is marked paused; - the next read still returns immediately with the snapshot it holds (`Load` stays non-blocking, as #1240 requires), and makes a non-blocking send that wakes the loop to recompute; reads after that get the fresh snapshot and the ticker resumes from there. **Default is off**, so behaviour is unchanged unless an operator turns it on. The trade-off for operators who do: the first read after a quiet spell can show a snapshot older than the interval (refreshed seconds later). It is documented in `config.example.json`. No existing issue covers this. ## Config example Off by default. To turn it on, add one key to the existing `analytics` block in `config.json`: ```json "analytics": { "defaultIntervalSeconds": 300, "pauseWhenIdle": true } ``` Nothing else changes: per-endpoint `recomputeIntervalSeconds` still apply while an endpoint is being read. There is no UI change, so no screenshot. ## Performance Staging A/B: the same 30 minutes of real production API traffic (5,640 requests from a day's nginx log, replayed at 2x against a copy of the production database with the live MQTT feed), on `master` and on this branch with `pauseWhenIdle: true`, each after a restart and warm-up. Allocation is the delta of `/debug/pprof/allocs` across the replay: | during the replay | master | this PR | |---|---|---| | allocated by `analyticsRecomputer.runOnce` | 459.7 MB | **53.4 MB (-88%)** | | total allocated by the process | 3,704 MB | 3,372 MB (-9%) | | GC cycles | 52 | 41 | | requests: p50 / p95 / errors | 9.9 ms / 57.0 ms / 0 | 10.4 ms / 60.1 ms / 0 | The replayed traffic included real analytics reads, so the paused recomputers were woken by them; the remaining 53 MB is that. Over 15 minutes each recomputer only ticks about three times, so on a longer quiet period the saving grows with the idle time. GC count and CPU on this shared host vary between runs on their own, so the allocation figures are the claim, not the GC count. There is no Go benchmark for this change: the behaviour is "how many computes happen in a period of no reads", which the tests below assert directly (16 computes on master vs at most 2 here over the same idle period). ## Tests - Red commit `24056c57`: config field, accessor stub and an unused recomputer flag, plus tests. Three fail on the stub: - `TestAnalyticsRecomputer_PauseWhenIdleSkipsUnreadTicks`: an unread recomputer with a 20 ms interval computes at most twice in 300 ms (16 times on the stub); - `TestConfig_AnalyticsPauseWhenIdle`: `pauseWhenIdle: true` enables it, absent or nil config leaves it off; - `TestStartAnalyticsRecomputers_AppliesPauseWhenIdle`: all eleven recomputers receive the flag. - Also added, passing on both commits on purpose: a read on a paused recomputer returns the existing snapshot and triggers a refresh; a recomputer that is being read keeps ticking; the default (off) keeps ticking unread. - Green commit `19de3636`: the behaviour, `config.example.json` entry with `_comment_pauseWhenIdle`, and the `Load` doc comment. - `go test -race` on the recomputer, warm-up (#1659) and config tests, three runs: pass. `make test`: 16 of 16 modules pass. The frontend tests that read `config.example.json` pass. ## Review notes - `roles` reads the nodes-clock-skew snapshot inside its own compute, which counts as a read, so that dependency keeps its source fresh while roles is in use. - The #1659 warm-up gate is untouched: a skipped tick never marks a first pass done. - Config documented per WORKFLOW.md section 7. Customizer (AGENTS.md rule 8): it is an operator setting, not a display value; no customizer entry proposed. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - No issues on the added lines. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:38:36 +00:00 · 2026-10-11 09:51:48 +02:00
parent 352a0ccd19
commit 4e91c5cf0b
5 changed files with 214 additions and 2 deletions
@@ -0,0 +1,142 @@
+package main
+
+import (
+	"encoding/json"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+// The analytics recomputers (#1240) recompute every endpoint on a fixed
+// interval whether or not anyone reads it. On a production instance the
+// topology, RF and hash-size endpoints were each read 23 times in 45 hours
+// while being recomputed every 5 minutes, and the recompute loop accounted for
+// 30% of all bytes allocated (121 GB in 45 h), which drives the collector.
+//
+// With pauseWhenIdle, a tick with no read since the previous compute is
+// skipped, and the next Load serves the existing snapshot and kicks a refresh.
+
+func startCountingRecomputer(t *testing.T, interval time.Duration, pause bool) (*analyticsRecomputer, *int64) {
+	t.Helper()
+	var runs int64
+	r := newAnalyticsRecomputer("test", interval, func() interface{} {
+		return atomic.AddInt64(&runs, 1)
+	})
+	r.pauseWhenIdle = pause
+	r.Start()
+	t.Cleanup(r.Stop)
+	return r, &runs
+}
+
+func TestAnalyticsRecomputer_PauseWhenIdleSkipsUnreadTicks(t *testing.T) {
+	_, runs := startCountingRecomputer(t, 20*time.Millisecond, true)
+	time.Sleep(300 * time.Millisecond) // ~15 ticks, no reads
+	if got := atomic.LoadInt64(runs); got > 2 {
+		t.Fatalf("expected the idle recomputer to stop after its first pass, got %d computes", got)
+	}
+}
+
+func TestAnalyticsRecomputer_ReadAfterPauseServesSnapshotAndRefreshes(t *testing.T) {
+	r, runs := startCountingRecomputer(t, 20*time.Millisecond, true)
+	time.Sleep(150 * time.Millisecond) // pause
+	before := atomic.LoadInt64(runs)
+
+	if v := r.Load(); v == nil {
+		t.Fatal("expected the existing snapshot while paused, got nil")
+	}
+	deadline := time.Now().Add(2 * time.Second)
+	for atomic.LoadInt64(runs) == before && time.Now().Before(deadline) {
+		time.Sleep(5 * time.Millisecond)
+	}
+	if atomic.LoadInt64(runs) == before {
+		t.Fatal("expected a read on a paused recomputer to start a refresh")
+	}
+	if got, ok := r.Load().(int64); !ok || got <= before {
+		t.Fatalf("expected a newer snapshot after the refresh, got %v", r.Load())
+	}
+}
+
+func TestAnalyticsRecomputer_KeepsTickingWhileRead(t *testing.T) {
+	r, runs := startCountingRecomputer(t, 20*time.Millisecond, true)
+	stop := time.Now().Add(300 * time.Millisecond)
+	for time.Now().Before(stop) {
+		r.Load()
+		time.Sleep(5 * time.Millisecond)
+	}
+	if got := atomic.LoadInt64(runs); got < 6 {
+		t.Fatalf("expected a recomputer that is being read to keep refreshing, got %d computes", got)
+	}
+}
+
+// Default behaviour is unchanged: without pauseWhenIdle it recomputes on every
+// tick, read or not.
+func TestAnalyticsRecomputer_DefaultKeepsTickingUnread(t *testing.T) {
+	_, runs := startCountingRecomputer(t, 20*time.Millisecond, false)
+	time.Sleep(300 * time.Millisecond)
+	if got := atomic.LoadInt64(runs); got < 6 {
+		t.Fatalf("expected the default recomputer to keep ticking, got %d computes", got)
+	}
+}
+
+func TestConfig_AnalyticsPauseWhenIdle(t *testing.T) {
+	var on, off Config
+	if err := json.Unmarshal([]byte(`{"analytics":{"pauseWhenIdle":true}}`), &on); err != nil {
+		t.Fatal(err)
+	}
+	if err := json.Unmarshal([]byte(`{"analytics":{"defaultIntervalSeconds":300}}`), &off); err != nil {
+		t.Fatal(err)
+	}
+	if !on.AnalyticsPauseWhenIdle() {
+		t.Error("expected pauseWhenIdle:true to enable it")
+	}
+	if off.AnalyticsPauseWhenIdle() {
+		t.Error("expected it to default to off")
+	}
+	if (*Config)(nil).AnalyticsPauseWhenIdle() {
+		t.Error("expected a nil config to default to off")
+	}
+}
+
+func TestStartAnalyticsRecomputers_AppliesPauseWhenIdle(t *testing.T) {
+	db := setupTestDB(t)
+	defer db.Close()
+	store := NewPacketStore(db, nil)
+	stop := store.StartAnalyticsRecomputers(time.Hour, AnalyticsRecomputeIntervals{PauseWhenIdle: true})
+	defer stop()
+	store.analyticsRecomputerMu.Lock()
+	all := store.analyticsRecomputersLocked()
+	store.analyticsRecomputerMu.Unlock()
+	for _, rc := range all {
+		if !rc.pauseWhenIdle {
+			t.Errorf("recomputer %q did not get pauseWhenIdle", rc.name)
+		}
+	}
+}
+
+// The node-health "Heard By" card reads the direct-heard snapshot through
+// loadDirectHeard, not through the recomputer's Load. With pauseWhenIdle on,
+// that read must still count, or direct-heard pauses after startup and never
+// recomputes again: nodes first heard later show "nobody hears this node".
+func TestDirectHeard_PausedRecomputerResumesWhenTheCardIsRead(t *testing.T) {
+	db := setupTestDB(t)
+	defer db.Close()
+	store := NewPacketStore(db, nil)
+	stop := store.StartAnalyticsRecomputers(20*time.Millisecond, AnalyticsRecomputeIntervals{PauseWhenIdle: true})
+	defer stop()
+	store.analyticsRecomputerMu.Lock()
+	rc := store.recompDirectHeard
+	store.analyticsRecomputerMu.Unlock()
+	if rc == nil {
+		t.Fatal("no direct-heard recomputer")
+	}
+	time.Sleep(150 * time.Millisecond) // unread: pauses
+	before := rc.computeRuns.Load()
+	deadline := time.Now().Add(300 * time.Millisecond)
+	for time.Now().Before(deadline) {
+		store.loadDirectHeard() // what the Heard By card calls
+		time.Sleep(10 * time.Millisecond)
+	}
+	if got := rc.computeRuns.Load() - before; got < 2 {
+		t.Errorf("direct-heard recomputed %d times in 300 ms of card reads at a 20 ms interval, want it to keep refreshing", got)
+	}
+}
@@ -45,6 +45,15 @@ type analyticsRecomputer struct {
 	startOnce sync.Once
 	stopOnce  sync.Once

+	// pauseWhenIdle: see AnalyticsConfig.PauseWhenIdle. A tick with no
+	// Load since the previous compute is skipped and the recomputer is
+	// marked paused; the next Load serves the existing snapshot and sends
+	// on kick, and the loop recomputes then.
+	pauseWhenIdle    bool
+	readSinceCompute atomic.Bool
+	paused           atomic.Bool
+	kick             chan struct{} // buffered(1): a paused recomputer was read
+
 	// Stats (atomic).
 	computeRuns   atomic.Int64
 	lastComputeNs atomic.Int64 // duration of last compute in nanoseconds
@@ -71,6 +80,7 @@ func newAnalyticsRecomputer(name string, interval time.Duration, compute func()
 		stop:         make(chan struct{}),
 		done:         make(chan struct{}),
 		recomputeReq: make(chan chan struct{}),
+		kick:         make(chan struct{}, 1),
 	}
 }

@@ -100,7 +110,16 @@ func (r *analyticsRecomputer) loop() {
 	for {
 		select {
 		case <-t.C:
+			if r.pauseWhenIdle && !r.readSinceCompute.Load() {
+				r.paused.Store(true)
+				continue
+			}
 			r.runOnce()
+		case <-r.kick:
+			if r.paused.Load() {
+				r.runOnce()
+				t.Reset(r.interval)
+			}
 		case ack := <-r.recomputeReq:
 			r.runOnce()
 			t.Reset(r.interval)
@@ -127,6 +146,9 @@ func (r *analyticsRecomputer) runOnce() {
 	// started on a partially loaded store must not end the warm-up,
 	// even if the load finishes while it runs.
 	ready := r.warmupReadyGateOpen_1659()
+	// Reads from here on count towards the next tick (pauseWhenIdle).
+	r.readSinceCompute.Store(false)
+	r.paused.Store(false)
 	t0 := time.Now()
 	result := r.compute()
 	r.lastComputeNs.Store(int64(time.Since(t0)))
@@ -190,10 +212,30 @@ func recomputeWhenLoaded(loaded, stop <-chan struct{}, rcs []*analyticsRecompute
 		len(rcs), time.Since(t0).Round(time.Millisecond), strings.Join(parts, " "))
 }

+// markRead counts a read for pauseWhenIdle: the next tick recomputes, and a
+// paused recomputer is kicked to refresh now. Readers that take the
+// snapshot some other way (direct-heard publishes its own) call it
+// directly. Never blocks; a no-op when pauseWhenIdle is off.
+func (r *analyticsRecomputer) markRead() {
+	if !r.pauseWhenIdle {
+		return
+	}
+	r.readSinceCompute.Store(true)
+	if r.paused.Load() {
+		select {
+		case r.kick <- struct{}{}:
+		default: // a refresh is already queued
+		}
+	}
+}
+
 // Load returns the most recently computed snapshot, or nil if Start
 // has not been called (or the very first compute returned nil).
-// Never blocks beyond a single atomic load.
+// Never blocks: beyond the atomic load, a paused recomputer (pauseWhenIdle)
+// gets a non-blocking send on kick, and the snapshot it already holds is
+// returned while the refresh runs.
 func (r *analyticsRecomputer) Load() interface{} {
+	r.markRead()
 	v := r.cache.Load()
 	if v == nil {
 		return nil
@@ -238,6 +280,9 @@ type AnalyticsRecomputeIntervals struct {
 	Roles              time.Duration
 	ObserversClockSkew time.Duration
 	NodesClockSkew     time.Duration
+
+	// PauseWhenIdle is applied to every recomputer (AnalyticsConfig.PauseWhenIdle).
+	PauseWhenIdle bool
 }

 func pickInterval(override, def time.Duration) time.Duration {
@@ -352,6 +397,9 @@ func (s *PacketStore) StartAnalyticsRecomputers(defaultInterval time.Duration, o
 		},
 	)
 	all := s.analyticsRecomputersLocked()
+	for _, rc := range all {
+		rc.pauseWhenIdle = ov.PauseWhenIdle
+	}
 	s.analyticsRecomputerMu.Unlock()

 	// Issue #1659 (PR #1688 r1, munger #5): wire the loader readiness
@@ -1138,6 +1138,11 @@ func (c *Config) IsObserverBlacklisted(id string) bool {
 type AnalyticsConfig struct {
 	DefaultIntervalSeconds   int            `json:"defaultIntervalSeconds,omitempty"`
 	RecomputeIntervalSeconds map[string]int `json:"recomputeIntervalSeconds,omitempty"`
+	// PauseWhenIdle skips a scheduled recompute when nothing has read that
+	// endpoint's snapshot since the previous recompute. The first read
+	// after a pause is served the existing snapshot at once and starts a
+	// refresh. Off by default (always recompute on the interval).
+	PauseWhenIdle bool `json:"pauseWhenIdle,omitempty"`
 	// LoRaPreset is the assumed PHY preset used by the relay-airtime-share
 	// metric to compute true Time-on-Air (issue #1768). Defaults to the
 	// EU MeshCore deployment: 869.6 MHz / BW 62.5 kHz / SF 8 / CR 4/5.
@@ -1164,11 +1169,16 @@ func (c *Config) AnalyticsDefaultRecomputeInterval() time.Duration {
 	return 5 * time.Minute
 }

+// AnalyticsPauseWhenIdle reports whether idle analytics recomputes are skipped.
+func (c *Config) AnalyticsPauseWhenIdle() bool {
+	return c != nil && c.Analytics != nil && c.Analytics.PauseWhenIdle
+}
+
 // AnalyticsRecomputeIntervals returns the per-endpoint override map.
 // Returns the zero value (all defaults) if the analytics block is
 // absent or empty.
 func (c *Config) AnalyticsRecomputeIntervals() AnalyticsRecomputeIntervals {
-	out := AnalyticsRecomputeIntervals{}
+	out := AnalyticsRecomputeIntervals{PauseWhenIdle: c.AnalyticsPauseWhenIdle()}
 	if c == nil || c.Analytics == nil || c.Analytics.RecomputeIntervalSeconds == nil {
 		return out
 	}
@@ -216,7 +216,17 @@ func (s *PacketStore) publishDirectHeard(idx directHeardIndex) {
 // compute has published one. A nil index simply yields empty direct rows:
 // the card degrades to "nobody hears this node" until the first pass lands,
 // never to a wrong attribution.
+//
+// It counts as a read of the direct-heard recomputer, so with pauseWhenIdle
+// on the card keeps it refreshing (the snapshot is read here, not through
+// the recomputer's Load).
 func (s *PacketStore) loadDirectHeard() directHeardIndex {
+	s.analyticsRecomputerMu.RLock()
+	rc := s.recompDirectHeard
+	s.analyticsRecomputerMu.RUnlock()
+	if rc != nil {
+		rc.markRead()
+	}
 	idx, _ := s.directHeardSnap.Load().(directHeardIndex)
 	return idx
 }
@@ -414,6 +414,8 @@

  "analytics": {
    "defaultIntervalSeconds": 300,
+    "pauseWhenIdle": false,
+    "_comment_pauseWhenIdle": "Opt-in. When true, a scheduled recompute is skipped if nothing has read that endpoint's snapshot since the previous recompute, so an instance whose analytics pages are rarely open stops recomputing (and allocating) for them every interval. The first read after a pause is served the existing snapshot at once, which can be older than the interval, and starts a refresh; reads after that see fresh data. Default false: every endpoint is recomputed on its interval whether read or not (#1240).",
    "recomputeIntervalSeconds": {
      "topology": 300,
      "rf": 300,