## Problem
`spTxIndex` kept every 2–8 hop run of every stored path as a map key,
with one transmission pointer per occurrence. On a busy regional
network's 7-day window (59k transmissions) that's about 0.9–1.07 M keys
and 130–160 MB of live heap, roughly a quarter of the store. It was also
rebuilt at the end of every startup, which is when memory peaks.
Two smaller issues came with it:
- Eviction never decremented `spTotalPaths`, so the percentages were
divided by paths that were no longer stored.
- Route Patterns took 1.2–3.6 s uncached, because the fast path still
split, resolved and re-joined about a million raw keys per request.
## Change
- **Count on request.** Subpaths are counted when the analytics are
asked for, from the stored paths. Results stay cached for `rfCacheTTL`,
as before.
- **Pointer-free tallies.** `tallyNamedSubpaths` counts runs in one
pointer-free `map[uint64]subpathTally`, keyed by a `maphash` of the
resolved name built in a reused buffer. The GC doesn't scan the map, and
counting allocates nothing but the map.
- **Names only for the top entries.** Each tally records the first
transmission and offset where its run appeared, so names and raw hops
are built only for each group's top entries. `topSubpathTallies` picks
those with a `container/heap` min-heap of `limit` entries instead of
sorting every name, and the existing `rankSubpaths` formats the
response, so no new response maps are added.
- **Hash collisions.** If two names shared a hash their counts would
merge; the odds are about 1 in 10⁷ for a few million names.
- **Subpath detail.** `GetSubpathDetail` finds the run with
`txsWithSubpath`, a scan of the stored paths.
- **Counts unchanged:** runs of 2–8 hops, counted once per occurrence,
case-folded. The tests pin this.
- **Totals.** `spTotalPaths` stays, and eviction now decrements it.
- **Memory estimate.** The O(path²) subpath term is dropped from the
estimate.
## Perf justification (AGENTS.md rule 0)
- **Complexity.** A cold Route Patterns request is O(Σ runs), about 1.5
M for this window: one pass over the stored paths, with one map lookup
per run against a reused buffer. Ranking is O(tallies · log limit).
Results are cached for `rfCacheTTL`, so this runs at most once per TTL
per group set.
- **Subpath detail** is O(stored hops), one pass with no allocation per
packet. It's an on-click endpoint, not a hot path.
- **Ingest and eviction** no longer touch a subpath index at all.
Removal used to be a linear search of every key's transmission list.
**Measured on a copy of that window**, upstream master vs this branch,
same queries:
| | master | this PR |
|---|---|---|
| peak RSS (`GOMEMLIMIT=750MiB`) | 948 MB | 767 MB |
| in-use heap after load (750MiB) | 592 MB | 456 MB |
| `/api/analytics/subpaths` cold (750MiB) | 3.56 s | 0.64 s |
| `/api/analytics/subpaths-bulk` cold (750MiB) | 3.33 s | 0.70 s |
| `/api/analytics/subpaths` cold (1GiB) | 1.18 s | 0.61 s |
| `/api/analytics/subpath-detail` | 0.6 ms | 8 ms |
| cached responses | 0.6 ms | 0.5 ms |
At 750 MiB, master's request time is mostly GC pressure from the index.
On the production-shaped fixture that `subpath_index_memory_test.go`
builds (41k transmissions):
| | master | this PR |
|---|---|---|
| retained by the store for subpaths | 197 MB | 0 MB |
| allocated per bulk request | 336 MB | 111 MB |
This has also run on a staging instance with live MQTT ingest for about
nine hours, with Route Patterns answering in about 1 s.
## Tests
- `subpaths_on_demand_test.go`:
- per-occurrence, case-folded counts;
- the 8-hop cap;
- bulk matches single;
- detail counts, including repeats, case and the length limits;
- eviction drops evicted paths from the totals.
- `TestSubpathIndexRetainedHeap`: the bound is now 5 MB (it was 225 MB).
- `TestSubpathRequestAllocations`: 150 MB per request.
- Each change has its red commit first, failing on assertions.
- Tests that reached into `spTxIndex` now assert through
`txsWithSubpath` and the public counts.
- `cd cmd/server && go test ./...` passes.
- SonarQube: quality gate passed, no issues on added lines.
**Overlap with #2181:** both edit the same memory-estimate comment block
in `store.go`. Whichever lands second needs a trivial rebase there;
there's no code dependency.
Backend only: no UI, API shape or config change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
