mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-10-15 12:29:52 +00:00
Part of #1870: this interns the three fields the issue names (`ObserverID`, `ObserverName`, `ObserverIATA`), plus `Direction`. ## Problem Each `StoreObs` kept the row scan's own copy of its observer ID (64 hex characters for most observers), observer name, IATA code and direction. A busy regional network's 7-day window holds about 400k observations from a few hundred observers, so a few hundred distinct values were stored hundreds of thousands of times. ## Change - `internStr` returns the canonical copy of each value from a map guarded by an `RWMutex`, so every observation shares one copy. - The copies are held for the life of the process. That is what makes the sharing survive garbage collections. An earlier version of this PR used `unique.Make`, which holds its copies weakly, so a value interned again after a GC came back as a second copy (thanks to the review for catching that). - Observer strings number in the hundreds. The map is capped at 65,536 entries as a guard (AGENTS.md: no unbounded structures); past the cap a string is returned as it came, so nothing breaks. - A hit takes only the read lock and allocates nothing. A new value is cloned, so it doesn't pin a larger buffer it may be a slice of. - All five places that build a `StoreObs` use it: `Load`, `loadChunk`, `scanAndMergeChunk`, and both ingest paths. - The ingest paths intern while scanning rows, **before** taking `s.mu`, so interning adds nothing to the time API readers wait on the store. - `cmd/server/go.mod` stays at `go 1.22`. The earlier 1.27 bump was only there for `unique`, and is reverted here; the module upgrade belongs in its own PR. ## Scaling `BenchmarkInternStr` (a hit, the steady state): | | alone | 12 cores (per call) | allocs | |---|---|---|---| | plain `Mutex` map (the first version of this PR) | 20 ns | 57 ns | 0 | | `unique` (the second version, withdrawn) | 48 ns | 8 ns | 0 | | **`RWMutex` map (this PR)** | 17 ns | 26 ns | 0 | ## Measured on a copy of that window (`GOMEMLIMIT=750MiB`) The same database and harness, one run each, the `unique` version against this one: | | `unique` version | this PR | |---|---|---| | live heap at the last GC after load | 581 MB | **535 MB** | | string bytes allocated by row scans still live (`bytealg.MakeNoZero`) | 106 MB | **80 MB** | | peak RSS | 777 MB | 781 MB | | load to ready | 15 s | 15 s | Peak RSS is set by the load itself, not by these strings. Against master, the first measurement of this PR found the strings retained from row scans went from 84 MB to 58 MB. An earlier version of this PR (the `unique` one) also ran on a staging instance (live MQTT ingest, `GOMEMLIMIT=1GiB`) through five back-to-back restarts and a two-hour soak, with no OOM kills and every request succeeding. This version has not had a staging soak yet. ## Tests - `observer_strings_intern_test.go`: - one copy of each value across 50 observations, through `Load`, `LoadChunked` and `loadChunk`; - **a value interned again after `runtime.GC()` is the same copy** (new: red against the `unique` version, green now); - 16 goroutines interning concurrently get the same copy; - a hit allocates nothing; - `BenchmarkInternStr` (serial and parallel). - Red commits first, then green. - `cd cmd/server && go test ./...` passes at `go 1.22`, and `-race` passes on the intern tests. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
51 lines
1.4 KiB
Go
51 lines
1.4 KiB
Go
package main
|
|
|
|
import (
|
|
"strings"
|
|
"sync"
|
|
)
|
|
|
|
// internMax bounds the intern table. Observer IDs, names, IATA codes and
|
|
// directions number in the hundreds on a busy instance, so the cap is only a
|
|
// guard against an unexpected flood of distinct values.
|
|
const internMax = 1 << 16
|
|
|
|
var (
|
|
internMu sync.RWMutex
|
|
internTable = make(map[string]string)
|
|
)
|
|
|
|
// internStr returns the canonical copy of s, so a store of hundreds of
|
|
// thousands of observations keeps one copy of each string that repeats
|
|
// across them (observer IDs, names, IATA codes, directions) rather than the
|
|
// one each row scan allocates.
|
|
//
|
|
// The table holds its copies for the life of the process: they are few, and
|
|
// a strongly held copy is what makes the sharing survive garbage collections
|
|
// (unique.Make holds its copies weakly, so a value interned again after a GC
|
|
// came back as a new copy). Past internMax entries a string is returned as
|
|
// it came: nothing is shared, nothing breaks. A hit takes only the read lock
|
|
// and does not allocate.
|
|
func internStr(s string) string {
|
|
if s == "" {
|
|
return ""
|
|
}
|
|
internMu.RLock()
|
|
c, ok := internTable[s]
|
|
internMu.RUnlock()
|
|
if ok {
|
|
return c
|
|
}
|
|
internMu.Lock()
|
|
defer internMu.Unlock()
|
|
if c, ok := internTable[s]; ok {
|
|
return c
|
|
}
|
|
if len(internTable) >= internMax {
|
|
return s
|
|
}
|
|
c = strings.Clone(s) // do not pin a larger buffer s may be a slice of
|
|
internTable[c] = c
|
|
return c
|
|
}
|