mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-12 11:38:28 +00:00
417c8dd00a243f29822f34e5c9aaadbd2e522213
19
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 n30nex
|
abad8a26b3 |
fix(liveness): require evidence from each observed relay frame (#2298)
Red commit: `46e3b082` ([regression CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203680455)). Fixes #2266 and #2267. Certain relay activity now requires an eligible path from the observation that supplied it. A representative transmission header, destination attribution or catalogue uniqueness alone cannot supply that evidence. Later valid FLOOD evidence survives observer/path deduplication. The fix preserves valid own packets and excludes failed signatures, planned DIRECT/TRACE paths, ambiguous/retired prefixes and missing or mismatched headers. Firmware reference: meshcore-dev/MeshCore `81dbae59`. - One eligibility byte replaces an unused raw-frame field. StoreObs remains 152 bytes in the protected 160-byte allocation class; no wire frames are retained. Existing dedup maps gain one pointer per slot only at 64+ observations. - Processing is linear in retained paths, bounded by 64 wire path bytes. No new SQL, schema, configuration or maps. - Tests cover five load/ingest paths, late duplicates, analytics/adaptive thresholds, positive controls and concurrent readers. Review correction `4ad6c5ad` preserves lazy decoded caches while retaining escaped/legacy signature rejection. - Full server suite, focused/race checks and frontend checks pass. Browser verified on http://127.0.0.1:18905: both upstream false-positive cases disappear and a genuine later FLOOD is recognized. Independent review findings are addressed. Local preflight checks passed; external home-directory bundle unavailable. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40205015381) passed on `4ad6c5ad`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
b3367f5aea |
fix(analytics): keep distance indexing lazy during startup (#2278)
Chunked startup now leaves the distance index lazy, as the legacy loader already does. Requests wait for the initial chunks to finish, then trigger one build and publish the refreshed result. A terminal initial-load failure returns a clear error instead of retrying forever; a completed hot window remains usable if later background filling fails. Fixes #2227. Validation: RED reproduces eager and premature builds plus the terminal-error retry loop. Full server and unchanged frontend suites pass, with focused race checks. The browser/API check observed 202 then 200, nonempty distances, and exactly one build. Performance evidence is the test-enforced absence of eager/duplicate distance construction, with required path indexes retained. No timing speedup is claimed. No new settings. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
efitenandClaude Opus 5.5
|
a08f2f4e7c |
fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR). ## Problem A repeater silent since 2026-10-18 showed **Active — Last relayed 4m ago** and **Last Heard 36m**, and its offline mail never fired. Every hit after that date was the 1-byte hop `67`, shared by nine nodes: - the live resolver's affinity tiebreak (`resolvePathForObs`) put those transmissions in `byNode`, which fed `stats.lastHeard` and `LastHeardMap`; - `last_relayed` folds in the 1-byte prefix bucket on purpose (#662), and drove the status in `roles.js` and the notify offline check. ## Fix The store keeps `certainHeard` per node: the newest transmission that can only be that node. | Evidence | Counts? | |---|---| | Own packet (`pubKey`, `srcPubKey`) | ✅ | | Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node | ✅ | | 1-byte hop shared with other repeaters | ❌ | | Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌ | | TRACE (path bytes are SNR) | ❌ | | Destination (`destPubKey`) | ❌ | A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, `Mesh::routeRecvPacket` appends at that size), so only uniqueness decides. The rule is applied in every load path through `indexResolvedPathHops` and the new `indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live ingest, late observations. Liveness reads only that: - `/api/nodes` and the node detail: `last_heard` is the newer of `last_seen` and the certain last heard. Before, it was a plain copy of `last_seen`. - health / bulk-health `stats.lastHeard`. - `LastHeardMap` → notify offline check. The separate relay input (`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed. - `roles.js getEffectiveHeardMs`: newest known timestamp, without `last_relayed`. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip. - New status **`uncertain`** (repeaters and rooms): nothing certain within the silent window, but the shared prefix (`last_relayed`) still in paths. Grey badge with explanation, an *Uncertain* filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen. - A prefix also held by a path-capable node in `inactive_nodes` is not unique; adverts with `signatureValid: false` are no evidence; `/api/nodes/search` reports the same `last_heard`. Attribution for analytics (`byNode`, packet counts, `last_relayed`, relay counters, usefulness) is unchanged. ## Performance Hot path: `noteCertainRelays` runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop. ``` BenchmarkNoteCertainRelays-16 10307640 116.2 ns/op 0 B/op 0 allocs/op (8-hop 2-byte path, 2,000 repeaters) BenchmarkLastHeardMap-16 1902 0.77 ms/op (5,000 nodes; was 3.7 ms scanning byNode) ``` Memory: one string per node ever certainly heard, dropped together with the node's `byNode` list on eviction. ## Tests - `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds `certainHeard`; each of the 8 call sites verified red with a nil prefix map. - `certain_heard_test.go`: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`, detail and health agree. - Regression for #2260: last certain packet on day X, 20 shared-prefix hops after it → `LastHeardMap` stays at X and `node.offline` goes bad after 72 h. Verified red with the uniqueness check mutated out. - Notify: `TestEvaluateCertainHeard` replaces the relay-input test. - Frontend: `last_relayed` alone no longer keeps a repeater or room active; newest timestamp wins (an old `_liveSeen` can't hide a fresh certain relay). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork with 457 extra commits. ## Validation Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose `last_relayed` is 3 min newer than anything certain keeps its certain time. **No browser screenshot**: Playwright isn't installed on this machine; CI's E2E covers the pages. ## Not changed - The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is. - The `lastHeard=` list filter still filters `last_seen` in SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
9529bf9d41 |
perf(store): read resolved_path pubkeys in place, one copy each (#2197)
Refs #1870. ## Problem `Load`, `loadChunk` and `scanAndMergeChunk` decode each observation's `resolved_path` only to collect its non-empty pubkeys. On live data that's a JSON array of pubkeys and nulls, about 420 characters, present on most observations. Decoding into `[]*string` allocated a string and a pointer for every hop, plus two slices. ## Change - `resolvedPubkeysFromJSON` reads the usual array of escape-free strings and nulls in place. Anything else falls back to decoding, so it returns exactly `extractResolvedPubkeys(unmarshalResolvedPath(s))`. - Relay pubkeys repeat across nearly every observation, so each one goes through `internStr` (from #2183) and is kept once. A pubkey seen before is found by substring, without copying, and the indexes it feeds (`byNode` and friends) share that copy. Interned keys live for the process, evicted nodes' included; the table is bounded at 64k entries. - The three bulk-load call sites use it. - **Eviction too.** `resolvedPubkeysForEvictionBatch` runs on every eviction batch, about once a minute in steady state, and spent about 130 MB per five minutes on a busy week. It now reads each pubkey in place and de-duplicates each packet's short list by scanning it instead of building a map per packet. - Other callers of `unmarshalResolvedPath` are unchanged. ## Measured | | before | after | |---|---|---| | allocations for 3 known pubkeys | 13 | 1 | | eviction, allocations per evicted packet (3 observations × 4 hops) | 51 | 15 | | this path's allocations over startup (copy of a busy week) | 236 MB | 61 MB | | server CPU over the run | 175.6 s | 170.6 s | The startup rows need that dataset and are from a single run. The benchmarks are rerunnable (`go test -run X -bench ResolvedPubkeys -benchmem ./cmd/server`, medians of 3): | | master | this PR | |---|---|---| | `BenchmarkResolvedPubkeysFromJSON` (5 relays + 3 nulls, pool of 2,000) | n/a | 619 ns, 128 B, 1 alloc | | `BenchmarkResolvedPubkeysForEvictionBatch` (1,000 packets × 3 observations) | 14.3 ms, 3.96 MB, 65,495 allocs | 6.7 ms, 1.36 MB, 18,501 allocs | ## Tests - `TestResolvedPubkeysFromJSONAllocations`: red commit first, with the bound tightened in green. - `TestResolvedPubkeysForEvictionBatch`: keys, order, and an allocation bound. It has its own red commit. - `TestResolvedPubkeysFromJSONMatchesDecoding` compares against decode-then-extract on 26 edge cases and 20,000 random inputs. - `go test ./cmd/server` passes. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
927a8eec8a |
fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211)
Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: | Range | Availability | Packets/day | |---|---|---| | 7d | 88.7% | 91 | | 30d | 42.6% | 265 | | All | 3.5% | 22 | Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - **UTC fix.** The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: | Range | Before | After | |---|---|---| | 7d | 82.7% | 83.3%, not partial | | 30d | 19.3% | 83.3%, partial from 2026-10-07 | | All | 1.6% | 83.3%, partial from 2026-10-07 | In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
92dd172f6e |
perf(store): bulk loaders reuse their row scan targets (#2201)
## Problem
`Load`, `loadChunk` and `scanAndMergeChunk` declared their ~20
`sql.Null*` scan targets, and built the `Scan` argument slice, inside
the row loop. Each target's address goes into the `[]interface{}`, so
each one escaped and was heap-allocated again on every row, along with
the slice and its appends. On a week of a busy network that was the
biggest flat cost inside `loadChunk`, about 330 MB.
## Change
- The targets and the argument slice are declared once, before the loop.
- `Scan` overwrites every target on each row.
- Nothing else takes their addresses; the `null*` helpers copy their
argument.
## Measured
| | before | after |
|---|---|---|
| allocations per row, fixture (`Load` / `LoadChunked` / `loadChunk`),
on current master | 85.9 / 87.1 / 86.0 | 63.9 / 65.1 / 64.0 |
| allocated over startup (copy of a busy week) | 5.41 GB | 5.05 GB |
| server CPU over the run | 175.6 s | 172.4 s |
The per-row figures are from the test on current master. The startup
figures were measured before #2183 and #2185 landed; this change saves
the same ~22 allocations per row on top of them.
**Scale:** this is the cold-load hot path, once per observation row
(about 400k rows on that week). The change removes per-row work and adds
none; complexity is unchanged.
## Tests
- `TestBulkLoadersReuseScanTargets` sets an allocations-per-row bound
for each loader, between the before and after values. Red commit first,
then green.
- `go vet ./...` is clean and `go test ./...` passes in `cmd/server`;
`-race -count=2` passes on the loader and observer-string tests.
- Local SonarQube: quality gate passed, no issues on added lines.
- `map[string]interface{}` in `cmd/server`: 934 on master and here.
No config or UI changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Alex BandClaude Opus 5.5
|
e53c055d6e |
perf(store): share one copy of each observer string across observations (#2183)
Part of #1870: this interns the three fields the issue names (`ObserverID`, `ObserverName`, `ObserverIATA`), plus `Direction`. ## Problem Each `StoreObs` kept the row scan's own copy of its observer ID (64 hex characters for most observers), observer name, IATA code and direction. A busy regional network's 7-day window holds about 400k observations from a few hundred observers, so a few hundred distinct values were stored hundreds of thousands of times. ## Change - `internStr` returns the canonical copy of each value from a map guarded by an `RWMutex`, so every observation shares one copy. - The copies are held for the life of the process. That is what makes the sharing survive garbage collections. An earlier version of this PR used `unique.Make`, which holds its copies weakly, so a value interned again after a GC came back as a second copy (thanks to the review for catching that). - Observer strings number in the hundreds. The map is capped at 65,536 entries as a guard (AGENTS.md: no unbounded structures); past the cap a string is returned as it came, so nothing breaks. - A hit takes only the read lock and allocates nothing. A new value is cloned, so it doesn't pin a larger buffer it may be a slice of. - All five places that build a `StoreObs` use it: `Load`, `loadChunk`, `scanAndMergeChunk`, and both ingest paths. - The ingest paths intern while scanning rows, **before** taking `s.mu`, so interning adds nothing to the time API readers wait on the store. - `cmd/server/go.mod` stays at `go 1.22`. The earlier 1.27 bump was only there for `unique`, and is reverted here; the module upgrade belongs in its own PR. ## Scaling `BenchmarkInternStr` (a hit, the steady state): | | alone | 12 cores (per call) | allocs | |---|---|---|---| | plain `Mutex` map (the first version of this PR) | 20 ns | 57 ns | 0 | | `unique` (the second version, withdrawn) | 48 ns | 8 ns | 0 | | **`RWMutex` map (this PR)** | 17 ns | 26 ns | 0 | ## Measured on a copy of that window (`GOMEMLIMIT=750MiB`) The same database and harness, one run each, the `unique` version against this one: | | `unique` version | this PR | |---|---|---| | live heap at the last GC after load | 581 MB | **535 MB** | | string bytes allocated by row scans still live (`bytealg.MakeNoZero`) | 106 MB | **80 MB** | | peak RSS | 777 MB | 781 MB | | load to ready | 15 s | 15 s | Peak RSS is set by the load itself, not by these strings. Against master, the first measurement of this PR found the strings retained from row scans went from 84 MB to 58 MB. An earlier version of this PR (the `unique` one) also ran on a staging instance (live MQTT ingest, `GOMEMLIMIT=1GiB`) through five back-to-back restarts and a two-hour soak, with no OOM kills and every request succeeding. This version has not had a staging soak yet. ## Tests - `observer_strings_intern_test.go`: - one copy of each value across 50 observations, through `Load`, `LoadChunked` and `loadChunk`; - **a value interned again after `runtime.GC()` is the same copy** (new: red against the `unique` version, green now); - 16 goroutines interning concurrently get the same copy; - a hit allocates nothing; - `BenchmarkInternStr` (serial and parallel). - Red commits first, then green. - `cd cmd/server && go test ./...` passes at `go 1.22`, and `-race` passes on the intern tests. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
8d39a9d465 |
perf(store): count subpaths per request instead of indexing every run (#2182)
## Problem `spTxIndex` kept every 2–8 hop run of every stored path as a map key, with one transmission pointer per occurrence. On a busy regional network's 7-day window (59k transmissions) that's about 0.9–1.07 M keys and 130–160 MB of live heap, roughly a quarter of the store. It was also rebuilt at the end of every startup, which is when memory peaks. Two smaller issues came with it: - Eviction never decremented `spTotalPaths`, so the percentages were divided by paths that were no longer stored. - Route Patterns took 1.2–3.6 s uncached, because the fast path still split, resolved and re-joined about a million raw keys per request. ## Change - **Count on request.** Subpaths are counted when the analytics are asked for, from the stored paths. Results stay cached for `rfCacheTTL`, as before. - **Pointer-free tallies.** `tallyNamedSubpaths` counts runs in one pointer-free `map[uint64]subpathTally`, keyed by a `maphash` of the resolved name built in a reused buffer. The GC doesn't scan the map, and counting allocates nothing but the map. - **Names only for the top entries.** Each tally records the first transmission and offset where its run appeared, so names and raw hops are built only for each group's top entries. `topSubpathTallies` picks those with a `container/heap` min-heap of `limit` entries instead of sorting every name, and the existing `rankSubpaths` formats the response, so no new response maps are added. - **Hash collisions.** If two names shared a hash their counts would merge; the odds are about 1 in 10⁷ for a few million names. - **Subpath detail.** `GetSubpathDetail` finds the run with `txsWithSubpath`, a scan of the stored paths. - **Counts unchanged:** runs of 2–8 hops, counted once per occurrence, case-folded. The tests pin this. - **Totals.** `spTotalPaths` stays, and eviction now decrements it. - **Memory estimate.** The O(path²) subpath term is dropped from the estimate. ## Perf justification (AGENTS.md rule 0) - **Complexity.** A cold Route Patterns request is O(Σ runs), about 1.5 M for this window: one pass over the stored paths, with one map lookup per run against a reused buffer. Ranking is O(tallies · log limit). Results are cached for `rfCacheTTL`, so this runs at most once per TTL per group set. - **Subpath detail** is O(stored hops), one pass with no allocation per packet. It's an on-click endpoint, not a hot path. - **Ingest and eviction** no longer touch a subpath index at all. Removal used to be a linear search of every key's transmission list. **Measured on a copy of that window**, upstream master vs this branch, same queries: | | master | this PR | |---|---|---| | peak RSS (`GOMEMLIMIT=750MiB`) | 948 MB | 767 MB | | in-use heap after load (750MiB) | 592 MB | 456 MB | | `/api/analytics/subpaths` cold (750MiB) | 3.56 s | 0.64 s | | `/api/analytics/subpaths-bulk` cold (750MiB) | 3.33 s | 0.70 s | | `/api/analytics/subpaths` cold (1GiB) | 1.18 s | 0.61 s | | `/api/analytics/subpath-detail` | 0.6 ms | 8 ms | | cached responses | 0.6 ms | 0.5 ms | At 750 MiB, master's request time is mostly GC pressure from the index. On the production-shaped fixture that `subpath_index_memory_test.go` builds (41k transmissions): | | master | this PR | |---|---|---| | retained by the store for subpaths | 197 MB | 0 MB | | allocated per bulk request | 336 MB | 111 MB | This has also run on a staging instance with live MQTT ingest for about nine hours, with Route Patterns answering in about 1 s. ## Tests - `subpaths_on_demand_test.go`: - per-occurrence, case-folded counts; - the 8-hop cap; - bulk matches single; - detail counts, including repeats, case and the length limits; - eviction drops evicted paths from the totals. - `TestSubpathIndexRetainedHeap`: the bound is now 5 MB (it was 225 MB). - `TestSubpathRequestAllocations`: 150 MB per request. - Each change has its red commit first, failing on assertions. - Tests that reached into `spTxIndex` now assert through `txsWithSubpath` and the public counts. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. **Overlap with #2181:** both edit the same memory-estimate comment block in `store.go`. Whichever lands second needs a trivial rebase there; there's no code dependency. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
dc7ebcd6f5 |
perf(store): check observation duplicates without per-transmission maps (#2181)
## Problem Every `StoreTx` carried two maps for observation de-duplication: - `obsKeys`, keyed by `observerID + "|" + pathJSON` (a concatenated string allocated for every observation); - `observerSet`, for `UniqueObserverCount`. On a busy regional network's 7-day window (64k transmissions, 400k observations), these maps and their concatenated keys held about 60 MB of heap. Most transmissions are heard a handful of times; the busiest in that week was heard 40 times. The maps cost a few hundred bytes per transmission to de-duplicate lists that are usually under ten long. ## Change - `hasObservation` and `hasObserver` scan `tx.Observations`, comparing the observer ID and path strings the observations already hold. These are the same two values the old map key was built from, so the answers are identical. - Once a transmission passes 64 observations (`obsIndexThreshold`), it builds the maps. They're keyed by a struct that shares the observations' strings, so no key bytes are allocated, and busy transmissions keep O(1) checks (the case #543 / #355 addressed). - The five places that appended an observation now call `addObservation`, which keeps `UniqueObserverCount` and `ObservationCount` in step. Those three lines were previously copied at each site. - The memory estimate no longer charges for the per-transmission maps or the concatenated keys. ## Perf justification (AGENTS.md rule 0) - **Complexity.** - Below the threshold, a duplicate check is O(k) for a transmission with k < 64 observations, so building it costs at most 63·64/2 = 2016 string-header comparisons. - At and above 64 observations, checks are O(1) map lookups, as before. - Nothing is O(n²) in the number of transmissions or observations overall: the scan is bounded per transmission. - **Scale.** Real data peaks at 40 observations per transmission. The scan compares string headers (length first, then bytes), and 64-character observer IDs differ early. `BenchmarkObsDedup` (a miss, the common case on ingest): | observations on tx | check used | time | |---|---|---| | 4 | scan | 15 ns | | 16 | scan | 70 ns | | 40 | scan | 151 ns | | 63 | scan | 233 ns | | 64+ | map (struct key) | 15 ns | For comparison, the old map lookup was about 15 ns too, plus building and allocating the `observerID|path` key string on every check. At 40 observations that's about 3 µs extra to build a whole transmission, against roughly 2 KB of maps no longer allocated for it. **Measured on a copy of that 7-day window** (`GOMEMLIMIT=750MiB`, same queries): | | before | after | |---|---|---| | in-use heap after load | 612 MB | 568 MB | | HeapAlloc after load | 660 MB | 564 MB | | peak RSS | 1077 MB | 1009 MB | | startup to ready | 31 s | 21 s | It has also run on a staging instance (live MQTT ingest) for about nine hours alongside other store changes, with no change in observation or observer counts. ## Tests - `obs_dedup_test.go`: - duplicates (same observer and path) are rejected, while a new path or a new observer is kept; - a transmission under the threshold carries no maps; - across the threshold the maps take over with the same answers; - empty observer IDs aren't counted as observers; - `BenchmarkObsDedup`. - Red commit first (stubs, failing on assertions), then green. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
b9fb3d244c |
feat(analytics): split adverts using recorded route evidence (#2088)
Red commits: `dc7dcbe`, `76e3217` ([assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37059977385)). Fixes #2041; supplies #2085's backend contract. Records at most two route bits per retained advert. Node flood counts now use the same evidence, including mixed and transport-flood adverts. Analytics failures log/count errors while core observation, path and liveness writes continue. The unrelated recent-advert limit clamp is removed. Direct adverts (empty path) describes the observed remaining path across valid hash widths. Firmware can forward an advert down to an empty path; this cannot establish original send mode or RF distance. The `zero_hop` API spelling remains compatible. Successful post-upgrade evidence writes survive observation replacement and restart. Older overwritten frames and failed writes can leave gaps. The UI explains automatic background backfill. ## Measured validation Synthetic Windows workloads; timings are not production guarantees: - 300K retained adverts, one-hour window: median 64.35→39.84 ms; 13.98 MB→145 KB allocated. Seven-day allocations unchanged; five paired samples. - 1M transmissions/11M observations: backfill 5m43s. Simple 10 Hz writer: all 3,423 observations persisted; p99 28 ms, maximum 323 ms. - Simulated pre-backfill cursor: retained evidence ready 4m21s; full catch-up 6m41s, 122 cache invalidations. Normal completed-migration startup bulk-loads current evidence. Portable opt-in scale tests and window benchmarks are included. All 183 frontend suites, full server tests, targeted races, lint and real-API desktop/mobile checks passed. Full [Linux CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37063210229) passed at `36def9c`; three independent reviews are clear. Local Windows testing hit the unchanged symlink-privilege test. No configuration changes. Preflight overrides: external script/OpenClaw profile unavailable; repository checks and Chromium used. |
||
|
Alex B
|
e6323ec587 |
fix(store): account path, decode-cache and dedup-key bytes so maxMemoryMB eviction triggers (#2035)
trackedBytes undercounted the packet store by about 2.5x, so packetStore.maxMemoryMB never triggered: a tx was charged at creation, before pickBestObservation set its path, so the byPathHop and spTxIndex costs were never added, and eviction then re-estimated with the path known and subtracted more than had been added, drifting the total downwards. The ParsedDecoded cache, the obsKeys dedup key and several per-observation strings were not estimated at all. StoreTx.accountedBytes now records what was charged, rechargeTx returns the delta after every pickBestObservation, and eviction subtracts accountedBytes instead of re-estimating. Measured by the author on a production database copy: trackedMB 151 against 402 MB of heap in use before, 351 against 396 MB after, with GC cycles dropping from ~0.71/s to ~0.011/s over 12 h on their instance. Reviewed by auditing the accounting rather than the arithmetic: all six production pickBestObservation sites recharge, all three subtraction sites read accountedBytes, every recharge site holds s.mu (Load from :857, the two ingest paths at :2778 and :3142 with deferred unlocks), observations are charged only after acceptance, and no charged tx is discarded during the chunk merge. That lock audit is the independent check, because CI's race job covers the ingestor only. Operator impact, both from the estimate growing rather than any limit moving: where maxMemoryMB is set, eviction now caps the real store size, and the cold load clamp drops about a third of the boot walk (124420 to 84374 packets at 650 MB). Where it is unset, which is the default, the change is inert. docs/go-migration.md claimed the Go server ignored the setting, which was never true, and is corrected here. Merged by the interim maintainer without a second human reviewer: CI (run 35258839322) plus the review above are the independent checks. |
||
|
efitenandClaude Opus 5
|
2d4019f719 |
fix(analytics): recompute once the store has fully loaded (#2025)
Refs #2023, #1659, #1724 ### Problem `main.go:258` waits only for the first load chunk, then `main.go:402` starts the analytics recomputers. `Start()` computes immediately on that chunk (`analytics_recomputer.go:86` on master) and the next compute waits a full interval (`:93`, 5 min default). The chunk loader walks by ascending id, so that chunk holds the oldest transmissions. - RF, topology, channels: the #1659 gate checked `LoadComplete()` after the compute (`analytics_warmup_1659.go:122`). `LoadComplete` flips at the end of the hot window (`chunked_load.go:489`), before the background fill (`store.go:1455`), so the gate could open on a snapshot without the background fill, and the 60 s force timeout (`:73`, `:162`) opened it on the first-chunk snapshot. In an end-to-end test on master, `/api/analytics/rf` returned 200 with 8 of 100 packets before the background fill ran. - Distance, hash-collisions, hash-sizes, roles, observers-clock-skew, nodes-clock-skew: no gate, partial snapshot served from the start. - Distance additionally served a snapshot from the previous index for up to one interval after each lazy index build. On a staging instance, a default analytics request returned 5,911 packets with hours-old last buckets until the next recompute (about 74k). ### Change - `StartupLoadDone()` (`chunked_load.go:108`): closed when `RunStartupLoad` returns, on every path (`chunked_load.go:202`). Closing it drops the hash-size info cache (15 s TTL) and the clock-skew engine throttle (30 s, `clock_skew.go:225`), both read by the post-load computes. - `recomputeWhenLoaded` (`analytics_recomputer.go:172`): on that signal, recompute each recomputer once, sequentially, via `RecomputeNow` (`:154`), which runs on the recomputer's own loop and restarts its ticker (`:106`). Order (`:255`): rf, topology, channels, distance, hash-collisions, hash-sizes, observers-clock-skew, nodes-clock-skew, roles (roles reads the nodes-clock-skew snapshot). Logs one line with per-recomputer durations. - Warm-up gate: now the same signal (`:343-354`), sampled before the compute starts (`:129`), so a pass that began on partial data never opens it. 503 + `Retry-After: 5` and the force timeout are unchanged; a forced-open snapshot is replaced by the post-load recompute. - Ungated endpoints: no new 503s (their API has none); snapshot replaced right after the load. - Distance: the lazy index build refreshes the distance recomputer before reporting built (`store.go:4476`). - Recompute intervals and config unchanged. ### Performance One extra compute per recomputer per process start, run sequentially so they do not all hold the store read lock at once. Ticker phases afterwards are offset by the cumulative post-load compute durations instead of all starting within the first-chunk compute window (relevant to #1724; the effect on lock waves is not measured). ### Tests `analytics_recompute_after_load_test.go`: signal open during background fill, closed after success and failure; cache drops; immediate and ordered post-load recompute; gate not opened by a pass started before the load; forced-open snapshot replaced on load; ticker restart; distance refresh before 202 ends; end to end with recomputers started before the background fill (RF 503 until load, then `totalTransmissions` equals the full store; six ungated endpoints 200 during load; all nine recomputed after load). 9 of these failed on master with stubs; 8 single-line mutations each caught. `go test ./...` in `cmd/server` passes. ### Staging validation Deployed together with the review follow-ups of #2015-#2023 (build `c646310f`), container restart: ``` 16:35:20 [store] first chunk ready (chunkSize=10000) 16:35:25 [store] LoadChunked complete ... starting background fill loader 16:36:58 [store] background load complete: 121120/121282 packets in memory (coverage=99.9%) 16:37:03 [analytics-recompute] startup load done: recomputed 10 snapshots in 5.155s (rf=955ms topology=1.684s channels=43ms distance=49ms hash-collisions=30ms hash-sizes=338ms observers-clock-skew=369ms nodes-clock-skew=692ms roles=2ms retransmissions=994ms) ``` Right after that line, `/api/analytics/rf` reported `totalTransmissions` 121,121 against 120,700 packets in memory, and the retransmissions default shape from #2023 covered the full 7 days. Before this change both waited for the next 5 minute tick. ### Merge order with #2023 #2023 adds a tenth recomputer. Whichever of the two merges second has to add `recompRetransmissions` to `analyticsRecomputersLocked`, wire it to `loadedGate` instead of `LoadComplete`, and change 9 to 10 in `TestAnalyticsRecomputers_PostLoadOrder`; the retransmissions gate test then calls `signalStartupLoadDone()` instead of setting `loadComplete`. That resolution is what ran on staging above. ### Not verified - Repeater-enrich recomputer and the region/window TTL caches (hash-collisions region results have a 1 h TTL) may also keep partial results after the load; not changed here. - Recompute order is tested structurally, not with roles/clock-skew data. - Whether this reduces the #1724 stalls; not measured. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
376c3e9f4a |
fix(packets): surface the transport region scope — detail pane row and a sortable Scope column (#1894)
## Summary `transmissions.scope_name` (#899) reached the database but never reached the UI. Two problems, one dead feature and one missing surface. ## 1. The detail pane's Scope row was dead `public/packets.js:3279` has rendered a **Scope** row since #899, gated on `pkt.scope_name != null`. It never fires in practice. `/api/packets` and `/api/packets/{id}` are served from the in-memory `PacketStore`. The store reads `scope_name` out of SQLite fine (`store.go:888`, `chunked_load.go:551` → `StoreTx.ScopeName`), but `txToMap()` did not put it in the JSON. Only packets old enough to have been evicted from the store — and thus served by the SQLite fallback in `db.go`, which does emit it — could ever show a scope. Verified against a live instance before the fix: ``` GET /api/packets/552e9687f1525537 → packet keys: ['_parsedPath','decoded_json','direction','first_seen','hash','id', 'observation_count','observations','observer_iata','observer_id', 'observer_name','path_json','payload_type','raw_hex','route_type','rssi','snr','timestamp'] ``` No `scope_name`. ### The NULL / "" distinction `StoreTx.ScopeName` was typed `string`, which collapses the two states the frontend distinguishes: | DB value | Meaning | UI | |---|---|---| | `NULL` | not transport-scoped | row hidden | | `""` | transport-scoped, region matched no configured key | muted "unknown scope" | | `"#be"` | matched region | the region name | `route_type` is **not** a usable proxy for that distinction: the ingestor writes NULL for a transport route whose `transport_code_1` is `0000` (`cmd/ingestor/db.go:1576` — `IsTransportScoped = route_type IN (0,3) AND Code1 ≠ "0000"`). So the field is now `*string`, with `nullStrPtr` preserving what `nullStrVal` collapsed. The two internal consumers (`TransportedScopes` #1751, `relayEntry.scope`) only care about non-empty named scopes and are unchanged in behaviour. ## 2. New: a Scope column on the packets table The scope was only reachable one packet at a time by opening the detail pane. It now has its own sortable column between Type and Observer, visible by default. The default view is **Group by Hash**, served by mappers that did not carry `scope_name` at all — so the column would have been empty in exactly the view most people look at. Both grouped paths now select and emit it: `groupedTxsToPage` in the store, and the dedicated grouped query in the DB fallback (v3 and legacy shapes). Rendering lives in `scopeCellHtml` (`public/app.js`, next to `transportBadge`) and is used on all three row-render sites — group header, expanded children, flat rows — so the column and the detail pane cannot drift apart. **Sorting** pins the empties last in both directions, as the nodes table already does for `default_scope`. Only ~8% of packets carry a scope, so an ascending sort would otherwise bury every scoped row under a wall of dashes. **Filtering**: `packet-filter.js` gains a `scope` field, so the cell is click-to-filter like Type and Observer, and `scope == "#be"` works in the filter bar. **Column prefs**: a `packets-known-cols` companion key. The `packets-visible-cols` array alone cannot distinguish "this column did not exist when you saved" from "you unchecked it", so any new column arrives silently hidden for every returning visitor. Keys absent from `known-cols` get the default treatment; keys the visitor actually hid stay hidden — there is a test for that second half specifically. ## Tests Each watched fail first. **Go** (`cmd/server/packet_scope_name_test.go`) - `txToMap` unit tests for all three states, including a JSON round-trip so a typed nil `*string` cannot pass as `null` - end-to-end through `/api/packets/{hash}` - `groupedTxsToPage` unit + end-to-end through `/api/packets?groupByHash=true`, across **both** the store-backed and DB-fallback paths - `transported_scopes_1751_test.go`: the "no scope" guard now covers both non-values (nil and a pointer to `""`) **Frontend** - `test-frontend-helpers.js`: `scopeCellHtml` three states + escaping - `test-packet-filter.js`: `scope` matching, case-insensitivity, and `FIELDS` registration - `test-packets-scope-column.js` (new Playwright e2e): header position, default visibility, one cell per row, em dash on non-transport rows, empties-last sorting, the Columns toggle, and the prefs backfill ## Verification Deployed and checked against a live instance: ``` /api/packets?groupByHash=true&limit=500 → scope_name present on 500/500, 59 with a matched region, 1 unknown-scope test-packets-scope-column.js → 7 passed, 0 failed cd cmd/server && go test ./... → ok ``` Two pre-existing failures, unrelated and equally red on an unmodified checkout: `test-e2e-playwright.js` "Customizer open does not overwrite server home config" and `test-observer-iata-1188-e2e.js` (timeout on `[data-loaded="true"]`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
fa15ab0a30 |
fix(#1809): gate background loader on LoadChunked completion (#1811)
Partial fix for #1809. Red commit: |
||
|
Michael J. Arcan
|
ae2e3933dd |
feat(server): store memory diagnostics + drop redundant obs.RawHex (#1773)
Drops the redundant per-observation RawHex (~98MB on a live store; reader already falls back to tx.RawHex #881) and adds an opt-in /api/perf?mem=1 memory breakdown (flood-forward share + per-component bytes). Profiled against a live instance. **Savings substantiation:** live-instance profiling shows ~1.66M observations in the store, each previously carrying its own per-observation `raw_hex` (avg ≈118 hex chars ≈59 bytes) that exactly duplicates the parent transmission's `raw_hex`. Dropping the duplicate on every load/ingest path eliminates ≈98 MB of redundant in-memory storage plus ~1.66M string allocations, with no data loss — the read path (`enrichObs`) already falls back to `tx.RawHex` when `obs.RawHex` is empty (verified by the new safety-gate test). The patched build cannot be run against the live instance here; instead the new opt-in `/api/perf?mem=1` diagnostic lets operators measure the real before/after (`trackedMB` and the per-component breakdown) directly after deploy. |
||
|
Michael J. Arcan
|
fc26fb6b3a |
feat(#1751): show transported region scopes in repeater sidebar (#1752)
Closes #1751. |
||
|
|
048143f54f |
fix(#1690): cold-load uses last_seen (effective recency) instead of first_seen (#1691)
## #1690 — cold-load uses wrong time axis (RED → GREEN) The on-disk DB has thousands of long-lived hashes with recent traffic. Prod's cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a column that is set once at insert time and never updated — so re-observation of an old hash does not move it into the hot window. Result: prod cold-loaded ~0.3% of the on-disk rows and flipped `backgroundLoadComplete=true` without ever walking the retention window (the `retentionHours - hotStartupHours <= 0` short-circuit at line 1353 of `cmd/server/store.go`). ### Three sub-fixes **A) Denormalize `transmissions.last_seen`** so cold-load can window on effective recency. - `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn` adds the column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both PREFLIGHT-annotated as cheap metadata-only ops). - `cmd/ingestor/db.go::OpenStoreWithInterval` schedules `tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` — `UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod). - Writer-side: `InsertTransmission` seeds `last_seen` on initial insert, and every observation insert bumps `last_seen = ?` via prepared statement `stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest never goes backwards). - Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and `cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY clauses to `t.last_seen` when the column is present (PRAGMA-detected via `DB.hasLastSeen`). Test/legacy DBs without the column fall back to `first_seen` so existing fixtures stay green. **B) Honest `backgroundLoadComplete` gating.** - Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod runs with both at 12h, which flipped Done=true immediately. - After the chunk loop, query `SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor` and compute `loadCoverageRatio = inMem / inDB`. Done=true only when `ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` + `backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000 rows (1000 in memory)"`). - `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the perf endpoint can read them without blocking the writer. **C) Perf exposure.** `PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`, `LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of the on-disk DB the in-memory store currently reflects, so operators can see the 0.3% case in `/api/perf` without reading the logs. ### TDD trail - **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests + 21-line store.go scaffolding. CI on this commit failed on assertions (intended). - **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite: `Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`, `Test1690_PerfStats_NewFields`, `TestHotStartup_*`, `TestIssue1690_LastSeenUpdatedOnObservation` — all pass. Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()` guard around `backgroundLoadDone.Store(true)` — `Test1690_BackgroundLoadHonesty` fails on the assertion `"backgroundLoadDone=true with only 1000/5000 packets loaded; must be false until coverage ≥ 90%"`. Restored. ### Async-migration preflight - `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both `// PREFLIGHT: async=true reason="..."` annotated. - `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`. - `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level UPDATE BY PRIMARY KEY, not a migration. ### Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=5K - check-async-migration: justified for `cmd/server/issue1690_cold_load_test.go` CREATE TABLE/INDEX statements — these build an in-memory test fixture DB (≤5000 rows, runs in <1s in CI), not a prod migration. Fixes #1690. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: bot <bot@example.com> |
||
|
|
938153dd92 |
fix(nodes): rebuild relay-hop history on startup from path_json (#1643)
## Problem A relay node's **activity timeline** — and its per-node `packetsToday` / observer counts — collapses to *"only the hour the server restarted"* after every restart. Before the restart the timeline shows only the node's own adverts (~1–2/hr); all of its relay activity piles into the single post-restart hour. ## Root cause All DB cold-load paths (`Load`, `loadChunk`, `scanAndMergeChunk`) index relay-hop attribution into `byNode` **only** from `observations.resolved_path`. But since #1287 the ingestor persists relay data as aggregate `neighbor_edges` and **never writes `resolved_path`** — it is `NULL` on every deployment (verified on a live DB: 0 of ~440k rows populated). So relay attribution is never reconstructed on startup; it only re-accumulates from live traffic (`IngestNew*`, which re-resolves from `path_json` + the neighbor graph), piling a relay node's whole history into the post-restart window. ## Fix Server read-side only — **no schema / ingestor / migration change**. When `resolved_path` is empty, re-resolve relay hops from the already-persisted `path_json` using the in-memory prefix map + neighbor graph (the same `resolvePathForObs` compute the live ingest path already runs). `main.go` now loads the persisted neighbor graph *before* the packet load so resolution has the graph available. Two correctness details worth a close look: 1. **Fetch the prefix-map/graph snapshot BEFORE opening each load cursor.** `getCachedNodesAndPM` issues its own DB query; doing so while a load cursor is open deadlocks on a single-connection SQLite pool (the test harness uses one). 2. **Index into `byNode` ONLY** — not the `resolved_path` / path-hop indexes. Those are cross-checked by `handleNodePaths` against the persisted `resolved_path` column (NULL here); populating them from an in-memory re-resolution would make that SQL confirmation fail and wrongly drop the tx from paths-through (#1352). ## Tests New coverage asserts a relay pubkey reachable *only* via `path_json` lands in `byNode` after a restart-style load, for both the hot-window (`LoadChunked`) and background-window (`loadChunk`) paths. Existing #1558 (`resolved_path`) and #1352 (paths-through) tests still pass. Full `cd cmd/server && go test ./...` is green under `-race`. ## Perf The fallback runs `resolvePathForObs` per observation with a non-empty `path_json` during cold load — the same per-packet compute the live ingest path already performs, so no new asymptotic cost. The prefix map + graph are snapshotted **once per load** (not per row); `getCachedNodesAndPM` is 30s-cached. In `loadChunk` the resolution runs in the existing lock-free scan and is accumulated locally, matching that function's "build local, merge under lock" design. ## Note on a pre-existing flaky test `TestDistanceConcurrentRequestsDuringBuildReturn202` is timing-fragile (fails ~1/15 on `master` without this change). It relies on the lazy distance build being slow because it's the first caller of `getCachedNodesAndPM` (cold cache). This PR pre-warms that cache during `Load`, narrowing the build window, so the test fails more often in **non-race** local runs. It passes reliably under `-race` (CI mode), where the build stays slow. Flagging in case you want to harden the test separately. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: openclaw-bot <openclaw-bot@users.noreply.github.com> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
|
bc1822e46c |
perf(load): chunked Load with early HTTP readiness (#1009) (#1596)
## What Switches the server's startup from a synchronous full-scan `PacketStore.Load()` to a chunked `LoadChunked(chunkSize)` that: 1. Streams transmissions+observations from SQLite in id-ordered chunks (default `chunkSize=10000`, configurable via `db.load.chunkSize`). 2. Closes `FirstChunkReady()` after the first chunk is merged — `main.go` binds the HTTP listener on that signal instead of blocking on the full multi-minute load. 3. Stamps `X-CoreScope-Load-Status: loading; progress=<rows>` on every response while LoadChunked is in flight, flipping to `ready` once it completes (via `loadStatusMiddleware`). 4. Preserves the existing retention/`hotStartupHours`/`maxMemoryMB` clamps and the post-load index rebuild (`pickBestObservation` / `buildSubpathIndex` / `buildPathHopIndex` / `buildDistanceIndex`). ## Why Per #1009: at 5M+ observations (Cascadia scale) the synchronous Load blocked HTTP for ~80s with a 2–3× steady-state RAM peak. With chunked load the listener binds within seconds; dashboards and probes can read partial data and see the `loading` status header until the background load finishes. ## Notes - `/api/healthz` readiness gate (`readiness` atomic, init `WaitGroup`) is unchanged — it still waits for neighbor-graph build + initial `pickBestObservation` before reporting `ready:true`. `LoadChunked` only changes when the listener BINDS, not when it advertises ready. - `cmd/server/main.go` waits for `FirstChunkReady` (or the full load on a tiny DB) before proceeding, and drains the load goroutine in the background with a logged error path. - Config Documentation Rule: `config.example.json` now documents `db.load.chunkSize` with a nested `_comment` describing the trade-off. ## Tests - `cmd/server/chunked_load_test.go` asserts: - (a) `FirstChunkReady` fires before `LoadChunked` returns - (b) `X-CoreScope-Load-Status` transitions `loading; progress=...` → `ready` - (c) `chunkSize` honored (2500 rows @ 1000 → 3 chunks via `OnChunkLoaded`) - (d) `Config.DBLoadChunkSize()` default 10000 + override - Red commit (`102a4c84`) lands the tests with stubs that fail on assertion — verified locally before the green commit. - Green commit (`35cecf16`) makes all four pass; full `cmd/server` suite green (47s locally). Closes #1009 ## TDD red-commit exemption The original red commit `f878e15e` ("test(load): failing tests for chunked Load + early HTTP readiness") fails to **compile** rather than failing on an assertion, because it references symbols (`store.LoadChunked`, `store.FirstChunkReady`, `store.OnChunkLoaded`, `Config.DBLoadChunkSize`, `loadStatusMiddleware`) that do not exist on master. Per `AGENTS.md` the bar is "MUST fail on an assertion ... A compile error is NOT a valid red commit." This is claimed under the **net-new surface** exemption with the following justification: - LoadChunked / FirstChunkReady / loadStatusMiddleware / DBLoadChunkSize are all introduced by this PR — no prior implementation existed to refactor. There is no behaviour on master that the red commit could meaningfully assert against without first declaring the new symbols. - The cheapest "proper" alternative (split the red into two commits: stub-first + assertion-fail) was deferred because the test file unambiguously fails on missing-symbol — there is no risk of the test becoming a tautology against a pre-existing stub. - **Behaviour gating IS proven elsewhere on this branch.** Commit `799bde49` ("test(load): red — LoadChunked must mark indexes ready + not flip Complete on error") is a proper assertion-fail red against the same package, and commit `92cadd1d` is the matching green. Reviewers can verify the red→green pattern there. If a future reviewer wants the strict pattern, the follow-up is mechanical: split `f878e15e` into a stub-only commit followed by the assertion commit. Not done here to keep the rework cost proportional to the risk (zero, in this case). ## Preflight overrides - check-async-migrations: justified — the flagged `CREATE TABLE`/`CREATE INDEX` statements live in `cmd/server/chunked_load_id_zero_test.go` and `cmd/server/chunked_load_oldest_test.go` only. They run against per-test `t.TempDir()` SQLite files (in-process, ~10 rows, lifetime = single test) — they are NOT production schema migrations. No prod table is touched. PREFLIGHT-MIGRATION-SCALE: <30s N=10 (per-test tempdir fixture). --------- Co-authored-by: CoreScope Bot <bot@corescope.local> Co-authored-by: clawbot <bot@noreply.example.com> Co-authored-by: Kpa-clawbot <bot@example.com> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot> |