mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-10-13 09:17:16 +00:00
efitenandClaude Opus 5.5
3e21b179ec
Refs #2146 (item: `/api/nodes/:pubkey/reach`). ## Situation The reach scan joins `observations` to `transmissions` and filters on `o.timestamp >= ?` plus one `o.path_json LIKE '%"XX"%'` per token. With the planner statistics `ANALYZE` produces on a large database, SQLite runs it as `SCAN t`: every transmission, with an index probe into `observations` (`idx_observations_tx_ts`) per row. The cost is then the size of `transmissions`, whatever node is asked for and whatever the window. ## My change in this PR `scanReachRows` uses `CROSS JOIN` for the transmissions join. In SQLite that fixes the join order: `observations` is the outer loop, read through `idx_observations_timestamp` for the window, and each row looks up its transmission by primary key. Results are the same rows; it is still an inner join. The query text moves into `reachScanSQL(nTokens)`, so a test can inspect the plan. ## Measurements Read-only on a database with 18.3M observations and 1.5M transmissions (Python `sqlite3` 3.45.1 on the host), 7-day window: | node (rows returned) | `JOIN`, first run | `JOIN`, warm | `CROSS JOIN`, both runs | |---|---|---|---| | busiest relay (54,387) | 10.29 s | 3.27 s | 1.95 s | | quartile (3,913) | 3.16 s | 3.15 s | 1.81 s | | median (1,310) | 3.17 s | 3.18 s | 1.82 s | | a repeater reported as timing out in the browser (34,209) | 14.11 s | 3.25 s | 1.93 s | On that instance `/api/nodes/:pubkey/reach` reports p95 188.6 s and max 192.6 s over 34 calls in `/api/perf`; a single uncached call from the host took 3.4 to 3.6 s, so most of that is waiting for the 2 build slots and the read pool. On a staging instance (smaller database), with unique cache keys and `days` 6 or 7, one cycle per build after the ingestor's startup `ANALYZE` had finished: | | without this PR | with this PR | |---|---|---| | sequential, 12 calls, p50 | 3402-3801 ms | 2078-2467 ms | | 2 concurrent clients (= `reachMaxConcurrentBuilds`), p50 / p95 | 3575-4087 / 4049-5291 ms | 2271-2462 / 2814-2928 ms | | calls completed in 120 s | 58-68 | 96-101 | The staging builds also contain #2147. ## Verification - `TestReachScanStartsFromObservationsTimeIndex` creates the two production indexes, loads the `sqlite_stat1` values from the 18.3M-observation database, and checks that the plan starts with `SEARCH o USING INDEX idx_observations_timestamp`. With the old `JOIN` it fails with the production plan (`SCAN t`, then `SEARCH o USING INDEX idx_observations_tx_ts`). - The existing reach tests (`TestScanReachRows_DecodesRows`, `TestScanReachRows_CapTruncates`, attribution tests) pass unchanged. - `cmd/server`: `go test ./...` green. ## Not in this PR - The scan still reads every observation in the window and applies the `LIKE` per row, so its cost grows with the window and with traffic. Removing the scan needs an index on hop tokens (for example a table the ingestor maintains). I have not measured what that would cost in storage. - The build-slot limit (2) and the 5-minute cache are unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>