mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 12:18:47 +00:00
7c00a42947
Refs #2146 (reach item: "an indexed hop table would remove it"). Follows #2155. ## What changes Reach finds the observations that relayed through a node with `path_json LIKE '%"XX"%'` over every observation in the window. The cost follows the window, not the node: a quiet repeater costs about as much as a busy one. When the database does not fit in the page cache this outlives the 60 s `WriteTimeout`, the request 500s, and nothing is cached. #2146 reports this endpoint at 14.9 s average and 74.7 s max on a 10.9 GB instance. Its 1.9 s figure after #2155 is a warm-cache timing; the scan reads the whole window, so cold it is far slower (below), and it holds one of the 4 read-pool connections the whole time. This PR adds an inverted index and uses it only when it is complete for the requested window: - `observation_hops(hop, timestamp, observation_id)`, `WITHOUT ROWID`. A token lookup is a seek and the window bound is a range on the same b-tree. - Triggers on `observations` maintain it (insert, delete, `path_json` update), so prune and path-repair need no changes. - `observation_hops_coverage.since` marks how far back the index is complete. If the window starts before `since`, or the table is missing (older DB, backfill unfinished), reach runs the existing `LIKE` scan unchanged. - The ingestor owns the schema, a resumable newest-first backfill, and a daily trim to `retention.hopIndexDays` (default 14). The server stays read-only and `AssertReady` does not require the table. The backfill starts after the ingest gate opens rather than inside `OpenStore`. My first version did the latter and held live ingest in the buffer for about 15 minutes on a 13.7 GB database (nothing dropped). Chunks are 15 minutes because one-hour chunks held the writer for 2 to 7 s per commit on EBS. ## Results ### Real data: vanilla vs this branch One snapshot of a real instance (11.5 GB, 10.7M observations over 14 days) served by both builds on the same machine (8 cores, 30 GB RAM, NVMe, warm cache). One cold request per cell, seconds, repeaters from busiest to quietest by 24 h relay count. | node (relays / 24 h) | 7 d vanilla | 7 d branch | 14 d vanilla | 14 d branch | |---|---|---|---|---| | 7,314 | 24.5 | 4.5 | 8.3 | 9.6 | | 7,307 | 1.7 | 2.4 | 1.3 | 3.1 | | 7,047 | 3.3 | 2.2 | 2.0 | 2.3 | | 2,151 | 20.0 | 0.44 | 29.6 | 2.0 | | 673 | 23.7 | 0.18 | 28.5 | 0.62 | | 240 | 21.8 | 0.01 | 27.8 | 0.04 | | 52 | 21.6 | 0.00 | 28.4 | 0.00 | - Nodes with few relays drop from 20 to 30 s to under a second. - The busiest relays are about the same, sometimes slower: both builds spend their time loading tens of thousands of matching rows by primary key, which the index does not speed up. - Single runs, and page-cache state moves them a lot: an earlier run, with the file freshly cached, had vanilla's quiet nodes at 4.5 to 5 s for 7 d, close to #2155's warm figure. Read orders of magnitude, not exact figures. Nothing failed on this fast machine. **Same answers.** Unit tests check indexed rows equal `LIKE` rows, fallback on partial coverage, and trigger upkeep. On real data, 21 node/window pairs (3 to 14 days) requested from both builds at the same moment were all identical once links were compared by public key. `LIKE` ignores case and the index does not; 0 of 10.1M observations in the window have a lower-case hop. Above `reachScanRowLimit` both paths truncate, and they can keep different subsets. ### Production instance t4g.large (2 vCPU, 8 GB, EBS), 13.7 GB database, about 4.5M observations per 7 days, so the cache held a fraction of it. - Before: `days=7` on a busy node returned 500 (`context canceled`), and `days=1` took 58.5 s. - With the index (38 probes of four nodes over 18 hours): `days=7` on that node returned 200 in 23 s. The busiest probe node had a median of 0.7 s and p95 18 s. The hottest (127k relay observations in 7 d) had a median of 1.3 s, p95 52 s, and one probe past 60 s at evening peak. The quietest had a median of 0.04 s. - On 2026-10-04 the host was resized to 4 vCPU and 16 GB so the cache holds the database. Same index afterwards (493 probes): hottest node median 1.4 s, p95 1.7 s, max 38 s; quietest median 0.05 s. (From 2026-10-03 the instance also ran a local change, not in this PR, that lets reach requests continue in the background, which only affects requests over 45 s.) So on a small cold host the index turned "always fails" into "usually works" but did not make the hottest relays reliably fast. That needs cheaper row loading (a compact side table measured 4 to 10x on cold reads; kept out of this PR) or background computation. ### Benchmarks (included) ``` go test ./cmd/server -run xxx -bench ScanReachRows -benchtime 5x # 600k observations, 11 hops avg, 7 d window in 14 d, warm cache hot_token/like 816 ms rare_token/like 486 ms hot_token/indexed 293 ms rare_token/indexed 1.7 ms go test ./internal/dbschema -run xxx -bench ObservationInsert -benchtime 20000x # one observation per transaction, WriterDSN (WAL, synchronous=FULL), 11 hops without_index 23 us/op with_index 127 us/op ``` The triggers add about 100 us per observation, roughly 0.1% of a core at the production instance's 8 observations per second. They cap one writer at about 8k observations per second instead of 43k. The benchmark ran in a temp directory, so it shows CPU and b-tree cost, not disk sync cost. Over 13 days on the production instance, 0 observations were dropped and newest-observation age was p99 4 s. Four of 623 samples were 20 s or more: 91 s and 20 s came within six minutes of a container restart, 31 s came 24 hours after one (when the ingestor's daily tickers fire), and 23 s had no restart nearby. Index coverage stayed between 14.0 and 15.2 days. **Disk.** On the snapshot (`dbstat`) the index is 1.98 GB for 10.1M observations in the window, about 196 bytes each, next to 6.81 GB for `observations`. It scales with hops per observation and `hopIndexDays`. Backfill of 77M rows took 15.5 min on the production instance. **Perf justification (AGENTS.md).** Reach goes from a scan over the whole window to a seek over the matching rows. Everything new is bounded: the index by `hopIndexDays` and the daily trim, the backfill by chunk size and a cancellable context, and the server adds no in-memory state. The ingest hot path pays the trigger cost above. ## Design notes and open questions - Why not the in-memory store: it defaults to 7 days (`retentionHours: 168`) and reach offers up to 30, so it could not answer the longer windows. The other option listed in #2146, driving the query from `observations.timestamp`, is what #2155 already did. - SQLite only: the triggers use `json_each`. The optional PostgreSQL backend in #2145 would need its own equivalent. - No off switch: the table and triggers are always installed and `hopIndexDays` only sets the window. If you would rather have this opt-in, or want a smaller default for small disks, I'll change it. ## Operator notes - Upgrade: the ingestor installs the table and triggers on start and backfills. Until a window is covered, reach behaves as today. - Rollback: an older binary keeps indexing new rows with no trim. To remove it with the ingestor stopped: `DROP TRIGGER trg_observation_hops_ai; DROP TRIGGER trg_observation_hops_ad; DROP TRIGGER trg_observation_hops_au; DROP TABLE observation_hops; DROP TABLE observation_hops_coverage;` - New key `retention.hopIndexDays`, documented in `config.example.json`. ## Tests `go test ./...` passes in `cmd/server`, `cmd/ingestor` and `internal/dbschema`. New: `node_reach_hops_test.go`, `observation_hops_test.go`, `observation_hops_backfill_test.go`, and the two benchmarks. ## Update after review (commit `2291bea2`) The text above is what was reviewed at `95e17976`. Where they differ, this section is current: - **Opt-in.** `retention.hopIndexDays` unset or 0 (the default, and the value in `config.example.json`) installs nothing. A value above 0 installs the table and triggers and backfills that many days. This replaces "default 14", the "No off switch" note and the Upgrade note above. - **Changing the setting** applies at the next ingestor start: raising it backfills the gap, lowering it trims, and 0 removes the triggers and both tables. On a 10.5M-observation copy the removal took 45 to 71 s, at startup before MQTT connects. - **Case.** Hops are stored and matched upper-cased, so the index ignores ASCII case the way LIKE does. This replaces the sentence on case under "Same answers". - **Rollback.** Set it to 0 and start the ingestor once, then roll back. The manual `DROP` list above still works with the ingestor stopped. - **Disk.** A full 14-day backfill on that copy built 2.17 GB for 9.68M observations, about 224 bytes each, in 13 min 37 s. - Backfill and trim are serialized, and the chunk SQL lives in `dbschema` next to the triggers. ## Update after review, round 2 (commit `95690e8c`) Where this differs from the two sections above, this one is current: - **Startup order.** The index is installed or removed in the startup block, after the MQTT subscription is up and before the ingest gate opens, so packets arriving during a removal are buffered (`ingestBufferSize`), not lost. This replaces "at startup before MQTT connects" in the first update. - **Mark moves cover their range.** Each backfill and trim transaction indexes or deletes everything between the old mark and the new one, so a row arriving between the chunk-picking read and the transaction is accounted for either way. - **Negative `hopIndexDays`** is a config error at startup; the ingestor refuses to run. This replaces "−5 is off" in the first update. - **Server.** If the index disappears between the coverage check and the scan, the request is answered from the `LIKE` scan instead of a 500. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>