Refs #2146 (reach item: "an indexed hop table would remove it"). Follows
#2155.
## What changes
Reach finds the observations that relayed through a node with `path_json
LIKE '%"XX"%'` over every observation in the window. The cost follows
the window, not the node: a quiet repeater costs about as much as a busy
one. When the database does not fit in the page cache this outlives the
60 s `WriteTimeout`, the request 500s, and nothing is cached. #2146
reports this endpoint at 14.9 s average and 74.7 s max on a 10.9 GB
instance. Its 1.9 s figure after #2155 is a warm-cache timing; the scan
reads the whole window, so cold it is far slower (below), and it holds
one of the 4 read-pool connections the whole time.
This PR adds an inverted index and uses it only when it is complete for
the requested window:
- `observation_hops(hop, timestamp, observation_id)`, `WITHOUT ROWID`. A
token lookup is a seek and the window bound is a range on the same
b-tree.
- Triggers on `observations` maintain it (insert, delete, `path_json`
update), so prune and path-repair need no changes.
- `observation_hops_coverage.since` marks how far back the index is
complete. If the window starts before `since`, or the table is missing
(older DB, backfill unfinished), reach runs the existing `LIKE` scan
unchanged.
- The ingestor owns the schema, a resumable newest-first backfill, and a
daily trim to `retention.hopIndexDays` (default 14). The server stays
read-only and `AssertReady` does not require the table.
The backfill starts after the ingest gate opens rather than inside
`OpenStore`. My first version did the latter and held live ingest in the
buffer for about 15 minutes on a 13.7 GB database (nothing dropped).
Chunks are 15 minutes because one-hour chunks held the writer for 2 to 7
s per commit on EBS.
## Results
### Real data: vanilla vs this branch
One snapshot of a real instance (11.5 GB, 10.7M observations over 14
days) served by both builds on the same machine (8 cores, 30 GB RAM,
NVMe, warm cache). One cold request per cell, seconds, repeaters from
busiest to quietest by 24 h relay count.
| node (relays / 24 h) | 7 d vanilla | 7 d branch | 14 d vanilla | 14 d
branch |
|---|---|---|---|---|
| 7,314 | 24.5 | 4.5 | 8.3 | 9.6 |
| 7,307 | 1.7 | 2.4 | 1.3 | 3.1 |
| 7,047 | 3.3 | 2.2 | 2.0 | 2.3 |
| 2,151 | 20.0 | 0.44 | 29.6 | 2.0 |
| 673 | 23.7 | 0.18 | 28.5 | 0.62 |
| 240 | 21.8 | 0.01 | 27.8 | 0.04 |
| 52 | 21.6 | 0.00 | 28.4 | 0.00 |
- Nodes with few relays drop from 20 to 30 s to under a second.
- The busiest relays are about the same, sometimes slower: both builds
spend their time loading tens of thousands of matching rows by primary
key, which the index does not speed up.
- Single runs, and page-cache state moves them a lot: an earlier run,
with the file freshly cached, had vanilla's quiet nodes at 4.5 to 5 s
for 7 d, close to #2155's warm figure. Read orders of magnitude, not
exact figures. Nothing failed on this fast machine.
**Same answers.** Unit tests check indexed rows equal `LIKE` rows,
fallback on partial coverage, and trigger upkeep. On real data, 21
node/window pairs (3 to 14 days) requested from both builds at the same
moment were all identical once links were compared by public key. `LIKE`
ignores case and the index does not; 0 of 10.1M observations in the
window have a lower-case hop. Above `reachScanRowLimit` both paths
truncate, and they can keep different subsets.
### Production instance
t4g.large (2 vCPU, 8 GB, EBS), 13.7 GB database, about 4.5M observations
per 7 days, so the cache held a fraction of it.
- Before: `days=7` on a busy node returned 500 (`context canceled`), and
`days=1` took 58.5 s.
- With the index (38 probes of four nodes over 18 hours): `days=7` on
that node returned 200 in 23 s. The busiest probe node had a median of
0.7 s and p95 18 s. The hottest (127k relay observations in 7 d) had a
median of 1.3 s, p95 52 s, and one probe past 60 s at evening peak. The
quietest had a median of 0.04 s.
- On 2026-10-04 the host was resized to 4 vCPU and 16 GB so the cache
holds the database. Same index afterwards (493 probes): hottest node
median 1.4 s, p95 1.7 s, max 38 s; quietest median 0.05 s. (From
2026-10-03 the instance also ran a local change, not in this PR, that
lets reach requests continue in the background, which only affects
requests over 45 s.)
So on a small cold host the index turned "always fails" into "usually
works" but did not make the hottest relays reliably fast. That needs
cheaper row loading (a compact side table measured 4 to 10x on cold
reads; kept out of this PR) or background computation.
### Benchmarks (included)
```
go test ./cmd/server -run xxx -bench ScanReachRows -benchtime 5x
# 600k observations, 11 hops avg, 7 d window in 14 d, warm cache
hot_token/like 816 ms rare_token/like 486 ms
hot_token/indexed 293 ms rare_token/indexed 1.7 ms
go test ./internal/dbschema -run xxx -bench ObservationInsert -benchtime 20000x
# one observation per transaction, WriterDSN (WAL, synchronous=FULL), 11 hops
without_index 23 us/op with_index 127 us/op
```
The triggers add about 100 us per observation, roughly 0.1% of a core at
the production instance's 8 observations per second. They cap one writer
at about 8k observations per second instead of 43k. The benchmark ran in
a temp directory, so it shows CPU and b-tree cost, not disk sync cost.
Over 13 days on the production instance, 0 observations were dropped and
newest-observation age was p99 4 s. Four of 623 samples were 20 s or
more: 91 s and 20 s came within six minutes of a container restart, 31 s
came 24 hours after one (when the ingestor's daily tickers fire), and 23
s had no restart nearby. Index coverage stayed between 14.0 and 15.2
days.
**Disk.** On the snapshot (`dbstat`) the index is 1.98 GB for 10.1M
observations in the window, about 196 bytes each, next to 6.81 GB for
`observations`. It scales with hops per observation and `hopIndexDays`.
Backfill of 77M rows took 15.5 min on the production instance.
**Perf justification (AGENTS.md).** Reach goes from a scan over the
whole window to a seek over the matching rows. Everything new is
bounded: the index by `hopIndexDays` and the daily trim, the backfill by
chunk size and a cancellable context, and the server adds no in-memory
state. The ingest hot path pays the trigger cost above.
## Design notes and open questions
- Why not the in-memory store: it defaults to 7 days (`retentionHours:
168`) and reach offers up to 30, so it could not answer the longer
windows. The other option listed in #2146, driving the query from
`observations.timestamp`, is what #2155 already did.
- SQLite only: the triggers use `json_each`. The optional PostgreSQL
backend in #2145 would need its own equivalent.
- No off switch: the table and triggers are always installed and
`hopIndexDays` only sets the window. If you would rather have this
opt-in, or want a smaller default for small disks, I'll change it.
## Operator notes
- Upgrade: the ingestor installs the table and triggers on start and
backfills. Until a window is covered, reach behaves as today.
- Rollback: an older binary keeps indexing new rows with no trim. To
remove it with the ingestor stopped: `DROP TRIGGER
trg_observation_hops_ai; DROP TRIGGER trg_observation_hops_ad; DROP
TRIGGER trg_observation_hops_au; DROP TABLE observation_hops; DROP TABLE
observation_hops_coverage;`
- New key `retention.hopIndexDays`, documented in `config.example.json`.
## Tests
`go test ./...` passes in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`. New: `node_reach_hops_test.go`,
`observation_hops_test.go`, `observation_hops_backfill_test.go`, and the
two benchmarks.
## Update after review (commit `2291bea2`)
The text above is what was reviewed at `95e17976`. Where they differ,
this section is current:
- **Opt-in.** `retention.hopIndexDays` unset or 0 (the default, and the
value in `config.example.json`) installs nothing. A value above 0
installs the table and triggers and backfills that many days. This
replaces "default 14", the "No off switch" note and the Upgrade note
above.
- **Changing the setting** applies at the next ingestor start: raising
it backfills the gap, lowering it trims, and 0 removes the triggers and
both tables. On a 10.5M-observation copy the removal took 45 to 71 s, at
startup before MQTT connects.
- **Case.** Hops are stored and matched upper-cased, so the index
ignores ASCII case the way LIKE does. This replaces the sentence on case
under "Same answers".
- **Rollback.** Set it to 0 and start the ingestor once, then roll back.
The manual `DROP` list above still works with the ingestor stopped.
- **Disk.** A full 14-day backfill on that copy built 2.17 GB for 9.68M
observations, about 224 bytes each, in 13 min 37 s.
- Backfill and trim are serialized, and the chunk SQL lives in
`dbschema` next to the triggers.
## Update after review, round 2 (commit `95690e8c`)
Where this differs from the two sections above, this one is current:
- **Startup order.** The index is installed or removed in the startup
block, after the MQTT subscription is up and before the ingest gate
opens, so packets arriving during a removal are buffered
(`ingestBufferSize`), not lost. This replaces "at startup before MQTT
connects" in the first update.
- **Mark moves cover their range.** Each backfill and trim transaction
indexes or deletes everything between the old mark and the new one, so a
row arriving between the chunk-picking read and the transaction is
accounted for either way.
- **Negative `hopIndexDays`** is a config error at startup; the ingestor
refuses to run. This replaces "−5 is off" in the first update.
- **Server.** If the index disappears between the coverage check and the
scan, the request is answered from the `LIKE` scan instead of a 500.
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
