2 Commits

Include renames

Author	SHA1	Message	Date
efitenandClaude Opus 5.5	abbe19531d	perf(paths): batch the resolved-path fallback for /nodes/{pk}/paths (#2223) (#2224) ... Fixes #2223. ## Problem `loadCanonicalResolvedPaths` reads the resolved path of each candidate's longest observation in batches (#2146). A transmission whose longest observation has no stored `resolved_path` then went through `bestResolvedPath` on its own, with two queries: - `fetchResolvedPathForObs` re-read that same observation, which the batch had just read; - `fetchResolvedPathsForTx` read all of the transmission's observations. On a production database, 24% of the observations with a path have no `resolved_path`, so most `/api/nodes/{pubkey}/paths` requests paid for this once per such transmission. ## Change - **`loadFallbackResolvedPaths` batches the fallback.** It reads those transmissions' stored paths in one query per `resolvedPathBatchSize` transmissions: ```sql SELECT id, transmission_id, resolved_path FROM observations WHERE resolved_path IS NOT NULL AND transmission_id IN (...) ``` On a production database the planner runs it as `SEARCH observations USING INDEX idx_observations_tx_ts (transmission_id=?)`. It then picks the observation `bestResolvedPath` would pick: the longest `path_json` with a stored path, the first in snapshot order on a tie. The result goes into the resolved-path LRU, as before. - **`longestStoredResolvedPath`** holds that choice. `bestResolvedPath` now uses it as well, so the rule exists in one place. - **A failed batch still falls back.** If the first-stage batch fails, it goes through `bestResolvedPath` per transmission, as before. A failed fallback batch does the same for its transmissions. The answer is the same as before, and no cache is added. ## Perf Staging, 9 nodes, requests made from the host itself (`localhost:3001`), so no proxy or TLS. Each node 3 times, after one warm-up round: | | median | p90 | max | total for 27 requests | |---|---|---|---|---| | before | 0.174 s | 0.604 s | 0.866 s | 6.95 s | | after | 0.079 s | 0.433 s | 0.484 s | 3.22 s | CPU profile, 30 s each, under the same request loop: - **Requests completed:** 180 before, 261 after. - **CPU in `handleNodePaths` per request:** 154 ms before (27.77 s / 180), 107 ms after (27.90 s / 261). - **Fallback cost:** `bestResolvedPath` took 13.56 s of `loadCanonicalResolvedPaths` before. The batched `loadFallbackResolvedPaths` takes 6.57 s. Caveats: - The before run was on a process that had been up longer than the after run, so the two LRU states differ. - One node (`060679ad…`, about 11k transmissions) did not change: 0.455 s against 0.451 s. Its time is elsewhere in the handler, and this PR does not address it. ## Tests - **`TestLoadCanonicalResolvedPathsBatchesFallback`:** 1,001 transmissions whose longest observation has no stored path. - On master: 2,002 per-transmission queries. - Now: 0 per-transmission queries and 6 batched ones (3 for the longest observations, 3 for the fallback), with every path taken from the shorter observation. - **Existing tests:** `TestLoadCanonicalResolvedPathsMatchesBestResolvedPath` still passes. It covers the longest observation without a path (#810), ties, nothing stored, an empty stored path, and an observation absent from the snapshot. The LRU and query-count tests also still pass. - `go vet` and `go test .` in `cmd/server` pass. No frontend change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-21 11:58:28 +02:00
efitenandClaude Opus 5.5	be9f382526	perf(server): /paths reads canonical paths in batches and skips redundant collision checks (#2157) ... Refs #2146 (item: `/api/nodes/:pubkey/paths`). ## Situation Per-step timing in a staging build (3 most-connected nodes, 2 calls each) showed where a `/paths` request spends its time: | Step | Time | |---|---| | build candidates under the read lock | 73-115 ms | | collision check: one `INSTR(LOWER(resolved_path), ?)` query per index hit, 13,367-16,790 per call | 1.3-2.7 s (first call after start: 13.5 s) | | canonical resolved_path per survivor (`bestResolvedPath`, one or two queries each) | 1.7-2.9 s | | aggregation under the read lock | 82-120 ms | So 3.2-4.1 s of a 3.4-5.1 s request was SQL. On a production database (busiest node, 27k transmissions, read-only): the collision check costs 1.85 s one by one and still 1.69 s batched, because SQLite reads and lowercases every stored path either way. Reading the canonical path (the longest observation's stored path) costs 0.18 s one by one and 0.09 s batched by id. ## My changes in this PR - **The collision check runs only where it decides something.** The aggregation keeps a transmission with a canonical path only if that path names the node (`if !containsTarget { continue }`). If no stored path names the node, the canonical one does not either. So for index hits with a canonical path the check changes nothing, and it now runs only for index hits without one, as before. - **Canonical paths are read in batches.** `loadCanonicalResolvedPaths` reads the stored path of each candidate's longest observation, one query per 500 transmissions. When that path is missing, empty or unparseable it calls `bestResolvedPath` for that transmission, so the result is the same as before. - The snapshot of observation IDs and `path_json` strings is taken in the existing candidate read lock (string copies, no parsing). The PR has two commits: a first version that batched both queries and did not get faster on staging, and the change described above, whose message has the measurements. Squash-merging is fine. ## Measurements Staging, after the ingestor's startup `ANALYZE`, same probe for both builds (ms): | | without this PR | with this PR | |---|---|---| | node 1 (degree 55), call 1 / call 2 | 22,237 / 3,364 | 1,793 / 1,686 | | node 2 (degree 47) | 4,427 / 3,615 | 1,828 / 1,809 | | node 3 (degree 46) | 5,171 / 4,035 | 3,050 / 2,408 | | 4 concurrent clients over 40 nodes, 120 s: calls / p50 / p95 | 73 / 6,717 / 15,160 | 125 / 2,861 / 10,630 | Both staging builds also contain #2147 and #2155. ## Tests - `TestLoadCanonicalResolvedPathsMatchesBestResolvedPath` compares the batched result with `bestResolvedPath` on seven cases: the #810 case (longest observation without a stored path), equal lengths, an empty stored path, case, an observation missing from the snapshot, longest wins, nothing stored. - `TestLoadCanonicalResolvedPathsBatchesQueries`: 1,001 transmissions take 3 batched queries and no per-transmission lookups. - `TestHandleNodePaths_CanonicalPathDecidesIndexHits`: the longest observation names A, a shorter one names B; both are index hits. A gets the transmission, B does not, with no per-transmission query. Run against master, the A/B results are the same and only the query-count check fails (1 there, 0 here). - The existing `/paths` tests pass unchanged (#810, #1278, #1352 collision and fallback cases, anchor bias). - `cmd/server`: `go test ./...` and `go test -race` green. ## Not in this PR - `nodeInResolvedPathViaIndex` still calls the per-transmission check; it has no production caller, only tests. - The concurrent p95 is still 10.6 s on staging. What is left there is the 4-connection read pool shared with other endpoints; a staging A/B with 8 connections gave no gain (`/paths` throughput dropped from 133-146 to 81-103 calls per 120 s), so the pool stays at 4. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 10:58:33 +02:00