5 Commits

Include renames

Author	SHA1	Message	Date
efiten	675c576fea	fix(scope-audit): bound the verifier's payload, and fix two tests that proved less than they claimed (#1993) ... Three leftovers from reviewing the scope-audit series (#1986, #1987, #1990). None is urgent; all three are the kind of thing that gets harder to explain the longer it sits. ## `scopeHMACInputs` accepted a payload `DecodePacket` rejects Its comment says it walks the same offsets as the decoder, and it does, minus the `maxPacketPayload` bound the decoder enforces. Unreachable in practice: such a packet never reaches the database with an empty `scope_name` in the first place, so the verifier never sees one. Worth closing anyway, because the comment claims the two agree. A verifier that accepts what the decoder refuses is a small divergence today and an hour of confusion on the day it matters. ## The corroboration test seeded the same packet twice The threshold of two rests on `code1` being two bytes: one match is 1/65536 by chance, two on the same region is (1/65536)². That argument needs two **independent** observations. The test seeded `realTransportFloodPacket` twice. Identical payloads derive identical codes, so it was one observation counted twice, and it would have passed just as happily against an implementation that counted rows rather than deriving anything. It now seeds the real captured packet plus a second one built for a different payload, both deriving to `#fm-112` on their own. **The feature was never wrong here.** `transmissions.hash` is unique and `ComputeContentHash` is path-independent, so two rows always mean two distinct payloads in production. Only the test failed to demonstrate the property it is named for. ## Naming at ingest and verifying at read time had no test together They were built separately, in #1990 and #1989, and the interaction between them is not exotic: with derived region keys enabled, a packet that used to be stored unnameable now arrives with a name. The audit must then report that region as observed by the ordinary route: - present in `agg.scopes` - absent from `notObserved` - **not** claimed by `regionEvidence`, which exists to explain regions that could only be established by verification Getting that wrong is quiet. The chip stays green while the reason underneath it is wrong, and a reader asking "how do we know this" gets the wrong story. ## Verification `cd cmd/server && go test ./...` passes (254s), `go vet` and `gofmt -l` clean. No production behaviour changes beyond the payload bound, which rejects input that cannot occur.	2026-09-10 22:30:24 +02:00
efiten	8c164c5315	feat(scope-audit): verify a declared region against the repeater's own traffic (#1990) ... Follow-up to #1987, and the point of counting that traffic in the first place. A region this instance holds no `hashRegions` key for is **unnameable, not absent**. #1987 says so with a caveat chip. This settles it wherever the evidence allows: derive `SHA256("#region")[:16]` from the repeater's own declaration and HMAC that repeater's own unmatched packets with it. Same computation the ingestor performs at ingest, with the candidate set narrowed from every configured key to this repeater's handful of declarations. Where it fires, a grey "declared but not observed" chip becomes a green one and the caveat count shrinks by the packets it explained. ## Two packets, not one `code1` is two bytes, so an unrelated name matches a given packet with probability 1/65536. Across ~400 unmatched packets and ~124 declared names, chance alone produces roughly one false match per refresh. Two matches on the same region for the same repeater is (1/65536)², about one in four billion. Lowering the threshold to one would not make this noisy, it would make it **unsound**, so the constant carries that arithmetic and a test rather than a comment. A region with exactly one hit stays grey and reports its single hit, so the page can say why it is still shown as not observed instead of leaving the reader to wonder. ## What it deliberately does not do **It writes nothing.** Read-time only. A wrong answer expires with the window instead of sitting in `transmissions.scope_name` until someone runs a repair, and `cmd/server` stays read-only per the invariant in AGENTS.md. **`notObserved` remains the single source of chip colour.** `regionEvidence` says only HOW a region was established. Two fields that can disagree about the same fact is how this column got confusing in the first place. ## Rule 0, including the part that was wrong at first The naive shape is `targets × names × packets` HMACs: 205 × 124 × 400 ≈ 10M. Caching per `(region, transmission)` pair cuts the HMACs to ~50k. **That measured 501ms**, because the HMACs had become a rounding error while the *iteration* stayed cubic at 10.2M map lookups. Re-keyed per region, holding the set of matching transmissions, it is **36ms** at the same worst-case shape: a region is HMACed over every packet once, and a target then asks one question per declared region instead of one per (region, packet). Most declared regions match nothing, so the common case is a single map lookup and no packet loop at all. `hmacCount` exists so a test can assert the first mistake cannot come back; the benchmark exists because only it caught the second. ## Both axes are bounded, because neither is bounded by the data The "~400 packets in a 7 day window" this was sized against is a property of one instance's configuration, not of the feature: the ingestor stores the unnameable state for every transport-scoped packet no configured key names, so an instance with few or no `hashRegions` entries — the stock state, and the one this helps most — has **every** scoped packet in that set. - the window query takes the 4096 most recent candidates and reports truncation, which the handler logs, so a grey chip on a sampled refresh is not read as "not forwarded" - the declared list is capped at 32 names per repeater: it arrives from a collector that validates each entry's shape but never how many entries there are - measured at the cap: **306ms** for 205 targets over 124 names, against 29ms for the shape a real network produces Because both caps make the evidence a sample, the response carries `observedUnmatchedSampled`. Without it a client subtracts a capped numerator from an uncapped total and overstates the unexplained traffic with no way to know it is doing so. The chip subtracts only evidence for regions **absent** from `notObserved` — a single-hit region the server refused to accept is not called explained either — and says "at most N" when the count was sampled. ## Verified on live data Six repeaters clear the threshold in a 7d window on a real instance. One of them: `nl-nb` green with 3 corroborating packets and the tooltip stating the count, `belml` still grey on 1, and the caveat chip reading 31 of 34 packets unexplained rather than 30. ## Tests `scope_verify_test.go` covers the HMAC-input walk against a real transport-flood packet captured from a live instance (a hand-built fixture would only prove the parser agrees with itself), that `regionCode` does not fold case, the threshold in both directions, the memo's HMAC count, both bounds with their truncation flag, and the benchmark at cap size. Handler-level tests cover a region verified into green, a single hit left grey with its count reported, and the sample-size field. `cd cmd/server && go test ./...` passes (77s), frontend 723 assertions pass, `go vet` and `gofmt -l` clean.	2026-09-09 17:58:09 +02:00
efiten	0605b1703a	feat(scope-audit): count and surface the traffic this instance cannot name (#1987) ... Follow-up to #1986, and the second half of the same problem. `ScopeAuditForwarding` drops rows whose `scope_name` is the empty string with a bare `continue`. That empty string is the ingestor's "transport-scoped, but no configured region key matched `code1`" state (`scopeNameForDB`), so those packets name no region and can never satisfy a declared one. The consequence is on the page: **a repeater forwarding a region this instance holds no `hashRegions` key for is reported exactly like a repeater forwarding nothing at all.** The audit presents a gap in the reader's own configuration as a finding about someone else's hardware. This counts them per target, exposes the count as `observedUnmatchedPackets`, and renders it as a caveat chip beside the scope chips. ## Why it is not a rare edge Measured on a live instance before this landed: of 613 `notObserved` entries across 205 repeaters, **260 named a region that never appeared under any name in the whole 7-day window**. Two of them (`behss`, `fm-112`) were hash-verified as genuinely forwarded traffic the instance simply could not name: packet `0a065d41d51f1f77` decodes to `code1=9209`, which is exactly the code `#fm-112` derives over that packet's own payload. That instance had 16 region keys configured against 124 distinct region names its repeaters declare. A stock install has fewer. ## What the counter is not It is deliberately **not** folded into `unscopedPackets`. The two are opposites: | | meaning | what governs it | |---|---|---| | `unscopedPackets` | the packet carried no scope at all (`scope_name` SQL NULL) | the `*` wildcard | | `observedUnmatchedPackets` | the packet IS scoped, this instance holds no key for that region | nothing the repeater declares | For the same reason the new count never feeds `wildcardContradiction`, which counts only plain unscoped floods. `scopeNameForDB` in the ingestor is the source of truth for that three-state encoding, and the comments point there rather than restating it. It is also distinct from `ambiguousHops`, and the distinction is the point of the chip: that one is a pubkey-prefix collision between two repeaters and is nobody's fault, this one is a missing entry in the reader's own configuration and they can act on it. Saying which is which is what stops someone investigating an innocent repeater. ## Frontend The chip reuses the muted dashed treatment of `.sa-chip-ambiguous` on purpose: both are caveats on the row's finding rather than findings themselves, and neither may compete visually with the red/green scope chips beside them. It renders nothing for a non-numeric count. The value is server-supplied, and a truthiness check would put the literal string `NaN forwarded packets` on the page if that ever stopped holding. ## Docs `docs/api-spec.md` had **no entry for `GET /api/scope-audit` at all**, so this adds one: query parameter, full response shape, and the notes a client needs (the three traps the per-node endpoint documents apply here identically, `*` is never a scope, and "never asked" is not "declared nothing"). The new field is documented there rather than in isolation. ## Tests - the counter on a last-hop and on a mid-path hop - an unmatched packet enters neither `agg.scopes` nor `unscopedPackets`, which is the confusion this field exists to prevent - the field on the API row - six frontend cases: zero renders nothing, a missing field renders nothing (older server), the chip carries its count and class, singular and plural are both grammatical, the title names the cause and the fix, and a non-numeric count renders nothing rather than `NaN` `cd cmd/server && go test ./...` passes, frontend 712 assertions pass, `go vet` and `gofmt -l` clean. Rule 0: the counter is one increment on a branch that already existed as a `continue`, inside a loop this PR does not change. No new query, no new pass over the data.	2026-09-09 17:05:29 +02:00
efiten	079e73aa4c	fix(scope-audit): attribute forwarding to every hop, and pay for the wider scan (#1986) ... The Scope Audit credits a transmission to `path[last]` only. On a flood-family route every forwarder appends its own hash to the END of the path (`internal/packetpath/route.go`), so `path[last]` does not mean "forwarded this packet", it means "was the transmission an uplinked observer heard directly". Every earlier hop forwarded the same packet and is discarded. The last-hop rule is genuinely required for DIRECT routes, which consume hops from the front, so their `path[last]` is the route’s far end rather than the transmitter. But `scopeAuditForwarderScanQuery` already restricts to `route_type IN (0, 1)` via `scopeConformanceForwarderRouteTypesSQL`, where that hazard cannot arise, so inside this query the restriction only throws evidence away. ## What it costs the page today Measured on a live-shaped instance, 206 declared repeaters, 965k transmissions, 7d window: | | before | after | |---|---|---| | repeaters with no attributable evidence of any kind | 133 of 205 (65%) | 30 of 206 (15%) | On a 1000-packet flood sample the mean path length is 7.08 hops, so the last-hop rule keeps 394 of 2789 hop observations (14%), and 85% of the nodes seen forwarding never appear as a last hop at all. Those repeaters have every region they declare reported as "declared, not observed", which is the page presenting a gap in our own attribution as a finding about someone else’s repeater. ## Rule 0: what widening it costs, and what pays for it Reading every hop multiplies the rows the scan returns: a 7d window yields **3,470,188 hop rows** from 1,368,761 observations carrying a path. Cold cost before this change was 16.7s for 7d and 4.0s for 24h, of which SQLite accounts for 2.7s. The rest was the Go side reading rows. Three changes, in order of what they bought: 1. **The scan carried `scope_name` and `first_seen` on every hop row.** Both are columns of `transmissions`, and at 43 hop rows per transmission the same two values were re-read that many times. They now come from one query over the same window keyed by transmission id, both inside one read transaction so a transmission arriving between them cannot appear in the hop scan with no metadata to attribute it by. The hop scan carries two columns instead of four. 2. **The hop is lower-cased into a stack buffer** instead of through `strings.ToLower`. 1,026,814 of the 1,284,897 hops in a 24h window are stored uppercase, because `packetpath.DecodePathFromRawHex` writes them that way, and the great majority match no declared target, so that allocation was paid millions of times to answer "no". The `(target, txID)` de-duplication key became a struct for the same reason. 3. **The compute ran outside the cache mutex**, so every request arriving on a cold window ran its own full scan concurrently. It now sits behind a singleflight, the same treatment `/api/observers` and `/api/nodes/{pubkey}/reach` already have, and the 7d window gets a 5 minute TTL while 1h and 24h keep 30s. At 30s a single reader with 7d open keeps the instance recomputing more than half the time, for an aggregate that moves at the pace of a week of traffic. Result, warm process: | window | before | after | |---|---|---| | 1h | 0.155s | 0.140s | | 24h | 4.04s | 2.79-2.89s across six samples | | 7d | 16.7s | 11.6s | | repeat inside TTL | ~1ms | ~1ms | **Rejected alternatives, measured on the same database**, so the next reader does not have to re-derive them: | approach | rows returned | time in SQLite | |---|---|---| | the query as written | 3,470,188 | 2.7s | | pre-filter on the declared targets’ first 4 hex chars | 1,971,126 | 20.9s | | `GROUP BY t.id, hop` | 965,025 | 38.0s | | `SELECT DISTINCT t.id, path_json` | 1,229,966 | 17.7s | The query plan is already index-driven (`idx_transmissions_first_seen`, then `idx_observations_tx_ts`), so there is no missing index behind this: the rows are inherent to the data. Note for anyone attempting a hop comparison in SQL: a case-sensitive comparison silently drops most attributable hops, per the 80% figure above. ## Tests - a mid-path hop is attributed (the case behind the 65% blind spot) - a DIRECT transmission whose `path[last]` **is** the target is still not attributed. With the last-hop rule gone this is the only thing standing between the audit and misattribution, so it gets its own test rather than relying on the route filter being obvious - one transmission counted once per target even when it appears on several hops of the same path, which the `(target, txID)` de-duplication now carries alone - a hop longer than the 4-char floor resolved by its own length, which nothing pinned before: every other test seeds 4-char hops - the per-window TTL, so collapsing it back to one constant has to delete the reason - a second request inside the TTL served from cache rather than recomputed. The cache path had no test at all `cd cmd/server && go test ./...` passes (168s), `go vet` and `gofmt -l` clean. Server-side only, no API shape change, no frontend change. Browser validation: run against a live instance carrying this change, the Scope Audit renders 220 rows matching the API row for row, and the per-node scopes page still answers with its route-type mix.	2026-09-09 16:00:15 +02:00
efiten	2c8c1161b5	feat(#1975): network-wide Scope Audit page, fed by confirmed scopes (#1976) ... One row per repeater whose configured region list is known, answering a question no other view answers: you declare these regions, but were you seen forwarding them? default_scope says what a node's adverts were observed under and transported_scopes (#1751) says what it carried, but nothing lined the declared list up against observed forwarding. The declared side merges every confirmed-scope source the database carries, newest answer per node wins, rather than naming one. On a stock install only nodes.configured_scope (#1865/#1971) exists and it degrades to the one-source case; deployments that collect the same fact another way keep working. Reading a single hard-coded source would have rendered an empty page on the very instance the evidence came from. Declared and observed are compared through normScope, so a leading "#" and a bare region name are one region. Unobserved regions render neutral, not red: absence over a short window is weak evidence, which the page header already states in words. Ported from a long-running fork deployment with its 17 server tests, rewired to the upstream data source, plus 14 frontend cases asserting rendered markup.	2026-09-06 23:04:42 +02:00