mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-30 03:58:01 +00:00
51a2a7dd2f7cfebcfa4e297feb14389764fef84f
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 efiten
|
296456f9c1 |
feat(map): colour and filter repeaters by scope-configuration state (#2006)
Closes #2001. Two review rounds plus a re-review; findings and evidence on the PR. Verified on a deployment against live data: the field over 1653 nodes, marker tints per filter state, the marker title and popup Scope row reaching the DOM, and the colorblind-preset cascade. The audit and the map are held to the same classification by an end-to-end test that fails when either side's wildcard handling drifts. |
||
|
efiten
|
8c164c5315 |
feat(scope-audit): verify a declared region against the repeater's own traffic (#1990)
Follow-up to #1987, and the point of counting that traffic in the first place. A region this instance holds no `hashRegions` key for is **unnameable, not absent**. #1987 says so with a caveat chip. This settles it wherever the evidence allows: derive `SHA256("#region")[:16]` from the repeater's own declaration and HMAC that repeater's own unmatched packets with it. Same computation the ingestor performs at ingest, with the candidate set narrowed from every configured key to this repeater's handful of declarations. Where it fires, a grey "declared but not observed" chip becomes a green one and the caveat count shrinks by the packets it explained. ## Two packets, not one `code1` is two bytes, so an unrelated name matches a given packet with probability 1/65536. Across ~400 unmatched packets and ~124 declared names, chance alone produces roughly one false match per refresh. Two matches on the same region for the same repeater is (1/65536)², about one in four billion. Lowering the threshold to one would not make this noisy, it would make it **unsound**, so the constant carries that arithmetic and a test rather than a comment. A region with exactly one hit stays grey and reports its single hit, so the page can say why it is still shown as not observed instead of leaving the reader to wonder. ## What it deliberately does not do **It writes nothing.** Read-time only. A wrong answer expires with the window instead of sitting in `transmissions.scope_name` until someone runs a repair, and `cmd/server` stays read-only per the invariant in AGENTS.md. **`notObserved` remains the single source of chip colour.** `regionEvidence` says only HOW a region was established. Two fields that can disagree about the same fact is how this column got confusing in the first place. ## Rule 0, including the part that was wrong at first The naive shape is `targets × names × packets` HMACs: 205 × 124 × 400 ≈ 10M. Caching per `(region, transmission)` pair cuts the HMACs to ~50k. **That measured 501ms**, because the HMACs had become a rounding error while the *iteration* stayed cubic at 10.2M map lookups. Re-keyed per region, holding the set of matching transmissions, it is **36ms** at the same worst-case shape: a region is HMACed over every packet once, and a target then asks one question per declared region instead of one per (region, packet). Most declared regions match nothing, so the common case is a single map lookup and no packet loop at all. `hmacCount` exists so a test can assert the first mistake cannot come back; the benchmark exists because only it caught the second. ## Both axes are bounded, because neither is bounded by the data The "~400 packets in a 7 day window" this was sized against is a property of one instance's configuration, not of the feature: the ingestor stores the unnameable state for every transport-scoped packet no configured key names, so an instance with few or no `hashRegions` entries — the stock state, and the one this helps most — has **every** scoped packet in that set. - the window query takes the 4096 most recent candidates and reports truncation, which the handler logs, so a grey chip on a sampled refresh is not read as "not forwarded" - the declared list is capped at 32 names per repeater: it arrives from a collector that validates each entry's shape but never how many entries there are - measured at the cap: **306ms** for 205 targets over 124 names, against 29ms for the shape a real network produces Because both caps make the evidence a sample, the response carries `observedUnmatchedSampled`. Without it a client subtracts a capped numerator from an uncapped total and overstates the unexplained traffic with no way to know it is doing so. The chip subtracts only evidence for regions **absent** from `notObserved` — a single-hit region the server refused to accept is not called explained either — and says "at most N" when the count was sampled. ## Verified on live data Six repeaters clear the threshold in a 7d window on a real instance. One of them: `nl-nb` green with 3 corroborating packets and the tooltip stating the count, `belml` still grey on 1, and the caveat chip reading 31 of 34 packets unexplained rather than 30. ## Tests `scope_verify_test.go` covers the HMAC-input walk against a real transport-flood packet captured from a live instance (a hand-built fixture would only prove the parser agrees with itself), that `regionCode` does not fold case, the threshold in both directions, the memo's HMAC count, both bounds with their truncation flag, and the benchmark at cap size. Handler-level tests cover a region verified into green, a single hit left grey with its count reported, and the sample-size field. `cd cmd/server && go test ./...` passes (77s), frontend 723 assertions pass, `go vet` and `gofmt -l` clean. |
||
|
efiten
|
0605b1703a |
feat(scope-audit): count and surface the traffic this instance cannot name (#1987)
Follow-up to #1986, and the second half of the same problem. `ScopeAuditForwarding` drops rows whose `scope_name` is the empty string with a bare `continue`. That empty string is the ingestor's "transport-scoped, but no configured region key matched `code1`" state (`scopeNameForDB`), so those packets name no region and can never satisfy a declared one. The consequence is on the page: **a repeater forwarding a region this instance holds no `hashRegions` key for is reported exactly like a repeater forwarding nothing at all.** The audit presents a gap in the reader's own configuration as a finding about someone else's hardware. This counts them per target, exposes the count as `observedUnmatchedPackets`, and renders it as a caveat chip beside the scope chips. ## Why it is not a rare edge Measured on a live instance before this landed: of 613 `notObserved` entries across 205 repeaters, **260 named a region that never appeared under any name in the whole 7-day window**. Two of them (`behss`, `fm-112`) were hash-verified as genuinely forwarded traffic the instance simply could not name: packet `0a065d41d51f1f77` decodes to `code1=9209`, which is exactly the code `#fm-112` derives over that packet's own payload. That instance had 16 region keys configured against 124 distinct region names its repeaters declare. A stock install has fewer. ## What the counter is not It is deliberately **not** folded into `unscopedPackets`. The two are opposites: | | meaning | what governs it | |---|---|---| | `unscopedPackets` | the packet carried no scope at all (`scope_name` SQL NULL) | the `*` wildcard | | `observedUnmatchedPackets` | the packet IS scoped, this instance holds no key for that region | nothing the repeater declares | For the same reason the new count never feeds `wildcardContradiction`, which counts only plain unscoped floods. `scopeNameForDB` in the ingestor is the source of truth for that three-state encoding, and the comments point there rather than restating it. It is also distinct from `ambiguousHops`, and the distinction is the point of the chip: that one is a pubkey-prefix collision between two repeaters and is nobody's fault, this one is a missing entry in the reader's own configuration and they can act on it. Saying which is which is what stops someone investigating an innocent repeater. ## Frontend The chip reuses the muted dashed treatment of `.sa-chip-ambiguous` on purpose: both are caveats on the row's finding rather than findings themselves, and neither may compete visually with the red/green scope chips beside them. It renders nothing for a non-numeric count. The value is server-supplied, and a truthiness check would put the literal string `NaN forwarded packets` on the page if that ever stopped holding. ## Docs `docs/api-spec.md` had **no entry for `GET /api/scope-audit` at all**, so this adds one: query parameter, full response shape, and the notes a client needs (the three traps the per-node endpoint documents apply here identically, `*` is never a scope, and "never asked" is not "declared nothing"). The new field is documented there rather than in isolation. ## Tests - the counter on a last-hop and on a mid-path hop - an unmatched packet enters neither `agg.scopes` nor `unscopedPackets`, which is the confusion this field exists to prevent - the field on the API row - six frontend cases: zero renders nothing, a missing field renders nothing (older server), the chip carries its count and class, singular and plural are both grammatical, the title names the cause and the fix, and a non-numeric count renders nothing rather than `NaN` `cd cmd/server && go test ./...` passes, frontend 712 assertions pass, `go vet` and `gofmt -l` clean. Rule 0: the counter is one increment on a branch that already existed as a `continue`, inside a loop this PR does not change. No new query, no new pass over the data. |
||
|
efiten
|
079e73aa4c |
fix(scope-audit): attribute forwarding to every hop, and pay for the wider scan (#1986)
The Scope Audit credits a transmission to `path[last]` only. On a
flood-family route every forwarder appends its own hash to the END of
the path (`internal/packetpath/route.go`), so `path[last]` does not mean
"forwarded this packet", it means "was the transmission an uplinked
observer heard directly". Every earlier hop forwarded the same packet
and is discarded.
The last-hop rule is genuinely required for DIRECT routes, which consume
hops from the front, so their `path[last]` is the route’s far end rather
than the transmitter. But `scopeAuditForwarderScanQuery` already
restricts to `route_type IN (0, 1)` via
`scopeConformanceForwarderRouteTypesSQL`, where that hazard cannot
arise, so inside this query the restriction only throws evidence away.
## What it costs the page today
Measured on a live-shaped instance, 206 declared repeaters, 965k
transmissions, 7d window:
| | before | after |
|---|---|---|
| repeaters with no attributable evidence of any kind | 133 of 205 (65%)
| 30 of 206 (15%) |
On a 1000-packet flood sample the mean path length is 7.08 hops, so the
last-hop rule keeps 394 of 2789 hop observations (14%), and 85% of the
nodes seen forwarding never appear as a last hop at all. Those repeaters
have every region they declare reported as "declared, not observed",
which is the page presenting a gap in our own attribution as a finding
about someone else’s repeater.
## Rule 0: what widening it costs, and what pays for it
Reading every hop multiplies the rows the scan returns: a 7d window
yields **3,470,188 hop rows** from 1,368,761 observations carrying a
path. Cold cost before this change was 16.7s for 7d and 4.0s for 24h, of
which SQLite accounts for 2.7s. The rest was the Go side reading rows.
Three changes, in order of what they bought:
1. **The scan carried `scope_name` and `first_seen` on every hop row.**
Both are columns of `transmissions`, and at 43 hop rows per transmission
the same two values were re-read that many times. They now come from one
query over the same window keyed by transmission id, both inside one
read transaction so a transmission arriving between them cannot appear
in the hop scan with no metadata to attribute it by. The hop scan
carries two columns instead of four.
2. **The hop is lower-cased into a stack buffer** instead of through
`strings.ToLower`. 1,026,814 of the 1,284,897 hops in a 24h window are
stored uppercase, because `packetpath.DecodePathFromRawHex` writes them
that way, and the great majority match no declared target, so that
allocation was paid millions of times to answer "no". The `(target,
txID)` de-duplication key became a struct for the same reason.
3. **The compute ran outside the cache mutex**, so every request
arriving on a cold window ran its own full scan concurrently. It now
sits behind a singleflight, the same treatment `/api/observers` and
`/api/nodes/{pubkey}/reach` already have, and the 7d window gets a 5
minute TTL while 1h and 24h keep 30s. At 30s a single reader with 7d
open keeps the instance recomputing more than half the time, for an
aggregate that moves at the pace of a week of traffic.
Result, warm process:
| window | before | after |
|---|---|---|
| 1h | 0.155s | 0.140s |
| 24h | 4.04s | 2.79-2.89s across six samples |
| 7d | 16.7s | 11.6s |
| repeat inside TTL | ~1ms | ~1ms |
**Rejected alternatives, measured on the same database**, so the next
reader does not have to re-derive them:
| approach | rows returned | time in SQLite |
|---|---|---|
| the query as written | 3,470,188 | 2.7s |
| pre-filter on the declared targets’ first 4 hex chars | 1,971,126 |
20.9s |
| `GROUP BY t.id, hop` | 965,025 | 38.0s |
| `SELECT DISTINCT t.id, path_json` | 1,229,966 | 17.7s |
The query plan is already index-driven (`idx_transmissions_first_seen`,
then `idx_observations_tx_ts`), so there is no missing index behind
this: the rows are inherent to the data. Note for anyone attempting a
hop comparison in SQL: a case-sensitive comparison silently drops most
attributable hops, per the 80% figure above.
## Tests
- a mid-path hop is attributed (the case behind the 65% blind spot)
- a DIRECT transmission whose `path[last]` **is** the target is still
not attributed. With the last-hop rule gone this is the only thing
standing between the audit and misattribution, so it gets its own test
rather than relying on the route filter being obvious
- one transmission counted once per target even when it appears on
several hops of the same path, which the `(target, txID)` de-duplication
now carries alone
- a hop longer than the 4-char floor resolved by its own length, which
nothing pinned before: every other test seeds 4-char hops
- the per-window TTL, so collapsing it back to one constant has to
delete the reason
- a second request inside the TTL served from cache rather than
recomputed. The cache path had no test at all
`cd cmd/server && go test ./...` passes (168s), `go vet` and `gofmt -l`
clean. Server-side only, no API shape change, no frontend change.
Browser validation: run against a live instance carrying this change,
the Scope Audit renders 220 rows matching the API row for row, and the
per-node scopes page still answers with its route-type mix.
|
||
|
efiten
|
2c8c1161b5 |
feat(#1975): network-wide Scope Audit page, fed by confirmed scopes (#1976)
One row per repeater whose configured region list is known, answering a question no other view answers: you declare these regions, but were you seen forwarding them? default_scope says what a node's adverts were observed under and transported_scopes (#1751) says what it carried, but nothing lined the declared list up against observed forwarding. The declared side merges every confirmed-scope source the database carries, newest answer per node wins, rather than naming one. On a stock install only nodes.configured_scope (#1865/#1971) exists and it degrades to the one-source case; deployments that collect the same fact another way keep working. Reading a single hard-coded source would have rendered an empty page on the very instance the evidence came from. Declared and observed are compared through normScope, so a leading "#" and a bare region name are one region. Unobserved regions render neutral, not red: absence over a short window is weak evidence, which the page header already states in words. Ported from a long-running fork deployment with its 17 server tests, rewired to the upstream data source, plus 14 frontend cases asserting rendered markup. |