From 696906315a04dd7fa50d2a01f385292f931fdc7d Mon Sep 17 00:00:00 2001 From: Alex B <142788550+A13xB0@users.noreply.github.com> Date: Sat, 10 Oct 2026 20:22:20 +0100 Subject: [PATCH] feat(scope-audit): transport view of every repeater seen forwarding (#2142, API) (#2167) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Backend half of #2142 (the page is the stacked PR #2169). `GET /api/scope-audit` lists only repeaters that have answered a declared-regions request: 17 of almost 200 nodes on the instance the request came from. This adds a transport view, `GET /api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per repeater **seen forwarding** in the window: - `transported`: the named region scopes it was seen carrying, each with packets and first/last seen (most packets first), plus unscoped, unmatched and ambiguous-hop counts; - `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt` where a declared answer exists; `asked: false` with null declared fields where it does not ("not asked" is not "declares nothing"); - with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every row and `carrying` / `notCarrying` counts, which is the rollout report the issue asks for. `mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400. Partial fix for #2142: this PR provides the API; the page is #2169 and the observer region filter #2171. The issue should only be considered done once all three are in. ## How it follows the issue's notes - **The scan does not change.** It is the same single `ScopeAuditForwarding` pass, with every repeater and room in `nodes` (plus any declared target) as a target instead of only the declared ones. Same attribution rules: FLOOD-family route types only, `minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches more than one repeater is credited to none of them. - **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities` runs one `IN` query; exercised at 1,200 keys in the benchmark below. - **Row count and response size:** measured below. - **Cache:** the transport view is cached per window with the declared view's TTLs (30 s; 5 min for 7d), under the same singleflight group with a `transport|` key. The scope filter is applied to a copy of the cached result, so one scan serves every scope. (`scope=`, not `region=`: across the API `region=` is the observer IATA filter.) - **Wording:** "transported" is documented as *seen carrying*, not configured for; the declared side stays separate. ## Performance `BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an instance with 1,179), 30k flood transmissions in the last 24h on 4-hop paths of 3-byte prefixes, a third region-scoped: ``` BenchmarkComputeScopeTransport-12 5 219253727 ns/op 425.4 response-KB 1200 rows ``` 219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a 425 KB uncompressed response for 1,200 rows. This is new code with no "before"; the declared view's numbers are untouched (same scan, same cache). ### On real data (a staging copy of a production database and its live feed) | request | rows | result | time | |---|---|---|---| | declared view, 24h | 16 | (unchanged) | | | `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them with a declared answer | | 1.5 s cold | | `mode=transport&scope=`, 7d | 664 | **51 carrying it, 613 not** | 14.4 s cold | | `mode=transport&scope=%23` (`#` and capitals), 7d, same window now cached | 664 | 237 carrying, 427 not | 0.35 s | The 7d cold compute is the same full-window hop scan the declared view already pays for 7d (its own TTL comment measures 16.7 s on a live-shaped database); like the declared view it is then cached for 5 minutes, and every scope filter is served from that one scan. - Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after opening; the real-data figures above were taken before the rename and are otherwise the same code. ## Tests - Red commit `2a9b1286`: response types and a `mode=transport` stub that returns no rows, plus tests. Four fail on the stub: - repeaters that never answered are listed with `asked: false` and null declared fields (and the answered one with its declared list and `notObserved`); - `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts them; no scope means no `carriesScope` or counts; - an unknown mode is a 400; - the transport and declared views are cached separately (requesting one does not serve the other). - Also added, passing on the stub by construction: a repeater with no traffic in the window is not listed; the node blacklist applies. - Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new `mode` and `scope` parameters, a Transport view section with the response shape), and the benchmark. - `make test`: 16 of 16 modules pass, including every existing scope-audit test. ## Review notes - Hidden-name and blacklist filtering use the same rules as the declared view. - `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use it (callers unchanged). - No new `map[string]interface{}`; the response is a named struct. - No config added. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the new `computeScopeTransport`. It is split into target, visibility, row and sort helpers; the response is the same and the transport tests are unchanged. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 --- cmd/server/area_filter_test.go | 2 +- cmd/server/routes.go | 4 + cmd/server/scope_audit.go | 288 ++++++++++++++++++++++++ cmd/server/scope_transport_2142_test.go | 244 ++++++++++++++++++++ docs/api-spec.md | 60 ++++- 5 files changed, 594 insertions(+), 4 deletions(-) create mode 100644 cmd/server/scope_transport_2142_test.go diff --git a/cmd/server/area_filter_test.go b/cmd/server/area_filter_test.go index 25c07d90..d76e4d82 100644 --- a/cmd/server/area_filter_test.go +++ b/cmd/server/area_filter_test.go @@ -74,7 +74,7 @@ func TestGetNodePubkeysInArea_Polygon(t *testing.T) { } // newTestStoreWithDB builds a minimal PacketStore wired to the given DB and config. -func newTestStoreWithDB(t *testing.T, db *DB, cfg *Config) *PacketStore { +func newTestStoreWithDB(t testing.TB, db *DB, cfg *Config) *PacketStore { t.Helper() return &PacketStore{ db: db, diff --git a/cmd/server/routes.go b/cmd/server/routes.go index 4d40bf6a..b741cf48 100644 --- a/cmd/server/routes.go +++ b/cmd/server/routes.go @@ -101,6 +101,10 @@ type Server struct { scopeAuditCache map[string]*ScopeAuditResponse scopeAuditCachedAt map[string]time.Time scopeAuditSF singleflight.Group + // #2142: the transport view (?mode=transport), cached per window with the + // same TTLs; computed under scopeAuditSF with a "transport|" key. + scopeTransportCache map[string]*ScopeTransportResponse + scopeTransportCachedAt map[string]time.Time // #1975: /api/scope-audit window cache and its single-flight guard, so a // burst of viewers on a cold cache recomputes the network-wide scan once diff --git a/cmd/server/scope_audit.go b/cmd/server/scope_audit.go index f4694d5e..69ff952c 100644 --- a/cmd/server/scope_audit.go +++ b/cmd/server/scope_audit.go @@ -849,6 +849,50 @@ type ScopeAuditResponse struct { Repeaters []ScopeAuditRow `json:"repeaters"` } +// ScopeTransportRow is one repeater seen forwarding in the window, in the +// transport view of the scope audit (#2142, ?mode=transport). Unlike +// ScopeAuditRow it does not require a declared-regions answer: "transported" +// means seen carrying, not configured for, and the declared side is reported +// where it exists (asked) and left null where it does not. +type ScopeTransportRow struct { + PublicKey string `json:"publicKey"` + Name *string `json:"name"` // null = no nodes row held for this key + Role *string `json:"role"` + + // Transported lists the named region scopes this repeater was seen + // forwarding in the window, most packets first. + Transported []ScopeObservation `json:"transported"` + UnscopedPackets int64 `json:"unscopedPackets"` // plain floods forwarded + UnmatchedPackets int64 `json:"unmatchedPackets"` // scoped, but no region key held here + AmbiguousHops int64 `json:"ambiguousHops"` // hops shared with another repeater's prefix, credited to neither + + // Asked is whether a declared-regions answer exists for this repeater. + // DeclaredRegions is null when it does not, [] when it answered with no + // named region. NotObserved (declared, not seen carried) is null when not + // asked. + Asked bool `json:"asked"` + DeclaredRegions []string `json:"declaredRegions"` + DeclaredWildcard bool `json:"declaredWildcard"` + ConfigState string `json:"configState,omitempty"` + DeclaredAt string `json:"declaredAt,omitempty"` + NotObserved []string `json:"notObserved"` + + // CarriesScope is set only when the request named a scope (?scope=): + // whether this repeater was seen carrying it in the window. + CarriesScope *bool `json:"carriesScope,omitempty"` +} + +// ScopeTransportResponse is GET /api/scope-audit?mode=transport. +type ScopeTransportResponse struct { + Mode string `json:"mode"` // "transport" + Window string `json:"window"` + Since string `json:"since"` + Scope string `json:"scope,omitempty"` // normalised ?scope= filter, when given + Carrying *int `json:"carrying,omitempty"` + NotCarry *int `json:"notCarrying,omitempty"` + Repeaters []ScopeTransportRow `json:"repeaters"` +} + // NodeScopesResponse is the payload for GET /api/nodes/{pubkey}/scopes: the // observed-forwarding side (ScopeConformance, embedded BY VALUE so its three // scope states sit at the JSON top level — unmatched and unscoped are @@ -998,6 +1042,16 @@ func (s *Server) handleScopeAudit(w http.ResponseWriter, r *http.Request) { sinceISO := time.Now().Add(-lookback).UTC().Format(time.RFC3339) + switch r.URL.Query().Get("mode") { + case "", "declared": + case "transport": + s.serveScopeTransport(w, r, window, sinceISO) + return + default: + writeError(w, 400, "mode must be declared or transport") + return + } + if cached, ok := s.scopeAuditCached(window); ok { writeJSON(w, cached) return @@ -1259,3 +1313,237 @@ func (s *Server) computeScopeAudit(window, sinceISO string) (*ScopeAuditResponse return resp, nil } + +// serveScopeTransport answers GET /api/scope-audit?mode=transport (#2142): +// the cached per-window transport view, with the optional region filter +// applied to a copy so the cached response is never changed. +func (s *Server) serveScopeTransport(w http.ResponseWriter, r *http.Request, window, sinceISO string) { + resp, ok := s.scopeTransportCached(window) + if !ok { + v, err, _ := s.scopeAuditSF.Do("transport|"+window, func() (interface{}, error) { + if cached, ok := s.scopeTransportCached(window); ok { + return cached, nil + } + fresh, cErr := s.computeScopeTransport(window, sinceISO) + if cErr != nil { + return nil, cErr + } + s.scopeTransportStore(window, fresh) + return fresh, nil + }) + if err != nil { + writeError(w, 500, err.Error()) + return + } + resp = v.(*ScopeTransportResponse) + } + // ?scope= names a region SCOPE (be, be-van). ?region= is kept for the + // observer IATA filter, as on every other endpoint. + if scope := strings.ToLower(normScope(strings.TrimSpace(r.URL.Query().Get("scope")))); scope != "" { + resp = withTransportScope(resp, scope) + } + writeJSON(w, resp) +} + +func (s *Server) scopeTransportCached(window string) (*ScopeTransportResponse, bool) { + s.scopeAuditMu.Lock() + defer s.scopeAuditMu.Unlock() + cached, ok := s.scopeTransportCache[window] + if !ok || time.Since(s.scopeTransportCachedAt[window]) >= scopeAuditTTLFor(window) { + return nil, false + } + return cached, true +} + +func (s *Server) scopeTransportStore(window string, resp *ScopeTransportResponse) { + s.scopeAuditMu.Lock() + defer s.scopeAuditMu.Unlock() + if s.scopeTransportCache == nil { + s.scopeTransportCache = make(map[string]*ScopeTransportResponse) + s.scopeTransportCachedAt = make(map[string]time.Time) + } + s.scopeTransportCache[window] = resp + s.scopeTransportCachedAt[window] = time.Now() +} + +// withTransportScope returns a copy of resp in which every row says whether +// it was seen carrying scope, with the two counts. scope is already +// normalised (no '#', lower case). +func withTransportScope(resp *ScopeTransportResponse, scope string) *ScopeTransportResponse { + out := *resp + out.Scope = scope + out.Repeaters = make([]ScopeTransportRow, len(resp.Repeaters)) + carrying, notCarrying := 0, 0 + for i, row := range resp.Repeaters { + carries := false + for _, so := range row.Transported { + if strings.ToLower(so.Scope) == scope { + carries = true + break + } + } + if carries { + carrying++ + } else { + notCarrying++ + } + c := carries + row.CarriesScope = &c + out.Repeaters[i] = row + } + out.Carrying, out.NotCarry = &carrying, ¬Carrying + return &out +} + +// repeaterPubkeys lists every repeater and room this instance holds a nodes +// row for: the candidates the transport view attributes forwarding hops to. +func (db *DB) repeaterPubkeys() ([]string, error) { + rows, err := db.conn.Query(`SELECT public_key FROM nodes WHERE role IN ('repeater', 'room')`) + if err != nil { + return nil, fmt.Errorf("scope transport repeaters: %w", err) + } + defer rows.Close() + var out []string + for rows.Next() { + var pk string + if rows.Scan(&pk) == nil { + out = append(out, strings.ToLower(pk)) + } + } + return out, rows.Err() +} + +// computeScopeTransport builds one window's transport view (#2142): the same +// single forwarding scan as the declared view (ScopeAuditForwarding, same +// FLOOD-family route types and minimum hop length, ambiguous hops credited to +// nobody), with every known repeater as a target instead of only the ones that +// answered a declared-regions request. A repeater is listed when the scan +// attributed any forwarding to it in the window. +func (s *Server) computeScopeTransport(window, sinceISO string) (*ScopeTransportResponse, error) { + declaredByPK, targets, err := s.scopeTransportTargets() + if err != nil { + return nil, err + } + forwarding := map[string]*scopeAuditTargetAgg{} + if s.store != nil { + forwarding, err = s.store.ScopeAuditForwarding(sinceISO, targets) + if err != nil { + return nil, err + } + } + + seen := make([]string, 0, len(forwarding)) + for pk, agg := range forwarding { + if len(agg.scopes) > 0 || agg.unscopedPackets > 0 || agg.unmatchedPackets > 0 { + seen = append(seen, pk) + } + } + identities := s.db.scopeAuditNodeIdentities(seen) + + resp := &ScopeTransportResponse{Mode: "transport", Window: window, Since: sinceISO, Repeaters: []ScopeTransportRow{}} + for _, pk := range seen { + id := identities[pk] + if s.scopeTransportHidden(pk, id) { + continue + } + resp.Repeaters = append(resp.Repeaters, scopeTransportRow(pk, id, forwarding[pk], declaredByPK)) + } + sortScopeTransportRows(resp.Repeaters) + return resp, nil +} + +// scopeTransportTargets is every repeater the scan should attribute +// forwarding to: those that answered a declared-regions request and every +// known repeater, sorted, with the declared answers by key. +func (s *Server) scopeTransportTargets() (map[string]DeclaredRegionsRow, []string, error) { + declared, err := s.db.AllCurrentDeclaredRegions() + if err != nil { + return nil, nil, err + } + declaredByPK := make(map[string]DeclaredRegionsRow, len(declared)) + targetSet := make(map[string]bool, len(declared)) + for _, d := range declared { + pk := strings.ToLower(d.Target) + declaredByPK[pk] = d + targetSet[pk] = true + } + repeaters, err := s.db.repeaterPubkeys() + if err != nil { + return nil, nil, err + } + for _, pk := range repeaters { + targetSet[pk] = true + } + targets := make([]string, 0, len(targetSet)) + for pk := range targetSet { + targets = append(targets, pk) + } + sort.Strings(targets) + return declaredByPK, targets, nil +} + +// scopeTransportHidden applies the blacklist and hidden name prefixes. +func (s *Server) scopeTransportHidden(pk string, id scopeAuditNodeIdentity) bool { + if s.cfg == nil { + return false + } + return s.cfg.IsBlacklisted(pk) || (id.Name != nil && s.cfg.IsNameHidden(*id.Name)) +} + +// scopeTransportRow is one repeater's row: the scopes it was seen carrying, +// busiest first, and, when it answered a declared-regions request, what it +// declares and which of those it was not seen carrying. +func scopeTransportRow(pk string, id scopeAuditNodeIdentity, agg *scopeAuditTargetAgg, declaredByPK map[string]DeclaredRegionsRow) ScopeTransportRow { + transported := make([]ScopeObservation, 0, len(agg.scopes)) + for _, so := range agg.scopes { + transported = append(transported, *so) + } + sort.Slice(transported, func(i, j int) bool { + if transported[i].Packets != transported[j].Packets { + return transported[i].Packets > transported[j].Packets + } + return transported[i].Scope < transported[j].Scope + }) + row := ScopeTransportRow{ + PublicKey: pk, + Name: id.Name, + Role: id.Role, + Transported: transported, + UnscopedPackets: agg.unscopedPackets, + UnmatchedPackets: agg.unmatchedPackets, + AmbiguousHops: agg.ambiguousHops, + } + d, ok := declaredByPK[pk] + if !ok { + return row + } + named, wildcard := splitDeclaredRegions(d.RegionsCSV) + row.Asked = true + row.DeclaredRegions = named + row.DeclaredWildcard = wildcard + row.ConfigState = scopeAuditConfigState(named, wildcard) + row.DeclaredAt = d.ObservedAt + row.NotObserved = []string{} + for _, rgn := range named { + if agg.scopes[rgn] == nil { + row.NotObserved = append(row.NotObserved, rgn) + } + } + return row +} + +// sortScopeTransportRows orders rows by name, falling back to the key. +func sortScopeTransportRows(rows []ScopeTransportRow) { + label := func(r ScopeTransportRow) string { + if r.Name != nil && *r.Name != "" { + return *r.Name + } + return r.PublicKey + } + sort.Slice(rows, func(i, j int) bool { + if an, bn := label(rows[i]), label(rows[j]); an != bn { + return an < bn + } + return rows[i].PublicKey < rows[j].PublicKey + }) +} diff --git a/cmd/server/scope_transport_2142_test.go b/cmd/server/scope_transport_2142_test.go new file mode 100644 index 00000000..7bd563bb --- /dev/null +++ b/cmd/server/scope_transport_2142_test.go @@ -0,0 +1,244 @@ +package main + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "io" + "log" + "net/http" + "net/http/httptest" + "os" + "strings" + "testing" + "time" + + "github.com/gorilla/mux" +) + +// #2142: the scope audit lists only repeaters that answered a declared-regions +// request (17 of ~200 nodes on the instance that asked). The transport view +// lists every repeater seen forwarding in the window, with the region scopes +// it carried, its declared regions where known, and a scope filter (?scope=) +// that splits the fleet into carriers and non-carriers for rollout tracking. + +var testFullPubkeyC = "cccc" + strings.Repeat("33", 30) + +func getScopeTransport(t *testing.T, router *mux.Router, query string) ScopeTransportResponse { + t.Helper() + req := httptest.NewRequest("GET", "/api/scope-audit?mode=transport"+query, nil) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", w.Code, w.Body.String()) + } + var got ScopeTransportResponse + if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { + t.Fatalf("decode: %v", err) + } + return got +} + +func transportRow(t *testing.T, rows []ScopeTransportRow, pk string) ScopeTransportRow { + t.Helper() + for _, r := range rows { + if r.PublicKey == pk { + return r + } + } + t.Fatalf("no row for %s in %d rows", pk[:8], len(rows)) + return ScopeTransportRow{} +} + +func addRepeater(t *testing.T, srv *Server, pk, name string) { + t.Helper() + if _, err := srv.db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES (?, ?, 'repeater') + ON CONFLICT(public_key) DO UPDATE SET name = excluded.name, role = 'repeater'`, pk, name); err != nil { + t.Fatal(err) + } +} + +// seedTwoRepeaters: A declared "be" and carries #be; B never answered and +// carries #fr twice; C is a repeater with no traffic in the window. +func seedTwoRepeaters(t *testing.T, srv *Server) { + t.Helper() + now := time.Now().UTC() + insertDeclared(t, srv, testFullPubkeyA, now.Format(time.RFC3339), "be", 0) + addRepeater(t, srv, testFullPubkeyA, "Alpha") + addRepeater(t, srv, testFullPubkeyB, "Bravo") + addRepeater(t, srv, testFullPubkeyC, "Charlie") + recent := now.Add(-time.Minute).Format(time.RFC3339) + seedTransmissionRouteAt(t, srv.store, testFullPubkeyA[:4], scopeMatched("#be"), RouteFlood, recent) + seedTransmissionRouteAt(t, srv.store, testFullPubkeyB[:4], scopeMatched("#fr"), RouteFlood, recent) + seedTransmissionRouteAt(t, srv.store, testFullPubkeyB[:4], scopeMatched("#fr"), RouteFlood, recent) +} + +func TestScopeTransport_ListsRepeatersThatNeverAnswered(t *testing.T) { + srv, router := setupScopeAuditServer(t) + seedTwoRepeaters(t, srv) + got := getScopeTransport(t, router, "") + if got.Mode != "transport" { + t.Fatalf("mode = %q, want transport", got.Mode) + } + a := transportRow(t, got.Repeaters, testFullPubkeyA) + if !a.Asked || len(a.DeclaredRegions) != 1 || a.DeclaredRegions[0] != "be" { + t.Errorf("A: asked=%v declared=%v, want asked with [be]", a.Asked, a.DeclaredRegions) + } + if len(a.Transported) != 1 || a.Transported[0].Scope != "be" || a.Transported[0].Packets != 1 { + t.Errorf("A: transported = %+v, want be x1", a.Transported) + } + if len(a.NotObserved) != 0 { + t.Errorf("A: notObserved = %v, want none", a.NotObserved) + } + b := transportRow(t, got.Repeaters, testFullPubkeyB) + if b.Asked || b.DeclaredRegions != nil || b.NotObserved != nil { + t.Errorf("B: asked=%v declared=%v notObserved=%v, want not asked with null declared fields", b.Asked, b.DeclaredRegions, b.NotObserved) + } + if len(b.Transported) != 1 || b.Transported[0].Scope != "fr" || b.Transported[0].Packets != 2 { + t.Errorf("B: transported = %+v, want fr x2", b.Transported) + } + if b.Name == nil || *b.Name != "Bravo" { + t.Errorf("B: name = %v, want Bravo", b.Name) + } +} + +func TestScopeTransport_OmitsRepeatersNotSeenForwarding(t *testing.T) { + srv, router := setupScopeAuditServer(t) + seedTwoRepeaters(t, srv) + for _, r := range getScopeTransport(t, router, "").Repeaters { + if r.PublicKey == testFullPubkeyC { + t.Fatalf("C forwarded nothing in the window and must not be listed: %+v", r) + } + } +} + +func TestScopeTransport_ScopeFilterSplitsCarriers(t *testing.T) { + srv, router := setupScopeAuditServer(t) + seedTwoRepeaters(t, srv) + for _, q := range []string{"&scope=fr", "&scope=%23fr", "&scope=FR"} { + got := getScopeTransport(t, router, q) + if got.Scope != "fr" { + t.Errorf("%s: scope = %q, want normalised \"fr\"", q, got.Scope) + } + a := transportRow(t, got.Repeaters, testFullPubkeyA) + b := transportRow(t, got.Repeaters, testFullPubkeyB) + if a.CarriesScope == nil || *a.CarriesScope || b.CarriesScope == nil || !*b.CarriesScope { + t.Errorf("%s: carriesScope A=%v B=%v, want A false, B true", q, a.CarriesScope, b.CarriesScope) + } + if got.Carrying == nil || *got.Carrying != 1 || got.NotCarry == nil || *got.NotCarry != 1 { + t.Errorf("%s: carrying=%v notCarrying=%v, want 1 and 1", q, got.Carrying, got.NotCarry) + } + } + plain := getScopeTransport(t, router, "") + if plain.Scope != "" || plain.Carrying != nil || transportRow(t, plain.Repeaters, testFullPubkeyB).CarriesScope != nil { + t.Error("without a scope filter, no carriesScope or counts may be reported") + } +} + +func TestScopeTransport_FiltersBlacklistedRepeater(t *testing.T) { + srv, router := setupScopeAuditServer(t) + seedTwoRepeaters(t, srv) + srv.cfg.NodeBlacklist = []string{testFullPubkeyB} + for _, r := range getScopeTransport(t, router, "").Repeaters { + if r.PublicKey == testFullPubkeyB { + t.Fatal("blacklisted repeater must be excluded") + } + } +} + +func TestScopeAudit_RejectsUnknownMode(t *testing.T) { + _, router := setupScopeAuditServer(t) + req := httptest.NewRequest("GET", "/api/scope-audit?mode=bogus", nil) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + if w.Code != http.StatusBadRequest { + t.Fatalf("status = %d, want 400 for an unknown mode", w.Code) + } +} + +func TestScopeTransport_CachedSeparatelyFromDeclaredView(t *testing.T) { + srv, router := setupScopeAuditServer(t) + seedTwoRepeaters(t, srv) + declared := getScopeAudit(t, router, "") + transport := getScopeTransport(t, router, "") + if len(declared.Repeaters) != 1 { + t.Fatalf("declared view = %d rows, want only the repeater that answered", len(declared.Repeaters)) + } + if len(transport.Repeaters) != 2 { + t.Fatalf("transport view = %d rows after the declared view was cached, want 2", len(transport.Repeaters)) + } +} + +// BenchmarkComputeScopeTransport: a fleet of 1,200 repeaters (#1975 quotes an +// instance with 1,179) and 30k flood transmissions in the last 24h, each +// forwarded along a 4-hop path of 3-byte prefixes, a third of them scoped to +// one of eight regions. Reports the response size too, as #2142 asks. +func BenchmarkComputeScopeTransport(b *testing.B) { + log.SetOutput(io.Discard) + b.Cleanup(func() { log.SetOutput(os.Stderr) }) + conn, err := sql.Open("sqlite3", ":memory:") + if err != nil { + b.Fatal(err) + } + conn.SetMaxOpenConns(1) + for _, ddl := range []string{ + `CREATE TABLE transmissions (id INTEGER PRIMARY KEY AUTOINCREMENT, raw_hex TEXT NOT NULL, hash TEXT NOT NULL UNIQUE, + first_seen TEXT NOT NULL, route_type INTEGER, payload_type INTEGER, code1 TEXT, code2 TEXT, scope_name TEXT)`, + `CREATE INDEX idx_transmissions_first_seen ON transmissions(first_seen)`, + `CREATE TABLE observations (id INTEGER PRIMARY KEY AUTOINCREMENT, transmission_id INTEGER NOT NULL, path_json TEXT, timestamp INTEGER NOT NULL)`, + `CREATE INDEX idx_obs_tx ON observations(transmission_id)`, + `CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, configured_scope TEXT, configured_scope_at TEXT)`, + } { + if _, err := conn.Exec(ddl); err != nil { + b.Fatal(err) + } + } + db := &DB{conn: conn} + if err := db.detectSchema(context.Background(), conn); err != nil { + b.Fatal(err) + } + const fleet = 1200 + pks := make([]string, fleet) + conn.Exec("BEGIN") + for i := range pks { + pks[i] = fmt.Sprintf("%06x", i*13+0x100000) + strings.Repeat("ab", 29) + scope := sql.NullString{} + if i%7 == 0 { + scope = sql.NullString{String: "be,be-van,*", Valid: true} + } + conn.Exec(`INSERT INTO nodes (public_key, name, role, configured_scope, configured_scope_at) VALUES (?, ?, 'repeater', ?, ?)`, + pks[i], fmt.Sprintf("Repeater %d", i), scope, time.Now().UTC().Format(time.RFC3339)) + } + regions := []string{"#be", "#be-van", "#be-ant", "#be-gnt", "#be-lge", "#nl", "#nl-ams", "#de"} + now := time.Now().UTC() + for i := 0; i < 30000; i++ { + var scope interface{} + route := RouteFlood + if i%3 == 0 { + route, scope = 0, regions[i%len(regions)] + } + res, _ := conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, code1, code2, scope_name) + VALUES ('AA', ?, ?, ?, 1, '1234', '00', ?)`, fmt.Sprintf("bench%d", i), now.Add(-time.Duration(i)*2*time.Second).Format(time.RFC3339), route, scope) + id, _ := res.LastInsertId() + path := fmt.Sprintf(`["%s","%s","%s","%s"]`, strings.ToUpper(pks[i%fleet][:6]), strings.ToUpper(pks[(i*7)%fleet][:6]), + strings.ToUpper(pks[(i*11)%fleet][:6]), strings.ToUpper(pks[(i*17)%fleet][:6])) + conn.Exec(`INSERT INTO observations (transmission_id, path_json, timestamp) VALUES (?, ?, ?)`, id, path, now.Unix()) + } + conn.Exec("COMMIT") + cfg := &Config{Port: 3000} + srv := NewServer(db, cfg, NewHub()) + srv.store = newTestStoreWithDB(b, db, cfg) + since := now.Add(-24 * time.Hour).Format(time.RFC3339) + b.ResetTimer() + var resp *ScopeTransportResponse + for i := 0; i < b.N; i++ { + if resp, err = srv.computeScopeTransport("24h", since); err != nil { + b.Fatal(err) + } + } + b.StopTimer() + body, _ := json.Marshal(resp) + b.ReportMetric(float64(len(resp.Repeaters)), "rows") + b.ReportMetric(float64(len(body))/1024, "response-KB") +} diff --git a/docs/api-spec.md b/docs/api-spec.md index 3a553748..04170e98 100644 --- a/docs/api-spec.md +++ b/docs/api-spec.md @@ -1890,9 +1890,11 @@ not being the same as "declared nothing"), which apply here identically. ### Query Parameters -| Param | Type | Default | Description | -|----------|--------|---------|---------------------------------| -| `window` | string | `24h` | Time window: `1h`, `24h`, `7d` | +| Param | Type | Default | Description | +|----------|--------|------------|---------------------------------| +| `window` | string | `24h` | Time window: `1h`, `24h`, `7d` | +| `mode` | string | `declared` | `declared` (this section) or `transport` (see [Transport view](#transport-view-modetransport)) | +| `scope` | string | — | `mode=transport` only: region scope to split the fleet by (`be`, `be-van`), with or without `#`, any case | ### Response `200` @@ -1932,6 +1934,12 @@ not being the same as "declared nothing"), which apply here identically. { "error": "window must be 1h, 24h, or 7d" } ``` +or, for an unknown `mode`: + +```json +{ "error": "mode must be declared or transport" } +``` + **Notes:** - Only repeaters with **at least one** declared-regions answer appear in `repeaters`. A repeater that was never successfully asked is absent, not shown as a row that declares @@ -2038,6 +2046,52 @@ not being the same as "declared nothing"), which apply here identically. --- + +### Transport view (`mode=transport`) + +Issue #2142. One row per repeater **seen forwarding** in the window, whether or not it has +ever answered a declared-regions request. Built from the same single forwarding scan as the +declared view, with the same attribution rules (FLOOD-family route types only, no 1-byte +hops, a hop whose prefix matches several repeaters is credited to none of them), but with +every repeater and room this instance knows as a target instead of only the declared ones. +"Transported" means **seen carrying**, not configured for; the declared side is reported +separately where it exists. Cached per window with the same TTLs as the declared view; the +`scope` filter is applied to the cached result. + +```jsonc +{ + "mode": "transport", + "window": string, + "since": string (ISO), + "scope": string | undefined, // normalised ?scope= filter ("#FR" -> "fr"); present only when given + "carrying": number | undefined, // rows with carriesScope: true; only with scope + "notCarrying": number | undefined, // rows with carriesScope: false; only with scope + "repeaters": [ + { + "publicKey": string, + "name": string | null, // null = no nodes row for this key + "role": string | null, + "transported": [ // named region scopes seen forwarded, most packets first + { "scope": string, "packets": number, "firstSeen": string (ISO), "lastSeen": string (ISO) } + ], + "unscopedPackets": number, // plain floods forwarded + "unmatchedPackets": number, // scoped packets whose region this instance holds no key for + "ambiguousHops": number, // hops shared with another repeater's prefix, credited to neither + "asked": boolean, // a declared-regions answer exists + "declaredRegions": [string] | null,// null when not asked; [] when it answered with no named region + "declaredWildcard": boolean, + "configState": string | undefined, // as in the declared view; only when asked + "declaredAt": string | undefined, // only when asked + "notObserved": [string] | null, // declared regions not seen carried; null when not asked + "carriesScope": boolean | undefined // only with scope + } + ] +} +``` + +A repeater with no attributed forwarding in the window (only ambiguous hops, or no traffic) +is not listed. Rows are sorted by name, then public key. + ## GET /api/scope-stats Scope-based packet statistics over a time window. Requires ingestor `scope_name_v1` migration to have run.