mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-10-22 23:01:16 +00:00
## Summary Backend half of #2142 (the page is the stacked PR #2169). `GET /api/scope-audit` lists only repeaters that have answered a declared-regions request: 17 of almost 200 nodes on the instance the request came from. This adds a transport view, `GET /api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per repeater **seen forwarding** in the window: - `transported`: the named region scopes it was seen carrying, each with packets and first/last seen (most packets first), plus unscoped, unmatched and ambiguous-hop counts; - `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt` where a declared answer exists; `asked: false` with null declared fields where it does not ("not asked" is not "declares nothing"); - with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every row and `carrying` / `notCarrying` counts, which is the rollout report the issue asks for. `mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400. Partial fix for #2142: this PR provides the API; the page is #2169 and the observer region filter #2171. The issue should only be considered done once all three are in. ## How it follows the issue's notes - **The scan does not change.** It is the same single `ScopeAuditForwarding` pass, with every repeater and room in `nodes` (plus any declared target) as a target instead of only the declared ones. Same attribution rules: FLOOD-family route types only, `minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches more than one repeater is credited to none of them. - **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities` runs one `IN` query; exercised at 1,200 keys in the benchmark below. - **Row count and response size:** measured below. - **Cache:** the transport view is cached per window with the declared view's TTLs (30 s; 5 min for 7d), under the same singleflight group with a `transport|` key. The scope filter is applied to a copy of the cached result, so one scan serves every scope. (`scope=`, not `region=`: across the API `region=` is the observer IATA filter.) - **Wording:** "transported" is documented as *seen carrying*, not configured for; the declared side stays separate. ## Performance `BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an instance with 1,179), 30k flood transmissions in the last 24h on 4-hop paths of 3-byte prefixes, a third region-scoped: ``` BenchmarkComputeScopeTransport-12 5 219253727 ns/op 425.4 response-KB 1200 rows ``` 219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a 425 KB uncompressed response for 1,200 rows. This is new code with no "before"; the declared view's numbers are untouched (same scan, same cache). ### On real data (a staging copy of a production database and its live feed) | request | rows | result | time | |---|---|---|---| | declared view, 24h | 16 | (unchanged) | | | `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them with a declared answer | | 1.5 s cold | | `mode=transport&scope=<scope A>`, 7d | 664 | **51 carrying it, 613 not** | 14.4 s cold | | `mode=transport&scope=%23<SCOPE B>` (`#` and capitals), 7d, same window now cached | 664 | 237 carrying, 427 not | 0.35 s | The 7d cold compute is the same full-window hop scan the declared view already pays for 7d (its own TTL comment measures 16.7 s on a live-shaped database); like the declared view it is then cached for 5 minutes, and every scope filter is served from that one scan. - Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after opening; the real-data figures above were taken before the rename and are otherwise the same code. ## Tests - Red commit `2a9b1286`: response types and a `mode=transport` stub that returns no rows, plus tests. Four fail on the stub: - repeaters that never answered are listed with `asked: false` and null declared fields (and the answered one with its declared list and `notObserved`); - `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts them; no scope means no `carriesScope` or counts; - an unknown mode is a 400; - the transport and declared views are cached separately (requesting one does not serve the other). - Also added, passing on the stub by construction: a repeater with no traffic in the window is not listed; the node blacklist applies. - Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new `mode` and `scope` parameters, a Transport view section with the response shape), and the benchmark. - `make test`: 16 of 16 modules pass, including every existing scope-audit test. ## Review notes - Hidden-name and blacklist filtering use the same rules as the declared view. - `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use it (callers unchanged). - No new `map[string]interface{}`; the response is a named struct. - No config added. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the new `computeScopeTransport`. It is split into target, visibility, row and sort helpers; the response is the same and the transport tests are unchanged. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8a3b923e81
commit
696906315a
5 files changed
+594
-4
No files matched your search
@@ -74,7 +74,7 @@ func TestGetNodePubkeysInArea_Polygon(t *testing.T) {
|
||||
}
|
||||
|
||||
// newTestStoreWithDB builds a minimal PacketStore wired to the given DB and config.
|
||||
func newTestStoreWithDB(t *testing.T, db *DB, cfg *Config) *PacketStore {
|
||||
func newTestStoreWithDB(t testing.TB, db *DB, cfg *Config) *PacketStore {
|
||||
t.Helper()
|
||||
return &PacketStore{
|
||||
db: db,
|
||||
|
||||
@@ -101,6 +101,10 @@ type Server struct {
|
||||
scopeAuditCache map[string]*ScopeAuditResponse
|
||||
scopeAuditCachedAt map[string]time.Time
|
||||
scopeAuditSF singleflight.Group
|
||||
// #2142: the transport view (?mode=transport), cached per window with the
|
||||
// same TTLs; computed under scopeAuditSF with a "transport|" key.
|
||||
scopeTransportCache map[string]*ScopeTransportResponse
|
||||
scopeTransportCachedAt map[string]time.Time
|
||||
|
||||
// #1975: /api/scope-audit window cache and its single-flight guard, so a
|
||||
// burst of viewers on a cold cache recomputes the network-wide scan once
|
||||
|
||||
@@ -849,6 +849,50 @@ type ScopeAuditResponse struct {
|
||||
Repeaters []ScopeAuditRow `json:"repeaters"`
|
||||
}
|
||||
|
||||
// ScopeTransportRow is one repeater seen forwarding in the window, in the
|
||||
// transport view of the scope audit (#2142, ?mode=transport). Unlike
|
||||
// ScopeAuditRow it does not require a declared-regions answer: "transported"
|
||||
// means seen carrying, not configured for, and the declared side is reported
|
||||
// where it exists (asked) and left null where it does not.
|
||||
type ScopeTransportRow struct {
|
||||
PublicKey string `json:"publicKey"`
|
||||
Name *string `json:"name"` // null = no nodes row held for this key
|
||||
Role *string `json:"role"`
|
||||
|
||||
// Transported lists the named region scopes this repeater was seen
|
||||
// forwarding in the window, most packets first.
|
||||
Transported []ScopeObservation `json:"transported"`
|
||||
UnscopedPackets int64 `json:"unscopedPackets"` // plain floods forwarded
|
||||
UnmatchedPackets int64 `json:"unmatchedPackets"` // scoped, but no region key held here
|
||||
AmbiguousHops int64 `json:"ambiguousHops"` // hops shared with another repeater's prefix, credited to neither
|
||||
|
||||
// Asked is whether a declared-regions answer exists for this repeater.
|
||||
// DeclaredRegions is null when it does not, [] when it answered with no
|
||||
// named region. NotObserved (declared, not seen carried) is null when not
|
||||
// asked.
|
||||
Asked bool `json:"asked"`
|
||||
DeclaredRegions []string `json:"declaredRegions"`
|
||||
DeclaredWildcard bool `json:"declaredWildcard"`
|
||||
ConfigState string `json:"configState,omitempty"`
|
||||
DeclaredAt string `json:"declaredAt,omitempty"`
|
||||
NotObserved []string `json:"notObserved"`
|
||||
|
||||
// CarriesScope is set only when the request named a scope (?scope=):
|
||||
// whether this repeater was seen carrying it in the window.
|
||||
CarriesScope *bool `json:"carriesScope,omitempty"`
|
||||
}
|
||||
|
||||
// ScopeTransportResponse is GET /api/scope-audit?mode=transport.
|
||||
type ScopeTransportResponse struct {
|
||||
Mode string `json:"mode"` // "transport"
|
||||
Window string `json:"window"`
|
||||
Since string `json:"since"`
|
||||
Scope string `json:"scope,omitempty"` // normalised ?scope= filter, when given
|
||||
Carrying *int `json:"carrying,omitempty"`
|
||||
NotCarry *int `json:"notCarrying,omitempty"`
|
||||
Repeaters []ScopeTransportRow `json:"repeaters"`
|
||||
}
|
||||
|
||||
// NodeScopesResponse is the payload for GET /api/nodes/{pubkey}/scopes: the
|
||||
// observed-forwarding side (ScopeConformance, embedded BY VALUE so its three
|
||||
// scope states sit at the JSON top level — unmatched and unscoped are
|
||||
@@ -998,6 +1042,16 @@ func (s *Server) handleScopeAudit(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
sinceISO := time.Now().Add(-lookback).UTC().Format(time.RFC3339)
|
||||
|
||||
switch r.URL.Query().Get("mode") {
|
||||
case "", "declared":
|
||||
case "transport":
|
||||
s.serveScopeTransport(w, r, window, sinceISO)
|
||||
return
|
||||
default:
|
||||
writeError(w, 400, "mode must be declared or transport")
|
||||
return
|
||||
}
|
||||
|
||||
if cached, ok := s.scopeAuditCached(window); ok {
|
||||
writeJSON(w, cached)
|
||||
return
|
||||
@@ -1259,3 +1313,237 @@ func (s *Server) computeScopeAudit(window, sinceISO string) (*ScopeAuditResponse
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// serveScopeTransport answers GET /api/scope-audit?mode=transport (#2142):
|
||||
// the cached per-window transport view, with the optional region filter
|
||||
// applied to a copy so the cached response is never changed.
|
||||
func (s *Server) serveScopeTransport(w http.ResponseWriter, r *http.Request, window, sinceISO string) {
|
||||
resp, ok := s.scopeTransportCached(window)
|
||||
if !ok {
|
||||
v, err, _ := s.scopeAuditSF.Do("transport|"+window, func() (interface{}, error) {
|
||||
if cached, ok := s.scopeTransportCached(window); ok {
|
||||
return cached, nil
|
||||
}
|
||||
fresh, cErr := s.computeScopeTransport(window, sinceISO)
|
||||
if cErr != nil {
|
||||
return nil, cErr
|
||||
}
|
||||
s.scopeTransportStore(window, fresh)
|
||||
return fresh, nil
|
||||
})
|
||||
if err != nil {
|
||||
writeError(w, 500, err.Error())
|
||||
return
|
||||
}
|
||||
resp = v.(*ScopeTransportResponse)
|
||||
}
|
||||
// ?scope= names a region SCOPE (be, be-van). ?region= is kept for the
|
||||
// observer IATA filter, as on every other endpoint.
|
||||
if scope := strings.ToLower(normScope(strings.TrimSpace(r.URL.Query().Get("scope")))); scope != "" {
|
||||
resp = withTransportScope(resp, scope)
|
||||
}
|
||||
writeJSON(w, resp)
|
||||
}
|
||||
|
||||
func (s *Server) scopeTransportCached(window string) (*ScopeTransportResponse, bool) {
|
||||
s.scopeAuditMu.Lock()
|
||||
defer s.scopeAuditMu.Unlock()
|
||||
cached, ok := s.scopeTransportCache[window]
|
||||
if !ok || time.Since(s.scopeTransportCachedAt[window]) >= scopeAuditTTLFor(window) {
|
||||
return nil, false
|
||||
}
|
||||
return cached, true
|
||||
}
|
||||
|
||||
func (s *Server) scopeTransportStore(window string, resp *ScopeTransportResponse) {
|
||||
s.scopeAuditMu.Lock()
|
||||
defer s.scopeAuditMu.Unlock()
|
||||
if s.scopeTransportCache == nil {
|
||||
s.scopeTransportCache = make(map[string]*ScopeTransportResponse)
|
||||
s.scopeTransportCachedAt = make(map[string]time.Time)
|
||||
}
|
||||
s.scopeTransportCache[window] = resp
|
||||
s.scopeTransportCachedAt[window] = time.Now()
|
||||
}
|
||||
|
||||
// withTransportScope returns a copy of resp in which every row says whether
|
||||
// it was seen carrying scope, with the two counts. scope is already
|
||||
// normalised (no '#', lower case).
|
||||
func withTransportScope(resp *ScopeTransportResponse, scope string) *ScopeTransportResponse {
|
||||
out := *resp
|
||||
out.Scope = scope
|
||||
out.Repeaters = make([]ScopeTransportRow, len(resp.Repeaters))
|
||||
carrying, notCarrying := 0, 0
|
||||
for i, row := range resp.Repeaters {
|
||||
carries := false
|
||||
for _, so := range row.Transported {
|
||||
if strings.ToLower(so.Scope) == scope {
|
||||
carries = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if carries {
|
||||
carrying++
|
||||
} else {
|
||||
notCarrying++
|
||||
}
|
||||
c := carries
|
||||
row.CarriesScope = &c
|
||||
out.Repeaters[i] = row
|
||||
}
|
||||
out.Carrying, out.NotCarry = &carrying, ¬Carrying
|
||||
return &out
|
||||
}
|
||||
|
||||
// repeaterPubkeys lists every repeater and room this instance holds a nodes
|
||||
// row for: the candidates the transport view attributes forwarding hops to.
|
||||
func (db *DB) repeaterPubkeys() ([]string, error) {
|
||||
rows, err := db.conn.Query(`SELECT public_key FROM nodes WHERE role IN ('repeater', 'room')`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("scope transport repeaters: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []string
|
||||
for rows.Next() {
|
||||
var pk string
|
||||
if rows.Scan(&pk) == nil {
|
||||
out = append(out, strings.ToLower(pk))
|
||||
}
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// computeScopeTransport builds one window's transport view (#2142): the same
|
||||
// single forwarding scan as the declared view (ScopeAuditForwarding, same
|
||||
// FLOOD-family route types and minimum hop length, ambiguous hops credited to
|
||||
// nobody), with every known repeater as a target instead of only the ones that
|
||||
// answered a declared-regions request. A repeater is listed when the scan
|
||||
// attributed any forwarding to it in the window.
|
||||
func (s *Server) computeScopeTransport(window, sinceISO string) (*ScopeTransportResponse, error) {
|
||||
declaredByPK, targets, err := s.scopeTransportTargets()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
forwarding := map[string]*scopeAuditTargetAgg{}
|
||||
if s.store != nil {
|
||||
forwarding, err = s.store.ScopeAuditForwarding(sinceISO, targets)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
seen := make([]string, 0, len(forwarding))
|
||||
for pk, agg := range forwarding {
|
||||
if len(agg.scopes) > 0 || agg.unscopedPackets > 0 || agg.unmatchedPackets > 0 {
|
||||
seen = append(seen, pk)
|
||||
}
|
||||
}
|
||||
identities := s.db.scopeAuditNodeIdentities(seen)
|
||||
|
||||
resp := &ScopeTransportResponse{Mode: "transport", Window: window, Since: sinceISO, Repeaters: []ScopeTransportRow{}}
|
||||
for _, pk := range seen {
|
||||
id := identities[pk]
|
||||
if s.scopeTransportHidden(pk, id) {
|
||||
continue
|
||||
}
|
||||
resp.Repeaters = append(resp.Repeaters, scopeTransportRow(pk, id, forwarding[pk], declaredByPK))
|
||||
}
|
||||
sortScopeTransportRows(resp.Repeaters)
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// scopeTransportTargets is every repeater the scan should attribute
|
||||
// forwarding to: those that answered a declared-regions request and every
|
||||
// known repeater, sorted, with the declared answers by key.
|
||||
func (s *Server) scopeTransportTargets() (map[string]DeclaredRegionsRow, []string, error) {
|
||||
declared, err := s.db.AllCurrentDeclaredRegions()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
declaredByPK := make(map[string]DeclaredRegionsRow, len(declared))
|
||||
targetSet := make(map[string]bool, len(declared))
|
||||
for _, d := range declared {
|
||||
pk := strings.ToLower(d.Target)
|
||||
declaredByPK[pk] = d
|
||||
targetSet[pk] = true
|
||||
}
|
||||
repeaters, err := s.db.repeaterPubkeys()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
for _, pk := range repeaters {
|
||||
targetSet[pk] = true
|
||||
}
|
||||
targets := make([]string, 0, len(targetSet))
|
||||
for pk := range targetSet {
|
||||
targets = append(targets, pk)
|
||||
}
|
||||
sort.Strings(targets)
|
||||
return declaredByPK, targets, nil
|
||||
}
|
||||
|
||||
// scopeTransportHidden applies the blacklist and hidden name prefixes.
|
||||
func (s *Server) scopeTransportHidden(pk string, id scopeAuditNodeIdentity) bool {
|
||||
if s.cfg == nil {
|
||||
return false
|
||||
}
|
||||
return s.cfg.IsBlacklisted(pk) || (id.Name != nil && s.cfg.IsNameHidden(*id.Name))
|
||||
}
|
||||
|
||||
// scopeTransportRow is one repeater's row: the scopes it was seen carrying,
|
||||
// busiest first, and, when it answered a declared-regions request, what it
|
||||
// declares and which of those it was not seen carrying.
|
||||
func scopeTransportRow(pk string, id scopeAuditNodeIdentity, agg *scopeAuditTargetAgg, declaredByPK map[string]DeclaredRegionsRow) ScopeTransportRow {
|
||||
transported := make([]ScopeObservation, 0, len(agg.scopes))
|
||||
for _, so := range agg.scopes {
|
||||
transported = append(transported, *so)
|
||||
}
|
||||
sort.Slice(transported, func(i, j int) bool {
|
||||
if transported[i].Packets != transported[j].Packets {
|
||||
return transported[i].Packets > transported[j].Packets
|
||||
}
|
||||
return transported[i].Scope < transported[j].Scope
|
||||
})
|
||||
row := ScopeTransportRow{
|
||||
PublicKey: pk,
|
||||
Name: id.Name,
|
||||
Role: id.Role,
|
||||
Transported: transported,
|
||||
UnscopedPackets: agg.unscopedPackets,
|
||||
UnmatchedPackets: agg.unmatchedPackets,
|
||||
AmbiguousHops: agg.ambiguousHops,
|
||||
}
|
||||
d, ok := declaredByPK[pk]
|
||||
if !ok {
|
||||
return row
|
||||
}
|
||||
named, wildcard := splitDeclaredRegions(d.RegionsCSV)
|
||||
row.Asked = true
|
||||
row.DeclaredRegions = named
|
||||
row.DeclaredWildcard = wildcard
|
||||
row.ConfigState = scopeAuditConfigState(named, wildcard)
|
||||
row.DeclaredAt = d.ObservedAt
|
||||
row.NotObserved = []string{}
|
||||
for _, rgn := range named {
|
||||
if agg.scopes[rgn] == nil {
|
||||
row.NotObserved = append(row.NotObserved, rgn)
|
||||
}
|
||||
}
|
||||
return row
|
||||
}
|
||||
|
||||
// sortScopeTransportRows orders rows by name, falling back to the key.
|
||||
func sortScopeTransportRows(rows []ScopeTransportRow) {
|
||||
label := func(r ScopeTransportRow) string {
|
||||
if r.Name != nil && *r.Name != "" {
|
||||
return *r.Name
|
||||
}
|
||||
return r.PublicKey
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool {
|
||||
if an, bn := label(rows[i]), label(rows[j]); an != bn {
|
||||
return an < bn
|
||||
}
|
||||
return rows[i].PublicKey < rows[j].PublicKey
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
|
||||
// #2142: the scope audit lists only repeaters that answered a declared-regions
|
||||
// request (17 of ~200 nodes on the instance that asked). The transport view
|
||||
// lists every repeater seen forwarding in the window, with the region scopes
|
||||
// it carried, its declared regions where known, and a scope filter (?scope=)
|
||||
// that splits the fleet into carriers and non-carriers for rollout tracking.
|
||||
|
||||
var testFullPubkeyC = "cccc" + strings.Repeat("33", 30)
|
||||
|
||||
func getScopeTransport(t *testing.T, router *mux.Router, query string) ScopeTransportResponse {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest("GET", "/api/scope-audit?mode=transport"+query, nil)
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200: %s", w.Code, w.Body.String())
|
||||
}
|
||||
var got ScopeTransportResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
func transportRow(t *testing.T, rows []ScopeTransportRow, pk string) ScopeTransportRow {
|
||||
t.Helper()
|
||||
for _, r := range rows {
|
||||
if r.PublicKey == pk {
|
||||
return r
|
||||
}
|
||||
}
|
||||
t.Fatalf("no row for %s in %d rows", pk[:8], len(rows))
|
||||
return ScopeTransportRow{}
|
||||
}
|
||||
|
||||
func addRepeater(t *testing.T, srv *Server, pk, name string) {
|
||||
t.Helper()
|
||||
if _, err := srv.db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES (?, ?, 'repeater')
|
||||
ON CONFLICT(public_key) DO UPDATE SET name = excluded.name, role = 'repeater'`, pk, name); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// seedTwoRepeaters: A declared "be" and carries #be; B never answered and
|
||||
// carries #fr twice; C is a repeater with no traffic in the window.
|
||||
func seedTwoRepeaters(t *testing.T, srv *Server) {
|
||||
t.Helper()
|
||||
now := time.Now().UTC()
|
||||
insertDeclared(t, srv, testFullPubkeyA, now.Format(time.RFC3339), "be", 0)
|
||||
addRepeater(t, srv, testFullPubkeyA, "Alpha")
|
||||
addRepeater(t, srv, testFullPubkeyB, "Bravo")
|
||||
addRepeater(t, srv, testFullPubkeyC, "Charlie")
|
||||
recent := now.Add(-time.Minute).Format(time.RFC3339)
|
||||
seedTransmissionRouteAt(t, srv.store, testFullPubkeyA[:4], scopeMatched("#be"), RouteFlood, recent)
|
||||
seedTransmissionRouteAt(t, srv.store, testFullPubkeyB[:4], scopeMatched("#fr"), RouteFlood, recent)
|
||||
seedTransmissionRouteAt(t, srv.store, testFullPubkeyB[:4], scopeMatched("#fr"), RouteFlood, recent)
|
||||
}
|
||||
|
||||
func TestScopeTransport_ListsRepeatersThatNeverAnswered(t *testing.T) {
|
||||
srv, router := setupScopeAuditServer(t)
|
||||
seedTwoRepeaters(t, srv)
|
||||
got := getScopeTransport(t, router, "")
|
||||
if got.Mode != "transport" {
|
||||
t.Fatalf("mode = %q, want transport", got.Mode)
|
||||
}
|
||||
a := transportRow(t, got.Repeaters, testFullPubkeyA)
|
||||
if !a.Asked || len(a.DeclaredRegions) != 1 || a.DeclaredRegions[0] != "be" {
|
||||
t.Errorf("A: asked=%v declared=%v, want asked with [be]", a.Asked, a.DeclaredRegions)
|
||||
}
|
||||
if len(a.Transported) != 1 || a.Transported[0].Scope != "be" || a.Transported[0].Packets != 1 {
|
||||
t.Errorf("A: transported = %+v, want be x1", a.Transported)
|
||||
}
|
||||
if len(a.NotObserved) != 0 {
|
||||
t.Errorf("A: notObserved = %v, want none", a.NotObserved)
|
||||
}
|
||||
b := transportRow(t, got.Repeaters, testFullPubkeyB)
|
||||
if b.Asked || b.DeclaredRegions != nil || b.NotObserved != nil {
|
||||
t.Errorf("B: asked=%v declared=%v notObserved=%v, want not asked with null declared fields", b.Asked, b.DeclaredRegions, b.NotObserved)
|
||||
}
|
||||
if len(b.Transported) != 1 || b.Transported[0].Scope != "fr" || b.Transported[0].Packets != 2 {
|
||||
t.Errorf("B: transported = %+v, want fr x2", b.Transported)
|
||||
}
|
||||
if b.Name == nil || *b.Name != "Bravo" {
|
||||
t.Errorf("B: name = %v, want Bravo", b.Name)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopeTransport_OmitsRepeatersNotSeenForwarding(t *testing.T) {
|
||||
srv, router := setupScopeAuditServer(t)
|
||||
seedTwoRepeaters(t, srv)
|
||||
for _, r := range getScopeTransport(t, router, "").Repeaters {
|
||||
if r.PublicKey == testFullPubkeyC {
|
||||
t.Fatalf("C forwarded nothing in the window and must not be listed: %+v", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopeTransport_ScopeFilterSplitsCarriers(t *testing.T) {
|
||||
srv, router := setupScopeAuditServer(t)
|
||||
seedTwoRepeaters(t, srv)
|
||||
for _, q := range []string{"&scope=fr", "&scope=%23fr", "&scope=FR"} {
|
||||
got := getScopeTransport(t, router, q)
|
||||
if got.Scope != "fr" {
|
||||
t.Errorf("%s: scope = %q, want normalised \"fr\"", q, got.Scope)
|
||||
}
|
||||
a := transportRow(t, got.Repeaters, testFullPubkeyA)
|
||||
b := transportRow(t, got.Repeaters, testFullPubkeyB)
|
||||
if a.CarriesScope == nil || *a.CarriesScope || b.CarriesScope == nil || !*b.CarriesScope {
|
||||
t.Errorf("%s: carriesScope A=%v B=%v, want A false, B true", q, a.CarriesScope, b.CarriesScope)
|
||||
}
|
||||
if got.Carrying == nil || *got.Carrying != 1 || got.NotCarry == nil || *got.NotCarry != 1 {
|
||||
t.Errorf("%s: carrying=%v notCarrying=%v, want 1 and 1", q, got.Carrying, got.NotCarry)
|
||||
}
|
||||
}
|
||||
plain := getScopeTransport(t, router, "")
|
||||
if plain.Scope != "" || plain.Carrying != nil || transportRow(t, plain.Repeaters, testFullPubkeyB).CarriesScope != nil {
|
||||
t.Error("without a scope filter, no carriesScope or counts may be reported")
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopeTransport_FiltersBlacklistedRepeater(t *testing.T) {
|
||||
srv, router := setupScopeAuditServer(t)
|
||||
seedTwoRepeaters(t, srv)
|
||||
srv.cfg.NodeBlacklist = []string{testFullPubkeyB}
|
||||
for _, r := range getScopeTransport(t, router, "").Repeaters {
|
||||
if r.PublicKey == testFullPubkeyB {
|
||||
t.Fatal("blacklisted repeater must be excluded")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopeAudit_RejectsUnknownMode(t *testing.T) {
|
||||
_, router := setupScopeAuditServer(t)
|
||||
req := httptest.NewRequest("GET", "/api/scope-audit?mode=bogus", nil)
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 for an unknown mode", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopeTransport_CachedSeparatelyFromDeclaredView(t *testing.T) {
|
||||
srv, router := setupScopeAuditServer(t)
|
||||
seedTwoRepeaters(t, srv)
|
||||
declared := getScopeAudit(t, router, "")
|
||||
transport := getScopeTransport(t, router, "")
|
||||
if len(declared.Repeaters) != 1 {
|
||||
t.Fatalf("declared view = %d rows, want only the repeater that answered", len(declared.Repeaters))
|
||||
}
|
||||
if len(transport.Repeaters) != 2 {
|
||||
t.Fatalf("transport view = %d rows after the declared view was cached, want 2", len(transport.Repeaters))
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkComputeScopeTransport: a fleet of 1,200 repeaters (#1975 quotes an
|
||||
// instance with 1,179) and 30k flood transmissions in the last 24h, each
|
||||
// forwarded along a 4-hop path of 3-byte prefixes, a third of them scoped to
|
||||
// one of eight regions. Reports the response size too, as #2142 asks.
|
||||
func BenchmarkComputeScopeTransport(b *testing.B) {
|
||||
log.SetOutput(io.Discard)
|
||||
b.Cleanup(func() { log.SetOutput(os.Stderr) })
|
||||
conn, err := sql.Open("sqlite3", ":memory:")
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
conn.SetMaxOpenConns(1)
|
||||
for _, ddl := range []string{
|
||||
`CREATE TABLE transmissions (id INTEGER PRIMARY KEY AUTOINCREMENT, raw_hex TEXT NOT NULL, hash TEXT NOT NULL UNIQUE,
|
||||
first_seen TEXT NOT NULL, route_type INTEGER, payload_type INTEGER, code1 TEXT, code2 TEXT, scope_name TEXT)`,
|
||||
`CREATE INDEX idx_transmissions_first_seen ON transmissions(first_seen)`,
|
||||
`CREATE TABLE observations (id INTEGER PRIMARY KEY AUTOINCREMENT, transmission_id INTEGER NOT NULL, path_json TEXT, timestamp INTEGER NOT NULL)`,
|
||||
`CREATE INDEX idx_obs_tx ON observations(transmission_id)`,
|
||||
`CREATE TABLE nodes (public_key TEXT PRIMARY KEY, name TEXT, role TEXT, configured_scope TEXT, configured_scope_at TEXT)`,
|
||||
} {
|
||||
if _, err := conn.Exec(ddl); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
db := &DB{conn: conn}
|
||||
if err := db.detectSchema(context.Background(), conn); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
const fleet = 1200
|
||||
pks := make([]string, fleet)
|
||||
conn.Exec("BEGIN")
|
||||
for i := range pks {
|
||||
pks[i] = fmt.Sprintf("%06x", i*13+0x100000) + strings.Repeat("ab", 29)
|
||||
scope := sql.NullString{}
|
||||
if i%7 == 0 {
|
||||
scope = sql.NullString{String: "be,be-van,*", Valid: true}
|
||||
}
|
||||
conn.Exec(`INSERT INTO nodes (public_key, name, role, configured_scope, configured_scope_at) VALUES (?, ?, 'repeater', ?, ?)`,
|
||||
pks[i], fmt.Sprintf("Repeater %d", i), scope, time.Now().UTC().Format(time.RFC3339))
|
||||
}
|
||||
regions := []string{"#be", "#be-van", "#be-ant", "#be-gnt", "#be-lge", "#nl", "#nl-ams", "#de"}
|
||||
now := time.Now().UTC()
|
||||
for i := 0; i < 30000; i++ {
|
||||
var scope interface{}
|
||||
route := RouteFlood
|
||||
if i%3 == 0 {
|
||||
route, scope = 0, regions[i%len(regions)]
|
||||
}
|
||||
res, _ := conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, code1, code2, scope_name)
|
||||
VALUES ('AA', ?, ?, ?, 1, '1234', '00', ?)`, fmt.Sprintf("bench%d", i), now.Add(-time.Duration(i)*2*time.Second).Format(time.RFC3339), route, scope)
|
||||
id, _ := res.LastInsertId()
|
||||
path := fmt.Sprintf(`["%s","%s","%s","%s"]`, strings.ToUpper(pks[i%fleet][:6]), strings.ToUpper(pks[(i*7)%fleet][:6]),
|
||||
strings.ToUpper(pks[(i*11)%fleet][:6]), strings.ToUpper(pks[(i*17)%fleet][:6]))
|
||||
conn.Exec(`INSERT INTO observations (transmission_id, path_json, timestamp) VALUES (?, ?, ?)`, id, path, now.Unix())
|
||||
}
|
||||
conn.Exec("COMMIT")
|
||||
cfg := &Config{Port: 3000}
|
||||
srv := NewServer(db, cfg, NewHub())
|
||||
srv.store = newTestStoreWithDB(b, db, cfg)
|
||||
since := now.Add(-24 * time.Hour).Format(time.RFC3339)
|
||||
b.ResetTimer()
|
||||
var resp *ScopeTransportResponse
|
||||
for i := 0; i < b.N; i++ {
|
||||
if resp, err = srv.computeScopeTransport("24h", since); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
b.StopTimer()
|
||||
body, _ := json.Marshal(resp)
|
||||
b.ReportMetric(float64(len(resp.Repeaters)), "rows")
|
||||
b.ReportMetric(float64(len(body))/1024, "response-KB")
|
||||
}
|
||||
+57
-3
@@ -1890,9 +1890,11 @@ not being the same as "declared nothing"), which apply here identically.
|
||||
|
||||
### Query Parameters
|
||||
|
||||
| Param | Type | Default | Description |
|
||||
|----------|--------|---------|---------------------------------|
|
||||
| `window` | string | `24h` | Time window: `1h`, `24h`, `7d` |
|
||||
| Param | Type | Default | Description |
|
||||
|----------|--------|------------|---------------------------------|
|
||||
| `window` | string | `24h` | Time window: `1h`, `24h`, `7d` |
|
||||
| `mode` | string | `declared` | `declared` (this section) or `transport` (see [Transport view](#transport-view-modetransport)) |
|
||||
| `scope` | string | — | `mode=transport` only: region scope to split the fleet by (`be`, `be-van`), with or without `#`, any case |
|
||||
|
||||
### Response `200`
|
||||
|
||||
@@ -1932,6 +1934,12 @@ not being the same as "declared nothing"), which apply here identically.
|
||||
{ "error": "window must be 1h, 24h, or 7d" }
|
||||
```
|
||||
|
||||
or, for an unknown `mode`:
|
||||
|
||||
```json
|
||||
{ "error": "mode must be declared or transport" }
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- Only repeaters with **at least one** declared-regions answer appear in `repeaters`. A
|
||||
repeater that was never successfully asked is absent, not shown as a row that declares
|
||||
@@ -2038,6 +2046,52 @@ not being the same as "declared nothing"), which apply here identically.
|
||||
|
||||
---
|
||||
|
||||
|
||||
### Transport view (`mode=transport`)
|
||||
|
||||
Issue #2142. One row per repeater **seen forwarding** in the window, whether or not it has
|
||||
ever answered a declared-regions request. Built from the same single forwarding scan as the
|
||||
declared view, with the same attribution rules (FLOOD-family route types only, no 1-byte
|
||||
hops, a hop whose prefix matches several repeaters is credited to none of them), but with
|
||||
every repeater and room this instance knows as a target instead of only the declared ones.
|
||||
"Transported" means **seen carrying**, not configured for; the declared side is reported
|
||||
separately where it exists. Cached per window with the same TTLs as the declared view; the
|
||||
`scope` filter is applied to the cached result.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"mode": "transport",
|
||||
"window": string,
|
||||
"since": string (ISO),
|
||||
"scope": string | undefined, // normalised ?scope= filter ("#FR" -> "fr"); present only when given
|
||||
"carrying": number | undefined, // rows with carriesScope: true; only with scope
|
||||
"notCarrying": number | undefined, // rows with carriesScope: false; only with scope
|
||||
"repeaters": [
|
||||
{
|
||||
"publicKey": string,
|
||||
"name": string | null, // null = no nodes row for this key
|
||||
"role": string | null,
|
||||
"transported": [ // named region scopes seen forwarded, most packets first
|
||||
{ "scope": string, "packets": number, "firstSeen": string (ISO), "lastSeen": string (ISO) }
|
||||
],
|
||||
"unscopedPackets": number, // plain floods forwarded
|
||||
"unmatchedPackets": number, // scoped packets whose region this instance holds no key for
|
||||
"ambiguousHops": number, // hops shared with another repeater's prefix, credited to neither
|
||||
"asked": boolean, // a declared-regions answer exists
|
||||
"declaredRegions": [string] | null,// null when not asked; [] when it answered with no named region
|
||||
"declaredWildcard": boolean,
|
||||
"configState": string | undefined, // as in the declared view; only when asked
|
||||
"declaredAt": string | undefined, // only when asked
|
||||
"notObserved": [string] | null, // declared regions not seen carried; null when not asked
|
||||
"carriesScope": boolean | undefined // only with scope
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A repeater with no attributed forwarding in the window (only ambiguous hops, or no traffic)
|
||||
is not listed. Rows are sorted by name, then public key.
|
||||
|
||||
## GET /api/scope-stats
|
||||
|
||||
Scope-based packet statistics over a time window. Requires ingestor `scope_name_v1` migration to have run.
|
||||
|
||||
Reference in new issue
Block a user