mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 20:37:41 +00:00
Part of #2238. Path hops through such a repeater (`/api/nodes`, `/api/resolve-hops`, the frontend hop resolver) still show a bare prefix; that part of the issue stays open. ## Summary A repeater that never sends an advert can still tell a nearby companion its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`. This PR stores those answers, resolves names with them, and shows these repeaters on the map, in the nodes list and in search, marked as self-reported. ### Ingestor: store the names - `367ecde1` adds a new `names` subtopic. - It is gated on `clientNodeNames.enabled`, default off. - Answers go into a new table, `client_node_names`: one row per observation, `UNIQUE(target, rx_pubkey, observed_at)`. - **Validation.** - `target` must be 64 hex. - The name is trimmed and rejected when it is empty, over 31 runes (firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8, or contains a control character. - **Retention.** `retention.clientNodeNamesDays` (0 = keep). `config.example.json` sets 30, like the other client tables: a companion only asks a repeater for its name when CoreScope has none (`nameMissing` in CoreDrive's `src/names.js`), so pruning is also how a renamed repeater gets its new name. - **Compatibility.** An ingestor without this change drops `/names` silently, so CoreDrive v1.20.0 works against either version. ### Server: resolve by the reported name - `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and `client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys` (the coverage tooltips) both use it. - **Which name wins.** The advert name, otherwise the newest reported name by `observed_at`. A prefix that matches different pubkeys across the two tables stays ambiguous. - **Only repeaters heard directly.** A reported name counts only once its target has at least one `discover` or `advert` reception in `client_receptions`, keyed by the full pubkey, or by its 8-byte prefix when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names` starts with that prefix (`clientNameHeard`). A name for a pubkey nobody heard adds no candidate, so it cannot make a real node's prefix ambiguous. Without the uniqueness check, a made-up pubkey that shares a heard repeater's first 8 bytes borrowed that repeater's receptions and passed (fixed in `4c569f91`). - **Filters.** The blacklist and the hidden name prefixes apply to both sources. - **Missing table.** A database without the table resolves from `nodes` alone, and the server checks for the table again until the ingestor has created it. - **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?, name?, ambiguous}`. ### Server: estimated position - `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`. It lists every repeater in `client_node_names` that never sent an advert (no row in `nodes` or `inactive_nodes`) and was heard directly at least once (see above). A node with an advert but no position stays out: it is already in `/api/nodes` under its advert name. - **Gated** like the other endpoints on `client_receptions`: 404 when `clientRxCoverage.enabled` is off, because the counts and the estimate come from companion GPS fixes. - **Which receptions count.** Only direct ones: - `discover` and `advert` rows, keyed by the full pubkey, or by its 8-byte prefix when no other known pubkey shares it; - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone. `geo` rows are left out. - **How the estimate is made.** - Receptions within 100 m form one spot, with the best SNR of that spot. - An estimate needs at least 3 spots, at least 200 m apart. - The position is the SNR-weighted centroid (`10^(snr/20)`). The radius is the distance to the farthest spot, at least 500 m. - The response is cached for 5 minutes, and the compute time is logged. ### Map - `5fcde554` adds `public/map-estimated.js`, which draws each estimate on its own layer: - a hollow, dashed "≈" marker and a dashed uncertainty circle; - a popup with an "Estimated position" badge, the method, the caveat that one-sided driving skews the estimate, and a link to the coverage map; - a Display toggle, "Estimated positions": on by default, kept in localStorage, and also hidden by the repeater role filter and in route view. ### Nodes page and search - `12e6b345` adds a "Self-reported repeaters" section under the nodes table, on the All and Repeaters tabs, filtered by the search box. - **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a self-reported detail view replaces "Node not found": - who reported the name; - the receptions; - the estimate, or why there is none; - links to the map and the coverage map. An unknown key still shows "Node not found". - **Header search** adds matching self-reported repeaters, tagged "self-reported". - **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as they are. These repeaters have no packets, so the packet filters, the live view and the channel sender lookup must not match them. ### Docs - `03580437` corrects the `/regions` section of `docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the switch, but the code uses `clientRegions.enabled`. - The new subtopic, endpoint and config keys are documented in the same file. ## Trust Any companion the broker lets publish can report a name for any repeater, and the newest report wins. This is the same model as `/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to linked companions. A report for a repeater nobody heard directly is stored but not used: it is not listed and adds no prefix candidate. A companion can still report a wrong name for a repeater that was heard. The same companion can also publish a fake `discover` reception for a made-up pubkey, so the "heard directly" check filters junk, not a malicious companion; `requireLinkedCompanion` is the control for that (its gate runs before the `/names` handler too). ## Performance - **`lookupPrefixes`** adds one term per key on `client_node_names`. - It matches the prefix as a range, `target >= ? AND target < ?` (upper bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX idx_cnn_target (target>? AND target<?)`, and the reception check is `SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`. - It is still one query per chunk of 200 keys: 400 compound terms, under SQLite's default limit of 500. - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks. - **`/api/nodes/reported`** makes three kinds of query: - one grouped query on `client_node_names`; - one `lookupPrefixes` call for the short hashes; - one `IN` query per 400 keys on `client_receptions`, served by `idx_client_recept_heard_geo`. The spread check is O(spots²) per repeater, with at most one spot per 100 m visited. The result is cached for 5 minutes; concurrent misses wait for one computation, and the response is written after the lock is released. - **The map does not wait for it.** It draws its markers first and adds the estimates when the list arrives. The map and the nodes page share one cached fetch (`ReportedNodes.load()`). - **Not benchmarked.** On the instance where this runs, `client_node_names` holds one row today. I expect at most tens of such repeaters per instance, but have not measured that. ## Notes - **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2 occurrences: the handler's `msg` parameter and its `gps` field. That is the decoded MQTT message, passed in the same shape as `handleClientPacket` and `handleClientRegions` already take it. No API response or internal structure uses it. - **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min TTL. They should move to the customizer later (rule 8). - **Map filters.** The estimates follow the "Estimated positions" toggle and the repeater role filter, not the area, region or last-heard filters. The toggle's tooltip says so. - **Name characters.** Names are rejected for control characters but not for Unicode format characters (bidi overrides, zero-width joiners), because emoji sequences in node names use the zero-width joiner. All output goes through `escapeHtml`. - **Not done:** the home-page search box (`home.js`, `/api/nodes/search` with a Claim button) does not list these repeaters. Claiming a node without adverts would have nothing to show. ## Tests On this branch (based on `550829af`): - `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and `cmd/ingestor`. - **New Go tests:** - handler and validation cases; - idempotent redelivery; - newest `observed_at` wins; - prune; - the gate on and off, without falling through to the observer path; - client-only node, nameless `nodes` row, advert name wins, ambiguity across the two sources; - hidden and blacklisted nodes; - absent-then-created table; - spot merging, threshold, weighted centroid and radius floor; - `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an advert position and a missing table; - an advert without a position, a never-heard name, a name heard only in `rxlog` paths, and the `clientRxCoverage` gate (404). These fail on `12e6b345` and pass at head. - a made-up pubkey sharing a heard repeater's 8-byte prefix, and two reported targets sharing one (`4c569f91`). Both fail without that commit. - **New frontend unit suites:** `tests/unit/test-map-estimated.js` and `tests/unit/test-reported-nodes.js` pass. Both load the real `escapeHtml` from `app.js`. - **`test-all.sh`** passes, all suites (with Python available for the XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the four changed frontend files. - **E2E**, against a static server with a mocked API: - `test-map-estimated-e2e.js`: 6/6, including the repeater role filter and route view hiding the estimates. Mutating either hide makes its step fail. - `test-reported-nodes-e2e.js`: 6/6. - `test-issue-1150-404-state-e2e.js`: 5/5. - **Real data, API only.** On the fork's staging, `/api/nodes/reported` lists one repeater: 5 spots, 17 receptions, an estimate with a radius of 2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays listed under the stricter rules. Not re-checked after `4c569f91`. - **Not done:** a check of that repeater in a browser. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
155 lines
7.1 KiB
Go
155 lines
7.1 KiB
Go
package main
|
|
|
|
import (
|
|
"encoding/json"
|
|
"testing"
|
|
)
|
|
|
|
// createClientNodeNames creates client_node_names the way the ingestor does
|
|
// (cmd/ingestor/db.go); the server never creates it.
|
|
func createClientNodeNames(t *testing.T, db *DB) {
|
|
t.Helper()
|
|
mustExecDB(t, db, `CREATE TABLE client_node_names (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT, target TEXT NOT NULL, rx_pubkey TEXT NOT NULL,
|
|
observed_at TEXT NOT NULL, ingested_at TEXT NOT NULL, name TEXT NOT NULL,
|
|
lat REAL, lon REAL, pos_acc_m REAL, repeater_clock INTEGER,
|
|
UNIQUE(target, rx_pubkey, observed_at))`)
|
|
mustExecDB(t, db, `CREATE INDEX idx_cnn_target ON client_node_names(target, observed_at)`)
|
|
}
|
|
|
|
// heardDirect records one discover reception of pk: a name only counts once the
|
|
// repeater was heard directly (clientNameHeard).
|
|
func heardDirect(t *testing.T, db *DB, pk string) {
|
|
t.Helper()
|
|
mustExecDB(t, db, `INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src)
|
|
VALUES ('aa11','`+pk[:16]+`',8,-5,51.0,5.38,'2026-10-16T10:00:00Z','x','discover')`)
|
|
}
|
|
|
|
func insClientName(t *testing.T, db *DB, target, name, observedAt string) {
|
|
t.Helper()
|
|
mustExecDB(t, db, `INSERT INTO client_node_names (target, rx_pubkey, observed_at, ingested_at, name)
|
|
VALUES ('`+target+`','aa11','`+observedAt+`','`+observedAt+`','`+name+`')`)
|
|
}
|
|
|
|
const (
|
|
cnnOnlyPK = "c0cd110000000000000000000000000000000000000000000000000000000000" // no nodes row
|
|
cnnNoNamePK = "c1c1220000000000000000000000000000000000000000000000000000000000" // nodes row, empty name
|
|
cnnAdvertPK = "c2c2330000000000000000000000000000000000000000000000000000000000" // nodes row with a name
|
|
cnnClashNode = "e470110000000000000000000000000000000000000000000000000000000000" // nodes, shares e470
|
|
cnnClashCli = "e470220000000000000000000000000000000000000000000000000000000000" // client only, shares e470
|
|
cnnHiddenPK = "c3c3440000000000000000000000000000000000000000000000000000000000"
|
|
cnnBlackPK = "c4c4550000000000000000000000000000000000000000000000000000000000"
|
|
cnnUnheard = "c2c2ff0000000000000000000000000000000000000000000000000000000000" // reported, never heard, shares c2c2
|
|
)
|
|
|
|
func seedClientNames(t *testing.T) *Server {
|
|
t.Helper()
|
|
db := seedCoverageDB(t)
|
|
createClientNodeNames(t, db)
|
|
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role) VALUES ('`+cnnNoNamePK+`','','repeater')`)
|
|
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role) VALUES ('`+cnnAdvertPK+`','FromAdvert','repeater')`)
|
|
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role) VALUES ('`+cnnClashNode+`','Aegis','repeater')`)
|
|
insClientName(t, db, cnnOnlyPK, "Old name", "2026-10-01T10:00:00.000Z")
|
|
insClientName(t, db, cnnOnlyPK, "Self Named", "2026-10-16T10:00:00.000Z")
|
|
insClientName(t, db, cnnNoNamePK, "Filled In", "2026-10-16T10:00:00.000Z")
|
|
insClientName(t, db, cnnAdvertPK, "Self Report", "2026-10-16T10:00:00.000Z")
|
|
insClientName(t, db, cnnClashCli, "Other", "2026-10-16T10:00:00.000Z")
|
|
insClientName(t, db, cnnHiddenPK, "🚫Secret", "2026-10-16T10:00:00.000Z")
|
|
insClientName(t, db, cnnBlackPK, "Blacklisted", "2026-10-16T10:00:00.000Z")
|
|
insClientName(t, db, cnnUnheard, "Invented", "2026-10-16T10:00:00.000Z")
|
|
for _, pk := range []string{cnnOnlyPK, cnnNoNamePK, cnnAdvertPK, cnnClashCli, cnnHiddenPK, cnnBlackPK} {
|
|
heardDirect(t, db, pk)
|
|
}
|
|
return &Server{db: db, cfg: &Config{
|
|
NodeBlacklist: []string{cnnBlackPK},
|
|
HiddenNamePrefixes: []string{"🚫"},
|
|
}}
|
|
}
|
|
|
|
func TestBatchResolveHeardKeysClientNames(t *testing.T) {
|
|
srv := seedClientNames(t)
|
|
got := srv.batchResolveHeardKeys([]string{"c0cd", "c1c122", "c2c2", "e470", "c3c3", "c4c4", cnnOnlyPK})
|
|
cases := map[string][2]string{
|
|
"c0cd": {cnnOnlyPK, "Self Named"}, // only known from client_node_names, newest observation
|
|
"c1c122": {cnnNoNamePK, "Filled In"}, // nodes row without a name: client name fills it
|
|
"c2c2": {cnnAdvertPK, "FromAdvert"}, // the advert name wins; the never-heard c2c2ff report adds no candidate
|
|
"e470": {"e470", ""}, // one candidate per source: still ambiguous
|
|
"c3c3": {"c3c3", ""}, // hidden-prefix client name
|
|
"c4c4": {"c4c4", ""}, // blacklisted pubkey
|
|
cnnOnlyPK: {cnnOnlyPK, "Self Named"}, // full pubkey
|
|
}
|
|
for k, want := range cases {
|
|
if got[k] != want {
|
|
t.Errorf("batchResolveHeardKeys[%q] = %v, want %v", k, got[k], want)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestResolvePrefixClientNames(t *testing.T) {
|
|
srv := seedClientNames(t)
|
|
resolve := func(pfx string) ResolvePrefixResp {
|
|
var r ResolvePrefixResp
|
|
json.Unmarshal(serveResolve(srv, "/api/nodes/resolve?prefix="+pfx).Body.Bytes(), &r)
|
|
return r
|
|
}
|
|
if r := resolve("c0cd"); r.Pubkey != cnnOnlyPK || r.Name != "Self Named" || r.Ambiguous {
|
|
t.Errorf("client-only node: %+v", r)
|
|
}
|
|
if r := resolve("c1c1"); r.Pubkey != cnnNoNamePK || r.Name != "Filled In" {
|
|
t.Errorf("nameless nodes row: %+v", r)
|
|
}
|
|
if r := resolve("c2c2"); r.Name != "FromAdvert" || r.Ambiguous {
|
|
t.Errorf("advert name must win, and a never-heard report must not make c2c2 ambiguous: %+v", r)
|
|
}
|
|
if r := resolve("c2c2ff"); r.Pubkey != "" || r.Ambiguous {
|
|
t.Errorf("a name for a repeater nobody heard directly must not resolve: %+v", r)
|
|
}
|
|
if r := resolve("e470"); !r.Ambiguous || r.Name != "" || r.Pubkey != "" {
|
|
t.Errorf("collision across sources must be ambiguous: %+v", r)
|
|
}
|
|
if r := resolve("c3c3"); r.Name != "" || r.Pubkey != "" || r.Ambiguous {
|
|
t.Errorf("hidden client name must resolve as not-found: %+v", r)
|
|
}
|
|
if r := resolve("c4c4"); r.Name != "" || r.Pubkey != "" || r.Ambiguous {
|
|
t.Errorf("blacklisted client-named node must resolve as not-found: %+v", r)
|
|
}
|
|
}
|
|
|
|
// A database without the table (an older ingestor) resolves from nodes alone,
|
|
// and picks the table up once the ingestor creates it, without a restart.
|
|
func TestClientNodeNamesTableAbsentThenCreated(t *testing.T) {
|
|
db := seedCoverageDB(t)
|
|
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role) VALUES ('`+cnnAdvertPK+`','FromAdvert','repeater')`)
|
|
srv := &Server{db: db, cfg: &Config{}}
|
|
|
|
got := srv.batchResolveHeardKeys([]string{"c2c2", "c0cd"})
|
|
if got["c2c2"] != [2]string{cnnAdvertPK, "FromAdvert"} || got["c0cd"] != [2]string{"c0cd", ""} {
|
|
t.Fatalf("without the table: %v", got)
|
|
}
|
|
|
|
createClientNodeNames(t, db)
|
|
insClientName(t, db, cnnOnlyPK, "Self Named", "2026-10-16T10:00:00.000Z")
|
|
heardDirect(t, db, cnnOnlyPK)
|
|
if got := srv.batchResolveHeardKeys([]string{"c0cd"}); got["c0cd"] != [2]string{cnnOnlyPK, "Self Named"} {
|
|
t.Fatalf("after the ingestor created the table: %v", got)
|
|
}
|
|
}
|
|
|
|
// Every key costs one query per chunk, whatever the number of keys: 250 keys
|
|
// span two chunks of at most 200, each within SQLite's 500-term compound limit.
|
|
func TestLookupPrefixesManyKeys(t *testing.T) {
|
|
srv := seedClientNames(t)
|
|
keys := make([]string, 0, 250)
|
|
for i := 0; i < 249; i++ {
|
|
keys = append(keys, "ffff"+string("0123456789abcdef"[i%16])+string("0123456789abcdef"[(i/16)%16]))
|
|
}
|
|
keys = append(keys, "c0cd")
|
|
hits := srv.lookupPrefixes(keys)
|
|
if len(hits) != len(keys) {
|
|
t.Fatalf("hits = %d, want %d (a chunk failed)", len(hits), len(keys))
|
|
}
|
|
if h := hits["c0cd"]; h.pubkey != cnnOnlyPK || h.name != "Self Named" {
|
|
t.Fatalf("c0cd in the second chunk: %+v", h)
|
|
}
|
|
}
|