From 914bd4cf0e01bc120617e30f2eb1b4d57eaca3ac Mon Sep 17 00:00:00 2001 From: efiten Date: Sun, 13 Sep 2026 20:36:34 +0200 Subject: [PATCH] feat(channels): show each message's region (#1851) (#2018) ## Show each channel message's region scope Each message in the Channels view now shows the region scope it was sent with, as a small chip in the meta line: the region name (for example `#be`), `unknown scope`, or nothing. This is the channel-message part of #1852 by @dborup, extracted as a focused change. #1852 was closed unmerged because it had grown to the whole fork diff. The implementation follows dborup's commits c686ae3f, 350bf7ee and a94d57ed on dborup/CoreScope, adapted to current master. dborup is co-author on the commit. ### What changed - `cmd/server/db.go:2099,2195`: `GetChannelMessages` selects `t.scope_name` when the column exists and returns it as `scope_name`. - `cmd/server/store.go:5654`: the in-memory `GetChannelMessages` returns `scope_name`, so the field does not depend on which path serves the endpoint. - `cmd/server/store.go:2966,3244`: both WebSocket broadcast builders carry `scope_name`, so a live message shows its region immediately. - `public/channels.js:342,2278`: `messageScopeChipHtml` renders the chip with the existing `.sa-chip-declared` / `.sa-chip-unmatched` styles from `scope-audit.css`. The name goes through `escapeHtml`. No new CSS. - `public/channels.js:678,695,1435,1487`: the decrypt path and the WebSocket path keep `scope_name` on the message. ### Differences from #1852 - The field is `scope_name`, the name `/api/packets` already uses. - No `routeType` field. `transmissions.scope_name` already tells the states apart: NULL means no transport code, an empty string means a transport code that no configured region key matched. The frontend uses `??`, not `||`, so the empty string is kept. - A chip instead of `scope: ` text. The area label from later #1852 commits is not included. ### Perf One extra column per observation row in the page query (at most `limit` transmissions), and one extra map entry per broadcast observation. No new queries, loops or API calls. ### Tests - `cmd/server/channel_message_scope_name_test.go`: the three states through the DB query, the store, `/api/channels/{hash}/messages` over both paths, a schema without the column, and both broadcast builders. 5 of its 6 tests fail without the change; the sixth guards the missing-column case and passes either way. - `test-issue-1851-channel-message-scope.js`: the REST, WebSocket and client-side decrypt paths, escaping, and the name / unknown / none render. 4/4 fail without the change. Registered in `test-all.sh` and the unit step of `deploy.yml`. - Mutation checks: returning `nil` for `scope_name` in the DB path fails the DB and endpoint tests; `||` instead of `??` in the WebSocket path fails the WebSocket test. - `go test ./...` in `cmd/server`: ok. gofmt and go vet clean. ### Browser validation On a staging instance with live traffic (build `e84d2da6`), in Chrome: - `/api/channels/{hash}/messages` carries the `scope_name` key on every message in the 19 channels whose results I read. `#hamradio`, latest 50: 34 named, 1 empty string, 15 NULL. - Opening `#hamradio` renders 104 chips: `#nl` 53, `#be` 32, `#de` 11, `#eu` 6, `#bx` 1 and `unknown scope` 1, and no chip on unscoped messages. Chip text `rgb(26, 26, 46)` on `rgb(238, 242, 255)` in the light theme. ### Not verified - Dark theme not checked. - The real-decrypt branch of `decryptCandidates` has no test and was not exercised in the browser; the already-decrypted branch is tested. - Messages already in the client decrypt cache show no chip until they are decrypted again. - `go test -race` and the Playwright E2E suite were not run locally. Fixes #1851 ## Review follow-up (commit `50346589`) An independent review found no correctness or XSS problem and confirmed DB, store and WebSocket agree on the value. Changed: - **Real decrypt branch tested.** A new test runs the real AES+HMAC decrypt branch in `decryptCandidates` with one packet per scope state; deleting `scope_name` there now fails 2 of 7 tests. - **Tooltip wording.** The unknown-scope tooltip now says the scope "could not be matched to a single region on this instance" (`public/channels.js:338-347`). The ingestor stores an empty name both when no key matches and when several match without exactly one operator-configured key (`cmd/ingestor/region_keys.go:364-393`), so "matches none of the configured keys" was wrong for the second case. - **Old decrypt cache.** Decrypted messages cached before this change had no `scope_name` key and stayed chipless as long as the candidate count did not change. A cached message missing the key now forces one full decrypt; a cache that has it still takes the delta path. A test covers each case. - **Docs.** `docs/api-spec.md` documents `scope_name` on the channel messages response, with the null / empty string / name semantics. Corrections to the description: - **Test counts.** With the `db.go` and `store.go` changes reverted, 4 of the 5 top-level Go tests fail (6 of 7 counting subtests); only the missing-column test passes. - **Broadcast payload.** `scope_name` is added to `pkt`, which is copied into `broadcastMap` and also nested as `packet` (`store.go` ~2974-2980, ~3252-3257), so the key appears twice per observation: 36 bytes for `null`, 48 bytes for `"#belgium"`. - **Side effect on the Packets page.** The live table reads `m.data.packet` (`packets.js` ~1316-1318), so flat rows and expanded group children now show Scope for live packets. In grouped mode a new group copies a fixed field list without `scope_name` (~1384-1395) and shows the empty placeholder until reload. Before this PR every live row showed that placeholder, so this is not a regression. The three copies of the three-state scope rendering (`app.js`, `packets.js`, `channels.js`) are left as they are. --------- Co-authored-by: dborup <3627142+dborup@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) --- cmd/server/channel_message_scope_name_test.go | 260 ++++++++++++++ cmd/server/db.go | 18 +- cmd/server/store.go | 3 + docs/api-spec.md | 13 +- public/channels.js | 26 +- test-all.sh | 1 + test-issue-1851-channel-message-scope.js | 330 ++++++++++++++++++ 7 files changed, 646 insertions(+), 5 deletions(-) create mode 100644 cmd/server/channel_message_scope_name_test.go create mode 100644 test-issue-1851-channel-message-scope.js diff --git a/cmd/server/channel_message_scope_name_test.go b/cmd/server/channel_message_scope_name_test.go new file mode 100644 index 00000000..3d2af471 --- /dev/null +++ b/cmd/server/channel_message_scope_name_test.go @@ -0,0 +1,260 @@ +package main + +import ( + "encoding/json" + "net/http/httptest" + "testing" + "time" + + "github.com/gorilla/mux" +) + +// #1851: the Channels view shows the region scope a message was sent under. +// transmissions.scope_name encodes three states (see StoreTx.ScopeName): +// SQL NULL = not transport-scoped, "" = transport-scoped but no configured +// region key matched, "#name" = matched region. /api/channels/{hash}/messages +// is served by the DB query when a DB is attached and by the in-memory store +// otherwise, and live messages arrive over the WebSocket broadcast, so all +// three paths must carry the same three states. + +const ( + chScopeHashUnscoped = "cccccccccccccc01" + chScopeHashUnknown = "cccccccccccccc02" + chScopeHashMatched = "cccccccccccccc03" +) + +var chScopeWant = map[string]interface{}{ + chScopeHashUnscoped: nil, + chScopeHashUnknown: "", + chScopeHashMatched: "#belgium", +} + +// setupChannelScopeDB seeds one channel with three messages, one per scope +// state, each heard by one observation. +func setupChannelScopeDB(t *testing.T) *DB { + t.Helper() + db := setupTestDB(t) + if _, err := db.conn.Exec(`ALTER TABLE transmissions ADD COLUMN scope_name TEXT DEFAULT NULL`); err != nil { + t.Fatalf("add scope_name column: %v", err) + } + db.hasScopeName = true + if _, err := db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer One', 'BRU')`); err != nil { + t.Fatalf("insert observer: %v", err) + } + now := time.Now().UTC() + rows := []struct { + hash string + routeType int + scope interface{} + text string + }{ + {chScopeHashUnscoped, 1, nil, "Alice: plain flood"}, + {chScopeHashUnknown, 0, "", "Bob: unknown region"}, + {chScopeHashMatched, 0, "#belgium", "Carol: matched region"}, + } + for i, r := range rows { + ts := now.Add(time.Duration(i-3) * time.Minute) + res, err := db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash, scope_name) + VALUES ('AABB', ?, ?, ?, 5, ?, '#scopetest', ?)`, + r.hash, ts.Format(time.RFC3339), r.routeType, + `{"type":"CHAN","channel":"#scopetest","text":"`+r.text+`"}`, r.scope) + if err != nil { + t.Fatalf("insert tx %s: %v", r.hash, err) + } + txID, _ := res.LastInsertId() + if _, err := db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp) + VALUES (?, 1, 9.5, -90, '[]', ?)`, txID, ts.Unix()); err != nil { + t.Fatalf("insert obs %s: %v", r.hash, err) + } + } + return db +} + +func assertChannelMessageScopes(t *testing.T, messages []map[string]interface{}) { + t.Helper() + if len(messages) != len(chScopeWant) { + t.Fatalf("expected %d messages, got %d", len(chScopeWant), len(messages)) + } + for _, m := range messages { + h, _ := m["packetHash"].(string) + want, known := chScopeWant[h] + if !known { + t.Errorf("unexpected packetHash %q", h) + continue + } + got, present := m["scope_name"] + if !present { + t.Errorf("%s: scope_name key missing", h) + continue + } + if got != want { + t.Errorf("%s: scope_name = %#v, want %#v", h, got, want) + } + } +} + +func TestDBGetChannelMessagesCarriesScopeName(t *testing.T) { + db := setupChannelScopeDB(t) + defer db.Close() + messages, _, err := db.GetChannelMessages("#scopetest", 100, 0) + if err != nil { + t.Fatal(err) + } + assertChannelMessageScopes(t, messages) +} + +func TestStoreGetChannelMessagesCarriesScopeName(t *testing.T) { + db := setupChannelScopeDB(t) + defer db.Close() + store := NewPacketStore(db, nil) + if err := store.Load(); err != nil { + t.Fatalf("store.Load: %v", err) + } + messages, _ := store.GetChannelMessages("#scopetest", 100, 0) + for i := range messages { + // The store keeps the value as a *string; normalise through JSON so + // the comparison is on what the browser receives. + b, err := json.Marshal(messages[i]) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var decoded map[string]interface{} + if err := json.Unmarshal(b, &decoded); err != nil { + t.Fatalf("unmarshal: %v", err) + } + messages[i] = decoded + } + assertChannelMessageScopes(t, messages) +} + +// A database the ingestor has not migrated yet has no scope_name column. The +// DB query must not select it (the query would fail) and every message reports +// null, the same as a non-transport-scoped message. +func TestDBGetChannelMessagesWithoutScopeNameColumn(t *testing.T) { + db := setupTestDB(t) + defer db.Close() + seedTestData(t, db) + messages, _, err := db.GetChannelMessages("#test", 100, 0) + if err != nil { + t.Fatalf("GetChannelMessages on a schema without scope_name: %v", err) + } + if len(messages) == 0 { + t.Fatal("expected messages for #test") + } + for _, m := range messages { + if got := m["scope_name"]; got != nil { + t.Errorf("scope_name = %#v, want nil without the column", got) + } + } +} + +func TestChannelMessagesEndpointExposesScopeName(t *testing.T) { + db := setupChannelScopeDB(t) + defer db.Close() + for _, withDB := range []bool{true, false} { + name := "db" + if !withDB { + name = "store" + } + t.Run(name, func(t *testing.T) { + store := NewPacketStore(db, nil) + if err := store.Load(); err != nil { + t.Fatalf("store.Load: %v", err) + } + srv := NewServer(db, &Config{Port: 3000}, NewHub()) + srv.store = store + if !withDB { + srv.db = nil + } + router := mux.NewRouter() + srv.RegisterRoutes(router) + + req := httptest.NewRequest("GET", "/api/channels/%23scopetest/messages?limit=50", nil) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + if w.Code != 200 { + t.Fatalf("expected 200, got %d (body: %s)", w.Code, w.Body.String()) + } + var body struct { + Messages []map[string]interface{} `json:"messages"` + } + if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil { + t.Fatalf("unmarshal: %v", err) + } + assertChannelMessageScopes(t, body.Messages) + }) + } +} + +// Live messages in the Channels view are built from the WebSocket broadcast, +// not from the REST response, so both broadcast builders must carry scope_name +// or a new message shows no region until the next REST refresh replaces it. +func TestBroadcastMapsCarryScopeName(t *testing.T) { + db := setupChannelScopeDB(t) + defer db.Close() + store := NewPacketStore(db, nil) + if err := store.Load(); err != nil { + t.Fatalf("store.Load: %v", err) + } + + assertBroadcast := func(t *testing.T, maps []map[string]interface{}, hash string, want interface{}) { + t.Helper() + found := false + for _, bm := range maps { + if bm["hash"] != hash { + continue + } + found = true + b, err := json.Marshal(bm) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var decoded map[string]interface{} + if err := json.Unmarshal(b, &decoded); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if got, ok := decoded["scope_name"]; !ok || got != want { + t.Errorf("top-level scope_name = %#v (present=%v), want %#v", got, ok, want) + } + pkt, _ := decoded["packet"].(map[string]interface{}) + if got, ok := pkt["scope_name"]; !ok || got != want { + t.Errorf("packet.scope_name = %#v (present=%v), want %#v", got, ok, want) + } + } + if !found { + t.Fatalf("no broadcast map for %s", hash) + } + } + + t.Run("IngestNewFromDB", func(t *testing.T) { + txMax := store.MaxTransmissionID() + now := time.Now().UTC() + res, err := db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash, scope_name) + VALUES ('AABB', 'cccccccccccccc04', ?, 0, 5, '{"type":"CHAN","channel":"#scopetest","text":"Dave: live"}', '#scopetest', '#belgium')`, + now.Format(time.RFC3339)) + if err != nil { + t.Fatalf("insert tx: %v", err) + } + txID, _ := res.LastInsertId() + if _, err := db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp) + VALUES (?, 1, 7.0, -95, '[]', ?)`, txID, now.Unix()); err != nil { + t.Fatalf("insert obs: %v", err) + } + maps, _ := store.IngestNewFromDB(txMax, 100) + assertBroadcast(t, maps, "cccccccccccccc04", "#belgium") + }) + + t.Run("IngestNewObservations", func(t *testing.T) { + obsMax := db.GetMaxObservationID() + var txID int + if err := db.conn.QueryRow(`SELECT id FROM transmissions WHERE hash = ?`, chScopeHashUnknown).Scan(&txID); err != nil { + t.Fatalf("lookup tx: %v", err) + } + if _, err := db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp) + VALUES (?, 1, 3.0, -110, '["aa"]', ?)`, txID, time.Now().Unix()); err != nil { + t.Fatalf("insert obs: %v", err) + } + maps := store.IngestNewObservations(obsMax, 100) + assertBroadcast(t, maps, chScopeHashUnknown, "") + }) +} diff --git a/cmd/server/db.go b/cmd/server/db.go index dac63be8..9ed1b9af 100644 --- a/cmd/server/db.go +++ b/cmd/server/db.go @@ -2094,10 +2094,16 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region . idPlaceholders[i] = "?" obsArgs[i] = id } + // #1851: scope_name lives on the transmission row, so appending it as the + // last selected column is safe for both query shapes. + scopeNameCol := "" + if db.hasScopeName { + scopeNameCol = ", t.scope_name" + } var obsSQL string if db.isV3 { obsSQL = `SELECT o.id, t.id, t.hash, t.decoded_json, t.first_seen, - obs.id, obs.name, o.snr, o.path_json, o.timestamp + obs.id, obs.name, o.snr, o.path_json, o.timestamp` + scopeNameCol + ` FROM observations o JOIN transmissions t ON t.id = o.transmission_id LEFT JOIN observers obs ON obs.rowid = o.observer_idx @@ -2105,7 +2111,7 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region . ORDER BY o.id ASC` } else { obsSQL = `SELECT o.id, t.id, t.hash, t.decoded_json, t.first_seen, - o.observer_id, o.observer_name, o.snr, o.path_json, o.timestamp + o.observer_id, o.observer_name, o.snr, o.path_json, o.timestamp` + scopeNameCol + ` FROM observations o JOIN transmissions t ON t.id = o.transmission_id WHERE t.id IN (` + strings.Join(idPlaceholders, ",") + `) @@ -2130,7 +2136,12 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region . var pktHash, dj, fs, obsID, obsName, pathJSON sql.NullString var snr sql.NullFloat64 var obsTs sql.NullInt64 - rows.Scan(&pktID, &txID, &pktHash, &dj, &fs, &obsID, &obsName, &snr, &pathJSON, &obsTs) + var scopeName sql.NullString + scanArgs := []interface{}{&pktID, &txID, &pktHash, &dj, &fs, &obsID, &obsName, &snr, &pathJSON, &obsTs} + if db.hasScopeName { + scanArgs = append(scanArgs, &scopeName) + } + rows.Scan(scanArgs...) if !dj.Valid { continue } @@ -2181,6 +2192,7 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region . "observers": []string{}, "hops": hops, "snr": nullFloat(snr), + "scope_name": nullStr(scopeName), }, Repeats: 1, } diff --git a/cmd/server/store.go b/cmd/server/store.go index 6b0b2546..5a0a84e0 100644 --- a/cmd/server/store.go +++ b/cmd/server/store.go @@ -2968,6 +2968,7 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac "path_json": strOrNil(obs.PathJSON), "direction": strOrNil(obs.Direction), "observation_count": tx.ObservationCount, + "scope_name": strPtrOrNil(tx.ScopeName), } // Use decode-window resolved path for broadcast (never from struct) if broadcastRP != nil { @@ -3245,6 +3246,7 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) []map[string] "path_json": strOrNil(obs.PathJSON), "direction": strOrNil(obs.Direction), "observation_count": tx.ObservationCount, + "scope_name": strPtrOrNil(tx.ScopeName), } // Use decode-window resolved path for broadcast if obsRPMap != nil { @@ -5665,6 +5667,7 @@ func (s *PacketStore) GetChannelMessages(channelHash string, limit, offset int, "observers": observers, "hops": hops, "snr": snrVal, + "scope_name": strPtrOrNil(tx.ScopeName), }, Repeats: 1, Observers: observers, diff --git a/docs/api-spec.md b/docs/api-spec.md index 3d2c19cd..09d38889 100644 --- a/docs/api-spec.md +++ b/docs/api-spec.md @@ -1169,13 +1169,24 @@ Messages for a specific channel. "repeats": number, // dedup count "observers": [string], // observer names "hops": number, - "snr": number | null + "snr": number | null, + "scope_name": string | null // region scope, see below } ], "total": number // total deduplicated messages } ``` +`scope_name` is the transmission's region scope (`transmissions.scope_name`), the same field `/api/packets` returns: + +| Value | Meaning | +|-------|---------| +| `null` | No transport code: the message was not region-scoped. Also `null` when the database has no `scope_name` column yet (ingestor migration not run). | +| `""` | Transport-scoped, but the ingestor could not match it to a single region: no region key matched, or several matched with no single operator-configured key among them. | +| `"#name"` | The matched region name. | + +The same field is on the WebSocket `packet` broadcast, both top-level and inside `packet`. + --- ## GET /api/analytics/rf diff --git a/public/channels.js b/public/channels.js index b4688932..964bfa87 100644 --- a/public/channels.js +++ b/public/channels.js @@ -335,6 +335,20 @@ return str.replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"').replace(/'/g, '''); } + // #1851: region scope from the packet's transport code (scope_name). + // null = no transport code, so no chip; '' = a transport code the ingestor + // could not match to a single region (no key matched, or several did with no + // operator-configured winner); otherwise the matched region name. Chip + // classes are the Scope Audit's (scope-audit.css), so a scope looks the same + // everywhere. + function messageScopeChipHtml(scopeName) { + if (scopeName == null) return ''; + if (scopeName === '') { + return 'unknown scope'; + } + return '' + escapeHtml(scopeName) + ''; + } + function truncate(str, len) { if (!str) return ''; return str.length > len ? str.slice(0, len) + '…' : str; @@ -579,7 +593,10 @@ // M5: Cache invalidation — if total candidate count changed, re-decrypt everything var totalCandidates = candidates.length; - var needFullDecrypt = (totalCandidates !== cachedCount) || opts.forceFullDecrypt; + // #1851: a cache written before messages carried scope_name would keep + // those messages chipless on the delta path, so decrypt them again. + var cacheLacksScope = cachedMsgs.some(function (m) { return !('scope_name' in m); }); + var needFullDecrypt = (totalCandidates !== cachedCount) || opts.forceFullDecrypt || cacheLacksScope; // M5: Delta fetch — only decrypt packets newer than lastTs if (!needFullDecrypt && cachedMsgs.length > 0 && lastTs) { @@ -663,6 +680,7 @@ packetHash: c.packet.hash, packetId: c.packet.id, hops: d.path_len || 0, snr: c.packet.snr || null, observers: c.packet.observer_name ? [c.packet.observer_name] : [], + scope_name: c.packet.scope_name ?? null, repeats: 1 }); continue; @@ -679,6 +697,7 @@ packetHash: c.packet.hash, packetId: c.packet.id, hops: 0, snr: c.packet.snr || null, observers: c.packet.observer_name ? [c.packet.observer_name] : [], + scope_name: c.packet.scope_name ?? null, repeats: 1 }); } else { @@ -1417,6 +1436,8 @@ var pktId = m.data?.id || null; var snr = m.data?.snr ?? m.data?.packet?.snr ?? payload.SNR ?? null; var observer = m.data?.packet?.observer_name || m.data?.observer || null; + // ?? not ||: '' (transport-scoped, region unmatched) must survive. + var scopeName = m.data?.scope_name ?? m.data?.packet?.scope_name ?? null; // Update channel list entry — only once per unique packet hash var isFirstObservation = pktHash && !seenHashes.has(pktHash + ':' + channelKey); @@ -1468,6 +1489,7 @@ observers: observer ? [observer] : [], hops: payload.path_len || 0, snr: snr, + scope_name: scopeName, // #1498: mark as WS-pushed so a later REST replacement // (selectChannel / refreshMessages) can merge instead of // stomp. Without this flag the REST response wipes any @@ -2258,6 +2280,8 @@ if (msg.observers?.length > 1) meta.push(`${msg.observers.length} observers`); if (msg.hops > 0) meta.push(`${msg.hops} hops`); if (msg.snr !== null && msg.snr !== undefined) meta.push(`SNR ${msg.snr}`); + const scopeChip = messageScopeChipHtml(msg.scope_name); + if (scopeChip) meta.push(scopeChip); const safeId = btoa(encodeURIComponent(sender)); // #1367: emit BOTH the new chat-app class names (.ch-message / diff --git a/test-all.sh b/test-all.sh index 9e15b728..2e5950a9 100755 --- a/test-all.sh +++ b/test-all.sh @@ -125,6 +125,7 @@ node test-issue-1836-crossnav-case-normalization.js node test-issue-1843-node-qr-quiet-zone.js node test-issue-1846-observers-width.js node test-issue-1849-trace-hashbytes.js +node test-issue-1851-channel-message-scope.js node test-issue-1868-control-decode.js node test-issue-1890-og-url.js node test-issue-1956-release-routing.js diff --git a/test-issue-1851-channel-message-scope.js b/test-issue-1851-channel-message-scope.js new file mode 100644 index 00000000..4352fa00 --- /dev/null +++ b/test-issue-1851-channel-message-scope.js @@ -0,0 +1,330 @@ +/* #1851: each channel message shows the region scope it was sent with. + * + * scope_name has three states (transmissions.scope_name): null = the packet + * carried no transport code, '' = it carried one the ingestor could not match + * to a single region, '#name' = matched region. A message reaches the Channels view by + * three routes (REST /api/channels/{hash}/messages, the WebSocket broadcast, + * and client-side decryption of /api/packets rows), so each route must keep + * the field, and the render must keep the '' state apart from null. + * + * Loads the real public/channels.js in a vm sandbox and reads the HTML it + * writes into #chMessages. + */ +'use strict'; +const vm = require('vm'); +const fs = require('fs'); +const assert = require('assert'); +const { createCipheriv, createHmac, createHash } = require('crypto'); + +let passed = 0, failed = 0; +const pending = []; +function test(name, fn) { + const done = (err) => { + if (err) { failed++; console.log(` ❌ ${name}: ${err.message}`); } + else { passed++; console.log(` ✅ ${name}`); } + }; + try { + const out = fn(); + if (out && typeof out.then === 'function') { pending.push(out.then(() => done(), done)); return; } + done(); + } catch (e) { done(e); } +} + +function makeChannelsSandbox(apiImpl) { + const dom = {}; + function makeEl(id) { + if (dom[id]) return dom[id]; + dom[id] = { + id, innerHTML: '', textContent: '', value: '', + scrollTop: 0, scrollHeight: 100, clientHeight: 80, + style: {}, dataset: {}, + classList: { add() {}, remove() {}, toggle() {}, contains() { return false; } }, + addEventListener() {}, removeEventListener() {}, + querySelector() { return null; }, querySelectorAll() { return []; }, + getBoundingClientRect() { return { left: 0, bottom: 0, width: 0 }; }, + setAttribute() {}, removeAttribute() {}, focus() {}, + }; + return dom[id]; + } + const headerText = { textContent: '' }; + makeEl('chHeader').querySelector = (sel) => (sel === '.ch-header-text' ? headerText : null); + ['chMessages', 'chList', 'chScrollBtn', 'chAriaLive', 'chBackBtn', 'chRegionFilter'].forEach(makeEl); + const layout = { classList: { add() {}, remove() {}, contains() { return false; } } }; + const appEl = { + innerHTML: '', + querySelector(sel) { return sel === '.ch-layout' ? layout : makeEl(sel); }, + addEventListener() {}, + }; + const storage = {}; + const ctx = { + window: { addEventListener() {}, dispatchEvent() {} }, + document: { + readyState: 'complete', + createElement: () => ({ id: '', textContent: '', innerHTML: '' }), + head: { appendChild() {} }, + body: { appendChild() {}, removeChild() {}, contains() { return false; } }, + documentElement: { getAttribute: () => null, setAttribute() {} }, + getElementById: makeEl, + querySelector: (sel) => (sel === '.ch-layout' ? layout : null), + querySelectorAll: () => [], + addEventListener() {}, removeEventListener() {}, + }, + console, Date, Math, Array, Object, String, Number, JSON, RegExp, Error, TypeError, + Map, Set, Promise, URLSearchParams, Infinity, parseInt, parseFloat, isNaN, isFinite, + encodeURIComponent, decodeURIComponent, + setTimeout: () => 0, clearTimeout() {}, setInterval: () => 0, clearInterval() {}, + performance: { now: () => Date.now() }, + localStorage: { + getItem: (k) => (k in storage ? storage[k] : null), + setItem: (k, v) => { storage[k] = String(v); }, + removeItem: (k) => { delete storage[k]; }, + }, + location: { hash: '' }, + getHashParams: () => new URLSearchParams(''), + history: { replaceState() {} }, + matchMedia: () => ({ matches: false }), + MutationObserver: function () { this.observe = () => {}; this.disconnect = () => {}; }, + RegionFilter: { init() {}, onChange() { return () => {}; }, offChange() {}, getRegionParam() { return ''; } }, + debouncedOnWS: (fn) => fn, onWS() {}, offWS() {}, + api: apiImpl, + CLIENT_TTL: { observers: 120000, channels: 15000, channelMessages: 10000, nodeDetail: 10000 }, + ROLE_EMOJI: {}, ROLE_LABELS: {}, + timeAgo: () => '1m ago', + registerPage: (name, handlers) => { ctx._pageHandlers = handlers; }, + btoa: (s) => Buffer.from(String(s), 'utf8').toString('base64'), + atob: (s) => Buffer.from(String(s), 'base64').toString('utf8'), + crypto: { subtle: require('crypto').webcrypto.subtle }, + TextEncoder, TextDecoder, Uint8Array, + }; + ctx.window.matchMedia = ctx.matchMedia; + vm.createContext(ctx); + for (const file of ['public/vendor/aes-ecb.js', 'public/channel-decrypt.js', 'public/channels.js']) { + vm.runInContext(fs.readFileSync(file, 'utf8'), ctx); + for (const k of Object.keys(ctx.window)) ctx[k] = ctx.window[k]; + } + ctx._pageHandlers.init(appEl); + return { ctx, dom }; +} + +function listApi(extra) { + return (path) => { + if (path.indexOf('/observers') === 0) return Promise.resolve({ observers: [] }); + if (path.indexOf('/channels') === 0 && path.indexOf('/messages') === -1) { + return Promise.resolve({ channels: [{ hash: 'general', name: 'general', messageCount: 4, lastActivity: null }] }); + } + return extra(path); + }; +} + +// Splits the rendered message list into one HTML chunk per message so each +// assertion is about a specific message, not about the page as a whole. +function messageChunks(html) { + return html.split('
').slice(1); +} + +// A GRP_TXT packet ChannelDecrypt can really verify and decrypt: AES-128-ECB +// under SHA-256(channelName)[:16], 2-byte HMAC-SHA256 MAC over the ciphertext +// with key + 16 zero bytes. Same construction as buildEncryptedGrpTxt in +// test-channel-live-decrypt.js, which runs on require and so cannot be imported. +function encryptedGrpTxt(channelName, sender, message) { + const key = createHash('sha256').update(channelName).digest().slice(0, 16); + const channelHash = createHash('sha256').update(key).digest()[0]; + const text = `${sender}: ${message}`; + const pt = Buffer.alloc(Math.ceil((5 + Buffer.byteLength(text, 'utf8') + 1) / 16) * 16); + pt.writeUInt32LE(1788256800, 0); + pt.write(text, 5, 'utf8'); + const cipher = createCipheriv('aes-128-ecb', key, null); + cipher.setAutoPadding(false); + const ct = Buffer.concat([cipher.update(pt), cipher.final()]); + const mac = createHmac('sha256', Buffer.concat([key, Buffer.alloc(16)])).update(ct).digest().slice(0, 2); + return { + keyHex: key.toString('hex'), + channelHash, + decodedJson: JSON.stringify({ type: 'GRP_TXT', channelHash, mac: mac.toString('hex'), encryptedData: ct.toString('hex') }), + }; +} + +// Three encrypted packets on one channel, one per scope_name state. +function encryptedChannelPackets(channelName) { + const rows = [ + { sender: 'Alice', text: 'secret matched', scope: '#belgium' }, + { sender: 'Bob', text: 'secret unmatched', scope: '' }, + { sender: 'Carol', text: 'secret plain', scope: null }, + ]; + let fx; + const packets = rows.map((r, i) => { + fx = encryptedGrpTxt(channelName, r.sender, r.text); + return { id: 10 + i, hash: 'e' + i, first_seen: `2026-09-01T10:0${i}:00Z`, scope_name: r.scope, decoded_json: fx.decodedJson }; + }); + return { packets, keyHex: fx.keyHex, channelHash: fx.channelHash, lastTs: packets[packets.length - 1].first_seen }; +} + +function packetsApi(packets) { + return listApi((path) => Promise.resolve(path.indexOf('/packets?') === 0 ? { packets } : {})); +} + +const SCOPE_CHIP =/]*>([^<]*)<\/span>/; +const UNKNOWN_CHIP = /]*>unknown scope<\/span>/; + +console.log('\n=== #1851: channel message region scope ==='); + +test('REST messages render a scope chip per state: name, unknown, none', async () => { + const { ctx, dom } = makeChannelsSandbox(listApi((path) => { + if (path.indexOf('/channels/general/messages') === 0) { + return Promise.resolve({ messages: [ + { sender: 'Alice', text: 'matched', timestamp: '2026-09-01T10:00:00Z', packetHash: 'h1', scope_name: '#belgium' }, + { sender: 'Bob', text: 'unmatched', timestamp: '2026-09-01T10:01:00Z', packetHash: 'h2', scope_name: '' }, + { sender: 'Carol', text: 'plain flood', timestamp: '2026-09-01T10:02:00Z', packetHash: 'h3', scope_name: null }, + { sender: 'Dave', text: 'older server', timestamp: '2026-09-01T10:03:00Z', packetHash: 'h4' }, + ] }); + } + return Promise.resolve({}); + })); + for (let i = 0; i < 10; i++) await Promise.resolve(); + await ctx.window._channelsSelectChannelForTest('general'); + const chunks = messageChunks(dom.chMessages.innerHTML); + assert.strictEqual(chunks.length, 4, 'expected 4 rendered messages'); + + const named = chunks[0].match(SCOPE_CHIP); + assert.ok(named, 'matched-region message must render a scope chip'); + assert.strictEqual(named[1], '#belgium'); + assert.ok(!UNKNOWN_CHIP.test(chunks[0]), 'matched-region message must not say unknown'); + + assert.ok(UNKNOWN_CHIP.test(chunks[1]), "scope_name '' must render the unknown-scope chip"); + assert.ok(!SCOPE_CHIP.test(chunks[1]), "scope_name '' must not render a named chip"); + // '' is stored both when no region key matched and when several matched + // with no operator-configured winner (cmd/ingestor/region_keys.go match), + // so the tooltip must not claim that no key matched. + const title = (chunks[1].match(/ch-msg-scope" title="([^"]*)"/) || [])[1]; + assert.strictEqual(title, 'Sent with a region scope that could not be matched to a single region on this instance'); + + assert.ok(!/ch-msg-scope/.test(chunks[2]), 'scope_name null must render no chip'); + assert.ok(!/ch-msg-scope/.test(chunks[3]), 'a message without scope_name must render no chip'); +}); + +test('scope name is escaped before it reaches innerHTML', async () => { + const { ctx, dom } = makeChannelsSandbox(listApi((path) => { + if (path.indexOf('/channels/general/messages') === 0) { + return Promise.resolve({ messages: [ + { sender: 'Eve', text: 'x', timestamp: '2026-09-01T10:00:00Z', packetHash: 'h1', scope_name: '' }, + ] }); + } + return Promise.resolve({}); + })); + for (let i = 0; i < 10; i++) await Promise.resolve(); + await ctx.window._channelsSelectChannelForTest('general'); + const html = dom.chMessages.innerHTML; + assert.ok(!html.includes(' { + const { ctx, dom } = makeChannelsSandbox(listApi(() => Promise.resolve({ messages: [] }))); + ctx.window._channelsSetStateForTest({ + selectedHash: 'general', + channels: [{ hash: 'general', name: 'general', messageCount: 0, lastActivityMs: 0 }], + messages: [], + }); + const packet = (hash, text, scope) => ({ + type: 'packet', + data: { + hash, scope_name: scope, + decoded: { header: { payloadTypeName: 'GRP_TXT' }, payload: { channel: 'general', text } }, + packet: { hash, observer_name: 'Obs', scope_name: scope }, + }, + }); + ctx.window._channelsProcessWSBatchForTest([ + packet('w1', 'Alice: live matched', '#belgium'), + packet('w2', 'Bob: live unmatched', ''), + packet('w3', 'Carol: live plain', null), + ], null); + const state = ctx.window._channelsGetStateForTest(); + assert.deepStrictEqual(Array.from(state.messages, (m) => m.scope_name), ['#belgium', '', null]); + const chunks = messageChunks(dom.chMessages.innerHTML); + assert.strictEqual(chunks.length, 3, 'expected 3 rendered messages'); + assert.strictEqual((chunks[0].match(SCOPE_CHIP) || [])[1], '#belgium'); + assert.ok(UNKNOWN_CHIP.test(chunks[1]), 'live message with scope_name "" must render the unknown chip'); + assert.ok(!/ch-msg-scope/.test(chunks[2]), 'live message with scope_name null must render no chip'); +}); + +test('client-side decrypted channel keeps scope_name from the /api/packets row', async () => { + const { ctx, dom } = makeChannelsSandbox(listApi((path) => { + if (path.indexOf('/packets?') === 0) { + return Promise.resolve({ packets: [ + { id: 1, hash: 'd1', first_seen: '2026-09-01T10:00:00Z', scope_name: '#belgium', + decoded_json: JSON.stringify({ type: 'CHAN', channel: '#priv', sender: 'Alice', text: 'Alice: decrypted' }) }, + { id: 2, hash: 'd2', first_seen: '2026-09-01T10:01:00Z', scope_name: '', + decoded_json: JSON.stringify({ type: 'CHAN', channel: '#priv', sender: 'Bob', text: 'Bob: decrypted' }) }, + ] }); + } + return Promise.resolve({}); + })); + for (let i = 0; i < 10; i++) await Promise.resolve(); + await ctx.window._channelsSelectChannelForTest('user:#priv', { + userKey: '00112233445566778899aabbccddeeff', channelHashByte: 1, channelName: '#priv', + }); + const state = ctx.window._channelsGetStateForTest(); + assert.deepStrictEqual(Array.from(state.messages, (m) => m.scope_name), ['#belgium', '']); + const chunks = messageChunks(dom.chMessages.innerHTML); + assert.strictEqual((chunks[0].match(SCOPE_CHIP) || [])[1], '#belgium'); + assert.ok(UNKNOWN_CHIP.test(chunks[1]), 'decrypted message with scope_name "" must render the unknown chip'); +}); + +test('packets decrypted with the stored key keep scope_name from the /api/packets row', async () => { + const fx = encryptedChannelPackets('#secret'); + const { ctx, dom } = makeChannelsSandbox(packetsApi(fx.packets)); + for (let i = 0; i < 10; i++) await Promise.resolve(); + await ctx.window._channelsSelectChannelForTest('user:#secret', { + userKey: fx.keyHex, channelHashByte: fx.channelHash, channelName: '#secret', + }); + const state = ctx.window._channelsGetStateForTest(); + // Sender and text only exist after a MAC check and AES decrypt, so these + // messages came through the encrypted branch, not the already-decrypted one. + assert.deepStrictEqual(Array.from(state.messages, (m) => m.sender + ': ' + m.text), + ['Alice: secret matched', 'Bob: secret unmatched', 'Carol: secret plain']); + assert.deepStrictEqual(Array.from(state.messages, (m) => m.scope_name), ['#belgium', '', null]); + const chunks = messageChunks(dom.chMessages.innerHTML); + assert.strictEqual(chunks.length, 3, 'expected 3 rendered messages'); + assert.strictEqual((chunks[0].match(SCOPE_CHIP) || [])[1], '#belgium'); + assert.ok(UNKNOWN_CHIP.test(chunks[1]), 'decrypted message with scope_name "" must render the unknown chip'); + assert.ok(!/ch-msg-scope/.test(chunks[2]), 'decrypted message with scope_name null must render no chip'); +}); + +test('a decrypt cache written before scope_name existed is decrypted again', async () => { + const fx = encryptedChannelPackets('#secret'); + const { ctx } = makeChannelsSandbox(packetsApi(fx.packets)); + // Same candidate count and last timestamp as the API returns, so without the + // missing key the delta path would find nothing new and serve this cache. + ctx.ChannelDecrypt.setCache('#secret', fx.packets.map((p, i) => ({ + sender: 'Old', text: 'cached ' + i, timestamp: p.first_seen, packetHash: p.hash, packetId: p.id, + hops: 0, snr: null, observers: [], repeats: 1, + })), fx.lastTs, fx.packets.length); + for (let i = 0; i < 10; i++) await Promise.resolve(); + await ctx.window._channelsSelectChannelForTest('user:#secret', { + userKey: fx.keyHex, channelHashByte: fx.channelHash, channelName: '#secret', + }); + const state = ctx.window._channelsGetStateForTest(); + assert.deepStrictEqual(Array.from(state.messages, (m) => m.scope_name), ['#belgium', '', null]); + assert.deepStrictEqual(Array.from(ctx.ChannelDecrypt.getCache('#secret').messages, (m) => m.scope_name), + ['#belgium', '', null], 'the rewritten cache must carry scope_name'); +}); + +test('a decrypt cache that already carries scope_name is still served by the delta path', async () => { + const fx = encryptedChannelPackets('#secret'); + const { ctx } = makeChannelsSandbox(packetsApi(fx.packets)); + ctx.ChannelDecrypt.setCache('#secret', fx.packets.map((p, i) => ({ + sender: 'Old', text: 'cached ' + i, timestamp: p.first_seen, packetHash: p.hash, packetId: p.id, + hops: 0, snr: null, observers: [], scope_name: null, repeats: 1, + })), fx.lastTs, fx.packets.length); + for (let i = 0; i < 10; i++) await Promise.resolve(); + await ctx.window._channelsSelectChannelForTest('user:#secret', { + userKey: fx.keyHex, channelHashByte: fx.channelHash, channelName: '#secret', + }); + const state = ctx.window._channelsGetStateForTest(); + assert.deepStrictEqual(Array.from(state.messages, (m) => m.text), ['cached 0', 'cached 1', 'cached 2']); +}); + +Promise.all(pending).then(() => { + console.log(`\n${passed} passed, ${failed} failed`); + if (failed > 0) process.exit(1); +});