feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728)

Implements #1727.

## What this adds

**Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage
feature. A roaming MeshCore **companion** radio (driven by the
open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA,
GPLv3) reports which nodes it heard directly, tagged with the phone's
GPS and the packet's SNR/RSSI. CoreScope ingests these into a new
`client_receptions` table and renders per-node **hex coverage** on the
Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`)
with a top-mobile-observers leaderboard.

Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only
node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by
the companion app for friendly names.

## Opt-in — default OFF (zero impact on existing deployments)

The whole feature is gated behind one config flag, **disabled by
default**:

```jsonc
"clientRxCoverage": { "enabled": false }
```

When disabled (the default): the ingestor writes **no**
`client_receptions`; the three coverage endpoints return a clean
**404**; the UI hides the Coverage nav link, the `#/rx-coverage` route,
and the Reach-page toggle. `/api/nodes/resolve` is always available (not
coverage-specific).

## How it works

```
companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets
                                                                      │
                                          ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key)
                                                                      │
              server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard
```

- **Direct-only capture:** records only what the companion heard itself
and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder)
for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded.
- **No new deps:** hexbins are a pure-Go pointy-top grid over Web
Mercator (`cmd/server/hexgrid.go`) computed at query time
(`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the
existing Leaflet.
- **Trust:** companion pubkey = identity; an EMQX ACL binds each client
to publish only to its own `meshcore/client/{pubkey}/packets` topic.
Payload contract in `docs/client-rx-coverage.md`.

## How to enable / try it

1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and
restart server + ingestor.
2. Point an EMQX (or any broker) listener so a client can publish to
`meshcore/client/<pubkey>/packets`; the ingestor already subscribes
under `meshcore/#`.
3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on
an Android phone paired (BLE) to a MeshCore companion — it captures
heard nodes + GPS and publishes.
4. View results: per-node Reach page → toggle **coverage**, or the
**Coverage** dashboard at `#/rx-coverage`.

## What's where

- **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go`
(`client_receptions` + `client_observers` schema), `main.go` (gated
dispatch), `config.go` (flag).
- **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go`
(endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid),
`node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go`
(wiring + flag + `/api/config/client` field).
- **Frontend:** `public/rx-coverage.js` (dashboard),
`node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach
toggle, flag-gated), `roles.js` (reads the flag, hides nav when off).
- **Docs:** `docs/client-rx-coverage.md`.

## Testing

- Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test
./...` — green, including new gate tests (`coverage_gate_test.go` in
both: off → no rows / 404, on → works) and the rx-coverage / resolve /
hexgrid suites.
- JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js`
(wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is
wired into the e2e job and **skips when `clientRxCoverage` is
disabled**, so it's safe under the default-off config.

## Notes for reviewers

- The four new routes are registered in
`cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness
ratchet), matching how other not-yet-spec'd routes are tracked. Happy to
write full OpenAPI spec entries instead if you prefer.
- Commits are split per layer (ingestor / server endpoints / resolve /
frontend / CI) for review.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Erwin Fiten <e.fiten@opteco.be>
This commit is contained in:
authored and GitHub committed 2026-06-19 11:37:16 -07:00
1 parent df28efaed9
commit 22fe929da2
41 files changed
+3887 -188

No files matched your search

+3
View File
@@ -138,6 +138,8 @@ jobs:
node test-issue-1509-nav-active-bg.js
node test-issue-1509-detect-preset.js
node test-live.js
node test-coverage-gate.js
node test-node-reach-coverage.js
node test-issue-1107-live-layout.js
node test-issue-1532-live-fullscreen.js
node test-issue-1619-feed-detail-card-draggable.js
@@ -464,6 +466,7 @@ jobs:
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-channels-ws-race-1498-e2e.js 2>&1 | tee -a e2e-output.txt
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1487-byop-modal-layout-e2e.js 2>&1 | tee -a e2e-output.txt
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1630-reach-mobile-e2e.js 2>&1 | tee -a e2e-output.txt
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-node-reach-coverage-e2e.js 2>&1 | tee -a e2e-output.txt
CHROMIUM_REQUIRE=1 BASE_URL=http://localhost:13581 node test-issue-1640-compare-discovery-e2e.js 2>&1 | tee -a e2e-output.txt
# #1616: slide-over focus-restore flake-gate. Runs the slide-over
+4
View File
@@ -33,3 +33,7 @@ corescope-server
cmd/server/server
# Local-only planning and design files
docs/superpowers/
# Environment-specific deploy scripts — live only on the deploy host, not tracked
deploy-live.sh
deploy-staging.sh
+223
View File
@@ -0,0 +1,223 @@
package main
import (
"log"
"regexp"
"strings"
"time"
"github.com/meshcore-analyzer/packetpath"
)
// clientPubkeyRe validates the companion pubkey taken from the MQTT topic
// (meshcore/client/<PUBLIC_KEY>/packets). A no-ACL broker would let a client
// publish under an arbitrary topic segment (e.g. "!@#$"), so we reject anything
// that is not lowercase hex before it reaches client_receptions/client_observers.
// Mirrors the server-side hexPrefixRe (cmd/server/node_resolve.go).
var clientPubkeyRe = regexp.MustCompile(`^[0-9a-f]{2,64}$`)
// handleClientPacket processes a packet from the mobile client RX topic
// (meshcore/client/{PUBLIC_KEY}/packets). Unlike observer packets, a roaming
// companion reports WHERE it directly heard a node, so we write a
// client_receptions row and never touch the observers/observations tables.
// rxPubkey is the companion pubkey from the topic (ACL-bound by the broker).
func handleClientPacket(store *Store, tag, rxPubkey string, msg map[string]interface{}, channelKeys map[string]string) {
// The companion identity IS the (ACL-bound) topic pubkey. Reject non-hex
// topic segments so a no-ACL broker can't pollute the coverage tables, and
// never fall back to a payload-supplied id (that would defeat the ACL trust
// model — see docs/client-rx-coverage.md).
rxPubkey = strings.ToLower(strings.TrimSpace(rxPubkey))
if !clientPubkeyRe.MatchString(rxPubkey) {
log.Printf("MQTT [%s] client: invalid pubkey %.8q, dropping", tag, rxPubkey)
return
}
rawHex, _ := msg["raw"].(string)
if rawHex == "" {
return
}
gps, ok := msg["gps"].(map[string]interface{})
if !ok {
return // a client packet without a GPS fix is not coverage; drop
}
lat, latOK := toFloat64(gps["lat"])
lon, lonOK := toFloat64(gps["lon"])
if !latOK || !lonOK {
return
}
var accPtr *float64
if acc, ok := toFloat64(gps["acc_m"]); ok {
accPtr = &acc
}
decoded, err := DecodePacket(rawHex, channelKeys, false)
if err != nil {
log.Printf("MQTT [%s] client decode error: %v", tag, err)
return
}
direction := ""
if v, ok := msg["direction"].(string); ok {
direction = v
} else if v, ok := msg["Direction"].(string); ok {
direction = v
}
var snrPtr *float64
if f, ok := toFloat64(firstPresent(msg, "SNR", "snr")); ok {
snrPtr = &f
}
var rssiPtr *int
if f, ok := toFloat64(firstPresent(msg, "RSSI", "rssi")); ok {
v := int(f)
rssiPtr = &v
}
rxAt, _ := resolveRxTime(msg, tag)
isAdvert := decoded.Header.PayloadTypeName == "ADVERT"
rec, ok := buildClientReception(
rxPubkey,
direction, decoded.Header.RouteType, decoded.Path.Hops, decoded.Payload.PubKey, isAdvert,
snrPtr, rssiPtr, lat, lon, accPtr, rxAt, time.Now().UTC().Format(time.RFC3339),
)
if !ok {
return
}
if _, err := store.InsertClientReception(rec); err != nil {
log.Printf("MQTT [%s] client_reception insert: %v", tag, err)
}
// Remember the companion's self-reported name (sent as "origin") so the
// leaderboard can show a name even if this companion never advertised.
if name := stringField(msg, "origin"); name != "" {
if err := store.UpsertClientObserver(rec.RxPubkey, name, time.Now().UTC().Format(time.RFC3339)); err != nil {
log.Printf("MQTT [%s] client_observer upsert: %v", tag, err)
}
}
}
// UpsertClientObserver records/updates a mobile client's self-reported name.
// All writes live in the ingestor (read/write invariant #1283).
func (s *Store) UpsertClientObserver(pubkey, name, ts string) error {
if pubkey == "" || name == "" {
return nil
}
_, err := s.db.Exec(`
INSERT INTO client_observers (pubkey, name, last_seen) VALUES (?,?,?)
ON CONFLICT(pubkey) DO UPDATE SET name = excluded.name, last_seen = excluded.last_seen`,
strings.ToLower(pubkey), name, ts)
return err
}
// firstPresent returns the first present value among the given keys.
func firstPresent(msg map[string]interface{}, keys ...string) interface{} {
for _, k := range keys {
if v, ok := msg[k]; ok {
return v
}
}
return nil
}
// stringField returns msg[key] as a string, or "" if absent/not a string.
func stringField(msg map[string]interface{}, key string) string {
if v, ok := msg[key].(string); ok {
return v
}
return ""
}
// ClientReception is one mobile RX coverage point: a companion (RxPubkey)
// directly heard a node (HeardKey) at a GPS position. Hex binning is done
// server-side from Lat/Lon at query time, so no cell id is stored here.
type ClientReception struct {
RxPubkey string
HeardKey string
HeardKeyLen int
RSSI *int
SNR *float64
Lat float64
Lon float64
PosAccM *float64
RxAt string
IngestedAt string
Src string
}
// deriveHeardKey applies the RX capture HARD RULE: record only what the
// companion heard itself and directly.
// - direction must be "rx".
// - hops present AND a FLOOD route → the directly-heard node is the LAST hop
// (path[len-1] = the forwarder that just transmitted; each FLOOD forwarder
// appends its hash to the end). 1-byte (2 hex char) prefixes are rejected.
// - hops present on a DIRECT route → NOT attributable: direct forwarders
// consume the next hop from the FRONT (firmware Mesh.cpp removeSelfFromPath),
// so path[len-1] is the route's destination-side end, not who was heard.
// - hops empty + isAdvert → the 0-hop advertiser, by its full pubkey.
// - otherwise → not attributable (ok=false).
//
// Returns (heardKey lowercased, keylenBytes, src, ok).
func deriveHeardKey(direction string, routeType int, hops []string, advertPubkey string, isAdvert bool) (string, int, string, bool) {
if !strings.EqualFold(direction, "rx") {
return "", 0, "", false
}
if len(hops) > 0 {
// FLOOD routes (TRANSPORT_FLOOD 0, FLOOD 1) APPEND each forwarder's hash to
// the END of the path, so path[last] is the immediate RF transmitter. DIRECT
// routes (2, 3) consume the next hop from the FRONT, so path[last] is the
// route's destination-side end, NOT who was heard.
if routeType != packetpath.RouteTransportFlood && routeType != packetpath.RouteFlood { // direct route: path[last] is not the transmitter
return "", 0, "", false
}
last := strings.ToLower(strings.TrimSpace(hops[len(hops)-1]))
keylen := len(last) / 2
if keylen < 2 { // exclude 1-byte (collision-prone), matching Reach
return "", 0, "", false
}
return last, keylen, "rxlog", true
}
if isAdvert && advertPubkey != "" {
pk := strings.ToLower(strings.TrimSpace(advertPubkey))
return pk, len(pk) / 2, "advert", true
}
return "", 0, "", false
}
// buildClientReception validates inputs and assembles a ClientReception, or
// returns ok=false when the packet is not attributable / out of range.
func buildClientReception(
rxPubkey, direction string, routeType int, hops []string, advertPubkey string, isAdvert bool,
snr *float64, rssi *int, lat, lon float64, posAccM *float64, rxAt, ingestedAt string,
) (*ClientReception, bool) {
if rxPubkey == "" || rxAt == "" {
return nil, false
}
if lat < -90 || lat > 90 || lon < -180 || lon > 180 {
return nil, false
}
heardKey, keylen, src, ok := deriveHeardKey(direction, routeType, hops, advertPubkey, isAdvert)
if !ok {
return nil, false
}
return &ClientReception{
RxPubkey: strings.ToLower(rxPubkey), HeardKey: heardKey, HeardKeyLen: keylen,
RSSI: rssi, SNR: snr, Lat: lat, Lon: lon, PosAccM: posAccM,
RxAt: rxAt, IngestedAt: ingestedAt, Src: src,
}, true
}
// InsertClientReception writes one coverage row. Idempotent via the
// UNIQUE(rx_pubkey, heard_key, rx_at) constraint; returns ins=false when the
// row already existed. All writes live in the ingestor (read/write invariant #1283).
func (s *Store) InsertClientReception(r *ClientReception) (bool, error) {
res, err := s.db.Exec(`
INSERT INTO client_receptions
(rx_pubkey, heard_key, heard_keylen, rssi, snr, lat, lon, pos_acc_m, rx_at, ingested_at, src)
VALUES (?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(rx_pubkey, heard_key, rx_at) DO NOTHING`,
r.RxPubkey, r.HeardKey, r.HeardKeyLen, r.RSSI, r.SNR, r.Lat, r.Lon, r.PosAccM, r.RxAt, r.IngestedAt, r.Src)
if err != nil {
return false, err
}
n, _ := res.RowsAffected()
return n > 0, nil
}
+334
View File
@@ -0,0 +1,334 @@
package main
import (
"database/sql"
"strings"
"testing"
"time"
"github.com/meshcore-analyzer/packetpath"
)
// TestPruneOldClientReceptions verifies the retention reaper bounds the coverage
// tables: rows older than the window (and stale companion names) are deleted,
// recent ones kept, and days=0 disables it.
func TestPruneOldClientReceptions(t *testing.T) {
s := newTestStore(t)
now := time.Now().UTC()
recent := now.AddDate(0, 0, -1).Format(time.RFC3339)
old := now.AddDate(0, 0, -40).Format(time.RFC3339)
const companion2 = "b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3"
s.InsertClientReception(&ClientReception{RxPubkey: testCompanionPK, HeardKey: "aabbcc", HeardKeyLen: 3, Lat: 51, Lon: 3.7, RxAt: recent, IngestedAt: "x", Src: "rxlog"})
s.InsertClientReception(&ClientReception{RxPubkey: testCompanionPK, HeardKey: "aabbcc", HeardKeyLen: 3, Lat: 51, Lon: 3.7, RxAt: old, IngestedAt: "x", Src: "rxlog"})
s.UpsertClientObserver(testCompanionPK, "Fresh", recent)
s.UpsertClientObserver(companion2, "Stale", old)
if n, _ := s.PruneOldClientReceptions(0); n != 0 {
t.Fatalf("days=0 must be a no-op, got %d", n)
}
n, err := s.PruneOldClientReceptions(7)
if err != nil {
t.Fatal(err)
}
if n != 1 {
t.Fatalf("expected 1 old reception pruned, got %d", n)
}
var recN, obsN int
s.db.QueryRow(`SELECT COUNT(*) FROM client_receptions`).Scan(&recN)
s.db.QueryRow(`SELECT COUNT(*) FROM client_observers`).Scan(&obsN)
if recN != 1 {
t.Fatalf("expected 1 reception remaining (recent), got %d", recN)
}
if obsN != 1 {
t.Fatalf("expected 1 observer remaining (fresh), got %d", obsN)
}
}
func TestClientReceptionsTableExists(t *testing.T) {
s := newTestStore(t)
cols := map[string]bool{}
rows, err := s.db.Query(`PRAGMA table_info(client_receptions)`)
if err != nil {
t.Fatalf("PRAGMA failed: %v", err)
}
defer rows.Close()
for rows.Next() {
var cid int
var name, ctype string
var notnull, pk int
var dflt any
if err := rows.Scan(&cid, &name, &ctype, &notnull, &dflt, &pk); err != nil {
t.Fatal(err)
}
cols[name] = true
}
for _, want := range []string{"id", "rx_pubkey", "heard_key", "heard_keylen", "rssi", "snr", "lat", "lon", "pos_acc_m", "rx_at", "ingested_at", "src"} {
if !cols[want] {
t.Errorf("missing column %q in client_receptions", want)
}
}
}
func crF(f float64) *float64 { return &f }
func crI(i int) *int { return &i }
// TestClientReceptionsCoverageQueryUsesIndex verifies #5/#18: the dominant
// per-node coverage query (sargable heard_key IN-list + bbox, mirroring
// cmd/server coverageHeardKeyCandidates) seeks the heard_key composite index
// rather than scanning the table. Without idx_client_recept_heard_geo the plan
// is "SCAN client_receptions".
func TestClientReceptionsCoverageQueryUsesIndex(t *testing.T) {
s := newTestStore(t)
q := `EXPLAIN QUERY PLAN SELECT lat, lon, snr, rssi, heard_key, rx_at
FROM client_receptions
WHERE heard_key IN (?,?,?) AND lat BETWEEN ? AND ? AND lon BETWEEN ? AND ?`
rows, err := s.db.Query(q, "aabbccddeeff00112233", "aabbcc", "aabb", 50.0, 52.0, 3.0, 4.0)
if err != nil {
t.Fatal(err)
}
defer rows.Close()
plan := ""
for rows.Next() {
var id, parent, notused int
var detail string
if err := rows.Scan(&id, &parent, &notused, &detail); err != nil {
t.Fatal(err)
}
plan += detail + "\n"
}
if !strings.Contains(plan, "USING INDEX idx_client_recept") {
t.Fatalf("coverage query should use a client_recept index, plan was:\n%s", plan)
}
if strings.Contains(plan, "SCAN client_receptions") {
t.Fatalf("coverage query should not full-scan, plan was:\n%s", plan)
}
}
// TestClientReceptionsRetentionUsesRxAtIndex verifies the retention reaper's
// DELETE ... WHERE rx_at < ? (and the leaderboard's rx_at window) seek the rx_at
// index rather than full-scanning under the writer lock (polish review).
func TestClientReceptionsRetentionUsesRxAtIndex(t *testing.T) {
s := newTestStore(t)
rows, err := s.db.Query(`EXPLAIN QUERY PLAN DELETE FROM client_receptions WHERE rx_at < ?`, "2026-01-01T00:00:00Z")
if err != nil {
t.Fatal(err)
}
defer rows.Close()
plan := ""
for rows.Next() {
var id, parent, notused int
var detail string
if err := rows.Scan(&id, &parent, &notused, &detail); err != nil {
t.Fatal(err)
}
plan += detail + "\n"
}
if !strings.Contains(plan, "idx_client_recept_rxat") {
t.Fatalf("retention DELETE should use idx_client_recept_rxat, plan was:\n%s", plan)
}
}
// TestRxLeaderboardQueryIsIndexBacked pins the planner choice for the leaderboard
// SELECT (the rx_at-windowed, rx_pubkey-grouped query in cmd/server/rx_dashboard.go).
// SQLite serves it from the UNIQUE(rx_pubkey,heard_key,rx_at) constraint index as a
// COVERING scan (not idx_client_recept_rxat, and not a table-heap scan). The table
// is retention-bounded, so a covering scan is acceptable; this test guards against a
// silent regression to a bare table scan under the writer lock when the schema is
// next tweaked. Representative form (no JOINs — they don't change whether `cr` is
// index-backed).
func TestRxLeaderboardQueryIsIndexBacked(t *testing.T) {
s := newTestStore(t)
rows, err := s.db.Query(`EXPLAIN QUERY PLAN
SELECT cr.rx_pubkey, COUNT(*), COUNT(DISTINCT cr.heard_key)
FROM client_receptions cr
WHERE cr.rx_at >= ?
GROUP BY cr.rx_pubkey
ORDER BY COUNT(*) DESC
LIMIT ?`, "2026-01-01T00:00:00Z", 100)
if err != nil {
t.Fatal(err)
}
defer rows.Close()
plan := ""
for rows.Next() {
var id, parent, notused int
var detail string
if err := rows.Scan(&id, &parent, &notused, &detail); err != nil {
t.Fatal(err)
}
plan += detail + "\n"
}
t.Logf("leaderboard plan:\n%s", plan)
// The concern is a bare table-heap scan, not which specific index wins. The
// plan must stay index-backed (covering or search) — a regression to a bare
// "SCAN cr" without an index fails here.
if !strings.Contains(plan, "INDEX") {
t.Fatalf("leaderboard SELECT must stay index-backed (no full table-heap scan), plan was:\n%s", plan)
}
}
func TestDeriveHeardKey(t *testing.T) {
full := "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789"
k, l, src, ok := deriveHeardKey("rx", packetpath.RouteFlood, nil, strings.ToUpper(full), true)
if !ok || l != 32 || src != "advert" || k != full {
t.Fatalf("0-hop advert: got k=%q l=%d src=%q ok=%v", k, l, src, ok)
}
k, l, src, ok = deriveHeardKey("rx", packetpath.RouteFlood, []string{"aa", "bbccdd"}, "", false)
if !ok || k != "bbccdd" || l != 3 || src != "rxlog" {
t.Fatalf("flood path: got k=%q l=%d src=%q ok=%v", k, l, src, ok)
}
// DIRECT route: path[last] is the route's far end, not the transmitter — must be rejected.
if _, _, _, ok = deriveHeardKey("rx", packetpath.RouteDirect, []string{"aa", "bbccdd"}, "", false); ok {
t.Fatalf("direct-route path must be rejected")
}
if _, _, _, ok = deriveHeardKey("rx", packetpath.RouteTransportDirect, []string{"aa", "bbccdd"}, "", false); ok {
t.Fatalf("transport-direct-route path must be rejected")
}
if _, _, _, ok = deriveHeardKey("rx", packetpath.RouteFlood, []string{"aa", "bb"}, "", false); ok {
t.Fatalf("1-byte last hop should be rejected")
}
if _, _, _, ok = deriveHeardKey("tx", packetpath.RouteFlood, []string{"aabbcc"}, "", false); ok {
t.Fatalf("tx must be rejected")
}
if _, _, _, ok = deriveHeardKey("rx", packetpath.RouteFlood, nil, "", false); ok {
t.Fatalf("no hops + non-advert must be rejected")
}
}
func TestBuildClientReception(t *testing.T) {
acc := 8.0
rec, ok := buildClientReception("companionpk", "rx", packetpath.RouteFlood, []string{"aa", "bbccdd"}, "", false,
crF(-7.5), crI(-92), 51.05, 3.72, &acc, "2026-06-09T12:00:00Z", "2026-06-09T12:00:01Z")
if !ok || rec.HeardKey != "bbccdd" || rec.HeardKeyLen != 3 || rec.Src != "rxlog" {
t.Fatalf("bad reception: %+v ok=%v", rec, ok)
}
if _, ok := buildClientReception("c", "rx", packetpath.RouteDirect, []string{"bbccdd"}, "", false,
crF(-7.5), crI(-92), 51.05, 3.72, nil, "t", "t"); ok {
t.Fatal("direct-route path must be rejected (not the transmitter)")
}
if _, ok := buildClientReception("c", "rx", packetpath.RouteFlood, []string{"bbccdd"}, "", false, nil, nil, 99.0, 3.72, nil, "t", "t"); ok {
t.Fatal("out-of-range lat must be rejected")
}
}
func TestInsertClientReceptionRoundTripAndIdempotent(t *testing.T) {
s := newTestStore(t)
rec := &ClientReception{
RxPubkey: "companionpk", HeardKey: "bbccdd", HeardKeyLen: 3, RSSI: crI(-92),
Lat: 51.05, Lon: 3.72, RxAt: "2026-06-09T12:00:00Z", IngestedAt: "2026-06-09T12:00:01Z", Src: "rxlog",
}
if ins, err := s.InsertClientReception(rec); err != nil || !ins {
t.Fatalf("first insert: ins=%v err=%v", ins, err)
}
if ins, err := s.InsertClientReception(rec); err != nil || ins {
t.Fatalf("second insert should be a no-op: ins=%v err=%v", ins, err)
}
var n int
s.db.QueryRow(`SELECT COUNT(*) FROM client_receptions`).Scan(&n)
if n != 1 {
t.Fatalf("expected 1 row, got %d", n)
}
}
func TestHandleClientPacketRelayedAdvertWritesReception(t *testing.T) {
s := newTestStore(t)
advertHex := "11451000D818206D3AAC152C8A91F89957E6D30CA51F36E28790228971C473B755F244F718754CF5EE4A2FD58D944466E42CDED140C66D0CC590183E32BAF40F112BE8F3F2BDF6012B4B2793C52F1D36F69EE054D9A05593286F78453E56C0EC4A3EB95DDA2A7543FCCC00B939CACC009278603902FC12BCF84B706120526F6F6620536F6C6172"
msg := map[string]interface{}{
"raw": advertHex,
"direction": "rx",
"timestamp": "2026-06-09T12:00:00Z",
"origin": "MyMob",
"SNR": -7.0,
"RSSI": -92.0,
"gps": map[string]interface{}{"lat": 51.05, "lon": 3.72, "acc_m": 8.0},
}
handleClientPacket(s, "test", testCompanionPK, msg, nil)
var obsName string
s.db.QueryRow(`SELECT name FROM client_observers WHERE pubkey=?`, testCompanionPK).Scan(&obsName)
if obsName != "MyMob" {
t.Fatalf("expected client_observers name 'MyMob', got %q", obsName)
}
// This fixture is a relayed advert (non-empty path), so by the capture HARD
// RULE we record the directly-heard LAST hop (multibyte), not the originator.
// The 0-hop advert→full-pubkey branch is covered by TestDeriveHeardKey.
var n, keylen int
var src string
if err := s.db.QueryRow(`SELECT COUNT(*), COALESCE(MAX(heard_keylen),0), COALESCE(MAX(src),'') FROM client_receptions WHERE rx_pubkey=?`, testCompanionPK).Scan(&n, &keylen, &src); err != nil {
t.Fatal(err)
}
if n != 1 || keylen < 2 || src != "rxlog" {
t.Fatalf("expected 1 rxlog reception (multibyte last hop), got n=%d keylen=%d src=%q", n, keylen, src)
}
// No GPS → no row.
const companion2 = "b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3"
handleClientPacket(s, "test", companion2, map[string]interface{}{"raw": advertHex, "direction": "rx"}, nil)
var n2 int
s.db.QueryRow(`SELECT COUNT(*) FROM client_receptions WHERE rx_pubkey=?`, companion2).Scan(&n2)
if n2 != 0 {
t.Fatalf("packet without gps must be dropped, got %d rows", n2)
}
}
// TestHandleClientPacketZeroHopAdvertWritesReception covers the #9 gap: the
// advert fixture used above is a RELAYED advert (non-empty path), so it exercises
// the rxlog last-hop branch, not the 0-hop src='advert' branch. Here we rebuild
// the same advert with zero hops — header (FLOOD ADVERT) + "00" (0 hops) + the
// same advert payload — so handleClientPacket stores the advertiser by its full
// pubkey with src='advert', and we assert gps/snr were captured too.
func TestHandleClientPacketZeroHopAdvertWritesReception(t *testing.T) {
s := newTestStore(t)
relayed := "11451000D818206D3AAC152C8A91F89957E6D30CA51F36E28790228971C473B755F244F718754CF5EE4A2FD58D944466E42CDED140C66D0CC590183E32BAF40F112BE8F3F2BDF6012B4B2793C52F1D36F69EE054D9A05593286F78453E56C0EC4A3EB95DDA2A7543FCCC00B939CACC009278603902FC12BCF84B706120526F6F6620536F6C6172"
// relayed = header(2) + path-descriptor(2) + 5*2-byte hops(20) + payload.
payload := relayed[24:]
zeroHop := "1100" + payload
advertPubkey := strings.ToLower(payload[:64]) // advert payload starts with the 32-byte pubkey
msg := map[string]interface{}{
"raw": zeroHop, "direction": "rx", "timestamp": "2026-06-09T12:00:00Z",
"origin": "MyMob", "SNR": -7.0, "RSSI": -92.0,
"gps": map[string]interface{}{"lat": 51.05, "lon": 3.72, "acc_m": 8.0},
}
handleClientPacket(s, "test", testCompanionPK, msg, nil)
var heardKey, src string
var keylen int
var snr sql.NullFloat64
var lat, lon float64
if err := s.db.QueryRow(`SELECT heard_key, heard_keylen, src, snr, lat, lon FROM client_receptions WHERE rx_pubkey=?`, testCompanionPK).
Scan(&heardKey, &keylen, &src, &snr, &lat, &lon); err != nil {
t.Fatalf("expected a 0-hop advert reception: %v", err)
}
if src != "advert" || keylen != 32 || heardKey != advertPubkey {
t.Fatalf("0-hop advert: want advert/32/%s, got %s/%d/%s", advertPubkey, src, keylen, heardKey)
}
if !snr.Valid || snr.Float64 != -7 || lat != 51.05 || lon != 3.72 {
t.Fatalf("gps/snr not captured: snr=%v lat=%f lon=%f", snr, lat, lon)
}
}
// TestHandleClientPacketRejectsNonHexPubkey verifies the #2 fix: a companion
// pubkey from the topic that isn't lowercase hex (a no-ACL broker could publish
// meshcore/client/!@#$/packets) writes nothing to either coverage table. Without
// the clientPubkeyRe guard this fixture would insert a polluting row.
func TestHandleClientPacketRejectsNonHexPubkey(t *testing.T) {
s := newTestStore(t)
advertHex := "11451000D818206D3AAC152C8A91F89957E6D30CA51F36E28790228971C473B755F244F718754CF5EE4A2FD58D944466E42CDED140C66D0CC590183E32BAF40F112BE8F3F2BDF6012B4B2793C52F1D36F69EE054D9A05593286F78453E56C0EC4A3EB95DDA2A7543FCCC00B939CACC009278603902FC12BCF84B706120526F6F6620536F6C6172"
for _, bad := range []string{"!@#$", "companionpk", "", "g0g0", "xyz"} {
msg := map[string]interface{}{
"raw": advertHex, "direction": "rx", "timestamp": "2026-06-09T12:00:00Z",
"origin": "Spoof", "SNR": -7.0, "RSSI": -92.0,
"gps": map[string]interface{}{"lat": 51.05, "lon": 3.72, "acc_m": 8.0},
}
handleClientPacket(s, "test", bad, msg, nil)
}
var nRecept, nObs int
s.db.QueryRow(`SELECT COUNT(*) FROM client_receptions`).Scan(&nRecept)
s.db.QueryRow(`SELECT COUNT(*) FROM client_observers`).Scan(&nObs)
if nRecept != 0 || nObs != 0 {
t.Fatalf("non-hex pubkey must write nothing, got %d receptions, %d observers", nRecept, nObs)
}
}
+40 -15
View File
@@ -43,21 +43,22 @@ type MQTTLegacy struct {
// Config holds the ingestor configuration, compatible with the Node.js config.json format.
type Config struct {
DBPath string `json:"dbPath"`
MQTT *MQTTLegacy `json:"mqtt,omitempty"`
MQTTSources []MQTTSource `json:"mqttSources,omitempty"`
LogLevel string `json:"logLevel,omitempty"`
ChannelKeysPath string `json:"channelKeysPath,omitempty"`
ChannelKeys map[string]string `json:"channelKeys,omitempty"`
HashChannels []string `json:"hashChannels,omitempty"`
HashRegions []string `json:"hashRegions,omitempty"`
Retention *RetentionConfig `json:"retention,omitempty"`
Metrics *MetricsConfig `json:"metrics,omitempty"`
Runtime *RuntimeConfig `json:"runtime,omitempty"`
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
ForeignAdverts *ForeignAdvertConfig `json:"foreignAdverts,omitempty"`
ValidateSignatures *bool `json:"validateSignatures,omitempty"`
DB *DBConfig `json:"db,omitempty"`
DBPath string `json:"dbPath"`
MQTT *MQTTLegacy `json:"mqtt,omitempty"`
MQTTSources []MQTTSource `json:"mqttSources,omitempty"`
LogLevel string `json:"logLevel,omitempty"`
ChannelKeysPath string `json:"channelKeysPath,omitempty"`
ChannelKeys map[string]string `json:"channelKeys,omitempty"`
HashChannels []string `json:"hashChannels,omitempty"`
HashRegions []string `json:"hashRegions,omitempty"`
Retention *RetentionConfig `json:"retention,omitempty"`
Metrics *MetricsConfig `json:"metrics,omitempty"`
Runtime *RuntimeConfig `json:"runtime,omitempty"`
ClientRxCoverage *ClientRxCoverageConfig `json:"clientRxCoverage,omitempty"`
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
ForeignAdverts *ForeignAdvertConfig `json:"foreignAdverts,omitempty"`
ValidateSignatures *bool `json:"validateSignatures,omitempty"`
DB *DBConfig `json:"db,omitempty"`
// ObserverIATAWhitelist restricts which observer IATA regions are processed.
// When non-empty, only observers whose IATA code (from the MQTT topic) matches
@@ -128,6 +129,17 @@ func (f *ForeignAdvertConfig) IsDropMode() bool {
return strings.EqualFold(strings.TrimSpace(f.Mode), "drop")
}
// ClientRxCoverageConfig controls the opt-in mobile client-RX coverage feature.
type ClientRxCoverageConfig struct {
Enabled bool `json:"enabled"`
}
// ClientRxCoverageEnabled reports whether the opt-in mobile client-RX coverage
// feature is on. Absent/nil ⇒ off (the safe default).
func (c *Config) ClientRxCoverageEnabled() bool {
return c.ClientRxCoverage != nil && c.ClientRxCoverage.Enabled
}
// RetentionConfig controls how long stale nodes are kept before being moved to inactive_nodes.
type RetentionConfig struct {
NodeDays int `json:"nodeDays"`
@@ -136,6 +148,10 @@ type RetentionConfig struct {
// PacketDays is the retention window for transmissions (#1283).
// Ownership moved from cmd/server to cmd/ingestor; 0 disables.
PacketDays int `json:"packetDays"`
// ClientRxDays is the retention window (by rx_at) for mobile client-RX
// coverage rows in client_receptions / client_observers; 0 disables. Bounds
// the table the opt-in coverage feature would otherwise grow without limit.
ClientRxDays int `json:"clientRxDays"`
}
// PacketDaysOrZero returns the configured retention.packetDays or 0
@@ -147,6 +163,15 @@ func (c *Config) PacketDaysOrZero() int {
return 0
}
// ClientRxDaysOrZero returns the configured retention.clientRxDays or 0
// (disabled) if not set.
func (c *Config) ClientRxDaysOrZero() int {
if c.Retention != nil && c.Retention.ClientRxDays > 0 {
return c.Retention.ClientRxDays
}
return 0
}
// MetricsConfig controls observer metrics collection.
type MetricsConfig struct {
SampleIntervalSec int `json:"sampleIntervalSec"`
+88
View File
@@ -0,0 +1,88 @@
package main
import "testing"
// testCompanionPK is a valid lowercase-hex companion pubkey for coverage tests.
// The topic segment must be hex (clientPubkeyRe) or handleClientPacket drops it.
const testCompanionPK = "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2"
// clientCoverageMsg builds a valid mobile client-RX coverage message on the
// dedicated topic meshcore/client/<pubkey>/packets. The raw hex is a relayed
// advert with GPS, so handleClientPacket would write exactly one
// client_receptions row when the feature is enabled (see
// TestHandleClientPacketAdvertWritesReception).
func clientCoverageMsg() *mockMessage {
advertHex := "11451000D818206D3AAC152C8A91F89957E6D30CA51F36E28790228971C473B755F244F718754CF5EE4A2FD58D944466E42CDED140C66D0CC590183E32BAF40F112BE8F3F2BDF6012B4B2793C52F1D36F69EE054D9A05593286F78453E56C0EC4A3EB95DDA2A7543FCCC00B939CACC009278603902FC12BCF84B706120526F6F6620536F6C6172"
payload := []byte(`{"raw":"` + advertHex + `","direction":"rx","timestamp":"2026-06-09T12:00:00Z","origin":"MyMob","SNR":-7.0,"RSSI":-92.0,"gps":{"lat":51.05,"lon":3.72,"acc_m":8.0}}`)
return &mockMessage{topic: "meshcore/client/" + testCompanionPK + "/packets", payload: payload}
}
func clientReceptionCount(t *testing.T, s *Store) int {
t.Helper()
var n int
if err := s.db.QueryRow(`SELECT COUNT(*) FROM client_receptions`).Scan(&n); err != nil {
t.Fatal(err)
}
return n
}
// TestClientRxCoverageEnabledDefault verifies the gate helper defaults OFF for
// nil/absent config and is only true when explicitly enabled.
func TestClientRxCoverageEnabledDefault(t *testing.T) {
if (&Config{}).ClientRxCoverageEnabled() {
t.Fatal("nil ClientRxCoverage must report disabled")
}
if (&Config{ClientRxCoverage: &ClientRxCoverageConfig{Enabled: false}}).ClientRxCoverageEnabled() {
t.Fatal("Enabled:false must report disabled")
}
if !(&Config{ClientRxCoverage: &ClientRxCoverageConfig{Enabled: true}}).ClientRxCoverageEnabled() {
t.Fatal("Enabled:true must report enabled")
}
}
// TestClientRxCoverageGateOff drives handleMessage with the feature OFF: the
// client-topic message must fall through and write no client_receptions rows.
func TestClientRxCoverageGateOff(t *testing.T) {
store := newTestStore(t)
source := MQTTSource{Name: "test"}
cfg := &Config{} // ClientRxCoverage nil ⇒ disabled
handleMessage(store, "test", source, clientCoverageMsg(), nil, nil, cfg)
if n := clientReceptionCount(t, store); n != 0 {
t.Fatalf("feature OFF: expected 0 client_receptions rows, got %d", n)
}
}
// TestClientRxCoverageGateOn drives handleMessage with the feature ON: the
// client-topic message must be dispatched and write exactly one row.
func TestClientRxCoverageGateOn(t *testing.T) {
store := newTestStore(t)
source := MQTTSource{Name: "test"}
cfg := &Config{ClientRxCoverage: &ClientRxCoverageConfig{Enabled: true}}
handleMessage(store, "test", source, clientCoverageMsg(), nil, nil, cfg)
if n := clientReceptionCount(t, store); n != 1 {
t.Fatalf("feature ON: expected 1 client_receptions row, got %d", n)
}
}
// TestClientRxCoverageBlacklistedDropped verifies the #1 fix: a blacklisted
// operator cannot skirt the observer blacklist via the client topic. With the
// feature ON but the companion pubkey blacklisted, no row is written. Without
// the gate the client dispatch runs before the blacklist check and inserts.
func TestClientRxCoverageBlacklistedDropped(t *testing.T) {
store := newTestStore(t)
source := MQTTSource{Name: "test"}
cfg := &Config{
ClientRxCoverage: &ClientRxCoverageConfig{Enabled: true},
ObserverBlacklist: []string{testCompanionPK},
}
handleMessage(store, "test", source, clientCoverageMsg(), nil, nil, cfg)
if n := clientReceptionCount(t, store); n != 0 {
t.Fatalf("blacklisted companion: expected 0 client_receptions rows, got %d", n)
}
}
+104
View File
@@ -0,0 +1,104 @@
package main
import (
"fmt"
"math/rand"
"strings"
"testing"
)
// coverageBenchSQL is the dominant per-node coverage query (mirrors
// cmd/server queryCoverageRows): a bbox range plus a full-key/2-3-byte-prefix
// match on the heard node.
const coverageBenchSQL = `SELECT lat, lon, snr, rssi, heard_key, rx_at
FROM client_receptions
WHERE lat BETWEEN ? AND ? AND lon BETWEEN ? AND ?
AND ( (heard_keylen = 32 AND heard_key = ?)
OR (heard_keylen IN (2,3) AND substr(?, 1, heard_keylen*2) = heard_key) )`
// BenchmarkCoverageQuery seeds ~1M receptions across a metro-area bbox and times
// the coverage query with the indexes (#5/#18) versus a forced full table scan.
// Run: go test -run x -bench BenchmarkCoverageQuery -benchtime 20x ./cmd/ingestor
func BenchmarkCoverageQuery(b *testing.B) {
const n = 1_000_000
const prefixPool = 2000 // distinct 3-byte heard_key prefixes
dir := b.TempDir()
s, err := OpenStore(dir + "/bench.db")
if err != nil {
b.Fatal(err)
}
defer s.Close()
rng := rand.New(rand.NewSource(1))
tx, err := s.db.Begin()
if err != nil {
b.Fatal(err)
}
stmt, err := tx.Prepare(`INSERT INTO client_receptions
(rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src)
VALUES (?,?,?,?,?,?,?,?,?)`)
if err != nil {
b.Fatal(err)
}
for i := 0; i < n; i++ {
hk := fmt.Sprintf("%06x", rng.Intn(prefixPool))
lat := 51.0 + rng.Float64()*0.4 // ~44 km metro span
lon := 3.5 + rng.Float64()*0.4
rxpk := fmt.Sprintf("%064x", rng.Intn(500))
// rx_at carries i so (rx_pubkey,heard_key,rx_at) stays unique.
if _, err := stmt.Exec(rxpk, hk, 3, -6.0, lat, lon, fmt.Sprintf("t%d", i), "x", "rxlog"); err != nil {
b.Fatal(err)
}
}
stmt.Close()
if err := tx.Commit(); err != nil {
b.Fatal(err)
}
// Target node whose 3-byte prefix (0003e8 = 1000) is in the pool, queried
// over a sub-bbox of the metro area.
target := "0003e8" + strings.Repeat("ab", 29) // 6 + 58 = 64 hex
// OR/substr query (original shape): bbox range OR'd with a non-sargable
// substr prefix match.
runOR := func(b *testing.B) {
for i := 0; i < b.N; i++ {
rows, err := s.db.Query(coverageBenchSQL, 51.1, 51.3, 3.6, 3.8, target, target)
if err != nil {
b.Fatal(err)
}
for rows.Next() {
}
rows.Close()
}
}
// IN-list query (sargable): the heard node's candidate keys are exactly the
// full pubkey and its 2/3-byte prefixes, so an IN-list seeks them via the
// heard_key-leading composite instead of scanning the bbox.
inListSQL := `SELECT lat, lon, snr, rssi, heard_key, rx_at FROM client_receptions
WHERE heard_key IN (?,?,?) AND lat BETWEEN ? AND ? AND lon BETWEEN ? AND ?`
runIN := func(b *testing.B) {
for i := 0; i < b.N; i++ {
rows, err := s.db.Query(inListSQL, target, target[:4], target[:6], 51.1, 51.3, 3.6, 3.8)
if err != nil {
b.Fatal(err)
}
for rows.Next() {
}
rows.Close()
}
}
b.Run("or_query_indexed", runOR)
b.Run("inlist_query_indexed", runIN)
// Drop the coverage indexes to measure the full-scan baseline.
for _, idx := range []string{"idx_client_recept_heard_geo", "idx_client_recept_latlon", "idx_client_recept_rxpk"} {
if _, err := s.db.Exec("DROP INDEX IF EXISTS " + idx); err != nil {
b.Fatal(err)
}
}
b.Run("or_query_table_scan", runOR)
}
+44 -1
View File
@@ -271,6 +271,50 @@ func applySchema(db *sql.DB) error {
-- the last_seen column exists (#1690) — keep it OUT of this base
-- schema block so legacy DBs (table-exists, column-missing) don't
-- trip on the CREATE INDEX before the ALTER runs.
-- Mobile client RX coverage: a roaming companion = a mobile observer
-- with a moving GPS position, so it gets its own table rather than
-- observations (which assumes a fixed observer/location).
CREATE TABLE IF NOT EXISTS client_receptions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
rx_pubkey TEXT NOT NULL,
heard_key TEXT NOT NULL,
heard_keylen INTEGER NOT NULL,
rssi INTEGER,
snr REAL,
lat REAL NOT NULL,
lon REAL NOT NULL,
pos_acc_m REAL,
rx_at TEXT NOT NULL,
ingested_at TEXT NOT NULL,
src TEXT NOT NULL,
UNIQUE(rx_pubkey, heard_key, rx_at)
);
-- Coverage queries filter by bbox AND match the heard node either by full
-- key (heard_keylen=32 AND heard_key=?) or by 2-3 byte prefix. The composite
-- (heard_key, heard_keylen, lat, lon) serves the heard_key-equality seek and
-- carries lat/lon so the bbox range is satisfied from the index; it also
-- supersedes the old single-column heard_key index. idx_client_recept_latlon
-- lets the planner instead drive from a selective bbox. (#5, #18)
CREATE INDEX IF NOT EXISTS idx_client_recept_heard_geo ON client_receptions(heard_key, heard_keylen, lat, lon);
CREATE INDEX IF NOT EXISTS idx_client_recept_latlon ON client_receptions(lat, lon);
-- rx_at backs both the retention reaper (DELETE WHERE rx_at < ?) and the
-- leaderboard, which range-scans WHERE rx_at >= ? and aggregates per
-- rx_pubkey in Go (see rxLeaderboard's frontier-weighted scoring). Without
-- this index either would full-scan the table under the writer lock
-- (verified by an EXPLAIN test). A dedicated rx_pubkey index stays
-- redundant — the leaderboard no longer groups by rx_pubkey in SQL.
CREATE INDEX IF NOT EXISTS idx_client_recept_rxat ON client_receptions(rx_at);
DROP INDEX IF EXISTS idx_client_recept_rxpk;
-- Self-reported name of each mobile client (companion), from the SELF_INFO
-- name the app sends as "origin". Lets the leaderboard show a name even
-- when the companion never advertised (so it isn't in the nodes table).
CREATE TABLE IF NOT EXISTS client_observers (
pubkey TEXT PRIMARY KEY,
name TEXT,
last_seen TEXT
);
`
if _, err := db.Exec(schema); err != nil {
return fmt.Errorf("base schema: %w", err)
@@ -1703,7 +1747,6 @@ func BuildPacketData(msg *MQTTPacketMessage, decoded *DecodedPacket, observerID,
return pd
}
// ─── Writer-lock instrumentation (issue #1340) ────────────────────────────
//
// Make SQLite writer-lock starvation visible to operators. Per-component
+42
View File
@@ -273,6 +273,18 @@ func main() {
}
}
// Client-RX coverage retention: bound the opt-in coverage tables (#1727).
// Independent of the feature flag, so data persists are reaped even after
// the feature is turned off. 0 = disabled.
clientRxDays := cfg.ClientRxDaysOrZero()
if clientRxDays > 0 {
if n, err := store.PruneOldClientReceptions(clientRxDays); err != nil {
log.Printf("[prune] error: %v", err)
} else if n > 0 {
log.Printf("[prune] startup pruned %d client_receptions older than %d days", n, clientRxDays)
}
}
vacuumPages := cfg.IncrementalVacuumPages()
store.RunIncrementalVacuum(vacuumPages)
@@ -335,6 +347,21 @@ func main() {
log.Printf("[prune] auto-prune enabled: packets older than %d days will be removed daily", packetDays)
}
// Daily ticker for client-RX coverage retention (#1727).
if clientRxDays > 0 {
clientRxRetentionTicker := time.NewTicker(24 * time.Hour)
go func() {
for range clientRxRetentionTicker.C {
if n, err := store.PruneOldClientReceptions(clientRxDays); err != nil {
log.Printf("[prune] error: %v", err)
} else if n > 0 {
store.RunIncrementalVacuum(vacuumPages)
}
}
}()
log.Printf("[prune] auto-prune enabled: client_receptions older than %d days will be removed daily", clientRxDays)
}
// Hourly WAL checkpoint to prevent unbounded WAL growth.
// TRUNCATE resets the WAL file to zero bytes when all frames are flushed;
// if the server's read connection holds frames, remaining pages stay in the
@@ -535,6 +562,21 @@ func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message,
return
}
// Mobile client RX coverage: dedicated topic meshcore/client/{PUBLIC_KEY}/packets.
// A roaming companion reports where it directly heard a node; handled in isolation
// from the observer/observations path. EMQX ACL binds parts[2] to the client's own key.
if cfg.ClientRxCoverageEnabled() && len(parts) >= 4 && parts[1] == "client" && parts[3] == "packets" {
// The observer blacklist (checked below) only runs on the observer path,
// so a blacklisted operator could otherwise skirt it via the client topic
// (#1). Enforce it here before any coverage write.
if cfg.IsObserverBlacklisted(parts[2]) {
log.Printf("MQTT [%s] client %.8s blacklisted, dropping", tag, parts[2])
return
}
handleClientPacket(store, tag, parts[2], msg, channelKeys)
return
}
// Skip status/connection topics
if topic == "meshcore/status" || topic == "meshcore/events/connection" {
return
+33
View File
@@ -48,6 +48,39 @@ func (s *Store) PruneOldPackets(days int) (int64, error) {
return n, nil
}
// PruneOldClientReceptions deletes mobile client-RX coverage rows older than
// `days` (by rx_at), and client_observers (companion names) whose last_seen has
// aged out. This bounds the otherwise-unbounded client_receptions table the
// opt-in coverage feature feeds. 0 disables. Owned by the ingestor writer
// (#1283). Returns the number of client_receptions rows deleted.
func (s *Store) PruneOldClientReceptions(days int) (int64, error) {
if days <= 0 {
return 0, nil
}
cutoff := time.Now().UTC().AddDate(0, 0, -days).Format(time.RFC3339)
var n int64
err := s.WriterTx("prune_client_receptions", func(tx *sql.Tx) error {
res, err := tx.Exec(`DELETE FROM client_receptions WHERE rx_at < ?`, cutoff)
if err != nil {
return fmt.Errorf("prune client_receptions: %w", err)
}
n, _ = res.RowsAffected()
// Drop companion name rows not refreshed within the window.
if _, err := tx.Exec(`DELETE FROM client_observers WHERE last_seen < ?`, cutoff); err != nil {
return fmt.Errorf("prune client_observers: %w", err)
}
return nil
})
if err != nil {
return 0, err
}
if n > 0 {
log.Printf("[prune] deleted %d client_receptions older than %d days", n, days)
}
return n, nil
}
// SoftDeleteBlacklistedObservers marks observers in the blacklist as
// inactive=1 so they are hidden from API responses. Owned by ingestor
// per #1287. Runs once at startup.
+27 -10
View File
@@ -133,7 +133,7 @@ type Config struct {
// Currently exposes runtime.maxMemoryMB which sets a soft memory limit
// (GOMEMLIMIT) via runtime/debug.SetMemoryLimit at startup. The
// GOMEMLIMIT environment variable, when set, takes precedence.
Runtime *RuntimeConfig `json:"runtime,omitempty"`
Runtime *RuntimeConfig `json:"runtime,omitempty"`
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
Areas map[string]AreaEntry `json:"areas,omitempty"`
@@ -160,7 +160,13 @@ type Config struct {
obsBlacklistSetCached map[string]bool
obsBlacklistOnce sync.Once
Compression *CompressionConfig `json:"compression,omitempty"`
Compression *CompressionConfig `json:"compression,omitempty"`
// ClientRxCoverage gates the opt-in mobile client-RX coverage feature
// (corescope-rx companions publishing GPS-tagged receptions). Absent/nil
// ⇒ off; see ClientRxCoverageEnabled.
ClientRxCoverage *ClientRxCoverageConfig `json:"clientRxCoverage,omitempty"`
ResolvedPath *ResolvedPathConfig `json:"resolvedPath,omitempty"`
NeighborGraph *NeighborGraphConfig `json:"neighborGraph,omitempty"`
@@ -250,6 +256,17 @@ func (c *Config) GZipEnabled() bool {
return c.Compression != nil && c.Compression.GZip
}
// ClientRxCoverageConfig gates the opt-in mobile client-RX coverage feature.
type ClientRxCoverageConfig struct {
Enabled bool `json:"enabled"`
}
// ClientRxCoverageEnabled reports whether the opt-in mobile client-RX coverage
// feature is on. Nil config or absent/nil section ⇒ off (the safe default).
func (c *Config) ClientRxCoverageEnabled() bool {
return c != nil && c.ClientRxCoverage != nil && c.ClientRxCoverage.Enabled
}
// WSCompressionEnabled returns true when WebSocket permessage-deflate is explicitly enabled.
func (c *Config) WSCompressionEnabled() bool {
return c.Compression != nil && c.Compression.Websocket
@@ -303,10 +320,10 @@ type RuntimeConfig struct {
}
type RetentionConfig struct {
NodeDays int `json:"nodeDays"`
ObserverDays int `json:"observerDays"`
PacketDays int `json:"packetDays"`
MetricsDays int `json:"metricsDays"`
NodeDays int `json:"nodeDays"`
ObserverDays int `json:"observerDays"`
PacketDays int `json:"packetDays"`
MetricsDays int `json:"metricsDays"`
}
// DBConfig is the shared SQLite vacuum/maintenance config (#919, #921).
@@ -601,7 +618,6 @@ func (c *Config) ResolveDBPath(baseDir string) string {
return filepath.Join(baseDir, "data", "meshcore.db")
}
func (c *Config) NormalizeTimestampConfig() {
defaults := defaultTimestampConfig()
if c.Timestamps == nil {
@@ -908,10 +924,11 @@ func (c *Config) IsObserverBlacklisted(id string) bool {
// data slowly." Lower values give fresher data at higher CPU cost.
//
// RecomputeIntervalSeconds keys (all optional):
// topology, rf, distance, channels, hashCollisions, hashSizes, roles, observersClockSkew, nodesClockSkew
//
// topology, rf, distance, channels, hashCollisions, hashSizes, roles, observersClockSkew, nodesClockSkew
type AnalyticsConfig struct {
DefaultIntervalSeconds int `json:"defaultIntervalSeconds,omitempty"`
RecomputeIntervalSeconds map[string]int `json:"recomputeIntervalSeconds,omitempty"`
DefaultIntervalSeconds int `json:"defaultIntervalSeconds,omitempty"`
RecomputeIntervalSeconds map[string]int `json:"recomputeIntervalSeconds,omitempty"`
}
// AnalyticsDefaultRecomputeInterval returns the configured default
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/gorilla/mux"
)
// gateTestServer builds a minimal *Server with the given client-RX coverage
// config and registers all routes, mirroring routes_test.go's setup but
// skipping the packet store (none of the gated routes need it for the 404 /
// registration assertions below).
func gateTestServer(t *testing.T, cov *ClientRxCoverageConfig) *mux.Router {
t.Helper()
db := setupTestDB(t)
cfg := &Config{Port: 3000, ClientRxCoverage: cov}
hub := NewHub()
srv := NewServer(db, cfg, hub)
router := mux.NewRouter()
srv.RegisterRoutes(router)
return router
}
func TestCoverageRoutesGatedOff(t *testing.T) {
router := gateTestServer(t, nil)
req := httptest.NewRequest("GET", "/api/rx-coverage", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusNotFound {
t.Fatalf("expected 404 for /api/rx-coverage when disabled, got %d", rr.Code)
}
creq := httptest.NewRequest("GET", "/api/config/client", nil)
crr := httptest.NewRecorder()
router.ServeHTTP(crr, creq)
if crr.Code != http.StatusOK {
t.Fatalf("config/client status %d body %s", crr.Code, crr.Body.String())
}
if !strings.Contains(crr.Body.String(), `"clientRxCoverage":false`) {
t.Fatalf("expected clientRxCoverage:false in config body, got %s", crr.Body.String())
}
}
func TestCoverageRoutesGatedOn(t *testing.T) {
router := gateTestServer(t, &ClientRxCoverageConfig{Enabled: true})
req := httptest.NewRequest("GET", "/api/rx-coverage", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code == http.StatusNotFound {
t.Fatalf("expected /api/rx-coverage to be registered when enabled, got 404")
}
creq := httptest.NewRequest("GET", "/api/config/client", nil)
crr := httptest.NewRecorder()
router.ServeHTTP(crr, creq)
if crr.Code != http.StatusOK {
t.Fatalf("config/client status %d body %s", crr.Code, crr.Body.String())
}
if !strings.Contains(crr.Body.String(), `"clientRxCoverage":true`) {
t.Fatalf("expected clientRxCoverage:true in config body, got %s", crr.Body.String())
}
}
+1 -1
View File
@@ -30,7 +30,7 @@ func setupTestDBv2(t *testing.T) *DB {
CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT,
lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, advert_count INTEGER DEFAULT 0,
battery_mv INTEGER, temperature_c REAL
battery_mv INTEGER, temperature_c REAL, foreign_advert INTEGER DEFAULT 0
);
CREATE TABLE observers (
id TEXT PRIMARY KEY, name TEXT, iata TEXT, last_seen TEXT, first_seen TEXT,
+119
View File
@@ -0,0 +1,119 @@
package main
import (
"fmt"
"math"
"strconv"
"strings"
)
// Pure-Go hexagonal binning for RX coverage display. We deliberately avoid the
// CGO-based uber/h3-go (this project builds with CGO_ENABLED=0). Points are
// projected to Web Mercator and snapped to a pointy-top hex grid whose size
// depends on the display resolution. Cell ids are "res:q:r" (axial coords).
// At city/region scale this looks like H3/mapme.sh coverage without any deps.
const hexEarthRadius = 6378137.0 // Web Mercator sphere radius (m)
// hexTargetPx is the desired on-screen hex size (point-to-point height) in CSS
// pixels. mercUPPZ0 is Web Mercator units per pixel at zoom 0 (world span / 256);
// Leaflet halves it each zoom level, independent of latitude. Sizing the hex in
// these units therefore renders it at a constant ~hexTargetPx at every zoom — the
// old fixed-meter buckets looked like specks when zoomed out (issue: hexes too small).
const hexTargetPx = 28.0
const mercUPPZ0 = 156543.03392
func hexMercator(lat, lon float64) (float64, float64) {
x := hexEarthRadius * lon * math.Pi / 180
y := hexEarthRadius * math.Log(math.Tan(math.Pi/4+lat*math.Pi/360))
return x, y
}
func hexInvMercator(x, y float64) (lat, lon float64) {
lon = x / hexEarthRadius * 180 / math.Pi
lat = (2*math.Atan(math.Exp(y/hexEarthRadius)) - math.Pi/2) * 180 / math.Pi
return lat, lon
}
// hexSizeForRes is the hex circumradius (center→corner) in Web Mercator units for a
// display resolution. Resolution equals the Leaflet zoom level (see zoomToHexRes), so
// the size scales as 2^-zoom and the hex keeps a constant ~hexTargetPx on-screen size
// regardless of zoom. hexCellAt (binning) and hexBoundary (drawing) both read this, so
// they stay consistent for a given cell id.
func hexSizeForRes(res int) float64 {
return (hexTargetPx / 2) * mercUPPZ0 / math.Pow(2, float64(res))
}
// hexMaxLat is the Web Mercator latitude limit. The projection (hexMercator)
// diverges toward ±90° — tan(π/4 + lat·π/360) → ∞ — so points beyond this would
// produce NaN cell rings via hexInvMercator. Coverage is therefore only defined
// within ±hexMaxLat; polar submissions are clamped to the edge (#17).
const hexMaxLat = 85.05112878
// hexCellAt returns a stable cell id ("res:q:r") for the lat/lon at res. Latitude
// is clamped to ±hexMaxLat so near-polar points bin to the edge instead of
// producing NaN geometry.
func hexCellAt(lat, lon float64, res int) string {
if lat > hexMaxLat {
lat = hexMaxLat
} else if lat < -hexMaxLat {
lat = -hexMaxLat
}
size := hexSizeForRes(res)
x, y := hexMercator(lat, lon)
q := (math.Sqrt(3)/3*x - 1.0/3*y) / size
r := (2.0 / 3 * y) / size
qi, ri := hexRound(q, r)
return fmt.Sprintf("%d:%d:%d", res, qi, ri)
}
// hexRound rounds fractional axial coords to the nearest hex via cube rounding.
func hexRound(q, r float64) (int, int) {
x, z := q, r
y := -x - z
rx, ry, rz := math.Round(x), math.Round(y), math.Round(z)
dx, dy, dz := math.Abs(rx-x), math.Abs(ry-y), math.Abs(rz-z)
switch {
case dx > dy && dx > dz:
rx = -ry - rz
case dy > dz:
ry = -rx - rz
default:
rz = -rx - ry
}
return int(rx), int(rz)
}
// hexBoundary returns the cell's 6 corners as a closed [lon,lat] ring (GeoJSON
// order), or nil if the cell id is malformed.
func hexBoundary(cellID string) [][2]float64 {
res, q, r, ok := parseHexCell(cellID)
if !ok {
return nil
}
size := hexSizeForRes(res)
cx := size * (math.Sqrt(3)*float64(q) + math.Sqrt(3)/2*float64(r))
cy := size * (1.5 * float64(r))
ring := make([][2]float64, 0, 7)
for i := 0; i < 6; i++ {
ang := math.Pi / 180 * float64(60*i-30)
lat, lon := hexInvMercator(cx+size*math.Cos(ang), cy+size*math.Sin(ang))
ring = append(ring, [2]float64{lon, lat})
}
ring = append(ring, ring[0]) // close the ring
return ring
}
func parseHexCell(id string) (res, q, r int, ok bool) {
p := strings.Split(id, ":")
if len(p) != 3 {
return 0, 0, 0, false
}
a, e1 := strconv.Atoi(p[0])
b, e2 := strconv.Atoi(p[1])
c, e3 := strconv.Atoi(p[2])
if e1 != nil || e2 != nil || e3 != nil {
return 0, 0, 0, false
}
return a, b, c, true
}
+54
View File
@@ -0,0 +1,54 @@
package main
import (
"math"
"testing"
)
// TestHexCellAtClampsPolarLatitude verifies #17: latitudes past the Web Mercator
// limit are clamped, so near-polar submissions bin to the edge cell and produce
// finite geometry instead of NaN rings.
func TestHexCellAtClampsPolarLatitude(t *testing.T) {
for _, lat := range []float64{89.9, 90.0, -89.9, -90.0} {
cell := hexCellAt(lat, 3.72, 9)
clamped := math.Copysign(hexMaxLat, lat)
if want := hexCellAt(clamped, 3.72, 9); cell != want {
t.Fatalf("lat %.1f should clamp to %q, got %q", lat, want, cell)
}
ring := hexBoundary(cell)
if ring == nil {
t.Fatalf("lat %.1f produced no ring", lat)
}
for _, pt := range ring {
if math.IsNaN(pt[0]) || math.IsNaN(pt[1]) || math.IsInf(pt[0], 0) || math.IsInf(pt[1], 0) {
t.Fatalf("lat %.1f produced non-finite ring point %v", lat, pt)
}
}
}
}
func TestHexCellAtStableAndDistinct(t *testing.T) {
a := hexCellAt(51.0500, 3.7200, 9)
b := hexCellAt(51.0500, 3.7200, 9)
if a == "" || a != b {
t.Fatalf("stable cell expected, got %q %q", a, b)
}
c := hexCellAt(51.2000, 3.7200, 9) // ~17 km away
if c == a {
t.Fatalf("distant point should differ, both %q", a)
}
}
func TestHexBoundaryClosedRing(t *testing.T) {
cell := hexCellAt(51.05, 3.72, 9)
ring := hexBoundary(cell)
if len(ring) != 7 {
t.Fatalf("expected 7 points (closed hex), got %d", len(ring))
}
if ring[0] != ring[6] {
t.Fatalf("ring not closed: %v vs %v", ring[0], ring[6])
}
if hexBoundary("garbage") != nil {
t.Fatalf("malformed cell should return nil")
}
}
+76
View File
@@ -0,0 +1,76 @@
package main
import (
"encoding/json"
"net/http"
"regexp"
"strings"
)
// ResolvePrefixResp is the tiny reply for /api/nodes/resolve — lets a client
// resolve a heard 2-3 byte path prefix (or full pubkey) to a node name without
// fetching the whole node list. Read-only.
type ResolvePrefixResp struct {
Prefix string `json:"prefix"`
Pubkey string `json:"pubkey,omitempty"`
Name string `json:"name,omitempty"`
Ambiguous bool `json:"ambiguous"`
}
var hexPrefixRe = regexp.MustCompile(`^[0-9a-f]{2,64}$`)
// minResolvePrefixHex is the shortest accepted prefix. 1-byte (2 hex) keys are
// never stored — the ingestor rejects heard keys shorter than 2 bytes — so the
// floor matches the data model and, by ruling out the 256 two-char prefixes,
// blunts trivial enumeration of every node name through this endpoint (#15).
const minResolvePrefixHex = 4
func (s *Server) handleResolvePrefix(w http.ResponseWriter, r *http.Request) {
pfx := strings.ToLower(strings.TrimSpace(r.URL.Query().Get("prefix")))
if !hexPrefixRe.MatchString(pfx) {
http.Error(w, "prefix must be hex", http.StatusBadRequest)
return
}
if len(pfx) < minResolvePrefixHex {
http.Error(w, "prefix must be at least 4 hex chars", http.StatusBadRequest)
return
}
if s.db == nil || s.db.conn == nil {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
// LIMIT 2: we only need to know unique vs ambiguous. nodes.public_key is the
// PK and stored lowercase; pfx is validated hex so the LIKE pattern is safe.
rows, err := s.db.conn.Query(`SELECT public_key, COALESCE(name,'') FROM nodes WHERE public_key LIKE ? LIMIT 2`, pfx+"%")
if err != nil {
http.Error(w, "query failed", http.StatusInternalServerError)
return
}
defer rows.Close()
var pks, names []string
for rows.Next() {
var pk, nm string
if err := rows.Scan(&pk, &nm); err != nil {
http.Error(w, "scan failed", http.StatusInternalServerError)
return
}
pks = append(pks, pk)
names = append(names, nm)
}
resp := ResolvePrefixResp{Prefix: pfx}
switch len(pks) {
case 1:
// Parity with /api/nodes/search and /api/resolve-hops: never reveal the
// identity of a blacklisted or hidden-prefix node (#1181). Report it as
// not-found rather than leaking the name the rest of the API hides.
if !s.cfg.IsBlacklisted(pks[0]) && !s.cfg.IsNameHidden(names[0]) {
resp.Pubkey = pks[0]
resp.Name = names[0]
}
default:
resp.Ambiguous = len(pks) > 1 // 0 → not found (name empty), >1 → ambiguous
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
}
+86
View File
@@ -0,0 +1,86 @@
package main
import (
"encoding/json"
"net/http/httptest"
"testing"
"github.com/gorilla/mux"
)
func serveResolve(srv *Server, path string) *httptest.ResponseRecorder {
router := mux.NewRouter()
router.HandleFunc("/api/nodes/resolve", srv.handleResolvePrefix).Methods("GET")
rr := httptest.NewRecorder()
router.ServeHTTP(rr, httptest.NewRequest("GET", path, nil))
return rr
}
func TestResolvePrefix(t *testing.T) {
db := setupTestDBv2(t)
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count)
VALUES ('efef7943505052b47f1809488ea4b4d3942d4ed72d2b1953b90a9f5e62a65fb5','NodeUnique','repeater','t','t',1)`)
// Two nodes sharing the 4-hex prefix aabb → ambiguous at the new minimum.
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count)
VALUES ('aabb110000000000000000000000000000000000000000000000000000000000','NodeA','repeater','t','t',1)`)
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count)
VALUES ('aabb220000000000000000000000000000000000000000000000000000000000','NodeB','repeater','t','t',1)`)
srv := &Server{db: db}
// unique 3-byte prefix → name
var r1 ResolvePrefixResp
json.Unmarshal(serveResolve(srv, "/api/nodes/resolve?prefix=efef79").Body.Bytes(), &r1)
if r1.Name != "NodeUnique" || r1.Ambiguous {
t.Fatalf("unique: %+v", r1)
}
// colliding 4-hex prefix (aabb…) → ambiguous, no name
var r2 ResolvePrefixResp
json.Unmarshal(serveResolve(srv, "/api/nodes/resolve?prefix=aabb").Body.Bytes(), &r2)
if !r2.Ambiguous || r2.Name != "" {
t.Fatalf("ambiguous: %+v", r2)
}
// not found → empty name, not ambiguous
var r3 ResolvePrefixResp
json.Unmarshal(serveResolve(srv, "/api/nodes/resolve?prefix=dead").Body.Bytes(), &r3)
if r3.Name != "" || r3.Ambiguous {
t.Fatalf("notfound: %+v", r3)
}
// non-hex prefix → 400
if serveResolve(srv, "/api/nodes/resolve?prefix=xyz").Code != 400 {
t.Fatal("non-hex prefix should be 400")
}
// #15: prefixes shorter than 4 hex are rejected (kills 256-prefix enumeration)
for _, short := range []string{"a", "aa", "abc"} {
if code := serveResolve(srv, "/api/nodes/resolve?prefix="+short).Code; code != 400 {
t.Fatalf("prefix %q (<4 hex) should be 400, got %d", short, code)
}
}
}
// TestResolvePrefixHidesBlacklistedAndHidden verifies the #15 parity fix: a
// unique match that is blacklisted or whose name is hidden (#1181) resolves as
// not-found, never leaking an identity the rest of the API hides.
func TestResolvePrefixHidesBlacklistedAndHidden(t *testing.T) {
db := setupTestDBv2(t)
const blPK = "bbcc110000000000000000000000000000000000000000000000000000000000"
const hidPK = "ddee220000000000000000000000000000000000000000000000000000000000"
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count)
VALUES ('`+blPK+`','BlacklistedNode','repeater','t','t',1)`)
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count)
VALUES ('`+hidPK+`','🚫HiddenNode','repeater','t','t',1)`)
srv := &Server{db: db, cfg: &Config{
NodeBlacklist: []string{blPK},
HiddenNamePrefixes: []string{"🚫"},
}}
var rb ResolvePrefixResp
json.Unmarshal(serveResolve(srv, "/api/nodes/resolve?prefix=bbcc11").Body.Bytes(), &rb)
if rb.Name != "" || rb.Pubkey != "" || rb.Ambiguous {
t.Fatalf("blacklisted node must resolve as not-found: %+v", rb)
}
var rh ResolvePrefixResp
json.Unmarshal(serveResolve(srv, "/api/nodes/resolve?prefix=ddee22").Body.Bytes(), &rh)
if rh.Name != "" || rh.Pubkey != "" || rh.Ambiguous {
t.Fatalf("hidden-prefix node must resolve as not-found: %+v", rh)
}
}
+4
View File
@@ -13,14 +13,18 @@
"/api/healthz",
"/api/known-channels",
"/api/nodes/clock-skew",
"/api/nodes/resolve",
"/api/nodes/{pubkey}/battery",
"/api/nodes/{pubkey}/clock-skew",
"/api/nodes/{pubkey}/reach",
"/api/nodes/{pubkey}/rx-coverage",
"/api/observers/clock-skew",
"/api/paths/inspect",
"/api/perf/io",
"/api/perf/sqlite",
"/api/perf/write-sources",
"/api/rx-coverage",
"/api/rx-leaderboard",
"/api/scope-stats",
"/api/spec"
]
+11 -3
View File
@@ -267,10 +267,17 @@ func (s *Server) RegisterRoutes(r *mux.Router) {
r.HandleFunc("/api/nodes/{pubkey}/clock-skew", s.handleNodeClockSkew).Methods("GET")
r.HandleFunc("/api/observers/clock-skew", s.handleObserverClockSkew).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/neighbors", s.handleNodeNeighbors).Methods("GET")
// Keep specific sub-routes (…/reach) registered BEFORE the catch-all
// /api/nodes/{pubkey} — mux matches in registration order, so reordering
// this below the catch-all would shadow it and break the route.
// Keep specific sub-routes (…/reach, …/rx-coverage) registered BEFORE the
// catch-all /api/nodes/{pubkey} — mux matches in registration order, so
// reordering these below the catch-all would shadow them and break the route.
r.HandleFunc("/api/nodes/{pubkey}/reach", s.handleNodeReach).Methods("GET")
// Coverage routes are always registered; each handler 404s when the opt-in
// clientRxCoverage flag is off (a clean 404 rather than the SPA fallback that
// an unregistered /api route would hit). See requireClientRxCoverage.
r.HandleFunc("/api/nodes/{pubkey}/rx-coverage", s.handleNodeRxCoverage).Methods("GET")
r.HandleFunc("/api/nodes/resolve", s.handleResolvePrefix).Methods("GET")
r.HandleFunc("/api/rx-coverage", s.handleRxCoverage).Methods("GET")
r.HandleFunc("/api/rx-leaderboard", s.handleRxLeaderboard).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}", s.handleNodeDetail).Methods("GET")
r.HandleFunc("/api/nodes", s.handleNodes).Methods("GET")
@@ -445,6 +452,7 @@ func (s *Server) handleConfigClient(w http.ResponseWriter, r *http.Request) {
MapDarkTileProvider: s.cfg.MapDarkTileProvider,
Tiles: s.cfg.Tiles,
Customizer: CustomizerClientConfig{DisabledTabs: disabledTabs},
ClientRxCoverage: s.cfg.ClientRxCoverageEnabled(),
})
}
+370
View File
@@ -0,0 +1,370 @@
package main
import (
"encoding/json"
"net/http"
"sort"
"strconv"
"strings"
"github.com/gorilla/mux"
)
// coverageRow is one raw reception read from client_receptions.
type coverageRow struct {
Lat, Lon float64
SNR *float64
RSSI *int
HeardKey string // directly-heard node key (2-3 byte prefix or full pubkey), lowercase
RxAt string // reception time (RFC3339); used to pick the latest SNR per node
}
// coverageFeatureCap bounds the number of hex cells returned in one response.
// A wide bbox at high zoom over the 30-day window could otherwise emit multi-MB
// GeoJSON; when more cells exist the densest are kept and Truncated is set (#12).
const coverageFeatureCap = 5000
// coverageCellNodeCap bounds the per-cell node breakdown shipped on the wire
// (the client only renders the top ~10). NodesTruncated flags that more were
// heard than returned (#11).
const coverageCellNodeCap = 25
// GeoJSON output (named structs, no map[string]interface{} — AGENTS.md).
// Truncated is a non-standard foreign member (ignored by GeoJSON consumers like
// Leaflet) that signals the cell list was capped at coverageFeatureCap.
type CoverageFeatureCollection struct {
Type string `json:"type"` // "FeatureCollection"
Features []CoverageFeature `json:"features"`
Truncated bool `json:"truncated,omitempty"`
// Per-node summary (set only by the per-node endpoint): total mobile-client
// receptions of this node and how many distinct companions heard it. Foreign
// members, omitempty so the global endpoint's payload is unchanged (#3).
MobileReceptions int `json:"mobile_receptions,omitempty"`
MobileClients int `json:"mobile_clients,omitempty"`
}
type CoverageFeature struct {
Type string `json:"type"` // "Feature"
Geometry CoveragePolygon `json:"geometry"`
Properties CoverageProperties `json:"properties"`
}
type CoveragePolygon struct {
Type string `json:"type"` // "Polygon"
Coordinates [][][2]float64 `json:"coordinates"` // one ring: [ [ [lon,lat], ... ] ]
}
type CoverageProperties struct {
Cell string `json:"cell"`
Count int `json:"count"`
BestSNR *float64 `json:"best_snr"`
HasSig bool `json:"has_sig"` // false → render grey (no signal metric)
Nodes []CoverageNode `json:"nodes"` // per-node breakdown, strongest latest-SNR first
NodesTruncated bool `json:"nodes_truncated,omitempty"` // true → more nodes heard than returned (#11)
}
// CoverageNode is one directly-heard node within a cell, with its latest SNR.
type CoverageNode struct {
Prefix string `json:"prefix"` // heard_key (resolved to Name when unique)
Name string `json:"name,omitempty"` // node name, empty if unknown/ambiguous prefix
SNR *float64 `json:"snr"` // latest SNR (by rx_at); nil → heard without signal
Count int `json:"count"`
}
type covAgg struct {
count int
bestSNR *float64
hasSig bool
nodes map[string]*covNodeAgg
}
// covNodeAgg tracks, per directly-heard node within a cell, its reception count and
// the SNR of its most recent reception (by rx_at). name/prefix are the resolved node
// name (when known) and a display prefix fallback. nameKeyLen records the heard_key
// length that set the current name, so the chosen identity is the most specific one
// regardless of row order (#20).
type covNodeAgg struct {
count int
latestAt string
latestSNR *float64
name string
nameKeyLen int
prefix string
}
// nodeResolver maps a heard_key (2-3 byte prefix or full pubkey) to a canonical
// identity key and a display name. A unique match returns (pubkey, name) so the same
// node heard under different prefix lengths collapses into one bucket; unknown or
// ambiguous keys return (heardKey, "") and stay distinct. nil disables resolution.
type nodeResolver func(heardKey string) (key, name string)
// aggregateCoverage bins raw rows into display-resolution hex cells, keeping the
// best (max) SNR per cell, and emits GeoJSON polygons. resolve (may be nil) collapses
// per-node receptions by resolved node identity.
func aggregateCoverage(rows []coverageRow, res int, resolve nodeResolver) CoverageFeatureCollection {
byCell := map[string]*covAgg{}
for _, row := range rows {
cell := hexCellAt(row.Lat, row.Lon, res)
a := byCell[cell]
if a == nil {
a = &covAgg{}
byCell[cell] = a
}
a.count++
if row.SNR != nil {
a.hasSig = true
if a.bestSNR == nil || *row.SNR > *a.bestSNR {
v := *row.SNR
a.bestSNR = &v
}
}
if row.HeardKey != "" {
if a.nodes == nil {
a.nodes = map[string]*covNodeAgg{}
}
key, name := row.HeardKey, ""
if resolve != nil {
if k, n := resolve(row.HeardKey); k != "" {
key, name = k, n
}
}
na := a.nodes[key]
if na == nil {
na = &covNodeAgg{prefix: row.HeardKey}
a.nodes[key] = na
}
// Lock the display identity to the MOST SPECIFIC (longest) heard_key
// that resolved to a non-empty name, tie-broken lexicographically, so
// the name no longer flaps with row/map order (#20). A full-pubkey
// reception thus outranks a short-prefix one for the same node.
if name != "" && (na.name == "" || len(row.HeardKey) > na.nameKeyLen ||
(len(row.HeardKey) == na.nameKeyLen && name < na.name)) {
na.name = name
na.nameKeyLen = len(row.HeardKey)
}
// Display-prefix fallback (shown when name is empty): same precedence so
// it is also order-independent.
if len(row.HeardKey) > len(na.prefix) ||
(len(row.HeardKey) == len(na.prefix) && row.HeardKey < na.prefix) {
na.prefix = row.HeardKey
}
na.count++
// rx_at is RFC3339, so lexical >= is chronological; keep the latest
// SNR. The first row always wins (latestAt starts "", and any value
// >= ""), so no separate count==1 guard is needed.
if row.RxAt >= na.latestAt {
na.latestAt = row.RxAt
na.latestSNR = row.SNR
}
}
}
fc := CoverageFeatureCollection{Type: "FeatureCollection", Features: []CoverageFeature{}}
for cell, a := range byCell {
ring := hexBoundary(cell)
if ring == nil {
continue
}
nodes, nodesTrunc := sortedCoverageNodes(a.nodes)
fc.Features = append(fc.Features, CoverageFeature{
Type: "Feature",
Geometry: CoveragePolygon{Type: "Polygon", Coordinates: [][][2]float64{ring}},
Properties: CoverageProperties{
Cell: cell, Count: a.count, BestSNR: a.bestSNR, HasSig: a.hasSig,
Nodes: nodes, NodesTruncated: nodesTrunc,
},
})
}
// Bound the response: when more cells exist than coverageFeatureCap, keep the
// densest (highest count) and flag the truncation, so a wide/zoomed-out query
// can't emit unbounded multi-MB GeoJSON (#12).
if len(fc.Features) > coverageFeatureCap {
sort.Slice(fc.Features, func(i, j int) bool {
ci, cj := fc.Features[i].Properties.Count, fc.Features[j].Properties.Count
if ci != cj {
return ci > cj // densest first
}
return fc.Features[i].Properties.Cell < fc.Features[j].Properties.Cell // deterministic tie-break
})
fc.Features = fc.Features[:coverageFeatureCap]
fc.Truncated = true
}
// Map iteration is randomized, so sort features by cell for a deterministic
// payload — stable ETag/caching and a non-flaky "first feature" in e2e (#8).
sort.Slice(fc.Features, func(i, j int) bool {
return fc.Features[i].Properties.Cell < fc.Features[j].Properties.Cell
})
return fc
}
// sortedCoverageNodes flattens the per-node aggregates into a slice sorted by latest
// SNR descending (nodes heard without a signal sort last), tie-broken by count then
// prefix for a stable order. The slice is capped at coverageCellNodeCap; truncated
// reports whether more nodes were heard in the cell than returned (#11).
func sortedCoverageNodes(m map[string]*covNodeAgg) (nodes []CoverageNode, truncated bool) {
out := make([]CoverageNode, 0, len(m))
for _, na := range m {
out = append(out, CoverageNode{Prefix: na.prefix, Name: na.name, SNR: na.latestSNR, Count: na.count})
}
sort.Slice(out, func(i, j int) bool {
si, sj := out[i].SNR, out[j].SNR
if (si == nil) != (sj == nil) {
return si != nil // signal before no-signal
}
if si != nil && *si != *sj {
return *si > *sj
}
if out[i].Count != out[j].Count {
return out[i].Count > out[j].Count
}
return out[i].Prefix < out[j].Prefix
})
if len(out) > coverageCellNodeCap {
return out[:coverageCellNodeCap], true
}
return out, false
}
type bbox struct{ MinLat, MinLon, MaxLat, MaxLon float64 }
// coverageHeardKeyCandidates returns the exact heard_key values that identify a
// node: its full pubkey (stored with heard_keylen 32) and the 2-byte (4 hex) and
// 3-byte (6 hex) prefixes a relay logs. Matching heard_key IN (these) is
// equivalent to the old "heard_keylen=32 AND heard_key=? OR heard_keylen IN (2,3)
// AND substr(?,1,keylen*2)=heard_key", but sargable — so the (heard_key, …)
// composite index seeks the few matching rows instead of scanning the bbox (#5).
func coverageHeardKeyCandidates(pubkey string) []string {
pk := strings.ToLower(pubkey)
seen := map[string]bool{}
out := make([]string, 0, 3)
for _, c := range []string{pk, prefixOrEmpty(pk, 6), prefixOrEmpty(pk, 4)} {
if c != "" && !seen[c] {
seen[c] = true
out = append(out, c)
}
}
return out
}
func prefixOrEmpty(s string, n int) string {
if len(s) >= n {
return s[:n]
}
return ""
}
// sqlPlaceholders returns "?,?,…" with n placeholders (n >= 1).
func sqlPlaceholders(n int) string {
if n <= 1 {
return "?"
}
return strings.Repeat("?,", n-1) + "?"
}
// queryCoverageRows returns raw coverage rows where the directly-heard node
// matches the target pubkey by its 2-3 byte prefix (or full pubkey), within the
// bbox. Read-only (server RO connection).
func (s *Server) queryCoverageRows(pubkey string, b bbox) ([]coverageRow, error) {
cands := coverageHeardKeyCandidates(pubkey)
args := make([]interface{}, 0, len(cands)+4)
for _, c := range cands {
args = append(args, c)
}
args = append(args, b.MinLat, b.MaxLat, b.MinLon, b.MaxLon)
rows, err := s.db.conn.Query(`
SELECT lat, lon, snr, rssi, heard_key, rx_at
FROM client_receptions
WHERE heard_key IN (`+sqlPlaceholders(len(cands))+`)
AND lat BETWEEN ? AND ? AND lon BETWEEN ? AND ?`, args...)
if err != nil {
return nil, err
}
defer rows.Close()
return scanCoverageRows(rows)
}
// mobileRxStats returns the total mobile-client receptions of a node (by its
// 2-3 byte prefix or full pubkey) and the number of distinct contributing clients.
func (s *Server) mobileRxStats(pubkey string) (count, clients int) {
if s.db == nil || s.db.conn == nil {
return 0, 0
}
cands := coverageHeardKeyCandidates(pubkey)
args := make([]interface{}, len(cands))
for i, c := range cands {
args[i] = c
}
s.db.conn.QueryRow(`
SELECT COUNT(*), COUNT(DISTINCT rx_pubkey) FROM client_receptions
WHERE heard_key IN (`+sqlPlaceholders(len(cands))+`)`, args...).Scan(&count, &clients)
return count, clients
}
// zoomToHexRes maps a Leaflet zoom level to the display resolution used for hex
// binning. Resolution == zoom (clamped to a sane range) so hex size tracks the map
// scale 1:1 and renders at a constant ~hexTargetPx (see hexSizeForRes). The clamp also
// guards the missing-param case (z parses to 0).
func zoomToHexRes(z int) int {
switch {
case z < 3:
return 3
case z > 18:
return 18
default:
return z
}
}
func parseBBox(s string) (bbox, bool) {
p := strings.Split(s, ",")
if len(p) != 4 {
return bbox{}, false
}
v := make([]float64, 4)
for i := range p {
f, err := strconv.ParseFloat(strings.TrimSpace(p[i]), 64)
if err != nil {
return bbox{}, false
}
v[i] = f
}
return bbox{MinLat: v[0], MinLon: v[1], MaxLat: v[2], MaxLon: v[3]}, true
}
// handleNodeRxCoverage serves per-node mobile RX coverage as a GeoJSON hex grid.
func (s *Server) handleNodeRxCoverage(w http.ResponseWriter, r *http.Request) {
if !s.requireClientRxCoverage(w, r) {
return
}
pubkey := strings.ToLower(mux.Vars(r)["pubkey"])
// Mirror handleNodeReach's gate at this same {pubkey}: reject malformed keys,
// and 404 blacklisted / hidden-prefix nodes. Hiding only the node *name* (via
// heardKeyResolver) still leaked the GPS hex bins and mobile_receptions /
// mobile_clients counts for a node the rest of the API hides (#1727 r2).
if !isHexPubkey(pubkey) {
http.Error(w, "invalid pubkey: expected 64 hex chars", http.StatusBadRequest)
return
}
if (s.cfg != nil && s.cfg.IsBlacklisted(pubkey)) || s.isPubkeyHidden(pubkey) {
http.NotFound(w, r)
return
}
b, ok := parseBBox(r.URL.Query().Get("bbox"))
if !ok {
http.Error(w, "bbox required as minLat,minLon,maxLat,maxLon", http.StatusBadRequest)
return
}
if s.db == nil || s.db.conn == nil {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
z, _ := strconv.Atoi(r.URL.Query().Get("z"))
rows, err := s.queryCoverageRows(pubkey, b)
if err != nil {
http.Error(w, "query failed", http.StatusInternalServerError)
return
}
fc := aggregateCoverage(rows, zoomToHexRes(z), s.heardKeyResolverFor(rows))
// Attach the node-wide reception/contributor totals (#3): the bbox limits the
// hex features to the current view, but these summarise all of this node's
// mobile coverage so the UI can show "heard by N clients" regardless of pan.
fc.MobileReceptions, fc.MobileClients = s.mobileRxStats(pubkey)
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(fc)
}
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"encoding/json"
"net/http/httptest"
"testing"
"github.com/gorilla/mux"
)
func seedCoverageDB(t *testing.T) *DB {
db := setupTestDBv2(t)
mustExecDB(t, db, `CREATE TABLE client_receptions (
id INTEGER PRIMARY KEY AUTOINCREMENT, rx_pubkey TEXT, heard_key TEXT, heard_keylen INTEGER,
rssi INTEGER, snr REAL, lat REAL, lon REAL, pos_acc_m REAL, rx_at TEXT, ingested_at TEXT, src TEXT)`)
mustExecDB(t, db, `CREATE TABLE client_observers (pubkey TEXT PRIMARY KEY, name TEXT, last_seen TEXT)`)
return db
}
func TestQueryCoverageRowsByPrefixAndBBox(t *testing.T) {
db := seedCoverageDB(t)
mustExecDB(t, db, `INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src)
VALUES ('comp','aabbcc',3,-6,51.05,3.72,'t','t','rxlog')`)
srv := &Server{db: db, cfg: &Config{ClientRxCoverage: &ClientRxCoverageConfig{Enabled: true}}}
rows, err := srv.queryCoverageRows("aabbccddeeff00112233", bbox{MinLat: 50, MinLon: 3, MaxLat: 52, MaxLon: 4})
if err != nil {
t.Fatal(err)
}
if len(rows) != 1 {
t.Fatalf("expected 1 row by prefix, got %d", len(rows))
}
rows, _ = srv.queryCoverageRows("aabbccddeeff00112233", bbox{MinLat: 0, MinLon: 0, MaxLat: 1, MaxLon: 1})
if len(rows) != 0 {
t.Fatalf("bbox filter failed, got %d", len(rows))
}
}
func TestMobileRxStats(t *testing.T) {
db := seedCoverageDB(t)
mustExecDB(t, db, `INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src) VALUES ('compA','aabbcc',3,-6,51.05,3.72,'t1','t','rxlog')`)
mustExecDB(t, db, `INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src) VALUES ('compB','aabbcc',3,-8,51.06,3.73,'t2','t','rxlog')`)
mustExecDB(t, db, `INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src) VALUES ('compA','ffeedd',3,-5,51.07,3.74,'t3','t','rxlog')`)
srv := &Server{db: db, cfg: &Config{ClientRxCoverage: &ClientRxCoverageConfig{Enabled: true}}}
c, cl := srv.mobileRxStats("aabbccddeeff00112233")
if c != 2 || cl != 2 {
t.Fatalf("got count=%d clients=%d, want 2/2", c, cl)
}
}
func serveRxCoverage(srv *Server, path string) *httptest.ResponseRecorder {
router := mux.NewRouter()
router.HandleFunc("/api/nodes/{pubkey}/rx-coverage", srv.handleNodeRxCoverage).Methods("GET")
req := httptest.NewRequest("GET", path, nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
return rr
}
// nodePK is a full 64-hex pubkey whose 3-byte prefix is the seeded heard_key.
const nodePK = "aabbcc0000000000000000000000000000000000000000000000000000000000"
func TestRxCoverageEndpointGeoJSON(t *testing.T) {
db := seedCoverageDB(t)
mustExecDB(t, db, `INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src)
VALUES ('comp','aabbcc',3,-6,51.05,3.72,'t','t','rxlog')`)
srv := &Server{db: db, cfg: &Config{ClientRxCoverage: &ClientRxCoverageConfig{Enabled: true}}}
rr := serveRxCoverage(srv, "/api/nodes/"+nodePK+"/rx-coverage?bbox=50,3,52,4&z=12")
if rr.Code != 200 {
t.Fatalf("status %d body %s", rr.Code, rr.Body.String())
}
var fc CoverageFeatureCollection
if err := json.Unmarshal(rr.Body.Bytes(), &fc); err != nil {
t.Fatalf("decode: %v", err)
}
if fc.Type != "FeatureCollection" || len(fc.Features) != 1 {
t.Fatalf("unexpected fc: %+v", fc)
}
// #3: the per-node response carries the node-wide mobile totals (wired in
// from mobileRxStats). One reception from one companion → 1/1.
if fc.MobileReceptions != 1 || fc.MobileClients != 1 {
t.Fatalf("want mobile_receptions=1 mobile_clients=1, got %d/%d", fc.MobileReceptions, fc.MobileClients)
}
if serveRxCoverage(srv, "/api/nodes/"+nodePK+"/rx-coverage").Code != 400 {
t.Fatal("missing bbox should be 400")
}
// Non-hex pubkey is rejected up front (parity with handleNodeReach).
if serveRxCoverage(srv, "/api/nodes/nothex/rx-coverage?bbox=50,3,52,4").Code != 400 {
t.Fatal("non-hex pubkey should be 400")
}
}
// TestNodeRxCoverageHidesBlacklistedAndHidden verifies #1727 r2 must-fix #1: the
// per-node coverage endpoint must 404 for blacklisted or hidden-prefix nodes, so
// their GPS hex bins / counts aren't retrievable at a pubkey the rest of the API
// hides — not just the node name.
func TestNodeRxCoverageHidesBlacklistedAndHidden(t *testing.T) {
const hidPK = "ddee110000000000000000000000000000000000000000000000000000000000"
db := seedCoverageDB(t)
mustExecDB(t, db, `INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src)
VALUES ('comp','aabbcc',3,-6,51.05,3.72,'t','t','rxlog')`)
mustExecDB(t, db, `INSERT INTO nodes (public_key,name,role,last_seen,first_seen,advert_count) VALUES ('`+hidPK+`','🚫Secret','repeater','t','t',1)`)
srv := &Server{db: db, cfg: &Config{
ClientRxCoverage: &ClientRxCoverageConfig{Enabled: true},
NodeBlacklist: []string{nodePK},
HiddenNamePrefixes: []string{"🚫"},
}}
if code := serveRxCoverage(srv, "/api/nodes/"+nodePK+"/rx-coverage?bbox=50,3,52,4").Code; code != 404 {
t.Fatalf("blacklisted node coverage should be 404, got %d", code)
}
if code := serveRxCoverage(srv, "/api/nodes/"+hidPK+"/rx-coverage?bbox=50,3,52,4").Code; code != 404 {
t.Fatalf("hidden-prefix node coverage should be 404, got %d", code)
}
}
+241
View File
@@ -0,0 +1,241 @@
package main
import (
"encoding/json"
"fmt"
"math"
"testing"
)
// TestAggregateCoverageCapsNodesPerCell verifies #11: a cell that heard more than
// coverageCellNodeCap distinct nodes ships at most that many, with NodesTruncated set.
func TestAggregateCoverageCapsNodesPerCell(t *testing.T) {
rows := make([]coverageRow, 0, coverageCellNodeCap+5)
for i := 0; i < coverageCellNodeCap+5; i++ {
rows = append(rows, coverageRow{
Lat: 51.05, Lon: 3.72, SNR: covF(float64(-i)),
HeardKey: fmt.Sprintf("aa%06x", i), RxAt: "2026-06-01T10:00:00Z",
})
}
fc := aggregateCoverage(rows, 9, nil)
if len(fc.Features) != 1 {
t.Fatalf("expected 1 cell, got %d", len(fc.Features))
}
p := fc.Features[0].Properties
if len(p.Nodes) != coverageCellNodeCap || !p.NodesTruncated {
t.Fatalf("want %d nodes + truncated, got %d nodes truncated=%v", coverageCellNodeCap, len(p.Nodes), p.NodesTruncated)
}
}
// TestAggregateCoverageCapsFeatures verifies #12: a query spanning more than
// coverageFeatureCap cells is bounded to that many features with Truncated set,
// and a smaller query is not truncated.
func TestAggregateCoverageCapsFeatures(t *testing.T) {
// 0.1° spacing >> a res-9 cell (~4 km), so each point lands in its own cell.
rows := make([]coverageRow, 0, coverageFeatureCap+200)
side := 75 // 75*75 = 5625 > 5000
for i := 0; i < side*side; i++ {
lat := 10.0 + float64(i/side)*0.1
lon := 10.0 + float64(i%side)*0.1
rows = append(rows, coverageRow{Lat: lat, Lon: lon, SNR: covF(-5)})
}
fc := aggregateCoverage(rows, 9, nil)
if len(fc.Features) != coverageFeatureCap || !fc.Truncated {
t.Fatalf("want %d features + truncated, got %d truncated=%v", coverageFeatureCap, len(fc.Features), fc.Truncated)
}
// Still sorted by cell after truncation.
for i := 1; i < len(fc.Features); i++ {
if fc.Features[i-1].Properties.Cell > fc.Features[i].Properties.Cell {
t.Fatalf("truncated features not sorted by cell at %d", i)
}
}
// A small query is not truncated.
small := aggregateCoverage(rows[:10], 9, nil)
if small.Truncated {
t.Fatalf("small query should not be truncated")
}
}
func covF(f float64) *float64 { return &f }
func TestAggregateCoverageBucketsBestSNR(t *testing.T) {
rows := []coverageRow{
{Lat: 51.05000, Lon: 3.72000, SNR: covF(-12)},
{Lat: 51.05001, Lon: 3.72001, SNR: covF(-6)}, // same cell, stronger
}
fc := aggregateCoverage(rows, 9, nil)
if len(fc.Features) != 1 {
t.Fatalf("expected 1 cell, got %d", len(fc.Features))
}
if p := fc.Features[0].Properties; p.BestSNR == nil || *p.BestSNR != -6 || p.Count != 2 || !p.HasSig {
t.Fatalf("bad props: %+v", fc.Features[0].Properties)
}
if g := fc.Features[0].Geometry; g.Type != "Polygon" || len(g.Coordinates) != 1 {
t.Fatalf("bad geometry: %+v", g)
}
if _, err := json.Marshal(fc); err != nil {
t.Fatalf("marshal: %v", err)
}
}
func TestAggregateCoverageGreyWhenNoSignal(t *testing.T) {
fc := aggregateCoverage([]coverageRow{{Lat: 51.05, Lon: 3.72}}, 9, nil)
if len(fc.Features) != 1 || fc.Features[0].Properties.HasSig {
t.Fatalf("expected one grey (no-sig) cell, got %+v", fc.Features)
}
}
// TestAggregateCoverageNodeBreakdown covers the per-cell node list: each heard node
// keeps its latest SNR (by rx_at) and reception count, sorted strongest-first with
// heard-without-signal nodes last.
func TestAggregateCoverageNodeBreakdown(t *testing.T) {
rows := []coverageRow{
// node A: two receptions; the later one (t2) has the weaker SNR -10.
{Lat: 51.05, Lon: 3.72, SNR: covF(-4), HeardKey: "aabb", RxAt: "2026-06-01T10:00:00Z"},
{Lat: 51.05001, Lon: 3.72001, SNR: covF(-10), HeardKey: "aabb", RxAt: "2026-06-02T10:00:00Z"},
// node B: single reception, strongest latest SNR.
{Lat: 51.05, Lon: 3.72, SNR: covF(-6), HeardKey: "ccdd", RxAt: "2026-06-01T10:00:00Z"},
// node C: heard without a signal metric.
{Lat: 51.05, Lon: 3.72, HeardKey: "eeff", RxAt: "2026-06-01T10:00:00Z"},
}
fc := aggregateCoverage(rows, 9, nil)
if len(fc.Features) != 1 {
t.Fatalf("expected 1 cell, got %d", len(fc.Features))
}
nodes := fc.Features[0].Properties.Nodes
if len(nodes) != 3 {
t.Fatalf("expected 3 nodes, got %d (%+v)", len(nodes), nodes)
}
if nodes[0].Prefix != "ccdd" || nodes[0].SNR == nil || *nodes[0].SNR != -6 {
t.Errorf("node[0] want ccdd@-6 (strongest), got %+v", nodes[0])
}
if nodes[1].Prefix != "aabb" || nodes[1].SNR == nil || *nodes[1].SNR != -10 || nodes[1].Count != 2 {
t.Errorf("node[1] want aabb latest -10 count 2, got %+v", nodes[1])
}
if nodes[2].Prefix != "eeff" || nodes[2].SNR != nil {
t.Errorf("node[2] want eeff no-signal (last), got %+v", nodes[2])
}
}
// TestResolveHeardKey covers heard_key → (pubkey, name) resolution: a unique match
// returns the canonical pubkey + name; an ambiguous prefix (>1 node) and an
// unknown/empty key return the key itself with an empty name.
func TestResolveHeardKey(t *testing.T) {
db := seedCoverageDB(t)
mustExecDB(t, db, `INSERT INTO nodes (public_key,name,role) VALUES ('aabbccdd11223344','Alice','repeater')`)
mustExecDB(t, db, `INSERT INTO nodes (public_key,name,role) VALUES ('aabbcc99887766aa','Bob','repeater')`)
srv := &Server{db: db}
if k, n := srv.resolveHeardKey("aabbccdd"); k != "aabbccdd11223344" || n != "Alice" {
t.Errorf("unique prefix → (pubkey,Alice), got (%q,%q)", k, n)
}
if k, n := srv.resolveHeardKey("aabbcc"); k != "aabbcc" || n != "" {
t.Errorf("ambiguous prefix → (key,\"\"), got (%q,%q)", k, n)
}
if k, n := srv.resolveHeardKey("ffff"); k != "ffff" || n != "" {
t.Errorf("unknown prefix → (key,\"\"), got (%q,%q)", k, n)
}
if k, n := srv.resolveHeardKey(""); k != "" || n != "" {
t.Errorf("empty prefix → (\"\",\"\"), got (%q,%q)", k, n)
}
}
// TestAggregateCoverageMergesResolvedNodes verifies that the same node heard under
// two different heard_keys (e.g. a 3-byte prefix and the full pubkey) collapses into a
// single entry — summed count, latest SNR — when the resolver maps both to one node.
func TestAggregateCoverageMergesResolvedNodes(t *testing.T) {
rows := []coverageRow{
{Lat: 51.05, Lon: 3.72, SNR: covF(-4), HeardKey: "aabbcc", RxAt: "2026-06-01T10:00:00Z"},
{Lat: 51.05, Lon: 3.72, SNR: covF(-9), HeardKey: "aabbccdd11223344", RxAt: "2026-06-03T10:00:00Z"},
{Lat: 51.05, Lon: 3.72, SNR: covF(-7), HeardKey: "aabbcc", RxAt: "2026-06-02T10:00:00Z"},
}
resolve := func(hk string) (string, string) { return "aabbccdd11223344", "Alice" }
fc := aggregateCoverage(rows, 9, resolve)
if len(fc.Features) != 1 {
t.Fatalf("expected 1 cell, got %d", len(fc.Features))
}
nodes := fc.Features[0].Properties.Nodes
if len(nodes) != 1 {
t.Fatalf("expected 1 merged node, got %d (%+v)", len(nodes), nodes)
}
n := nodes[0]
if n.Name != "Alice" || n.Count != 3 || n.SNR == nil || *n.SNR != -9 {
t.Errorf("merged node want Alice count 3 latest -9, got %+v (snr=%v)", n, n.SNR)
}
}
// TestAggregateCoverageDeterministicFeatureOrder verifies #8: features come out
// sorted by cell regardless of Go's randomized map iteration, so the GeoJSON is
// stable (cacheable / non-flaky e2e).
func TestAggregateCoverageDeterministicFeatureOrder(t *testing.T) {
rows := []coverageRow{
{Lat: 51.0, Lon: 3.0, SNR: covF(-5)},
{Lat: 48.0, Lon: 2.0, SNR: covF(-5)},
{Lat: 52.0, Lon: 4.0, SNR: covF(-5)},
{Lat: 40.0, Lon: -3.0, SNR: covF(-5)},
}
fc := aggregateCoverage(rows, 9, nil)
if len(fc.Features) < 2 {
t.Fatalf("expected multiple cells, got %d", len(fc.Features))
}
for i := 1; i < len(fc.Features); i++ {
if fc.Features[i-1].Properties.Cell > fc.Features[i].Properties.Cell {
t.Fatalf("features not sorted by cell at %d: %q > %q", i,
fc.Features[i-1].Properties.Cell, fc.Features[i].Properties.Cell)
}
}
}
// TestAggregateCoverageNamePrecedenceOrderIndependent verifies #20: when two
// heard_keys resolve to the same node but the resolver returns different display
// names, the most specific (longest) heard_key wins regardless of row order, so
// the name no longer depends on map/row iteration.
func TestAggregateCoverageNamePrecedenceOrderIndependent(t *testing.T) {
resolve := func(hk string) (string, string) {
if hk == "aabbccdd11223344" {
return "aabbccdd11223344", "Alice"
}
return "aabbccdd11223344", "AliceShortPrefix"
}
full := coverageRow{Lat: 51.05, Lon: 3.72, SNR: covF(-5), HeardKey: "aabbccdd11223344", RxAt: "2026-06-01T10:00:00Z"}
prefix := coverageRow{Lat: 51.05, Lon: 3.72, SNR: covF(-6), HeardKey: "aabbcc", RxAt: "2026-06-02T10:00:00Z"}
for _, order := range [][]coverageRow{{full, prefix}, {prefix, full}} {
fc := aggregateCoverage(order, 9, resolve)
nodes := fc.Features[0].Properties.Nodes
if len(nodes) != 1 {
t.Fatalf("expected 1 merged node, got %d (%+v)", len(nodes), nodes)
}
if nodes[0].Name != "Alice" {
t.Fatalf("name precedence flapped with row order: got %q, want Alice", nodes[0].Name)
}
}
}
func TestZoomToHexRes(t *testing.T) {
// Resolution tracks zoom 1:1 within [3,18], clamped at the edges (z=0 is the
// missing-param case).
cases := map[int]int{0: 3, 3: 3, 8: 8, 16: 16, 18: 18, 25: 18}
for z, want := range cases {
if got := zoomToHexRes(z); got != want {
t.Fatalf("zoomToHexRes(%d)=%d, want %d", z, got, want)
}
}
}
// TestHexSizeRendersConstantPx verifies the core fix: a hex sized for resolution
// res renders at a constant ~hexTargetPx on screen at the corresponding zoom level,
// instead of the old fixed-meter buckets that were ~2px when zoomed out.
func TestHexSizeRendersConstantPx(t *testing.T) {
for res := 4; res <= 16; res++ {
// On-screen point-to-point height = 2*circumradius / mercUnitsPerPixel(zoom),
// where mercUnitsPerPixel = mercUPPZ0 / 2^zoom and zoom == res.
px := 2 * hexSizeForRes(res) * math.Pow(2, float64(res)) / mercUPPZ0
if math.Abs(px-hexTargetPx) > 0.001 {
t.Fatalf("res %d renders %.2fpx, want %.2fpx", res, px, hexTargetPx)
}
// Size must halve each zoom step (finer grid as you zoom in).
if ratio := hexSizeForRes(res) / hexSizeForRes(res+1); math.Abs(ratio-2) > 1e-9 {
t.Fatalf("res %d→%d size ratio %.4f, want 2", res, res+1, ratio)
}
}
}
+421
View File
@@ -0,0 +1,421 @@
package main
import (
"context"
"database/sql"
"encoding/json"
"log"
"net/http"
"sort"
"strconv"
"strings"
"time"
)
// scanCoverageRows reads (lat,lon,snr,rssi,heard_key,rx_at) rows into coverageRow values.
func scanCoverageRows(rows *sql.Rows) ([]coverageRow, error) {
out := []coverageRow{}
for rows.Next() {
var lat, lon float64
var snr sql.NullFloat64
var rssi sql.NullInt64
var heardKey, rxAt sql.NullString
if err := rows.Scan(&lat, &lon, &snr, &rssi, &heardKey, &rxAt); err != nil {
return nil, err
}
cr := coverageRow{Lat: lat, Lon: lon, HeardKey: strings.ToLower(heardKey.String), RxAt: rxAt.String}
if snr.Valid {
v := snr.Float64
cr.SNR = &v
}
if rssi.Valid {
v := int(rssi.Int64)
cr.RSSI = &v
}
out = append(out, cr)
}
return out, rows.Err()
}
// heardKeyResolverFor builds a nodeResolver for exactly the distinct heard_keys
// present in rows, resolving them all in one batched query instead of one query
// per key (the previous per-key resolver was N+1 on the read connection). Maps a
// heard_key to (pubkey, name) on a unique, non-hidden match; to (heardKey, "")
// otherwise. nil when there's no DB.
func (s *Server) heardKeyResolverFor(rows []coverageRow) nodeResolver {
if s.db == nil || s.db.conn == nil {
return nil
}
keys := make([]string, 0, len(rows))
seen := map[string]bool{}
for _, r := range rows {
if r.HeardKey != "" && !seen[r.HeardKey] {
seen[r.HeardKey] = true
keys = append(keys, r.HeardKey)
}
}
resolved := s.batchResolveHeardKeys(keys)
return func(heardKey string) (string, string) {
if v, ok := resolved[heardKey]; ok {
return v[0], v[1]
}
return heardKey, ""
}
}
// batchResolveHeardKeys resolves many heard_keys (2-3 byte prefixes or full
// pubkeys) to their canonical (pubkey, name) in a single round-trip per chunk: a
// UNION ALL of one LIMIT-2 prefix lookup each, so per-key work stays bounded
// (2 rows) and the whole set costs one query, not N. A unique match returns
// [pubkey, name]; unknown / ambiguous / blacklisted / hidden-prefix keys (#15,
// #1181) return [heardKey, ""].
func (s *Server) batchResolveHeardKeys(keys []string) map[string][2]string {
res := make(map[string][2]string, len(keys))
valid := make([]string, 0, len(keys))
seen := map[string]bool{}
for _, k := range keys {
if k == "" || seen[k] {
continue
}
seen[k] = true
if !hexPrefixRe.MatchString(k) {
res[k] = [2]string{k, ""}
continue
}
valid = append(valid, k)
}
// SQLITE_MAX_COMPOUND_SELECT is 500 by default; chunk well under it.
const chunk = 200
for i := 0; i < len(valid); i += chunk {
end := i + chunk
if end > len(valid) {
end = len(valid)
}
batch := valid[i:end]
parts := make([]string, len(batch))
args := make([]interface{}, 0, len(batch)*2)
for j, k := range batch {
// Parameterized: the prefix flows in as bound args, never interpolated,
// so this stays injection-safe regardless of how hexPrefixRe later
// evolves. The per-prefix LIMIT 2 lives in a subquery because a bare
// LIMIT on a UNION ALL term is a SQLite syntax error.
parts[j] = "SELECT * FROM (SELECT ? AS pfx, public_key, COALESCE(name,'') AS nm FROM nodes WHERE public_key LIKE ? LIMIT 2)"
args = append(args, k, k+"%")
}
rows, err := s.db.conn.Query(strings.Join(parts, " UNION ALL "), args...)
if err != nil {
// Don't fail the request, but don't fail silently either: a swallowed
// error here presents as "every name is ambiguous" with no signal.
log.Printf("WARN batchResolveHeardKeys: %v", err)
for _, k := range batch {
res[k] = [2]string{k, ""}
}
continue
}
type agg struct {
pk, name string
cnt int
}
acc := map[string]*agg{}
for rows.Next() {
var pfx, pk, nm string
if err := rows.Scan(&pfx, &pk, &nm); err != nil {
continue
}
a := acc[pfx]
if a == nil {
a = &agg{}
acc[pfx] = a
}
a.cnt++
a.pk, a.name = pk, nm
}
rows.Close()
for _, k := range batch {
a := acc[k]
if a != nil && a.cnt == 1 && !s.cfg.IsBlacklisted(a.pk) && !s.cfg.IsNameHidden(a.name) {
res[k] = [2]string{a.pk, a.name}
} else {
res[k] = [2]string{k, ""}
}
}
}
return res
}
// resolveHeardKey resolves a single heard_key (2-3 byte prefix or full pubkey)
// to its canonical (pubkey, name), or (heardKey, "") when unknown / ambiguous /
// hidden. Thin wrapper over batchResolveHeardKeys so there is one code path.
func (s *Server) resolveHeardKey(heardKey string) (string, string) {
v := s.batchResolveHeardKeys([]string{heardKey})[heardKey]
if v[0] == "" && v[1] == "" {
return heardKey, "" // empty/unresolved → echo the key
}
return v[0], v[1]
}
// queryCoverageFiltered returns coverage rows within a bbox, optionally filtered
// by heard node (prefix/pubkey), contributing client (rx_pubkey), and time window
// (days; 0 = all time). Powers the global and per-observer coverage maps.
func (s *Server) queryCoverageFiltered(node, rx string, days int, b bbox) ([]coverageRow, error) {
where := []string{"lat BETWEEN ? AND ?", "lon BETWEEN ? AND ?"}
args := []interface{}{b.MinLat, b.MaxLat, b.MinLon, b.MaxLon}
if node != "" {
// Sargable heard_key IN-list (see coverageHeardKeyCandidates) so the
// (heard_key, …) composite index is used instead of a substr() scan (#5).
cands := coverageHeardKeyCandidates(node)
where = append(where, "heard_key IN ("+sqlPlaceholders(len(cands))+")")
for _, c := range cands {
args = append(args, c)
}
}
if rx != "" {
where = append(where, "rx_pubkey = ?")
args = append(args, strings.ToLower(rx))
}
if days > 0 {
since := time.Now().UTC().AddDate(0, 0, -days).Format(time.RFC3339)
where = append(where, "rx_at >= ?")
args = append(args, since)
}
rows, err := s.db.conn.Query("SELECT lat, lon, snr, rssi, heard_key, rx_at FROM client_receptions WHERE "+strings.Join(where, " AND "), args...)
if err != nil {
return nil, err
}
defer rows.Close()
return scanCoverageRows(rows)
}
// handleRxCoverage serves global (or per-observer via ?rx=) coverage as GeoJSON
// hexbins, over a time window. ?node= also works (same as the per-node endpoint).
// requireClientRxCoverage writes a 404 and returns false when the opt-in
// client-RX coverage feature is disabled, so the coverage endpoints read as
// "not found" instead of serving data on deployments that haven't enabled it.
func (s *Server) requireClientRxCoverage(w http.ResponseWriter, r *http.Request) bool {
// Routes are registered unconditionally, so guard against a nil server/cfg
// (e.g. handlers exercised in isolation) rather than panicking (#4).
// ClientRxCoverageEnabled is itself nil-receiver-safe.
if s == nil || s.cfg == nil || !s.cfg.ClientRxCoverageEnabled() {
http.NotFound(w, r)
return false
}
return true
}
func (s *Server) handleRxCoverage(w http.ResponseWriter, r *http.Request) {
if !s.requireClientRxCoverage(w, r) {
return
}
b, ok := parseBBox(r.URL.Query().Get("bbox"))
if !ok {
http.Error(w, "bbox required as minLat,minLon,maxLat,maxLon", http.StatusBadRequest)
return
}
if s.db == nil || s.db.conn == nil {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
days := clampDays(atoiDefault(r.URL.Query().Get("days"), 7))
z, _ := strconv.Atoi(r.URL.Query().Get("z"))
rows, err := s.queryCoverageFiltered(r.URL.Query().Get("node"), r.URL.Query().Get("rx"), days, b)
if err != nil {
http.Error(w, "query failed", http.StatusInternalServerError)
return
}
fc := aggregateCoverage(rows, zoomToHexRes(z), s.heardKeyResolverFor(rows))
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(fc)
}
// --- Leaderboard (top mobile observers) ---
type LeaderObserver struct {
Pubkey string `json:"pubkey"`
Name string `json:"name"`
Receptions int `json:"receptions"`
Nodes int `json:"nodes"`
Cells int `json:"cells"` // distinct fixed-res hex cells covered
Score float64 `json:"score"` // frontier-weighted coverage score
}
type RxLeaderboardResp struct {
Days int `json:"days"`
Observers []LeaderObserver `json:"observers"`
}
// leaderboardHexRes is the fixed hex resolution used to bucket receptions into
// "cells visited" for the frontier-weighted score. ~150 m ground cells at our
// latitude: coarse enough that a parked node's GPS jitter stays in one cell,
// fine enough that real driving paints many. Independent of the coverage map's
// zoom-dependent render resolution so the ranking is stable across views.
const leaderboardHexRes = 13
// leaderboardScanCap bounds how many rows the leaderboard aggregates in memory.
// The endpoint is unauthenticated (only requireClientRxCoverage), and the Go-side
// rarity weighting can't push the GROUP BY into SQLite, so without a cap a wide
// window on a busy network would stream the whole table into maps. At the cap we
// log and return a partial (best-effort) ranking rather than OOM (#review r2).
const leaderboardScanCap = 500000
// rxLeaderboard ranks mobile observers by frontier-weighted cell coverage over
// the time window. Each distinct cell an observer covers contributes
// 1/(observers covering that cell): a cell only they reached weighs 1.0, a cell
// shared by N observers weighs 1/N. This rewards expanding the map's edge and is
// spam-proof — a stationary node covers exactly one cell regardless of how many
// receptions it logs. Bucketing + the rarity weight can't be expressed in SQL,
// so we aggregate the window's rows in Go (bounded by leaderboardScanCap).
func (s *Server) rxLeaderboard(ctx context.Context, days, limit int) ([]LeaderObserver, error) {
since := time.Now().UTC().AddDate(0, 0, -days).Format(time.RFC3339)
// Name preference: the node's advertised name, else the companion's
// self-reported name (client_observers), else empty (UI shows the prefix).
// Hard LIMIT bounds memory; ORDER BY rx_at DESC so a truncated window keeps
// the most recent receptions.
rows, err := s.db.conn.QueryContext(ctx, `
SELECT cr.rx_pubkey, COALESCE(NULLIF(n.name,''), NULLIF(co.name,''), ''),
cr.lat, cr.lon, cr.heard_key
FROM client_receptions cr
LEFT JOIN nodes n ON n.public_key = cr.rx_pubkey
LEFT JOIN client_observers co ON co.pubkey = cr.rx_pubkey
WHERE cr.rx_at >= ?
ORDER BY cr.rx_at DESC
LIMIT ?`, since, leaderboardScanCap)
if err != nil {
return nil, err
}
defer rows.Close()
type agg struct {
name string
receptions int
cells map[string]struct{}
nodes map[string]struct{}
}
obsAgg := map[string]*agg{}
cellObservers := map[string]map[string]struct{}{} // cell -> set of rx_pubkey
scanned := 0
for rows.Next() {
// Honour client cancellation/timeout on the long scan (checked in batches
// to avoid a per-row context mutex on up to 500k rows).
if scanned&2047 == 0 && ctx.Err() != nil {
return nil, ctx.Err()
}
var pk, name, heardKey string
var lat, lon float64
if err := rows.Scan(&pk, &name, &lat, &lon, &heardKey); err != nil {
return nil, err
}
scanned++
a := obsAgg[pk]
if a == nil {
a = &agg{name: name, cells: map[string]struct{}{}, nodes: map[string]struct{}{}}
obsAgg[pk] = a
}
a.receptions++
a.nodes[heardKey] = struct{}{}
cell := hexCellAt(lat, lon, leaderboardHexRes)
a.cells[cell] = struct{}{}
set := cellObservers[cell]
if set == nil {
set = map[string]struct{}{}
cellObservers[cell] = set
}
set[pk] = struct{}{}
}
if err := rows.Err(); err != nil {
return nil, err
}
if scanned >= leaderboardScanCap {
log.Printf("[rx-leaderboard] scan hit cap %d over %dd window; ranking is partial (most-recent rows)", leaderboardScanCap, days)
}
// Per-cell observer counts EXCLUDING blacklisted contributors, so an operator
// of a blacklisted node parked in a cell can't silently dilute everyone else's
// frontier weight (#review r2). Name-hidden (not blacklisted) observers are
// legitimate contributors and still count.
cellCount := make(map[string]int, len(cellObservers))
for cell, set := range cellObservers {
n := 0
for pk := range set {
if !s.cfg.IsObserverBlacklisted(pk) && !s.cfg.IsBlacklisted(pk) {
n++
}
}
cellCount[cell] = n
}
out := make([]LeaderObserver, 0, len(obsAgg))
for pk, a := range obsAgg {
var score float64
for cell := range a.cells {
if c := cellCount[cell]; c > 0 {
score += 1.0 / float64(c)
}
}
out = append(out, LeaderObserver{
Pubkey: pk,
Name: a.name,
Receptions: a.receptions,
Nodes: len(a.nodes),
Cells: len(a.cells),
Score: score,
})
}
// Rank by frontier score; ties broken by raw receptions then pubkey so the
// order is deterministic (keeps same-location fixtures stable).
sort.Slice(out, func(i, j int) bool {
if out[i].Score != out[j].Score {
return out[i].Score > out[j].Score
}
if out[i].Receptions != out[j].Receptions {
return out[i].Receptions > out[j].Receptions
}
return out[i].Pubkey < out[j].Pubkey
})
// Identity hiding parity (#1727 r2): drop observer-blacklisted contributors,
// blank node-blacklisted / hidden-prefix names, cap at limit. nil cfg ⇒ no-ops.
filtered := make([]LeaderObserver, 0, limit)
for _, o := range out {
if s.cfg.IsObserverBlacklisted(o.Pubkey) {
continue
}
if s.cfg.IsBlacklisted(o.Pubkey) || s.cfg.IsNameHidden(o.Name) {
o.Name = ""
}
filtered = append(filtered, o)
if len(filtered) >= limit {
break
}
}
return filtered, nil
}
func (s *Server) handleRxLeaderboard(w http.ResponseWriter, r *http.Request) {
if !s.requireClientRxCoverage(w, r) {
return
}
if s.db == nil || s.db.conn == nil {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
days := clampDays(atoiDefault(r.URL.Query().Get("days"), 7))
limit := atoiDefault(r.URL.Query().Get("limit"), 20)
if limit < 1 || limit > 100 {
limit = 20
}
obs, err := s.rxLeaderboard(r.Context(), days, limit)
if err != nil {
http.Error(w, "query failed", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(RxLeaderboardResp{Days: days, Observers: obs})
}
func atoiDefault(s string, d int) int {
if n, err := strconv.Atoi(strings.TrimSpace(s)); err == nil {
return n
}
return d
}
+275
View File
@@ -0,0 +1,275 @@
package main
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"testing"
"time"
)
// TestRequireClientRxCoverageNilSafe verifies the #4 fix: coverage routes are
// registered unconditionally, so a nil server cfg (or nil *Config receiver)
// must 404 rather than panic.
func TestRequireClientRxCoverageNilSafe(t *testing.T) {
var nilCfg *Config
if nilCfg.ClientRxCoverageEnabled() {
t.Fatal("nil *Config must report disabled")
}
req := func(srv *Server) int {
rr := httptest.NewRecorder()
srv.handleRxCoverage(rr, httptest.NewRequest("GET", "/api/rx-coverage?bbox=50,3,52,4", nil))
return rr.Code
}
if code := req(&Server{}); code != http.StatusNotFound { // cfg nil → would panic without the guard
t.Fatalf("nil cfg: want 404, got %d", code)
}
if code := req(&Server{cfg: &Config{}}); code != http.StatusNotFound { // feature disabled
t.Fatalf("disabled: want 404, got %d", code)
}
}
func insRx(t *testing.T, db *DB, rx, hk, at string, lat, lon float64) {
mustExecDB(t, db, fmt.Sprintf(
`INSERT INTO client_receptions (rx_pubkey,heard_key,heard_keylen,snr,lat,lon,rx_at,ingested_at,src) VALUES ('%s','%s',3,-6,%f,%f,'%s','x','rxlog')`,
rx, hk, lat, lon, at))
}
func TestQueryCoverageFiltered(t *testing.T) {
db := seedCoverageDB(t)
now := time.Now().UTC()
recent := now.Format(time.RFC3339)
old := now.AddDate(0, 0, -40).Format(time.RFC3339)
insRx(t, db, "compa", "aabbcc", recent, 51.05, 3.72)
insRx(t, db, "compb", "ffeedd", recent, 51.06, 3.73)
insRx(t, db, "compa", "aabbcc", old, 51.05, 3.72)
srv := &Server{db: db}
bb := bbox{MinLat: 50, MinLon: 3, MaxLat: 52, MaxLon: 4}
if rows, _ := srv.queryCoverageFiltered("", "", 7, bb); len(rows) != 2 {
t.Fatalf("global 7d: want 2, got %d", len(rows))
}
if rows, _ := srv.queryCoverageFiltered("", "compa", 7, bb); len(rows) != 1 {
t.Fatalf("observer compa 7d: want 1, got %d", len(rows))
}
if rows, _ := srv.queryCoverageFiltered("", "", 0, bb); len(rows) != 3 {
t.Fatalf("global all-time: want 3, got %d", len(rows))
}
}
func TestRxLeaderboard(t *testing.T) {
db := seedCoverageDB(t)
recent := time.Now().UTC().Format(time.RFC3339)
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count) VALUES ('compa','MyCompanion','companion','t','t',1)`)
// compc is NOT in nodes, but reported its name via client_observers (fallback).
mustExecDB(t, db, `INSERT INTO client_observers (pubkey, name, last_seen) VALUES ('compc','MobOnly','t')`)
for i := 0; i < 3; i++ {
insRx(t, db, "compa", fmt.Sprintf("aabb%02d", i), recent, 51.05, 3.72)
}
insRx(t, db, "compc", "ddee00", recent, 51.05, 3.72)
insRx(t, db, "compc", "ddee01", recent, 51.05, 3.72)
insRx(t, db, "compb", "aabbcc", recent, 51.05, 3.72) // no name anywhere
srv := &Server{db: db}
obs, err := srv.rxLeaderboard(context.Background(), 7, 10)
if err != nil {
t.Fatal(err)
}
byPk := map[string]LeaderObserver{}
for _, o := range obs {
byPk[o.Pubkey] = o
}
if byPk["compa"].Name != "MyCompanion" || byPk["compa"].Receptions != 3 {
t.Fatalf("compa (nodes name): %+v", byPk["compa"])
}
if byPk["compc"].Name != "MobOnly" || byPk["compc"].Receptions != 2 {
t.Fatalf("compc (client_observers fallback): %+v", byPk["compc"])
}
if byPk["compb"].Name != "" {
t.Fatalf("compb should have no name: %+v", byPk["compb"])
}
}
// TestBatchResolveHeardKeys verifies the N+1 fix: many heard_keys resolve in one
// batched call with the same unique/ambiguous/unknown/hidden semantics as the
// single-key path.
func TestBatchResolveHeardKeys(t *testing.T) {
db := setupTestDBv2(t)
mustExecDB(t, db, `INSERT INTO nodes (public_key,name,role) VALUES ('aabbccdd11223344','Alice','repeater')`)
mustExecDB(t, db, `INSERT INTO nodes (public_key,name,role) VALUES ('aabbcc99887766aa','Bob','repeater')`)
mustExecDB(t, db, `INSERT INTO nodes (public_key,name,role) VALUES ('ddee110000000000','🚫Hidden','repeater')`)
srv := &Server{db: db, cfg: &Config{HiddenNamePrefixes: []string{"🚫"}}}
got := srv.batchResolveHeardKeys([]string{"aabbccdd", "aabbcc", "ffff", "ddee11", "aabbccdd"})
cases := map[string][2]string{
"aabbccdd": {"aabbccdd11223344", "Alice"}, // unique
"aabbcc": {"aabbcc", ""}, // ambiguous (Alice + Bob)
"ffff": {"ffff", ""}, // unknown
"ddee11": {"ddee11", ""}, // unique but hidden-prefix → not surfaced
}
for k, want := range cases {
if got[k] != want {
t.Errorf("batchResolveHeardKeys[%q] = %v, want %v", k, got[k], want)
}
}
}
// TestRxLeaderboardHidesBlacklistedAndHidden verifies #1727 r2 must-fix #2: the
// leaderboard must drop observer-blacklisted contributors and blank the name of
// node-blacklisted or hidden-prefix identities (pre-PR / post-blacklist rows).
func TestRxLeaderboardHidesBlacklistedAndHidden(t *testing.T) {
db := seedCoverageDB(t)
recent := time.Now().UTC().Format(time.RFC3339)
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count) VALUES ('aa01','GoodGuy','companion','t','t',1)`)
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count) VALUES ('cc03','BadNode','companion','t','t',1)`)
mustExecDB(t, db, `INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count) VALUES ('dd04','🚫Hidden','companion','t','t',1)`)
insRx(t, db, "aa01", "aabb01", recent, 51.05, 3.72) // normal → kept with name
insRx(t, db, "bb02", "aabb02", recent, 51.05, 3.72) // observer-blacklisted → dropped
insRx(t, db, "cc03", "aabb03", recent, 51.05, 3.72) // node-blacklisted → name blanked
insRx(t, db, "dd04", "aabb04", recent, 51.05, 3.72) // hidden prefix → name blanked
srv := &Server{db: db, cfg: &Config{
ObserverBlacklist: []string{"bb02"},
NodeBlacklist: []string{"cc03"},
HiddenNamePrefixes: []string{"🚫"},
}}
obs, err := srv.rxLeaderboard(context.Background(), 7, 100)
if err != nil {
t.Fatal(err)
}
byPk := map[string]LeaderObserver{}
for _, o := range obs {
byPk[o.Pubkey] = o
}
if _, ok := byPk["bb02"]; ok {
t.Fatalf("observer-blacklisted contributor must be dropped, got %+v", byPk["bb02"])
}
if byPk["aa01"].Name != "GoodGuy" {
t.Fatalf("normal contributor name should be kept: %+v", byPk["aa01"])
}
if _, ok := byPk["cc03"]; !ok || byPk["cc03"].Name != "" {
t.Fatalf("node-blacklisted contributor should remain with a blanked name: %+v", byPk["cc03"])
}
if _, ok := byPk["dd04"]; !ok || byPk["dd04"].Name != "" {
t.Fatalf("hidden-prefix contributor should remain with a blanked name: %+v", byPk["dd04"])
}
}
// TestRxLeaderboardLimitSurvivesBlacklistDrop verifies #1727 r2 must-fix #2: the
// SQL LIMIT runs before the Go-side observer-blacklist drop, so the leaderboard
// must over-fetch and still return `limit` non-blacklisted rows even when the
// top contributors are blacklisted (not limit-minus-dropped).
func TestRxLeaderboardLimitSurvivesBlacklistDrop(t *testing.T) {
db := seedCoverageDB(t)
recent := time.Now().UTC().Format(time.RFC3339)
// Reception counts strictly descending so ORDER BY COUNT(*) DESC is deterministic:
// the two blacklisted observers are the top two, then five good ones.
counts := []struct {
pk string
n int
}{
{"bk1", 10}, {"bk2", 9}, // observer-blacklisted (top of the board)
{"g1", 8}, {"g2", 7}, {"g3", 6}, {"g4", 5}, {"g5", 4},
}
for _, c := range counts {
for i := 0; i < c.n; i++ {
insRx(t, db, c.pk, fmt.Sprintf("%s%04d", c.pk, i), recent, 51.05, 3.72)
}
}
srv := &Server{db: db, cfg: &Config{ObserverBlacklist: []string{"bk1", "bk2"}}}
obs, err := srv.rxLeaderboard(context.Background(), 7, 3)
if err != nil {
t.Fatal(err)
}
if len(obs) != 3 {
t.Fatalf("expected exactly 3 rows after dropping 2 blacklisted from the top, got %d: %+v", len(obs), obs)
}
want := []string{"g1", "g2", "g3"}
for i, o := range obs {
if o.Pubkey == "bk1" || o.Pubkey == "bk2" {
t.Fatalf("blacklisted observer %q leaked into the leaderboard", o.Pubkey)
}
if o.Pubkey != want[i] {
t.Fatalf("row %d = %q, want %q (top-3 non-blacklisted by count)", i, o.Pubkey, want[i])
}
}
}
// TestRxLeaderboardFrontierScore verifies the leaderboard ranks by frontier-
// weighted cell coverage, not raw reception count: a roaming observer with FEWER
// receptions but MORE distinct cells outranks a stationary spammer, a cell only
// one observer covers weighs 1.0, and a cell shared by N observers weighs 1/N.
func TestRxLeaderboardFrontierScore(t *testing.T) {
db := seedCoverageDB(t)
recent := time.Now().UTC().Format(time.RFC3339)
// "park": 5 receptions all at ONE spot → 1 cell (stationary spammer).
for i := 0; i < 5; i++ {
insRx(t, db, "park", fmt.Sprintf("pk%04d", i), recent, 51.05, 3.72)
}
// "roam": 3 receptions at 3 far-apart spots → 3 distinct cells. The first
// coincides with park's cell, so that cell is shared by 2 observers.
insRx(t, db, "roam", "rm0001", recent, 51.05, 3.72) // shared with park
insRx(t, db, "roam", "rm0002", recent, 51.06, 3.72) // unique to roam
insRx(t, db, "roam", "rm0003", recent, 51.07, 3.72) // unique to roam
srv := &Server{db: db}
obs, err := srv.rxLeaderboard(context.Background(), 7, 10)
if err != nil {
t.Fatal(err)
}
if len(obs) != 2 {
t.Fatalf("want 2 observers, got %d: %+v", len(obs), obs)
}
// Roamer outranks the parked spammer despite fewer receptions.
if obs[0].Pubkey != "roam" || obs[1].Pubkey != "park" {
t.Fatalf("ranking: want [roam park], got [%s %s]", obs[0].Pubkey, obs[1].Pubkey)
}
byPk := map[string]LeaderObserver{}
for _, o := range obs {
byPk[o.Pubkey] = o
}
// park: 1 cell shared with roam → score 0.5; 5 receptions retained.
if byPk["park"].Cells != 1 || byPk["park"].Receptions != 5 {
t.Fatalf("park: %+v", byPk["park"])
}
if d := byPk["park"].Score - 0.5; d > 1e-9 || d < -1e-9 {
t.Fatalf("park score: want 0.5, got %v", byPk["park"].Score)
}
// roam: shared cell (0.5) + 2 unique cells (1.0 each) = 2.5; 3 cells.
if byPk["roam"].Cells != 3 {
t.Fatalf("roam cells: want 3, got %d", byPk["roam"].Cells)
}
if d := byPk["roam"].Score - 2.5; d > 1e-9 || d < -1e-9 {
t.Fatalf("roam score: want 2.5, got %v", byPk["roam"].Score)
}
}
// TestRxLeaderboardScoreNotDilutedByBlacklisted verifies the #review-r2 fix: a
// blacklisted observer sharing a cell must NOT dilute a legitimate observer's
// frontier score. Without excluding blacklisted pubkeys from the per-cell count,
// the legit observer's only cell would weigh 1/2 = 0.5 instead of 1.0.
func TestRxLeaderboardScoreNotDilutedByBlacklisted(t *testing.T) {
db := seedCoverageDB(t)
recent := time.Now().UTC().Format(time.RFC3339)
// Legit "good" and blacklisted "bad" both cover the same ~150 m cell.
insRx(t, db, "good", "aabb01", recent, 51.05, 3.72)
insRx(t, db, "bad", "aabb02", recent, 51.05, 3.72)
srv := &Server{db: db, cfg: &Config{ObserverBlacklist: []string{"bad"}}}
obs, err := srv.rxLeaderboard(context.Background(), 7, 100)
if err != nil {
t.Fatal(err)
}
byPk := map[string]LeaderObserver{}
for _, o := range obs {
byPk[o.Pubkey] = o
}
if _, leaked := byPk["bad"]; leaked {
t.Fatalf("blacklisted observer must not appear: %+v", byPk["bad"])
}
if d := byPk["good"].Score - 1.0; d > 1e-9 || d < -1e-9 {
t.Fatalf("blacklisted observer diluted the score: got %v, want 1.0", byPk["good"].Score)
}
}
+100 -99
View File
@@ -55,21 +55,21 @@ type TimeBucket struct {
// ─── Stats ─────────────────────────────────────────────────────────────────────
type StatsResponse struct {
TotalPackets int `json:"totalPackets"`
TotalTransmissions *int `json:"totalTransmissions"`
TotalObservations int `json:"totalObservations"`
TotalNodes int `json:"totalNodes"`
TotalNodesAllTime int `json:"totalNodesAllTime"`
TotalObservers int `json:"totalObservers"`
PacketsLastHour int `json:"packetsLastHour"`
PacketsLast24h int `json:"packetsLast24h"`
Engine string `json:"engine"`
Version string `json:"version"`
Commit string `json:"commit"`
BuildTime string `json:"buildTime"`
Counts RoleCounts `json:"counts"`
SignatureDrops int64 `json:"signatureDrops,omitempty"`
HashMigrationComplete bool `json:"hashMigrationComplete"`
TotalPackets int `json:"totalPackets"`
TotalTransmissions *int `json:"totalTransmissions"`
TotalObservations int `json:"totalObservations"`
TotalNodes int `json:"totalNodes"`
TotalNodesAllTime int `json:"totalNodesAllTime"`
TotalObservers int `json:"totalObservers"`
PacketsLastHour int `json:"packetsLastHour"`
PacketsLast24h int `json:"packetsLast24h"`
Engine string `json:"engine"`
Version string `json:"version"`
Commit string `json:"commit"`
BuildTime string `json:"buildTime"`
Counts RoleCounts `json:"counts"`
SignatureDrops int64 `json:"signatureDrops,omitempty"`
HashMigrationComplete bool `json:"hashMigrationComplete"`
// Memory accounting (issue #832). All values in MB.
//
@@ -207,31 +207,31 @@ type EndpointStatsResp struct {
}
type PacketStoreIndexes struct {
ByHash int `json:"byHash"`
ByObserver int `json:"byObserver"`
ByNode int `json:"byNode"`
ByHash int `json:"byHash"`
ByObserver int `json:"byObserver"`
ByNode int `json:"byNode"`
AdvertByObserver int `json:"advertByObserver"`
}
type PerfPacketStoreStats struct {
TotalLoaded int `json:"totalLoaded"`
TotalObservations int `json:"totalObservations"`
Evicted int `json:"evicted"`
Inserts int64 `json:"inserts"`
Queries int64 `json:"queries"`
InMemory int `json:"inMemory"`
SqliteOnly bool `json:"sqliteOnly"`
MaxPackets int `json:"maxPackets"`
EstimatedMB float64 `json:"estimatedMB"`
TrackedMB float64 `json:"trackedMB"`
AvgBytesPerPacket int64 `json:"avgBytesPerPacket"`
MaxMB int `json:"maxMB"`
Indexes PacketStoreIndexes `json:"indexes"`
HotStartupHours float64 `json:"hotStartupHours"`
BackgroundLoadComplete bool `json:"backgroundLoadComplete"`
BackgroundLoadFailed bool `json:"backgroundLoadFailed"`
BackgroundLoadProgress int64 `json:"backgroundLoadProgress"`
BackgroundLoadError string `json:"backgroundLoadError,omitempty"`
TotalLoaded int `json:"totalLoaded"`
TotalObservations int `json:"totalObservations"`
Evicted int `json:"evicted"`
Inserts int64 `json:"inserts"`
Queries int64 `json:"queries"`
InMemory int `json:"inMemory"`
SqliteOnly bool `json:"sqliteOnly"`
MaxPackets int `json:"maxPackets"`
EstimatedMB float64 `json:"estimatedMB"`
TrackedMB float64 `json:"trackedMB"`
AvgBytesPerPacket int64 `json:"avgBytesPerPacket"`
MaxMB int `json:"maxMB"`
Indexes PacketStoreIndexes `json:"indexes"`
HotStartupHours float64 `json:"hotStartupHours"`
BackgroundLoadComplete bool `json:"backgroundLoadComplete"`
BackgroundLoadFailed bool `json:"backgroundLoadFailed"`
BackgroundLoadProgress int64 `json:"backgroundLoadProgress"`
BackgroundLoadError string `json:"backgroundLoadError,omitempty"`
// #1690: surface retention + coverage so operators can see how much
// of the on-disk DB the in-memory store currently reflects.
RetentionHours float64 `json:"retentionHours"`
@@ -288,24 +288,24 @@ type GoRuntimeStats struct {
// ─── Packets ───────────────────────────────────────────────────────────────────
type TransmissionResp struct {
ID int `json:"id"`
RawHex interface{} `json:"raw_hex"`
Hash string `json:"hash"`
FirstSeen string `json:"first_seen"`
Timestamp string `json:"timestamp"`
RouteType interface{} `json:"route_type"`
PayloadType interface{} `json:"payload_type"`
PayloadVersion interface{} `json:"payload_version,omitempty"`
DecodedJSON interface{} `json:"decoded_json"`
ObservationCount int `json:"observation_count"`
ObserverID interface{} `json:"observer_id"`
ObserverName interface{} `json:"observer_name"`
ObserverIATA interface{} `json:"observer_iata"`
SNR interface{} `json:"snr"`
RSSI interface{} `json:"rssi"`
PathJSON interface{} `json:"path_json"`
Direction interface{} `json:"direction"`
Score interface{} `json:"score,omitempty"`
ID int `json:"id"`
RawHex interface{} `json:"raw_hex"`
Hash string `json:"hash"`
FirstSeen string `json:"first_seen"`
Timestamp string `json:"timestamp"`
RouteType interface{} `json:"route_type"`
PayloadType interface{} `json:"payload_type"`
PayloadVersion interface{} `json:"payload_version,omitempty"`
DecodedJSON interface{} `json:"decoded_json"`
ObservationCount int `json:"observation_count"`
ObserverID interface{} `json:"observer_id"`
ObserverName interface{} `json:"observer_name"`
ObserverIATA interface{} `json:"observer_iata"`
SNR interface{} `json:"snr"`
RSSI interface{} `json:"rssi"`
PathJSON interface{} `json:"path_json"`
Direction interface{} `json:"direction"`
Score interface{} `json:"score,omitempty"`
Observations []ObservationResp `json:"observations,omitempty"`
}
@@ -374,18 +374,18 @@ type DecodeResponse struct {
// ─── Nodes ─────────────────────────────────────────────────────────────────────
type NodeResp struct {
PublicKey string `json:"public_key"`
Name interface{} `json:"name"`
Role interface{} `json:"role"`
Lat interface{} `json:"lat"`
Lon interface{} `json:"lon"`
LastSeen interface{} `json:"last_seen"`
FirstSeen interface{} `json:"first_seen"`
AdvertCount int `json:"advert_count"`
HashSize interface{} `json:"hash_size,omitempty"`
HashSizeInconsistent bool `json:"hash_size_inconsistent,omitempty"`
HashSizesSeen []int `json:"hash_sizes_seen,omitempty"`
LastHeard interface{} `json:"last_heard,omitempty"`
PublicKey string `json:"public_key"`
Name interface{} `json:"name"`
Role interface{} `json:"role"`
Lat interface{} `json:"lat"`
Lon interface{} `json:"lon"`
LastSeen interface{} `json:"last_seen"`
FirstSeen interface{} `json:"first_seen"`
AdvertCount int `json:"advert_count"`
HashSize interface{} `json:"hash_size,omitempty"`
HashSizeInconsistent bool `json:"hash_size_inconsistent,omitempty"`
HashSizesSeen []int `json:"hash_sizes_seen,omitempty"`
LastHeard interface{} `json:"last_heard,omitempty"`
}
type NodeListResponse struct {
@@ -669,7 +669,7 @@ type TopologyResponse struct {
HopDistribution []TopologyHopDist `json:"hopDistribution"`
TopRepeaters []TopRepeater `json:"topRepeaters"`
TopPairs []TopPair `json:"topPairs"`
HopsVsSnr []HopsVsSnr `json:"hopsVsSnr"`
HopsVsSnr []HopsVsSnr `json:"hopsVsSnr"`
Observers []ObserverRef `json:"observers"`
PerObserverReach map[string]*ObserverReach `json:"perObserverReach"`
MultiObsNodes []MultiObsNode `json:"multiObsNodes"`
@@ -761,12 +761,12 @@ type DistOverTimeEntry struct {
}
type DistanceAnalyticsResponse struct {
Summary DistanceSummary `json:"summary"`
TopHops []DistanceHop `json:"topHops"`
TopPaths []DistancePath `json:"topPaths"`
CatStats map[string]*CategoryDistStats `json:"catStats"`
DistHistogram *Histogram `json:"distHistogram"`
DistOverTime []DistOverTimeEntry `json:"distOverTime"`
Summary DistanceSummary `json:"summary"`
TopHops []DistanceHop `json:"topHops"`
TopPaths []DistancePath `json:"topPaths"`
CatStats map[string]*CategoryDistStats `json:"catStats"`
DistHistogram *Histogram `json:"distHistogram"`
DistOverTime []DistOverTimeEntry `json:"distOverTime"`
}
// ─── Analytics — Hash Sizes ────────────────────────────────────────────────────
@@ -795,11 +795,11 @@ type MultiByteNode struct {
}
type HashSizeAnalyticsResponse struct {
Total int `json:"total"`
Distribution map[string]int `json:"distribution"`
Hourly []HashSizeHourly `json:"hourly"`
TopHops []HashSizeHop `json:"topHops"`
MultiByteNodes []MultiByteNode `json:"multiByteNodes"`
Total int `json:"total"`
Distribution map[string]int `json:"distribution"`
Hourly []HashSizeHourly `json:"hourly"`
TopHops []HashSizeHop `json:"topHops"`
MultiByteNodes []MultiByteNode `json:"multiByteNodes"`
}
// ─── Analytics — Subpaths ──────────────────────────────────────────────────────
@@ -933,10 +933,10 @@ type SnrDistributionEntry struct {
}
type ObserverAnalyticsResponse struct {
Timeline []TimeBucket `json:"timeline"`
PacketTypes map[string]int `json:"packetTypes"`
NodesTimeline []TimeBucket `json:"nodesTimeline"`
SnrDistribution []SnrDistributionEntry `json:"snrDistribution"`
Timeline []TimeBucket `json:"timeline"`
PacketTypes map[string]int `json:"packetTypes"`
NodesTimeline []TimeBucket `json:"nodesTimeline"`
SnrDistribution []SnrDistributionEntry `json:"snrDistribution"`
RecentPackets []map[string]interface{} `json:"recentPackets"`
}
@@ -999,24 +999,25 @@ type MapConfigResponse struct {
}
type ClientConfigResponse struct {
Roles interface{} `json:"roles"`
HealthThresholds interface{} `json:"healthThresholds"`
Map interface{} `json:"map"`
Tiles interface{} `json:"tiles,omitempty"` // deprecated
SnrThresholds interface{} `json:"snrThresholds"`
DistThresholds interface{} `json:"distThresholds"`
MaxHopDist interface{} `json:"maxHopDist"`
Limits interface{} `json:"limits"`
PerfSlowMs interface{} `json:"perfSlowMs"`
WsReconnectMs interface{} `json:"wsReconnectMs"`
CacheInvalidateMs interface{} `json:"cacheInvalidateMs"`
ExternalUrls interface{} `json:"externalUrls"`
PropagationBufferMs float64 `json:"propagationBufferMs"`
LiveMapMaxNodes int `json:"liveMapMaxNodes"`
Timestamps TimestampConfig `json:"timestamps"`
DebugAffinity bool `json:"debugAffinity,omitempty"`
MapDarkTileProvider string `json:"mapDarkTileProvider,omitempty"` // deprecated. TODO: remove after v3.5.0
Roles interface{} `json:"roles"`
HealthThresholds interface{} `json:"healthThresholds"`
Map interface{} `json:"map"`
Tiles interface{} `json:"tiles,omitempty"` // deprecated
SnrThresholds interface{} `json:"snrThresholds"`
DistThresholds interface{} `json:"distThresholds"`
MaxHopDist interface{} `json:"maxHopDist"`
Limits interface{} `json:"limits"`
PerfSlowMs interface{} `json:"perfSlowMs"`
WsReconnectMs interface{} `json:"wsReconnectMs"`
CacheInvalidateMs interface{} `json:"cacheInvalidateMs"`
ExternalUrls interface{} `json:"externalUrls"`
PropagationBufferMs float64 `json:"propagationBufferMs"`
LiveMapMaxNodes int `json:"liveMapMaxNodes"`
Timestamps TimestampConfig `json:"timestamps"`
DebugAffinity bool `json:"debugAffinity,omitempty"`
MapDarkTileProvider string `json:"mapDarkTileProvider,omitempty"` // deprecated. TODO: remove after v3.5.0
Customizer CustomizerClientConfig `json:"customizer"`
ClientRxCoverage bool `json:"clientRxCoverage"`
}
// CustomizerClientConfig is the operator-side customizer-modal knobs that
+4 -1
View File
@@ -11,7 +11,8 @@
"nodeDays": 7,
"observerDays": 14,
"packetDays": 30,
"_comment": "nodeDays: nodes not seen in N days moved to inactive_nodes (default 7). observerDays: observers not sending data in N days are removed (-1 = keep forever, default 14). packetDays: transmissions older than N days are deleted (0 = disabled). NOTE (#1283): all four retention fields are consumed by the INGESTOR process. The server is read-only and never prunes."
"clientRxDays": 30,
"_comment": "nodeDays: nodes not seen in N days moved to inactive_nodes (default 7). observerDays: observers not sending data in N days are removed (-1 = keep forever, default 14). packetDays: transmissions older than N days are deleted (0 = disabled). clientRxDays: mobile client-RX coverage rows (client_receptions/client_observers) older than N days are deleted (0 = disabled) — bounds the opt-in coverage tables (#1727). NOTE (#1283): all retention fields are consumed by the INGESTOR process. The server is read-only and never prunes."
},
"db": {
"vacuumOnStartup": false,
@@ -357,6 +358,8 @@
]
},
"_comment_compression": "Opt-in HTTP gzip middleware + WebSocket permessage-deflate. Both default to false — enable ONLY when your upstream reverse proxy is NOT already compressing. gzip: enables the gzipMiddleware wrapper around the HTTP handler. websocket: sets gorilla websocket Upgrader.EnableCompression. level: gzip compression level 1..9 (1=BestSpeed, 9=BestCompression, default 6). minSizeBytes: advisory minimum response size below which compression would not pay off. contentTypes: MIME allow-list — only responses with these Content-Type values are compressed. Already-compressed types (image/*, video/*, audio/*, application/zip, application/x-gzip, application/pdf, application/octet-stream) are always skipped, as are responses whose handler already set Content-Encoding. Omit contentTypes to use the built-in default allow-list.",
"clientRxCoverage": { "enabled": false },
"_comment_clientRxCoverage": "Opt-in mobile client-RX coverage (corescope-rx companions publishing GPS-tagged receptions to meshcore/client/<pubkey>/packets). Default OFF: when disabled the ingestor writes no client_receptions, the /api/rx-coverage|rx-leaderboard|nodes/{pubkey}/rx-coverage endpoints 404, and the UI hides the Coverage dashboard + Reach overlay. Set enabled=true to turn it on. SINGLE FLAG, BOTH PROCESSES: the ingestor and server each parse this same config.json, so this one clientRxCoverage.enabled entry gates both the ingest write path and the read endpoints — set it once, not per-process. TRUST: the feature requires an ACL-capable broker binding meshcore/client/{pubkey}/packets to that publisher; without ACLs the companion GPS is spoofable (see docs/client-rx-coverage.md). Retention: see retention.clientRxDays. Companion app + setup: https://github.com/efiten/corescope-rx.",
"_comment_channelKeys": "Hex keys for decrypting channel messages. Key name = channel display name. public channel key is well-known.",
"_comment_hashChannels": "Channel names whose keys are derived via SHA256. Key = SHA256(name)[:16]. Listed here so the ingestor can auto-derive keys.",
"hashRegions": [
-27
View File
@@ -1,27 +0,0 @@
#!/bin/bash
set -e
DEPLOY_DIR="$(cd "$(dirname "$0")" && pwd)"
MATOMO_COMMIT="38c30f9"
cd "$DEPLOY_DIR"
echo "[deploy] Fetching latest from origin..."
git fetch origin
echo "[deploy] Resetting to origin/master..."
git reset --hard origin/master
echo "[deploy] Building Docker image..."
docker build -t meshcore-analyzer .
echo "[deploy] Stopping old container (30s grace period)..."
docker stop -t 30 meshcore-analyzer && docker rm meshcore-analyzer
docker run -d --name meshcore-analyzer \
--restart unless-stopped \
-p 3000:3000 \
-v "$(pwd)/config.json:/app/config.json:ro" \
-v meshcore-data:/app/data \
meshcore-analyzer
echo "[deploy] Done. Live at https://analyzer.on8ar.eu"
-30
View File
@@ -1,30 +0,0 @@
#!/bin/bash
set -e
DEPLOY_DIR="$(cd "$(dirname "$0")" && pwd)"
cd "$DEPLOY_DIR"
echo "[staging] Fetching latest from origin..."
git fetch origin
BRANCH="${1:-master}"
echo "[staging] Checking out $BRANCH..."
git reset --hard "origin/$BRANCH"
echo "[staging] Building Docker image..."
docker build -t meshcore-analyzer-staging .
echo "[staging] Stopping old container (30s grace period)..."
docker stop -t 30 meshcore-staging 2>/dev/null || true
docker rm meshcore-staging 2>/dev/null || true
echo "[staging] Starting new container..."
docker run -d --name meshcore-staging \
--restart unless-stopped \
-p 3001:3000 \
-v "$(pwd)/config.json:/app/config.json:ro" \
-v meshcore-staging-data:/app/data \
meshcore-analyzer-staging
echo "[staging] Done. Live at https://staging.on8ar.eu"
+210
View File
@@ -0,0 +1,210 @@
# Client RX Coverage
Crowdsourced RF coverage from mobile clients: a phone connects over BLE to a MeshCore
*companion* radio, captures which nodes the companion hears (with SNR/RSSI), tags each reception
with the phone's GPS position, and publishes it to MQTT. CoreScope ingests these into
`client_receptions` and renders per-node H3-style hex coverage on the Reach page.
## Companion app — where to get it
The mobile capture side is **[corescope-rx](https://github.com/efiten/corescope-rx)** — an
open-source (GPL-3.0) Android PWA. Operators who enable coverage point their users at it: it connects
over BLE to a MeshCore companion radio, captures directly-heard nodes + the phone's GPS, and publishes
the payload defined below. It's self-hostable and generic — a runtime `config.json` aims it at your
own MQTT broker + CoreScope instance (see its README).
## Enabling coverage (operators)
Coverage is **off by default**. To turn it on:
1. In CoreScope's `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart the server
and ingestor. This is a **single flag read by both processes** — the ingestor and server each parse
the same `config.json`, so you set `clientRxCoverage.enabled` once and it gates both the ingest write
path and the read endpoints. There is no separate per-process flag.
2. **Required: an ACL-capable broker.** Bind `meshcore/client/{PUBLIC_KEY}/packets` so each client may
publish **only** under its own pubkey (e.g. an EMQX ACL keyed on the connected client's identity).
This is the trust boundary, not an optimization — see [Trust](#trust). The ingestor already
subscribes under `meshcore/#`.
3. Optionally set `retention.clientRxDays` to bound the coverage tables (see
[Storage](#storage--client_receptions-ingestor-owned)).
4. Point your users at [corescope-rx](https://github.com/efiten/corescope-rx) and they start
contributing. Results show on each node's Reach page (coverage toggle) and the `#/rx-coverage`
dashboard. **Warn them first that their contribution is world-readable and a per-observer view can
reconstruct their movements — see [Privacy](#privacy--contributor-location-is-public).**
The rest of this document is the MQTT payload contract the companion app implements.
## Companion BLE source (verified against firmware)
The mobile app's RX data comes from the companion's **`PUSH_CODE_LOG_RX_DATA` (0x88)** BLE frame:
`[0x88][snr×4 int8][rssi int8][raw packet bytes]`. This is emitted for **every** received
packet (promiscuous, incl. overheard flood traffic), not just messages addressed to the device:
- `src/Dispatcher.cpp:198` calls `logRxRaw(getLastSNR(), getLastRSSI(), raw, len)` in `checkRecv()`
**unconditionally** — NOT behind `#if MESH_PACKET_LOGGING`. So it works on stock firmware.
- `examples/companion_radio/MyMesh.cpp:283` overrides it to write the 0x88 frame whenever the app
is connected over BLE (`_serial->isConnected()`).
So per received packet the app gets SNR + RSSI + the raw bytes. It decodes the raw packet (standard
MeshCore format) to derive the directly-heard node (`path[last]` or 0-hop advert pubkey) and pairs it
with the phone's GPS. The bare advert push (`PUSH_CODE_ADVERT` 0x80) carries only a pubkey (no SNR/
RSSI/path) and is NOT used — 0x88 already covers adverts (the raw advert is in its payload).
Caveats: 0x88 is only sent while the app is BLE-connected; packets larger than `MAX_FRAME_SIZE` are
skipped; the firmware doc labels 0x88 "can be ignored" (messaging-app view) — for coverage it is the
primary frame. GPS is always the phone's, never the companion's.
## MQTT topic & payload
Topic: `meshcore/client/{PUBLIC_KEY}/packets` — `{PUBLIC_KEY}` is the companion's pubkey. The
broker (EMQX) should ACL-restrict each client to publish only under its own pubkey, which is how
"a connected companion may only inject under the keys that apply" is enforced.
Payload — meshcoretomqtt-compatible packet, plus a `gps` object:
```json
{
"origin": "<companion name>",
"origin_id": "<companion pubkey hex>",
"timestamp": "2026-06-09T12:00:00Z",
"type": "PACKET",
"direction": "rx",
"raw": "<packet hex>",
"SNR": -7,
"RSSI": -92,
"gps": { "lat": 51.05, "lon": 3.72, "acc_m": 8 }
}
```
- The discriminator is the `gps` object. A packet without `gps` is dropped (coverage needs a position).
- `raw` is decoded server-side to derive the directly-heard node and the path; `hash`/`path` fields
are not required.
- Subscription: the ingestor's default subscription (`meshcore/#`) already covers this topic. Sources
configured with an explicit topic list must add `meshcore/client/+/packets`.
## Capture HARD RULE — only what was heard directly
The app and ingestor record **only the node the companion physically received**, never upstream
relayers:
- **FLOOD** packet **with a path** (≥1 hop) → record `path[len-1]` (the last forwarder = the
immediate RF transmitter). Confirmed against firmware `Mesh.cpp` (`routeRecvPacket` appends the
forwarder's hash to the END of the path) and CoreScope's `neighbor_builder.go:226-228`.
- **DIRECT** packet **with a path** → **NOT attributable, discarded.** Direct forwarders consume the
next hop from the FRONT (`Mesh.cpp removeSelfFromPath`), so `path[len-1]` is the route's
destination-side end, NOT the node we heard. Attributing it credits the SNR to the wrong (often
far-away) node. Only FLOOD routes (0,1) are recorded from a path.
- Packet **with no path** (0 hops) **and** an advert → record the advertiser's full pubkey.
- `direction` must be `rx`. 1-byte (2 hex char) prefixes are excluded (collision-prone, like Reach).
- The RSSI/SNR belong to the directly-received transmission, so they attach to the recorded node.
- The rest of the path is discarded for coverage.
## Storage — `client_receptions` (ingestor-owned)
A roaming companion is a mobile observer with a moving position, so it gets its own table (not
`observations`, which assumes a fixed observer location). Per the #1283 read/write invariant, the
table and all writes live in `cmd/ingestor/`.
```
client_receptions(
id, rx_pubkey, heard_key, heard_keylen, rssi, snr,
lat, lon, pos_acc_m, rx_at, ingested_at, src,
UNIQUE(rx_pubkey, heard_key, rx_at)) -- idempotent re-ingest
```
`heard_keylen` is 32 for a full pubkey (0-hop advert) or 2/3 for a multibyte prefix. `src` is
`advert` or `rxlog`. No hex cell is stored — binning is computed server-side from lat/lon.
Indexes: a composite `(heard_key, heard_keylen, lat, lon)` and a `(lat, lon)` index back the coverage
queries; the per-node query matches a sargable `heard_key IN (pubkey, prefix6, prefix4)` list so the
composite is used instead of a table scan (see the benchmark in `cmd/ingestor`).
Retention: the table grows on every submission, so set `retention.clientRxDays` (ingestor) to delete
rows older than N days (and stale `client_observers`); `0` disables it. Without it the table is
unbounded.
## Read API — coverage GeoJSON
`GET /api/nodes/{pubkey}/rx-coverage?bbox={minLat,minLon,maxLat,maxLon}&z={zoom}`
Returns a GeoJSON `FeatureCollection` of hexagons covering where clients heard the node, aggregated
server-side (read-only). Each feature:
```json
{ "type": "Feature",
"geometry": { "type": "Polygon", "coordinates": [[[lon,lat], ...]] },
"properties": { "cell": "9:123:-45", "count": 7, "best_snr": -6, "has_sig": true,
"nodes": [{ "prefix": "aabbcc", "name": "Alice", "snr": -6, "count": 3 }],
"nodes_truncated": false } }
```
- Hex binning is a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`). We do **not**
use `uber/h3-go` because it is CGO and the project builds with `CGO_ENABLED=0`. Latitude is only
defined within ±85.05° (Web Mercator limit) and is clamped to that range.
- `z` (Leaflet zoom) selects the hex resolution (zoom-adaptive). Raw points never leave the server
(privacy: contributors' tracks are not exposed).
- `best_snr` / `has_sig` drive the colour: green→orange by best SNR, grey when no signal metric.
- Features are sorted by `cell` for a deterministic (cacheable) payload.
- **Bounds:** the per-cell `nodes` list is capped (with `nodes_truncated`), and the collection is
capped at a fixed feature count — when exceeded, the densest cells are kept and the top-level
`truncated` flag is set. The per-node endpoint also returns `mobile_receptions` and `mobile_clients`
totals (node-wide, independent of the bbox).
## Frontend
Shown only in the Reach view (`#/nodes/{pubkey}/reach`), as a toggleable hex layer drawn on the
existing Leaflet map (`public/node-reach-coverage.js`), deep-linked via `?coverage=1`. No new
frontend dependencies. Colours come from CSS variables in `public/node-reach.css`
(`--nq-cov-strong|mid|weak|grey`).
## Trust
Identity = the companion pubkey (`rx_pubkey`), taken from the `{PUBLIC_KEY}` topic segment.
**The feature requires an ACL-capable broker.** The reported GPS position is the contributor's own
claim, so the only thing anchoring a reception to a real identity is the broker ACL binding
`meshcore/client/{PUBLIC_KEY}/packets` to the client that holds that key. **Without such an ACL, the
topic — and therefore the GPS and the heard-node attribution — is spoofable:** anyone who can publish
to the broker could inject coverage under any pubkey. Do not enable this feature on an open/no-ACL
broker if you trust the resulting map.
Server/ingestor-side defense-in-depth (these reduce blast radius but do **not** replace the ACL):
- The ingestor rejects any topic pubkey that is not lowercase hex before writing, and never falls back
to a payload-supplied id (`cmd/ingestor/client_reception.go`, #2/#10).
- A blacklisted operator cannot contribute via the client topic (the blacklist is enforced before the
coverage write, #1).
- The frontend HTML-escapes the pubkey it renders, so a junk pubkey can't inject markup (#14).
- `/api/nodes/resolve` and coverage tooltips never reveal blacklisted or hidden-prefix node identities
(#15).
## Privacy — contributor location is public
⚠️ **Enabling coverage publishes contributors' GPS-tagged receptions, and the per-observer view can
reconstruct a contributor's movements.** The hex map is read without authentication. The leaderboard
exposes each companion's pubkey, and clicking one filters the map to that single companion
(`/api/rx-coverage?rx=<pubkey>`); at high zoom over the retention window this is effectively a public
movement trail (home / work / commute) of whoever carries that companion. **A pseudonymous companion
name does not mitigate this** — the *locations themselves* are identifying (overnight clustering = home),
and all of one contributor's points are linked by the pubkey.
This is an accepted tradeoff of the feature, not a bug: fine resolution is what makes the aggregate
coverage map useful, the feature is opt-in and OFF by default, and contributors choose to run the
companion. But the consent must be **informed**:
- **Operators:** tell your users, before they contribute, that their coverage (including a per-observer
view of their own track) is world-readable for as long as `retention.clientRxDays` keeps it.
- **Contributors:** do not contribute from a device you carry on your person if a public record of where
you have been is a concern. Use a dedicated/stationary node, or accept that the trail is public.
Operators who want to harden this further can lower `retention.clientRxDays`, run the dashboard behind
their own auth/proxy, or (future hardening) coarsen stored coordinates / apply a k-anonymity threshold
to the per-observer view.
Optional future hardening: have the companion sign a broker-issued token (the firmware exposes
on-device signing) — not required for the MVP, tracked as a follow-up.
## Configurable values (future customizer)
Hardcoded initially, tracked for the customizer per AGENTS.md rule 8: hex resolution per zoom
(`zoomToHexRes`), colour SNR thresholds (`coverageColorVar`), and any `rx_at` max-age validation.
+3
View File
@@ -40,6 +40,7 @@
<link rel="stylesheet" href="home.css?v=__BUST__">
<link rel="stylesheet" href="live.css?v=__BUST__">
<link rel="stylesheet" href="node-reach.css?v=__BUST__">
<link rel="stylesheet" href="node-reach-coverage.css?v=__BUST__">
<link rel="stylesheet" href="bottom-nav.css?v=__BUST__">
<link rel="stylesheet" href="nav-drawer.css?v=__BUST__">
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css"
@@ -218,7 +219,9 @@
<script src="compare.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-analytics.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-reach-map.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-reach-coverage.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="node-reach.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="rx-coverage.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="perf.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="mobile-page-actions.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
</body>
+41
View File
@@ -0,0 +1,41 @@
/* Client-RX coverage styles (fork-only feature). Kept in a DEDICATED file —
separate from node-reach.css — so upstream's periodic node-reach.css rewrites
don't drop the coverage tier colours + leaderboard layout (as happened in the
v3.9.0 merge). Consumed by rx-coverage.js (standalone Coverage dashboard) and,
once re-grafted, the per-node Reach page coverage overlay. */
/* RX coverage hex layer (mobile client receptions) */
:root {
--nq-cov-strong: #2ecc71; /* SF8: SNR ≥ −5 dB (good margin) */
--nq-cov-mid: #e67e22; /* SF8: −9..−5 dB (near the limit) */
--nq-cov-weak: #e74c3c; /* SF8: < −9 dB (poor, packet loss likely) */
--nq-cov-grey: #95a5a6; /* heard, no SNR metric */
}
/* Dark-theme variants: the saturated mid-luminance defaults glare on dark
basemaps. Mirrors the rest of the dashboard's theme-aware tokens (#polish). */
[data-theme="dark"] {
--nq-cov-strong: #3fb950;
--nq-cov-mid: #d29922;
--nq-cov-weak: #f85149;
--nq-cov-grey: #8b949e;
}
.nq-cov-legend { display:flex; gap:12px; align-items:center; font-size:11px; color:var(--text-muted, #57606a); margin:4px 0 10px; }
/* Toggled by node-reach.js applyCoverage via class, not inline style, so CSS
(print rules, themes) can still override visibility (#19). */
.nq-cov-legend.is-hidden { display:none !important; }
.nq-cov-legend i { width:12px; height:12px; border-radius:2px; display:inline-block; margin-right:4px; vertical-align:middle; }
/* Mobile RX coverage hub — leaderboard */
.rxb { display:flex; flex-direction:column; gap:3px; }
.rxb-row { display:flex; align-items:center; gap:10px; padding:7px 10px; background:var(--section-bg, #f6f8fa); border:1px solid var(--border, #d0d7de); border-radius:6px; font-size:13px; cursor:pointer; }
.rxb-row.rxb-head { font-size:10px; text-transform:uppercase; color:var(--text-muted, #57606a); cursor:default; }
.rxb-row.sel { outline:2px solid var(--accent, #2ecc71); }
/* Visible keyboard focus for the now-focusable (#polish) leaderboard rows. */
.rxb-row[data-rx]:focus-visible { outline:2px solid var(--link, #0969da); outline-offset:1px; }
.rxb-rank { width:24px; text-align:right; color:var(--text-muted, #57606a); }
.rxb-name { flex:1; overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }
.rxb-rec, .rxb-nodes, .rxb-cells, .rxb-score { width:56px; text-align:right; font-variant-numeric:tabular-nums; }
/* Sortable column headers in the leaderboard head row. */
.rxb-head .rxb-sort { cursor:pointer; user-select:none; }
.rxb-head .rxb-sort:hover { color:var(--link, #0969da); }
.rxb-head .rxb-sort:focus-visible { outline:2px solid var(--link, #0969da); outline-offset:1px; }
+73
View File
@@ -0,0 +1,73 @@
/* === CoreScope — node-reach-coverage.js ===
Draws per-node mobile RX coverage as an H3-style hex layer on the existing
Reach Leaflet map, from the /api/nodes/{pubkey}/rx-coverage GeoJSON.
No external deps; colours via CSS variables. */
'use strict';
(function () {
// coverageColorVar maps a GeoJSON feature's properties to a CSS variable name.
// Grey = received but no signal metric; otherwise SF8 SNR thresholds: ≥ −5 green
// (good margin), −9..−5 orange (near the limit), < −9 red (packet loss likely).
function coverageColorVar(props) {
if (!props || !props.has_sig || props.best_snr == null) return '--nq-cov-grey';
var s = Number(props.best_snr);
if (s >= -5) return '--nq-cov-strong';
if (s >= -9) return '--nq-cov-mid';
return '--nq-cov-weak';
}
function cssColor(varName) {
try {
var v = getComputedStyle(document.documentElement).getPropertyValue(varName).trim();
return v || '#888';
} catch (e) { return '#888'; }
}
// coverageFillOpacity gives the SNR tier a redundant, non-hue cue (stronger =
// more opaque) so colour-blind users can distinguish tiers (#a11y).
function coverageFillOpacity(props) {
switch (coverageColorVar(props)) {
case '--nq-cov-strong': return 0.6;
case '--nq-cov-mid': return 0.48;
case '--nq-cov-weak': return 0.34;
default: return 0.22;
}
}
// addLayer fetches coverage for the current map bbox/zoom and draws hex
// polygons. Returns a handle with off() so the caller can remove it.
function addLayer(map, pubkey) {
var group = L.layerGroup().addTo(map);
function refresh() {
var b = map.getBounds();
var bbox = [b.getSouth(), b.getWest(), b.getNorth(), b.getEast()].join(',');
var url = '/api/nodes/' + encodeURIComponent(pubkey) + '/rx-coverage?bbox=' + bbox + '&z=' + map.getZoom();
fetch(url).then(function (r) { return r.json(); }).then(function (fc) {
group.clearLayers();
(fc.features || []).forEach(function (f) {
var ring = (f.geometry.coordinates[0] || []).map(function (c) { return [c[1], c[0]]; }); // [lon,lat]→[lat,lon]
var col = cssColor(coverageColorVar(f.properties));
L.polygon(ring, { color: col, weight: 1, fillColor: col, fillOpacity: coverageFillOpacity(f.properties) })
.addTo(group)
.bindTooltip('n=' + f.properties.count +
(f.properties.best_snr != null ? ' · SNR ' + f.properties.best_snr : ' · no signal'));
});
}).catch(function (e) {
// Leave the layer empty on error; never crash the reach page.
console.warn('node-reach-coverage: coverage fetch failed', e);
});
}
// Debounce pan/zoom redraws so dragging doesn't storm the coverage endpoint
// (#6). Keep the reference so off() can unbind the same handler.
var debouncedRefresh = (typeof debounce === 'function') ? debounce(refresh, 200) : refresh;
map.on('moveend zoomend', debouncedRefresh);
refresh();
return {
off: function () {
map.off('moveend zoomend', debouncedRefresh);
try { map.removeLayer(group); } catch (e) {}
}
};
}
window.NodeReachCoverage = { coverageColorVar: coverageColorVar, coverageFillOpacity: coverageFillOpacity, addLayer: addLayer };
})();
+49 -1
View File
@@ -8,6 +8,16 @@
var current = null;
var loadGen = 0; // bumped per load + on destroy; guards against in-flight races
var DEFAULT_DAYS = 7; // single JS source for the default window (mirrors the server default)
var coverageOn = false; // mobile-client RX coverage hex layer (deep-linked ?coverage=1)
var covHandle = null; // handle from NodeReachCoverage.addLayer (off() to remove)
// setCoverageHash reflects the coverage toggle in the URL hash without
// re-triggering the router (history.replaceState, not location.hash =).
function setCoverageHash(on) {
var h = location.hash.replace(/([?&])coverage=1/, '$1').replace(/[?&]$/, '');
if (on) { h += (h.indexOf('?') >= 0 ? '&' : '?') + 'coverage=1'; }
try { history.replaceState(null, '', h || location.pathname + location.search); } catch (e) {}
}
// Single source of the bottleneck tiers: colour + threshold + colour-blind
// glyph + legend text. The map legend and the table both read from this.
@@ -104,7 +114,14 @@
// pass false so the current report stays on screen until the swap (no flash).
async function load(container, pubkey, days, isInitial) {
var myGen = ++loadGen;
// Wait for server config so the coverage flag (read just below and in the
// actions bar markup) is populated even on a direct land on this route — a
// race that otherwise hides the coverage toggle (#13).
try { await window.MeshConfigReady; } catch (e) {}
if (myGen !== loadGen) return; // superseded by a newer load while waiting
current = { pubkey: pubkey, days: days };
coverageOn = window.MC_CLIENT_RX_COVERAGE === true && (typeof getHashParams === 'function' && getHashParams().get('coverage') === '1');
if (covHandle) { try { covHandle.off(); } catch (e) {} covHandle = null; }
if (qmap) { qmap.destroy(); qmap = null; }
if (isInitial) {
container.innerHTML = '<div class="nq-load">Loading reach…</div>';
@@ -172,9 +189,16 @@
'<label for="nqIncoming"><input type="checkbox" id="nqIncoming"> incoming <span class="nq-dir">(we hear them)</span></label>' +
'<label for="nqOutgoing"><input type="checkbox" id="nqOutgoing"> outgoing <span class="nq-dir">(they hear us)</span></label>' +
'</fieldset>' +
(window.MC_CLIENT_RX_COVERAGE ? '<label for="nqCoverage"><input type="checkbox" id="nqCoverage"' + (coverageOn ? ' checked' : '') + '> coverage <span class="nq-dir">(where clients heard it)</span></label>' : '') +
'<span id="nqCount" class="nq-count" aria-live="polite"></span>' +
'<button id="nqPrintBtn" class="btn-primary nq-print-btn">Print / PDF</button>' +
'</div>' +
(window.MC_CLIENT_RX_COVERAGE ? '<div class="nq-cov-legend nq-noprint' + (coverageOn ? '' : ' is-hidden') + '" id="nqCovLegend">' +
'<span><i style="background:var(--nq-cov-strong)"></i>strong</span>' +
'<span><i style="background:var(--nq-cov-mid)"></i>medium</span>' +
'<span><i style="background:var(--nq-cov-weak)"></i>weak</span>' +
'<span><i style="background:var(--nq-cov-grey)"></i>no signal</span>' +
'</div>' : '') +
'<div id="nqNoGps" class="nq-nogps"></div>' +
'<div id="nqMap" class="nq-map"></div>' +
'<table class="nq-table"><thead><tr><th>#</th><th>Neighbour</th><th>we hear</th>' +
@@ -187,6 +211,7 @@
qmap = window.NodeReachMap.render('nqMap', n, TIERS);
}
var lastList = []; // most recent filtered link list (so the coverage toggle can restore lines)
// Two-way links are always shown; the two checkboxes add the asymmetric ones.
function paint() {
var inc = document.getElementById('nqIncoming').checked;
@@ -205,13 +230,35 @@
var noGps = list.filter(function (l) { return l.lat == null || l.lon == null; }).length;
document.getElementById('nqNoGps').textContent =
noGps ? noGps + ' link' + (noGps === 1 ? '' : 's') + ' have no location and are not drawn on the map.' : '';
if (qmap) qmap.setLinks(list);
lastList = list;
if (qmap) qmap.setLinks(coverageOn ? [] : list);
}
paint();
document.getElementById('nqIncoming').addEventListener('change', paint);
document.getElementById('nqOutgoing').addEventListener('change', paint);
document.getElementById('nqPrintBtn').addEventListener('click', printReport);
// Coverage overlay (fork): toggles the mobile-client RX hex layer in place and
// hides the link lines under it (declutter) — the table still lists every link.
var covLegend = document.getElementById('nqCovLegend');
function applyCoverage() {
if (covLegend) covLegend.classList.toggle('is-hidden', !coverageOn);
if (coverageOn) {
if (qmap && window.NodeReachCoverage && !covHandle) covHandle = window.NodeReachCoverage.addLayer(qmap.map, pubkey);
} else if (covHandle) {
try { covHandle.off(); } catch (e) {}
covHandle = null;
}
if (qmap) qmap.setLinks(coverageOn ? [] : lastList);
}
var covCb = document.getElementById('nqCoverage');
if (covCb) covCb.addEventListener('change', function (e) {
coverageOn = e.target.checked;
setCoverageHash(coverageOn);
applyCoverage();
});
if (coverageOn) applyCoverage(); // deep-linked ?coverage=1: add layer + hide lines now
wireTimeRange(container, pubkey);
}
@@ -225,6 +272,7 @@
function destroy() {
loadGen++; // invalidate any in-flight load so it won't mutate a foreign container
if (covHandle) { try { covHandle.off(); } catch (e) {} covHandle = null; }
if (qmap) { qmap.destroy(); qmap = null; }
current = null;
}
+16
View File
@@ -533,6 +533,22 @@
// ─── Fetch server overrides ───
window.MeshConfigReady = fetch('/api/config/client').then(function (r) { return r.json(); }).then(function (cfg) {
window.MC_CLIENT_RX_COVERAGE = cfg.clientRxCoverage === true;
// Coverage is opt-in: the nav link is NOT in static HTML (so the default-off
// nav matches upstream and the nav-overflow tests). Inject it after Analytics
// only when enabled, then nudge applyNavPriority (it re-runs on 'resize').
if (window.MC_CLIENT_RX_COVERAGE && !document.querySelector('.nav-links [data-route="rx-coverage"]')) {
var navAnchor = document.querySelector('.nav-links [data-route="analytics"]');
if (navAnchor) {
var covLink = document.createElement('a');
covLink.href = '#/rx-coverage';
covLink.className = 'nav-link';
covLink.setAttribute('data-route', 'rx-coverage');
covLink.innerHTML = '<svg class="ph-icon" aria-hidden="true" focusable="false"><use href="/icons/phosphor-sprite.svg#ph-broadcast"></use></svg> Coverage';
navAnchor.insertAdjacentElement('afterend', covLink);
window.dispatchEvent(new Event('resize'));
}
}
if (cfg.roles) {
if (cfg.roles.colors) {
// #1407 — ROLE_COLORS is now a live getter; merge into the override map.
+263
View File
@@ -0,0 +1,263 @@
/* === CoreScope — rx-coverage.js ===
Mobile RX coverage hub (route #/rx-coverage):
- global H3-style hex coverage map (all mobile observers), time-windowed
- leaderboard of top mobile observers (companion name + counts)
- click an observer to filter the map to just their coverage
Fork-only feature; isolated page (no changes to the core map). */
'use strict';
(function () {
var map = null, covLayer = null, days = 7, selectedRx = '', selectedName = '', boardCache = [], destroyed = false;
function cssColor(varName) {
try { return getComputedStyle(document.documentElement).getPropertyValue(varName).trim() || '#888'; }
catch (e) { return '#888'; }
}
// SF8 SNR thresholds: ≥ −5 good margin, −9..−5 near the limit, < −9 packet loss
// likely. Grey = heard but no SNR metric.
function colorVar(p) {
if (!p || !p.has_sig || p.best_snr == null) return '--nq-cov-grey';
var s = Number(p.best_snr);
if (s >= -5) return '--nq-cov-strong';
if (s >= -9) return '--nq-cov-mid';
return '--nq-cov-weak';
}
function dayBtn(d) { return '<button data-days="' + d + '"' + (d === days ? ' class="active"' : '') + ' aria-pressed="' + (d === days ? 'true' : 'false') + '">' + (d === 1 ? '24h' : d + 'd') + '</button>'; }
function pageHtml() {
return '<div style="max-width:1100px;margin:0 auto;padding:12px 16px">' +
'<h2 style="margin:4px 0 2px;font-size:18px">🗺️ Mobile RX coverage</h2>' +
'<div style="color:var(--text-muted);font-size:11px">Where roaming CoreScope-RX clients heard nodes. Colour = best signal per cell. <a href="https://github.com/efiten/corescope-rx" target="_blank" rel="noopener">Get the companion app →</a></div>' +
'<div class="analytics-time-range" id="rxDays" style="margin:8px 0">' + dayBtn(1) + dayBtn(7) + dayBtn(14) + dayBtn(30) + '</div>' +
'<div class="nq-cov-legend"><span><i style="background:var(--nq-cov-strong)"></i>strong</span><span><i style="background:var(--nq-cov-mid)"></i>medium</span><span><i style="background:var(--nq-cov-weak)"></i>weak</span><span><i style="background:var(--nq-cov-grey)"></i>no signal</span></div>' +
'<div id="rxMap" style="height:60vh;min-height:360px;border:1px solid var(--border,#d0d7de);border-radius:6px;margin:8px 0"></div>' +
'<div class="nq-group-h">Top mobile observers</div>' +
'<div id="rxBoard" class="rxb"></div>' +
'</div>';
}
// coverageNodeRow renders one heard node: name (or heard_key prefix) + latest SNR + count.
function coverageNodeRow(n) {
var label = n.name ? escapeHtml(n.name) : '<code>' + escapeHtml(n.prefix || '?') + '</code>';
var snr = (n.snr != null) ? Number(n.snr).toFixed(1) + ' dB' : 'no sig';
return '<div style="display:flex;gap:8px;justify-content:space-between;font-size:12px;line-height:1.6">' +
'<span>' + label + '</span>' +
'<span style="color:var(--text-muted)">' + snr + ' · ×' + n.count + '</span></div>';
}
// coverageNodesHtml lists the nodes directly heard in a cell (properties.nodes:
// {prefix, name, snr, count}, strongest latest-SNR first; prefix shown when the
// name is unresolved). Rendered in the hover tooltip; capped at 10 rows with a
// "(N more)" footer so dense cells don't produce an unwieldy tooltip.
var COVERAGE_NODE_CAP = 10;
function coverageNodesHtml(p) {
var nodes = (p && p.nodes) || [];
var head = '<div style="font-weight:600;margin-bottom:4px">' +
nodes.length + (nodes.length === 1 ? ' node heard here' : ' nodes heard here') + '</div>';
if (!nodes.length) return head + '<div style="color:var(--text-muted)">n=' + (p ? p.count : 0) + '</div>';
var rows = nodes.slice(0, COVERAGE_NODE_CAP).map(coverageNodeRow).join('');
var more = (nodes.length > COVERAGE_NODE_CAP)
? '<div style="color:var(--text-muted);font-size:11px;margin-top:3px">(' + (nodes.length - COVERAGE_NODE_CAP) + ' more)</div>'
: '';
return head + '<div style="min-width:180px">' + rows + '</div>' + more;
}
// fillOpacityFor adds a redundant, non-hue cue to the SNR tier so the map is
// distinguishable for colour-blind users (orange vs red): stronger signal =
// more opaque. Pairs with the hue and the per-cell SNR in the tooltip (#a11y).
function fillOpacityFor(p) {
switch (colorVar(p)) {
case '--nq-cov-strong': return 0.6;
case '--nq-cov-mid': return 0.48;
case '--nq-cov-weak': return 0.34;
default: return 0.22;
}
}
function drawCoverage() {
if (!map || destroyed) return;
var b = map.getBounds();
var bbox = [b.getSouth(), b.getWest(), b.getNorth(), b.getEast()].join(',');
var url = '/api/rx-coverage?bbox=' + bbox + '&z=' + map.getZoom() + '&days=' + days + (selectedRx ? '&rx=' + encodeURIComponent(selectedRx) : '');
fetch(url).then(function (r) { return r.json(); }).then(function (fc) {
if (destroyed || !covLayer) return;
covLayer.clearLayers();
(fc.features || []).forEach(function (f) {
var ring = (f.geometry.coordinates[0] || []).map(function (c) { return [c[1], c[0]]; });
var col = cssColor(colorVar(f.properties));
L.polygon(ring, { color: col, weight: 1, fillColor: col, fillOpacity: fillOpacityFor(f.properties) }).addTo(covLayer)
.bindTooltip(coverageNodesHtml(f.properties));
});
}).catch(function (e) { console.warn('rx-coverage: coverage fetch failed', e); });
}
// Leaderboard sort state. Default = frontier score, descending. The rank (#)
// column is not sortable (it just reflects the current order). Numeric columns
// default to descending on first click; the name column to ascending.
var boardSort = { key: 'score', dir: 'desc' };
var BOARD_COLS = [
{ key: 'name', label: 'Observer (companion)', cls: 'rxb-name' },
{ key: 'score', label: 'score', cls: 'rxb-score',
title: 'Score telt je gedekte cellen, waarbij elke cel zwaarder weegt naarmate minder andere waarnemers ze bereikt hebben — grensverleggende dekking weegt meer dan drukke zones opnieuw afrijden.' },
{ key: 'cells', label: 'cells', cls: 'rxb-cells',
title: 'Aantal unieke ~150 m-cellen waar deze waarnemer iets hoorde.' },
{ key: 'nodes', label: 'nodes', cls: 'rxb-nodes' },
{ key: 'receptions', label: 'pkts', cls: 'rxb-rec' }
];
function sortBoard() {
var k = boardSort.key, dir = boardSort.dir === 'asc' ? 1 : -1;
boardCache.sort(function (a, b) {
if (k === 'name') {
var an = (a.name || a.pubkey).toLowerCase(), bn = (b.name || b.pubkey).toLowerCase();
return an < bn ? -dir : an > bn ? dir : 0;
}
return (Number(a[k]) - Number(b[k])) * dir;
});
}
function boardHeadHtml() {
var cells = BOARD_COLS.map(function (c) {
var arrow = boardSort.key === c.key ? (boardSort.dir === 'asc' ? ' ▲' : ' ▼') : '';
return '<span class="' + c.cls + ' rxb-sort" data-sort="' + c.key + '"' +
(c.title ? ' title="' + escapeHtml(c.title) + '"' : '') +
' role="button" tabindex="0">' + escapeHtml(c.label) + arrow + '</span>';
}).join('');
return '<div class="rxb-row rxb-head"><span class="rxb-rank">#</span>' + cells + '</div>';
}
function renderBoard() {
var el = document.getElementById('rxBoard');
if (!el) return;
if (!boardCache.length) { el.innerHTML = '<div class="muted" style="color:var(--text-muted);font-size:13px">No mobile observers in this window yet.</div>'; return; }
sortBoard();
var rows = boardCache.map(function (o, i) {
var nm = o.name ? escapeHtml(o.name) : (escapeHtml(o.pubkey.slice(0, 10)) + '…');
return '<div class="rxb-row' + (o.pubkey === selectedRx ? ' sel' : '') + '" data-rx="' + escapeHtml(o.pubkey) + '" data-name="' + escapeHtml(o.name || '') + '"' +
' role="button" tabindex="0" aria-pressed="' + (o.pubkey === selectedRx ? 'true' : 'false') + '" aria-label="Show coverage for ' + escapeHtml(o.name || o.pubkey.slice(0, 10)) + '">' +
'<span class="rxb-rank">' + (i + 1) + '</span><span class="rxb-name">' + nm + '</span>' +
'<span class="rxb-score">' + Number(o.score).toFixed(1) + '</span>' +
'<span class="rxb-cells">' + o.cells + '</span>' +
'<span class="rxb-nodes">' + o.nodes + '</span>' +
'<span class="rxb-rec">' + o.receptions + '</span></div>';
}).join('');
el.innerHTML = (selectedRx ? '<button id="rxAll" class="btn-primary" style="margin:0 0 8px">← Show all observers</button>' : '') +
boardHeadHtml() + rows;
// Column sort handlers (click + keyboard).
el.querySelectorAll('.rxb-sort[data-sort]').forEach(function (h) {
function applySort() {
var k = h.dataset.sort;
if (boardSort.key === k) {
boardSort.dir = boardSort.dir === 'asc' ? 'desc' : 'asc';
} else {
boardSort.key = k;
boardSort.dir = (k === 'name') ? 'asc' : 'desc';
}
renderBoard();
}
h.addEventListener('click', applySort);
h.addEventListener('keydown', function (e) {
if (e.key === 'Enter' || e.key === ' ' || e.key === 'Spacebar') { e.preventDefault(); applySort(); }
});
});
// Row click-to-filter (preserved from the original).
el.querySelectorAll('.rxb-row[data-rx]').forEach(function (r) {
function activate() {
selectedRx = r.dataset.rx; selectedName = r.dataset.name || '';
renderBoard(); fitToObserver(); syncHash();
}
r.addEventListener('click', activate);
r.addEventListener('keydown', function (e) {
if (e.key === 'Enter' || e.key === ' ' || e.key === 'Spacebar') { e.preventDefault(); activate(); }
});
});
var all = document.getElementById('rxAll');
if (all) all.addEventListener('click', function () { selectedRx = ''; selectedName = ''; renderBoard(); drawCoverage(); syncHash(); });
}
// fitToObserver zooms the map to the selected observer's full coverage extent
// (fetched with a world bbox so it's independent of the current view), then the
// resulting moveend redraws the hexes at the fitted resolution.
function fitToObserver() {
if (!map || !selectedRx) { drawCoverage(); return; }
var url = '/api/rx-coverage?bbox=-90,-180,90,180&z=' + Math.max(8, map.getZoom()) + '&days=' + days + '&rx=' + encodeURIComponent(selectedRx);
fetch(url).then(function (r) { return r.json(); }).then(function (fc) {
if (destroyed || !map) return;
var minLat = 90, minLon = 180, maxLat = -90, maxLon = -180, any = false;
(fc.features || []).forEach(function (f) {
(f.geometry.coordinates[0] || []).forEach(function (c) {
any = true;
if (c[1] < minLat) minLat = c[1]; if (c[1] > maxLat) maxLat = c[1];
if (c[0] < minLon) minLon = c[0]; if (c[0] > maxLon) maxLon = c[0];
});
});
if (!any) { drawCoverage(); return; } // observer has no data in window → keep view
map.fitBounds([[minLat, minLon], [maxLat, maxLon]], { padding: [30, 30], maxZoom: 15 });
drawCoverage(); // fitBounds may not fire moveend if the view is unchanged
}).catch(function (e) { console.warn('rx-coverage: observer extent fetch failed', e); drawCoverage(); });
}
function loadBoard() {
fetch('/api/rx-leaderboard?days=' + days + '&limit=25').then(function (r) { return r.json(); })
.then(function (d) { if (destroyed) return; boardCache = d.observers || []; renderBoard(); })
.catch(function (e) {
console.warn('rx-coverage: leaderboard fetch failed', e);
var el = document.getElementById('rxBoard');
if (el) el.innerHTML = '<div class="muted" style="color:var(--text-muted);font-size:13px">Could not load mobile observers.</div>';
});
}
function setDays(d) {
days = d;
var bar = document.getElementById('rxDays');
if (bar) bar.querySelectorAll('button').forEach(function (b) { b.classList.toggle('active', +b.dataset.days === d); });
loadBoard(); drawCoverage(); syncHash();
}
function syncHash() {
var q = 'days=' + days + (selectedRx ? '&rx=' + selectedRx : '');
try { history.replaceState(null, '', '#/rx-coverage?' + q); } catch (e) {}
}
function init(container) {
destroyed = false;
// A direct land on #/rx-coverage can run before MeshConfigReady resolves, at
// which point MC_CLIENT_RX_COVERAGE is still undefined and the page would
// wrongly show "not enabled". Defer until server config is loaded (#13).
Promise.resolve(window.MeshConfigReady).then(function () {
if (!destroyed) start(container);
});
}
function start(container) {
if (!window.MC_CLIENT_RX_COVERAGE) {
container.innerHTML = '<div class="nq-msg">Coverage is not enabled on this deployment.</div>';
return;
}
selectedRx = ''; selectedName = ''; days = 7; boardCache = [];
try {
var p = (typeof getHashParams === 'function') ? getHashParams() : null;
if (p) { var dd = parseInt(p.get('days'), 10); if ([1, 7, 14, 30].indexOf(dd) >= 0) days = dd; selectedRx = (p.get('rx') || '').toLowerCase(); }
} catch (e) {}
container.innerHTML = pageHtml();
map = L.map('rxMap', { zoomControl: true, attributionControl: false }).setView([51.0, 4.8], 8);
if (typeof window._applyTilesToNodeMap === 'function') window._applyTilesToNodeMap(map);
else L.tileLayer('https://tile.openstreetmap.org/{z}/{x}/{y}.png', { maxZoom: 19 }).addTo(map);
covLayer = L.layerGroup().addTo(map);
// Debounce pan/zoom redraws so dragging the map doesn't fire a storm of
// /api/rx-coverage requests (#6). Direct calls (setDays, fit) stay immediate.
map.on('moveend zoomend', debounce(drawCoverage, 200));
var bar = document.getElementById('rxDays');
if (bar) bar.addEventListener('click', function (e) { var b = e.target.closest('button[data-days]'); if (b) setDays(+b.dataset.days); });
setTimeout(function () { if (!destroyed && map) { map.invalidateSize(); if (selectedRx) fitToObserver(); else drawCoverage(); } }, 150);
loadBoard();
}
function destroy() {
destroyed = true;
if (map) { try { map.remove(); } catch (e) {} map = null; }
covLayer = null;
}
registerPage('rx-coverage', { init: init, destroy: destroy });
})();
+65
View File
@@ -0,0 +1,65 @@
'use strict';
// Unit test for the client-RX coverage frontend gate (SP2).
//
// The coverage toggle (#nqCoverage) and its legend (#nqCovLegend) are built
// inline inside node-reach.js's load() — a large async function that depends on
// api()/Leaflet, so it can't be invoked headless. Instead we extract the exact
// actions-HTML concatenation block from the real source and evaluate it in a vm
// sandbox with both values of window.MC_CLIENT_RX_COVERAGE. This exercises the
// real source markup logic (no hand-copied duplicate) for the gate.
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const src = fs.readFileSync(path.join(__dirname, 'public', 'node-reach.js'), 'utf8');
// Slice the actions-HTML expression: from the '<div class="nq-actions ...' line
// through the table that closes the block (ends with '</div></div>';).
const startMarker = "'<div class=\"nq-actions nq-noprint\">'";
const startIdx = src.indexOf(startMarker);
assert.ok(startIdx >= 0, 'could not locate nq-actions block start in node-reach.js');
const endMarker = "'</div></div>';";
const endIdx = src.indexOf(endMarker, startIdx);
assert.ok(endIdx >= 0, 'could not locate nq-actions block end in node-reach.js');
// Drop the trailing ";" so we can wrap the concatenation as a single expression.
let block = src.slice(startIdx, endIdx + endMarker.length).replace(/;\s*$/, '');
// Render the actions HTML for a given flag value via a controlled sandbox.
function renderActions(flag) {
const sandbox = {
window: { MC_CLIENT_RX_COVERAGE: flag },
coverageOn: false,
statsHtml: '',
};
vm.createContext(sandbox);
return vm.runInContext('(' + block + ')', sandbox);
}
// Flag OFF ⇒ no coverage checkbox, no legend.
const off = renderActions(false);
assert.ok(!off.includes('id="nqCoverage"'),
'flag false: actions HTML must NOT contain id="nqCoverage"');
assert.ok(!off.includes('id="nqCovLegend"'),
'flag false: actions HTML must NOT contain id="nqCovLegend"');
// Flag ON ⇒ coverage checkbox and legend present.
const on = renderActions(true);
assert.ok(on.includes('id="nqCoverage"'),
'flag true: actions HTML MUST contain id="nqCoverage"');
assert.ok(on.includes('id="nqCovLegend"'),
'flag true: actions HTML MUST contain id="nqCovLegend"');
// #19: the legend visibility is class-driven (.is-hidden), not an inline
// style="display:..." that CSS can't override. coverageOn is false in this
// sandbox, so the legend must carry is-hidden and no inline display style.
assert.ok(/class="nq-cov-legend[^"]*\bis-hidden\b/.test(on),
'flag true + coverage off: legend must use the is-hidden class');
assert.ok(!on.includes('style="display:'),
'legend must not use an inline display style (#19)');
// Sanity: the non-gated controls render regardless of the flag.
assert.ok(off.includes('id="nqIncoming"') && on.includes('id="nqIncoming"'),
'incoming filter must render irrespective of the coverage flag');
console.log('coverage gate (node-reach actions HTML) OK');
+73
View File
@@ -0,0 +1,73 @@
'use strict';
// Unit test for #6: pan/zoom coverage redraws must be debounced so dragging the
// map fires at most one /api/...rx-coverage request per settle, not one per
// moveend. We load node-reach-coverage.js in a vm sandbox with controllable
// timers + the real debounce, bind the layer, fire a burst of map events, then
// advance the fake clock and assert exactly one extra fetch happened.
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const code = fs.readFileSync(path.join(__dirname, 'public', 'node-reach-coverage.js'), 'utf8');
// Controllable timer queue.
let now = 0;
let nextId = 1;
let timers = [];
function setTimeoutStub(fn, ms) { const id = nextId++; timers.push({ id: id, fn: fn, at: now + (ms || 0) }); return id; }
function clearTimeoutStub(id) { timers = timers.filter(function (t) { return t.id !== id; }); }
function advance(ms) {
now += ms;
const due = timers.filter(function (t) { return t.at <= now; });
timers = timers.filter(function (t) { return t.at > now; });
due.forEach(function (t) { t.fn(); });
}
let fetchCount = 0;
function fakeFetch() {
fetchCount++;
// Chainable stub whose then/catch never invoke callbacks (we only count calls).
const chain = { then: function () { return chain; }, catch: function () { return chain; } };
return chain;
}
const fakeGroup = { addTo: function () { return fakeGroup; }, clearLayers: function () {}, };
const map = {
handlers: {},
on: function (ev, fn) { this.handlers[ev] = fn; },
off: function () {},
getBounds: function () { return { getSouth: function () { return 0; }, getWest: function () { return 0; }, getNorth: function () { return 1; }, getEast: function () { return 1; } }; },
getZoom: function () { return 10; },
removeLayer: function () {},
};
const sandbox = {
window: {},
console: { warn: function () {} },
setTimeout: setTimeoutStub,
clearTimeout: clearTimeoutStub,
fetch: fakeFetch,
L: { layerGroup: function () { return fakeGroup; } },
getComputedStyle: function () { return { getPropertyValue: function () { return ''; } }; },
document: { documentElement: {} },
};
vm.createContext(sandbox);
// Define debounce IN the context so it uses the controllable timers above.
vm.runInContext('function debounce(fn, ms){var t; return function(){var a=arguments, c=this; clearTimeout(t); t=setTimeout(function(){fn.apply(c,a);}, ms);};}', sandbox);
vm.runInContext(code, sandbox);
const handle = sandbox.window.NodeReachCoverage.addLayer(map, 'aabbccddeeff');
assert.strictEqual(fetchCount, 1, 'addLayer should fetch once initially');
// Burst of pan/zoom events.
const fire = map.handlers['moveend zoomend'];
assert.strictEqual(typeof fire, 'function', 'moveend/zoomend handler must be bound');
for (let i = 0; i < 6; i++) fire();
assert.strictEqual(fetchCount, 1, 'burst must not fetch immediately (debounced)');
advance(200);
assert.strictEqual(fetchCount, 2, 'after settle, exactly one coalesced fetch (got ' + fetchCount + ')');
handle.off();
console.log('node-reach-coverage debounce OK');
+65
View File
@@ -0,0 +1,65 @@
// E2E for the RX coverage hex layer on the Reach page (#/nodes/<pubkey>/reach?coverage=1).
// Defaults to localhost:3000 — NEVER point at prod (AGENTS.md). CI sets BASE_URL.
const { chromium } = require('playwright');
const BASE = process.env.BASE_URL || 'http://localhost:3000';
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
// Coverage is opt-in (config flag, default off). Skip when the deployment under
// test hasn't enabled it — the endpoints 404 and the UI toggle is absent by design.
const clientCfg = await (await page.request.get(BASE + '/api/config/client')).json();
if (clientCfg.clientRxCoverage !== true) {
console.log('node-reach-coverage E2E SKIP (clientRxCoverage disabled on this deployment)');
await browser.close();
return;
}
const nodes = await (await page.request.get(BASE + '/api/nodes?role=repeater&limit=1')).json();
if (!nodes.nodes || !nodes.nodes.length) {
console.log('node-reach-coverage E2E SKIP (no repeater in dataset)');
await browser.close();
return;
}
const pk = nodes.nodes[0].public_key;
// 1. The coverage endpoint returns a GeoJSON FeatureCollection.
const cov = await (await page.request.get(
BASE + '/api/nodes/' + pk + '/rx-coverage?bbox=-90,-180,90,180&z=10')).json();
if (cov.type !== 'FeatureCollection' || !Array.isArray(cov.features)) {
throw new Error('rx-coverage must return a FeatureCollection with a features array');
}
// 2. Bad bbox → 400.
const bad = await page.request.get(BASE + '/api/nodes/' + pk + '/rx-coverage');
if (bad.status() !== 400) throw new Error('missing bbox should be 400, got ' + bad.status());
// 3. The Reach page exposes the coverage toggle.
await page.goto(BASE + '/#/nodes/' + pk + '/reach');
await page.waitForSelector('.nq-head', { timeout: 20000 });
const reach = await (await page.request.get(BASE + '/api/nodes/' + pk + '/reach?days=7')).json();
if (reach.reliable_tokens && reach.reliable_tokens.length && (await page.locator('#nqRows').count())) {
await page.waitForSelector('#nqCoverage');
// 4. Enabling coverage issues a request to rx-coverage, shows the legend, and deep-links.
const waitCov = page.waitForRequest((r) => r.url().includes('/rx-coverage'), { timeout: 15000 });
await page.check('#nqCoverage');
await waitCov;
await page.waitForSelector('#nqCovLegend', { state: 'visible' });
if (!/coverage=1/.test(await page.evaluate(() => location.hash))) {
throw new Error('coverage toggle did not deep-link ?coverage=1');
}
// 5. Toggling off hides the legend.
await page.uncheck('#nqCoverage');
await page.waitForSelector('#nqCovLegend', { state: 'hidden' });
}
const errors = [];
page.on('console', (m) => { if (m.type() === 'error') errors.push(m.text()); });
if (errors.length) throw new Error('console errors: ' + errors.join('; '));
console.log('node-reach-coverage E2E OK');
await browser.close();
})().catch((e) => { console.error('node-reach-coverage E2E FAIL:', e.message); process.exit(1); });
+38
View File
@@ -0,0 +1,38 @@
'use strict';
// Unit test for node-reach-coverage.js color buckets. Loads the browser IIFE in
// a vm sandbox (pattern from test-frontend-helpers.js) and exercises the pure
// coverageColorVar mapping.
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const code = fs.readFileSync(path.join(__dirname, 'public', 'node-reach-coverage.js'), 'utf8');
const sandbox = { window: {}, document: {}, getComputedStyle: function () { return { getPropertyValue: function () { return ''; } }; } };
vm.createContext(sandbox);
vm.runInContext(code, sandbox);
const { coverageColorVar } = sandbox.window.NodeReachCoverage;
// SF8 SNR thresholds: ≥ −5 strong, −9..−5 mid, < −9 weak.
assert.strictEqual(coverageColorVar({ has_sig: false }), '--nq-cov-grey', 'no-sig → grey');
assert.strictEqual(coverageColorVar({ has_sig: true, best_snr: null }), '--nq-cov-grey', 'null snr → grey');
assert.strictEqual(coverageColorVar({ has_sig: true, best_snr: -3 }), '--nq-cov-strong', 'strong');
assert.strictEqual(coverageColorVar({ has_sig: true, best_snr: -5 }), '--nq-cov-strong', 'boundary strong (≥ −5)');
assert.strictEqual(coverageColorVar({ has_sig: true, best_snr: -6 }), '--nq-cov-mid', 'just below −5 → mid');
assert.strictEqual(coverageColorVar({ has_sig: true, best_snr: -9 }), '--nq-cov-mid', 'boundary mid (≥ −9)');
assert.strictEqual(coverageColorVar({ has_sig: true, best_snr: -10 }), '--nq-cov-weak', 'below −9 → weak');
assert.strictEqual(coverageColorVar({ has_sig: true, best_snr: -18 }), '--nq-cov-weak', 'weak');
assert.strictEqual(coverageColorVar(null), '--nq-cov-grey', 'null props → grey');
// #a11y: fill opacity is a redundant, monotonic non-hue cue for the SNR tier so
// colour-blind users can tell tiers apart. Must strictly decrease strong→grey.
const { coverageFillOpacity } = sandbox.window.NodeReachCoverage;
const oStrong = coverageFillOpacity({ has_sig: true, best_snr: -3 });
const oMid = coverageFillOpacity({ has_sig: true, best_snr: -6 });
const oWeak = coverageFillOpacity({ has_sig: true, best_snr: -10 });
const oGrey = coverageFillOpacity({ has_sig: false });
assert.ok(oStrong > oMid && oMid > oWeak && oWeak > oGrey,
'opacity must ramp strong>mid>weak>grey, got ' + [oStrong, oMid, oWeak, oGrey].join(','));
console.log('node-reach-coverage color buckets + opacity ramp OK');
+43
View File
@@ -0,0 +1,43 @@
'use strict';
// Unit test for #13: a direct land on #/rx-coverage must not read the coverage
// feature flag before MeshConfigReady resolves. init() should defer its
// enabled/disabled decision until the config promise settles, instead of
// synchronously rendering "not enabled" when the flag is still undefined.
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const code = fs.readFileSync(path.join(__dirname, 'public', 'rx-coverage.js'), 'utf8');
let page = null;
let resolveConfig;
const sandbox = {
window: { MeshConfigReady: new Promise(function (r) { resolveConfig = r; }) }, // MC_CLIENT_RX_COVERAGE undefined for now
document: { getElementById: function () { return null; } },
registerPage: function (name, obj) { page = obj; },
console: { warn: function () {} },
Promise: Promise,
setTimeout: function () {}, clearTimeout: function () {},
fetch: function () { var c = { then: function () { return c; }, catch: function () { return c; } }; return c; },
L: {}, getComputedStyle: function () { return { getPropertyValue: function () { return ''; } }; },
};
vm.createContext(sandbox);
vm.runInContext(code, sandbox);
assert.ok(page && typeof page.init === 'function', 'registerPage should expose init');
(async function () {
const container = { innerHTML: '' };
page.init(container);
// Before MeshConfigReady resolves, init must NOT have decided yet. The old
// code read the (undefined) flag synchronously and rendered "not enabled".
assert.strictEqual(container.innerHTML, '', 'init must defer until MeshConfigReady resolves');
// Resolve config with the feature OFF → only now should it render not-enabled.
sandbox.window.MC_CLIENT_RX_COVERAGE = false;
resolveConfig();
await new Promise(function (r) { setTimeout(r, 10); });
assert.ok(/not enabled/i.test(container.innerHTML), 'after config (off) resolves, shows not-enabled');
console.log('rx-coverage config race OK');
})().catch(function (e) { console.error(e); process.exit(1); });
+61
View File
@@ -0,0 +1,61 @@
'use strict';
// Unit test for #14: the mobile RX coverage leaderboard must HTML-escape the
// pubkey it interpolates into the row markup (data-rx="..." and the truncated
// fallback label), not only the name. A no-ACL broker / pre-validation rows
// could carry a non-hex pubkey, and the rest of the row is built by string
// concatenation, so an unescaped pubkey is an HTML-injection vector.
//
// Like test-coverage-gate.js we slice the real row-building expression out of
// public/rx-coverage.js and evaluate it in a vm sandbox — no hand-copied
// duplicate — so the test tracks the actual source.
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const src = fs.readFileSync(path.join(__dirname, 'public', 'rx-coverage.js'), 'utf8');
// Slice from `var nm = o.name ...` through the end of the returned row string.
const startMarker = 'var nm = o.name ? escapeHtml(o.name)';
const endMarker = "o.nodes + '</span></div>';";
const startIdx = src.indexOf(startMarker);
assert.ok(startIdx >= 0, 'could not locate row-builder start in rx-coverage.js');
const endIdx = src.indexOf(endMarker, startIdx);
assert.ok(endIdx >= 0, 'could not locate row-builder end in rx-coverage.js');
const block = src.slice(startIdx, endIdx + endMarker.length);
// Canonical escapeHtml (public/app.js).
function escapeHtml(s) {
if (s == null) return '';
return String(s).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;').replace(/'/g, '&#39;');
}
function renderRow(o) {
const sandbox = { o: o, i: 0, selectedRx: '', escapeHtml: escapeHtml };
vm.createContext(sandbox);
return vm.runInContext('(function () { ' + block + ' })()', sandbox);
}
// Malicious pubkey that would break out of the data-rx attribute and inject a
// tag if interpolated raw. With escaping, no raw '<', '>' or attribute-closing
// '"' survives.
const evil = '"><img src=x onerror=alert(1)>';
// Case 1: no name → pubkey used as the visible label fallback too.
const row1 = renderRow({ pubkey: evil, name: '', receptions: 1, nodes: 1 });
assert.ok(row1.indexOf('<img') === -1, 'raw <img must not appear in row (label fallback): ' + row1);
assert.ok(row1.indexOf('data-rx="' + evil + '"') === -1, 'raw pubkey must not appear unescaped in data-rx');
assert.ok(row1.indexOf('&lt;img') !== -1 || row1.indexOf('&quot;&gt;') !== -1, 'pubkey should be HTML-escaped: ' + row1);
// Case 2: name present → label is the (escaped) name, but data-rx still carries
// the pubkey and must be escaped.
const row2 = renderRow({ pubkey: evil, name: 'Mob', receptions: 2, nodes: 3 });
assert.ok(row2.indexOf('<img') === -1, 'raw <img must not appear in row (named): ' + row2);
// #a11y: the clickable row must be keyboard-operable (role/tabindex) and expose
// pressed state, so it isn't a mouse-only <div>.
assert.ok(/role="button"/.test(row2), 'row must have role="button"');
assert.ok(/tabindex="0"/.test(row2), 'row must be focusable (tabindex)');
assert.ok(/aria-pressed="(true|false)"/.test(row2), 'row must expose aria-pressed');
console.log('rx-coverage pubkey escaping + row a11y OK');