dandri/meshcore-analyzer
1
0 Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-11-11 11:57:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,126 Commits 85 Branches 42 Tags
417c8dd00a243f29822f34e5c9aaadbd2e522213
Commit Graph
478 Commits
Author SHA1 Message Date
TeTeHackoandClaude Opus 5.5 417c8dd00a feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300) 
Closes #2270, closes #2271.

The ingestor took only the confirmed region scopes out of an observer's
`/neighbors` report (#1865) and dropped the rest. Nothing could
therefore say when an observer last reported, or whom it heard.

## Ingestor

`observer_neighbor_reports` (created in `dbschema.Apply`) keeps the
latest report of each observer, one row per observer, replaced by every
live report.

The report is unauthenticated, so `neighbors[]` is stored sanitised:

- only 64-hex keys
- `snr` and `heard_secs_ago` finite and in range
- `status` limited to `[a-z_]`
- `scopes` clamped
- at most 500 rows

`total_neighbors` is believed only when it is not below the number of
rows that arrived.

A **retained replay** only fills a missing row, and it is dated by the
report's own `timestamp`. After an ingestor restart, a long-dead
observer therefore does not look as if it just reported.

## API

The table is optional for the server (no `AssertReady` change); without
it the fields are simply absent.

- `GET /api/observers`: `neighbors_report_at` and `neighbors_count`,
from one batch query in the existing cached builder.
- `GET /api/observers/{id}`: also `neighbor_report`, which is
`{received_at, reported_at, neighbor_count, queried_neighbors,
self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr,
heard_secs_ago, status, scopes}]}`. Names and roles come from the node
table in one query.

## Frontend

`public/observer-neighbors.js` is shared by both pages and escapes
everything.

- **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h
ago · 12`, or `—` when the observer never reported. The same value
appears in the slide-over.
- **#2270, observer page:** **Neighbors Report** and **Neighbors** stat
cards.
- **#2271, observer page:** a **Neighbors** table listing each neighbor
by name (a link to the node) or key prefix, with SNR, heard, scope query
status and scopes. The heading says when the list is shorter than the
count, e.g. `(2 of 3 listed)`.

## Tests

- **ingestor:** sanitising, the cap, live replace, the retained rules
(fill only, dated by the report, undated ignored) and the MQTT topic end
to end. The existing #1865 scope tests are unchanged and still pass.
- **server:** the list summary, the detail with names, and both
endpoints without the table.
- **frontend:**
  - unit tests of the helpers
- an XSS case for every rendered field; I checked that it fails when
escaping is removed
- `test-observers-headings` now expects the new column after `Last
Packet`
- **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer
endpoints, since the fixture has no reports. It is listed in
`scripts/non-unit-tests.json` and on shard 2.

Gates that pass:

- `go test -race` in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`
- `sh test-all.sh`
- `eslint@8`, with 0 errors
- the XSS diff gate
- these E2E suites: the new one,
`test-issue-1639-observers-sort-e2e.js`,
`test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js`

Performance: the list adds one primary-key-sized query per cache fill,
and the detail adds two small queries per request.

Also fixes #2330 (folded in from #2337, second commit).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 22:36:53 +01:00
n30nexandn30nex 1ed5e4ab3f fix(compare): exclude observer TX-only packets from overlap (#2346) 
Red commit: `39295da8` ([intended CI assertion
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)).
Green commit: `d1bda496`.

Observer comparison counted an observer's TX-only packets as receptions,
producing false overlap. It now requests `excludeTx=true`: the selected
observer and non-TX direction must match the same observation in memory
and SQLite v2/v3. Normal packet lists and stored TX records remain
unchanged; missing directions retain legacy compatibility.

Refs #667. This completes a comparison slice, not the issue's remaining
observer counters or #671's fleet/SNR feature. Representative display
fields and existing time-filter semantics are unchanged. API/OpenAPI
docs describe the filter and multi-node bypass.

Validation:
- RED: Go tests compiled with 32 intended failures/16 control passes;
unit and browser regressions reproduced false overlap. CI reached the
new unit assertion.
- GREEN: all 48 packet cases pass, plus grouped-cache isolation and
OpenAPI tests. The added cache regression caught a malformed-radio
cache-key collision before correction.
- Full local server suite: 174s; ingestor: 185s; `npm run test:unit`:
pass. Final focused packet/grouped/OpenAPI checks: pass.
- E2E assertion added to
`tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks
pass, including correct 1/2 reception totals and 50% overlap.

Performance: still two capped bulk requests, indexed candidate lookups
retained, one direction check within the existing observation scan/SQL
predicate, no new persistent structures. No configuration/customizer
changes.

Final-head CI passed; independent reviews found no must-fix items.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
 2026-10-29 18:34:06 +01:00
n30nexandn30nex c211ef7643 feat(nodes): show relay activity from certain forwarding evidence (#2336) 
Fixes #2331 with the additive API option.

The Last Relayed row shows green **actively relaying** only when
`last_certain_relay` falls within the existing
`healthThresholds.relayActiveHours` window. Missing, invalid, future or
expired evidence adds no idle/alive claim. Existing `relay_active`,
attributed timestamps and counts retain their meanings.

The optional node list/detail field uses the existing certain-relay
path: an eligible observed frame, matching hop, unique prefix, no
retired namesake and existing signature checks. Originating an advert
does not count. Forwarding another node's advert can count, even with
zero non-advert attributions. Adverts with an unknown origin cannot
establish forwarding.

Evidence uses transmission `first_seen`, not late observation receipt
time, and follows node-membership eviction. Snapshots take one read lock
over requested keys, O(page size), with no packet scan or new database
query. Work per qualifying hop stays bounded. No new dependency or
setting; configuration documentation and OpenAPI explain the
distinction.

The test-first history includes assertion-failing RED commits and GREEN
fixes. The initial RED's nil-return compile stub is explicitly permitted
by `docs/agents/TDD.md`. Local native server/ingestor suites, focused
race checks, 809 frontend helper assertions and 70 browser cases pass.
Browser coverage includes both themes and full/side/tablet detail views.

[Review regression
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339)
proves both malformed-origin failures. All three independent review
roles pass at `d1fe79cb`. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728)
passes on that exact commit, including all three browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
 2026-10-29 18:33:59 +01:00
n30nexandn30nex ef17d78909 docs(api): document node resolution and battery history (#2338) 
Partial follow-up to #1670; the broader documentation issue stays open.

Documents the existing node-prefix resolution and node battery-history
GET APIs: parameters, normalization and lookup rules, defaults,
ambiguity/visibility, ordered samples, nullable latest values and error
media types. The generator now has concrete success schemas for both.
Error statuses remain documented in prose under its existing convention.

Removes exactly these two entries from the known-gap list: 18 → 16, with
115 of 131 routes described. No handler, query, configuration or
dependency changes.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426035729/job/121931981058)
reaches the missing-documentation assertions. GREEN `521fb3ef` passes
both full native CGO suites and focused OpenAPI/real-handler contract
tests. Parent browser acceptance passed 28 checks through the rendered
Swagger UI and actual endpoints, including defaults, errors and
empty/null history.

All three independent review roles passed. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427081025)
passes on this exact head, including all browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
 2026-10-29 18:33:41 +01:00
efitenandClaude Opus 5.5 b65412ab83 feat(observer-logins): turn on neighbour reports in the firmware block of the credentials mail (#2349) 
The firmware block of the observer credentials mail now ends with `set
mqtt.neighbors on`, so firmware observers send the same `/neighbors`
report that the OpenHop block already turns on ("Publish Neighbours",
#2347). The ingestor reads that report for region scopes (#1865).

## Changes

- `cmd/server/observer_mail.go`: one extra line in the firmware block,
and one sentence in its note. The sentence says the command sends the
neighbour table and region scopes once a day, and that older firmware
does not know it and should skip it.
- `cmd/server/observer_mail_test.go`: the full firmware block assertion
includes the new line, and the slot test checks the line and the note.
- `docs/observer-mqtt-logins.md`: the mail section describes the new
line.

## Tests

- `go test ./...` in `cmd/server`: ok. Skipped
`TestSaveGeoFilterPreservesFileMode`, which needs Unix file modes; this
run was on Windows.
- `TestObserverCredentialsMail*` failed before the change, on the
missing line, and passes after it.

## Not checked

- The command name and the 24-hour default come from #1865. I did not
read the observer firmware source.
- The wording of the error older firmware gives for an unknown command
is not quoted in the mail. The note only says to skip the line.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 06:25:42 +01:00
efitenandClaude Opus 5.5 cadc666bd9 feat(observer-logins): OpenHop block in the credentials mail (#2347) 
The observer credentials mail (#2306) had blocks for meshcoretomqtt, the
Gessaman/Dutch-MeshCore firmware, Home Assistant and plain values. An
operator running OpenHop (repeater software with its own MQTT uploader)
could not tell which of these matched its broker form, and picked the
format by trial.

## Changes

- `cmd/server/observer_mail.go`: a fifth block, "4. OpenHop (repeater
software)", before the plain values. It uses the field and option labels
of the OpenHop broker form:
- `Format: Letsmesh`, `Transport: TCP` or `Websockets` (from
`observerLogins.transport`), host, port, username, password, `TLS:
on/off`, `Use JWT Auth: off`, `Retain Status: on`, `Publish Neighbours:
on`.
- The note says Meshcoretomqtt works too, but the format labelled
"openHop" (value `mqtt`) does not: it publishes to
`meshcore/repeater/<name>`, which a personal login may not use.
- Publish Neighbours is on because the ingestor reads the `neighbors`
topic for region scopes (#1865).
- `openapi.go` and `docs/observer-mqtt-logins.md`: "four formats"
becomes five.

## Sources

Read from OpenHop, not tested on a device by me:
- format family and topics:
`openhop_repeater/repeater/data_acquisition/mqtt_handler.py`
(`MC2MQTT_FORMATS = ("meshcoretomqtt", "letsmesh", "waev")`, the `mqtt`
format's `meshcore/repeater/<node_name>` base topic, transport values
`tcp`/`websockets`);
- labels: the built web UI (`Configuration-*.js`: "Add Broker", option
labels `Letsmesh`, `Meshcoretomqtt`, `openHop`, `TCP`, `Websockets`,
"Retain Status", "Publish Neighbours", "Use JWT Auth").

An operator on our instance set it up from a screenshot of that form
with format Letsmesh, TCP, Retain Status and Publish Neighbours on; the
observer publishes, and its `neighbors` report set `configured_scope` on
its node.

## Tests

- `TestObserverCredentialsMailOpenHopBlock`: the block text for TCP
without TLS and for Websockets with TLS, the "openHop" warning, HTML
escaping, and the block order.
- `TestObserverCredentialsMailBlockTitles`: five titles in order.
- `cmd/server` suite green locally (the two Windows-only failures
skipped, as on master).

## Not in this PR

- The websocket path: OpenHop uses paho's default, the doc already tells
operators to match the listener path.
- Anything with the other fields of the `neighbors` report (SNR,
heard_secs_ago, default_scope): we are measuring how many observers send
it first.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-28 22:31:43 +01:00
efitenandClaude Opus 5.5 d29957fbdc feat(observer-logins): lock the shared login out of migrated observers (#2348) 
Follow-up to #2306 / #2319. With personal observer logins, a migrated
observer still is not safe from the shared login: the hook answers
`ignore` for the shared login, so EMQX's built-in database lets it
publish under any observer's topic, including one whose owner has moved
to a personal login. This PR lets an operator close that per migrated
observer, before the shared login is removed altogether.

## Changes

**users.db v8.** `observer_bindings.confirmed_at`: the moment the
owner's personal login first published under that pubkey.
- An auto-bind (the owner's first publish) is confirmed at once.
- An admin assignment stays unconfirmed until the owner's login
publishes under that pubkey; that publish sets it.
- Reassigning to another user or unbinding clears it; reassigning to the
same owner keeps it.
- Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin`
rows stay NULL.

**Config.** `userManagement.observerLogins.sharedUsernames` (default
empty): the broker's shared login names. A name starting with `obs_` is
refused at startup.

**Authz hook.** For a login in `sharedUsernames`, a publish under
`meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is
confirmed, whatever the number of topic segments (the ingestor
attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey).
Logged as `publish denied: user=<name> observer=<8 hex>: confirmed
owner`. Every other request from a shared login keeps `ignore`, so the
built-in database decides as before. Personal logins are unchanged.

An orphaned binding (owner account deleted) does not count as confirmed.

**Admin.** The observers table of the Observer logins tab gets a
Confirmed column; the bindings JSON gets `confirmedAt`.

**Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out
of migrated observers" (what it denies, how to see `confirmedAt`, the
risk below). `config.example.json` has the key.

## Risk for operators

An observer that is confirmed but still runs a second uploader on the
shared login loses that second uploader once its name is in
`sharedUsernames`. The deny log line names the observer.

## Tests

- `internal/users`: v7 to v8 migration on a real v7 database, auto
confirm, admin assignment then owner confirm, non-owner and repeated
confirm, reassign clears, orphan reassign.
- `cmd/server`: the decision table (confirmed, unconfirmed, unbound,
non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty
and `client` iata, lowercase key; owner publish confirms, confirm
failure still allows and retries; reassign, unbind and user removal
clear the flag; the index load against a real store; config validation;
the hook and admin endpoints end to end.
- `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an
auto-bound observer.
- Local: `internal/users`, `cmd/server` (two Windows-only tests skipped,
as on master) and `cmd/ingestor` green. The E2E assertion was not run
locally; CI runs it.

## Not in this PR

- Using `confirmed_at` for verified `/neighbors` reports: a separate PR,
built on #2300.
- Removing the shared login: still an operator step after migration.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-28 19:10:41 +01:00
TeTeHackoandClaude Opus 5.5 660cbbf5ff deps: golang.org/x/net v0.61.0, x/crypto v0.58.0, paho.mqtt.golang v1.5.1; runtime alpine:3.24 (#2297) 
A Trivy scan of the published image found **19 HIGH and 20 MEDIUM** in
Go dependencies, plus a runtime base past its support window. There is
no Dependabot here, so this collects the bumps in one PR.

| dependency | from | to | why |
|---|---|---|---|
| `golang.org/x/net` (ingestor) | v0.27.0 | v0.61.0 | CVE-2024-45338 and
later advisories, fixed up to v0.60.0 |
| `golang.org/x/crypto` (server, `internal/users`) | v0.31.0 | v0.58.0 |
CVE-2025-22869, CVE-2025-47913 and later; `x/sys` follows to v0.49.0 |
| `github.com/eclipse/paho.mqtt.golang` | v1.5.0 | v1.5.1 |
CVE-2025-10543, an integer overflow in the UTF-8 string encoding |
| runtime image | `alpine:3.20` | `alpine:3.24` | 3.20 is out of
support; its musl 1.2.5-r3 carries CVE-2026-107891 |

The Alpine change covers `Dockerfile`, `Dockerfile.go`, the arm64 smoke
step in `deploy.yml` and the comment in
`docker/test-select-service-user.sh`.

**What changes besides the versions:**

- The new `x/` modules declare `go 1.26.0`, so the `go` directive of
`cmd/server`, `cmd/ingestor` and `internal/users` moves up from 1.22.
The builder image and CI already run 1.27.
- `x/sync` follows paho to v0.17.0.

**What is left:** only `GO-2026-5932`. It flags `x/crypto/openpgp` as
unmaintained; nothing here imports that package and no fix exists.

## Checks

- Every module passes `go mod tidy`, `go build` and `go vet`.
- `go test -race` passes in `internal/users`, `cmd/ingestor` and
`cmd/server`.
- A local image build on alpine 3.24.3:
  - ships mosquitto 2.1.2, caddy v2.11.4 and supervisor 4.3.0
- keeps the caddy and mosquitto accounts at the same uid:gid (100:101,
101:102), so `test-select-service-user.sh` still describes the image
(116 assertions pass)
- comes up healthy, ingests an MQTT status through mosquitto and serves
the frontend

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 21:09:01 +01:00
TeTeHackoandClaude Opus 5.5 77a5f14925 feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292) 
Part of #1869.

## Problem

`observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer
reported. Once an observer is retuned (say from SF7 to the EU narrow SF8
setting to listen to a neighbouring mesh), every packet it ever heard
reads as heard on the new setting, and there is no way to tell an SF7
packet from an SF8 one.

## What this does

As #1869 proposes, the packet data model is untouched: the setting of an
observation is resolved from its observer and time.

**Ingestor** (all schema in `dbschema.Apply`)

- `observer_radios`: one row per stretch of time an observer, and each
radio of a multi-radio node, stayed on one setting. Every live status
extends the current row and a change starts a new one; retained replays
never extend a row. On the first run it is seeded with every observer's
current radio, which is what the UI already showed for the past.
- Settings are stored canonical through `dbschema.CanonicalRadio`, so
`869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting.
- `observation_radios` (`observation_id → rx_radio_id`): multi-radio
openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id,
radio}]` in their status, so their observations resolve per radio.
- The table is sparse: single-radio nodes never send the field, so
`observations` and every query that scans it stay untouched.
  - The row id comes from the observation upsert via `RETURNING id`.
- Rows are pruned with their transmissions, and a purged observer takes
its history along.

**Server** (read-only)

- No `AssertReady` change: an absent table reads as empty and is probed
again later, the same rule as `advert_route_evidence`.
- The history is held in memory. The live poller checks it on every tick
(two primary-key lookups), so a retune or a new `rx_radio_id` is known
before the observation is broadcast. A full reload runs every 10 min.
- `radio` is added to observations, packets, grouped rows (taken from
the header observation) and the live feed.
- `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting.
- Combined with `observer=`, it means *that* observer heard it on that
setting.
- A radio query is served from memory instead of falling back to SQL,
which would return the window unfiltered.
- `GET /api/radios` (named struct) and `radio_history` on `GET
/api/observers/{id}`.

**Frontend**

- Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8`
or `freq > 869.5 && bw == 62.5`.
- A Radio row in the packet detail and a Radio column in its
observations table (only when the server reports one).
- A radio history table on the observer page once there is more than one
stretch.
- `RfParams.radioText` formats the setting. It escapes all five
characters and returns non-numeric input as it came, escaped.

## Performance

`txMatchesRadio` runs once per observation inside `filterPackets`.

- Each observation costs one map lookup by observer id.
- Observers with a single stretch (nearly all) answer from that lookup
directly.
- Retuned observers add a binary search over their few stretches.
- Multi-radio observers add one lookup by observation id.
- The snapshot is loaded once per filter call.

`BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20
observers, 2 of them retuned):

```
BenchmarkTxMatchesRadio-24    602    2056924 ns/op    0 B/op    0 allocs/op
```

Memory is one entry per stretch plus one per multi-radio observation;
nothing is added per observation otherwise.

## Tests

- **dbschema:** seed runs once, canonical spellings, the tables exist
without `observers.radio`.
- **ingestor:**
  - stretches extend and split; canonical spellings do not split
- two radios of one node hearing one transmission over two paths each
keep their own radio
  - a retained status does not write history
  - prune and purge
  - `rx_radio_id` reaches `PacketData`
- **server:**
- resolution before, between and after stretches, and with an unknown
radio id
  - `observer` + `radio`, and grouped rows carry `radio`
  - incremental refresh, `/api/radios`, `radio_history`, missing tables
  - the benchmark above
- **frontend:** the new filter fields in
`tests/unit/test-packet-filter.js`, and XSS cases through the real
`RfParams.radioText` and the history renderer.
- **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped
rows' radio, `sf ==` on the packets page, the detail Radio row and the
observer history. It is added to `scripts/non-unit-tests.json` and to
shard 2.

`go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema`
passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS
diff gate. The new E2E test and `test-e2e-playwright.js` pass against
the fixture.

## Browser validation

This runs on a live instance with six observers, one of them retuned to
869.618/62.5/SF8/CR8:

- The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR
4/5`.
- `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107;
the SF8 observer has not heard anything yet.
- `/api/radios` lists both settings.
- No page errors.

## Notes

- **The customizer (rule 8)** gets nothing: the only constants are
server-side refresh intervals.
- **Attribution timing:** packets heard between a retune and the
observer's next status message (minutes) are attributed to the previous
setting, because a stretch starts when the status is handled. This is
documented at `RecordObserverRadios`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 21:08:40 +01:00
n30nex 1dfcd0f5d3 fix(analytics): preserve distance invalidation during an active build (#2284) 
Red commit: `38c0abe1` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190273036)).

Fixes #2281. Background-fill completion could reset an active
`sync.Once`, causing a fatal mutex unlock or letting an older distance
build mark an incomplete index ready. Build admission now uses the
existing mutex, and a generation check prevents stale completion from
consuming invalidation. The loader also releases the store lock before
acquiring the distance gate.

- Deterministic regression drives the actual loader, recomputer and HTTP
handler; the old runtime crash is contained in a child process.
- Two cold observations stay below the normal 5% rebuild threshold. Ten
concurrent callers must share one rebuild, and the final response must
contain the cold path.
- Validation on `b8a2333b`: full server suite, focused distance suite,
ten repeated race runs plus a parent race rerun. Shared-base ingestor
suite passed.
- Browser verified: http://127.0.0.1:18855 returned 202 with
Retry-After, then 200 with 20 paths and 20 hops.
- Constant-time admission/invalidation and one fixed-size counter;
existing computation costs remain unchanged. No schema, settings or
API-shape changes.

Privacy, whitespace and scope checks passed. The external home-directory
preflight bundle is unavailable; applicable checks were performed
locally. Independent review found no outstanding fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191246966)
passed on `b8a2333b`, including all three browser shards and both
architecture builds.
 2026-10-26 15:42:42 +01:00
n30nex f66cd320da perf(graph): reuse indexed shortest-path workspaces (#2293) 
Red commit: `8b3cffd0` ([regression
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203043161)).

Fixes #2226. Bridge and coverage scoring now convert keys once and reuse
indexed distance/predecessor workspaces and at most one heap item per
vertex. Existing algorithms, normalization, snapshot publication and
synchronous startup behavior remain intact.

Five paired runs, deterministic 2,000-node/10,000-edge fixture, Go
1.27.2 on Windows/amd64, GOMAXPROCS=2:

| Calculation | Median time | Allocated bytes/run |
|---|---:|---:|
| Bridge | 8.237s → 0.819s | 2,083,295,328 → 2,517,664 |
| Coverage | 4.900s → 0.694s | 1,508,440,736 → 2,209,984 |

The desktop was shared; baseline bridge runs ranged 7.81–17.56s. These
are algorithm/allocation measurements, not startup/API latency or
retained memory. Rebuilding committed `5f34b5bf` produced the exact
measured binary hash.

- Independent path oracle, edge/epsilon/tie/disconnected cases,
allocation regression, race checks and full server/frontend suites pass.
- Browser verified: isolated native servers at http://127.0.0.1:18903
and http://127.0.0.1:18904 return identical nonzero graph scores.
- O(V+E) bounded workspace; O(V(V+E) log V) calculation. No persistent
cache, schema, dependency or configuration change.

Independent reviews and local preflight checks passed. External
home-directory preflight bundle unavailable; applicable checks were
performed locally. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204557795)
passed on `5f34b5bf`, including all three browser shards and both
architecture builds.
 2026-10-26 15:42:29 +01:00
n30nex 63d63ea90a perf(healthz): avoid scanning retained packets after startup (#2305) 
Fixes #2301.

Readiness probes currently scan every retained transmission while
holding the packet-store read lock. Once `StartupLoadDone` closes, this
reads the maintained observation counter beside the transmission count
under that same lock. Count extraction becomes O(1), with no new cache,
query or allocation.

Startup retains the original scan: the background loader publishes
indexes before its packet slice, so switching on readiness or
`LoadComplete` would be premature. Response fields, readiness status,
ingestion and configuration remain unchanged.

Whole-handler benchmark medians, microseconds per probe:

| Retained transmissions | Upstream | Candidate |
| ---: | ---: | ---: |
| 1,000 | 72.14 | 83.84 |
| 30,000 | 320.85 | 65.74 |
| 100,000 | 1,228.50 | 81.97 |

Four alternating pairs, 10,000 iterations/size, real SQLite-loaded
fixtures with three observations per transmission on average; Go 1.27.2,
Windows/amd64, two CPUs. All samples retained; shared-host variability
means no small-cache or production-latency improvement is claimed.
Allocations remain 36/probe. The included benchmark supports
reproduction against unchanged upstream production plus the identical
harness.

TDD exemption: pure performance refactor; **all pre-existing test files
are byte-unchanged**. Added characterization tests pass before and
after, covering loaders, deduplication, live updates, eviction, partial
startup failures, concurrent snapshots and liveness. Independent review
prompted test-only follow-up `2a0083b1`, which detects premature counter
use; production remains identical to measured `1b8e6680`.

Full server, race, ingestor and post-push browser/API checks pass
locally. [Final-head CI
passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40213482779),
including all browser shards and architecture builds. No customizer
changes.
 2026-10-26 15:42:15 +01:00
n30nex abad8a26b3 fix(liveness): require evidence from each observed relay frame (#2298) 
Red commit: `46e3b082` ([regression
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203680455)).

Fixes #2266 and #2267. Certain relay activity now requires an eligible
path from the observation that supplied it. A representative
transmission header, destination attribution or catalogue uniqueness
alone cannot supply that evidence. Later valid FLOOD evidence survives
observer/path deduplication.

The fix preserves valid own packets and excludes failed signatures,
planned DIRECT/TRACE paths, ambiguous/retired prefixes and missing or
mismatched headers. Firmware reference: meshcore-dev/MeshCore
`81dbae59`.

- One eligibility byte replaces an unused raw-frame field. StoreObs
remains 152 bytes in the protected 160-byte allocation class; no wire
frames are retained. Existing dedup maps gain one pointer per slot only
at 64+ observations.
- Processing is linear in retained paths, bounded by 64 wire path bytes.
No new SQL, schema, configuration or maps.
- Tests cover five load/ingest paths, late duplicates,
analytics/adaptive thresholds, positive controls and concurrent readers.
Review correction `4ad6c5ad` preserves lazy decoded caches while
retaining escaped/legacy signature rejection.
- Full server suite, focused/race checks and frontend checks pass.
Browser verified on http://127.0.0.1:18905: both upstream false-positive
cases disappear and a genuine later FLOOD is recognized.

Independent review findings are addressed. Local preflight checks
passed; external home-directory bundle unavailable. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40205015381)
passed on `4ad6c5ad`, including all three browser shards and both
architecture builds.
 2026-10-26 15:42:01 +01:00
n30nex 3a85a98cc5 docs(api): describe the supported packet-list query parameters (#2294) 
Red commit: `cee2a4bc` ([regression
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203052113)).

Partial follow-up for #1670. GET /api/packets now documents all 16
supported query parameters, including hash, route, node(s), region,
channel, area and expand. It removes unsupported sort/timeRange/search
entries, corrects numeric types, and describes actual defaults, limits
and memory/SQL/multi-node differences.

This changes only packet-route metadata. Runtime API behavior and the
schema generator stay unchanged.

- Served-spec regressions fail on upstream while actual-route controls
pass.
- Focused OpenAPI/SwaggerUI tests, full server suite and full standalone
frontend suite pass on `d8014193`.
- Browser verified: http://127.0.0.1:18906/api/spec exposes the 16
supported parameters and integer types; the running packet route
confirms default limit 50 and offset 0.
- No new requests, dependencies, settings, schema or untyped-map
occurrences.

The umbrella remains open: current route coverage stays 98/122 with 24
allowlisted gaps. Other endpoint parameters/schemas and a generic
parameter-presence gate are outside this PR.

Independent reviews and local preflight checks passed. External
home-directory preflight bundle unavailable; applicable checks were
performed locally. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204567683)
passed on `d8014193`, including all three browser shards and both
architecture builds.
 2026-10-26 15:41:20 +01:00
n30nex 041daecd1e docs(api): describe performance and clock diagnostics (#2303) 
Partial follow-up to #1670.

Six existing read-only diagnostics endpoints lacked usable OpenAPI
descriptions: process I/O, SQLite metrics, ingestor write counters, and
fleet/node/observer clock skew. This documents their parameters,
empty/error behavior and response root shapes. The known-gap list falls
from 24 to 18.

Descriptions preserve current behavior, including unknown/empty area
fallback, the PacketStore cache ratio, stale write-counter snapshots,
and the single-node 404. Schemas remain minimal; the existing generator
still declares only the generic 200 response, with 404 explained in
prose.

No runtime handler, authentication, configuration or customizer changes.
No added generic response maps.

Validation: regression tests exercise the served spec and real router,
including missing/malformed data, area selection and fallback, and
calibration responses. Full server and ingestor tests pass locally;
post-push Chromium verifies the served spec, six routes and Swagger
rendering. Three independent reviews passed.

Test-only RED `218a3bf3` produced the [six expected documentation
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212159661/job/121225534416).
GREEN `1e2d8c08` adds the documentation and [passes
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212678062),
including all browser shards and architecture builds.
 2026-10-26 15:41:05 +01:00
efitenandClaude Opus 5.5 e149568c6c feat(observer-logins): personal MQTT logins for observers (#2306) 
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.

Closes #2319.

The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.

## Changes in this PR

**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.

**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.

**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.

Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.

**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.

**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.

**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.

**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.

## Enabling it on an instance

Full steps in `docs/observer-mqtt-logins.md`. In short:

1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.

## Who can claim an observer

Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.

What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.

## Performance

`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.

## Verification

- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.

- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.

## Known behaviour

EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.

## Not in this PR

- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.

Note on cb9f4ff3: its message says the scheme problem was reported by an
operator. That report turned out to come from before the broker used the
hook, so it does not show this cause; the change rests on the firmware
source only.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 09:50:20 +01:00
n30nex b3367f5aea fix(analytics): keep distance indexing lazy during startup (#2278) 
Chunked startup now leaves the distance index lazy, as the legacy loader
already does. Requests wait for the initial chunks to finish, then
trigger one build and publish the refreshed result. A terminal
initial-load failure returns a clear error instead of retrying forever;
a completed hot window remains usable if later background filling fails.

Fixes #2227.

Validation: RED reproduces eager and premature builds plus the
terminal-error retry loop. Full server and unchanged frontend suites
pass, with focused race checks. The browser/API check observed 202 then
200, nonempty distances, and exactly one build.

Performance evidence is the test-enforced absence of eager/duplicate
distance construction, with required path indexes retained. No timing
speedup is claimed. No new settings. Common ingestor validation is
covered by the separate #2272 test-stability change.
 2026-10-25 12:11:09 +01:00
n30nex 77c0392947 fix(reach): avoid caching interrupted query results (#2277) 
Reach now propagates failed scope lookups, rejects cancelled results
before caching, and retains the prior complete degree snapshot when
scanning or row iteration fails. Missing rows and optional scope columns
still produce the legitimate empty result.

Fixes #2258.

Validation: RED reproduces partial degree snapshots and cached success
after failed/cancelled follow-up queries. The full server and unchanged
frontend suites pass; focused race and browser Reach checks pass.
Handler regressions verify that failed builds do not replace caches and
that a later request rebuilds successfully.

The shared-build cancellation policy and query count are unchanged. This
is independent of #2249's admission/queue work. Common ingestor
validation is covered by the separate #2272 test-stability change.
 2026-10-25 12:10:58 +01:00
n30nex c085ca8cba fix(rf-params): respect node visibility in responses (#2276) 
The RF endpoint now applies the current blacklist and hidden-name policy
to both exposed node metadata and deviations, including after cache
warm-up. Unhiding a node restores its data without waiting for cache
expiry.

Fixes #2268.

The shared RF snapshot and peer calculation remain unchanged; filtering
copies only the response data. This adds a linear node pass and bounded
deviation copies, with no additional database calls or cache
invalidation. The feature remains opt-in.

Validation: RED reproduces both response leaks. The full server and
standalone frontend suites pass, as do focused race tests and the RF
browser suite. Regressions cover warm/cold caches, policy updates,
unhide, visible controls and default-off behavior. Common ingestor
validation is covered by the separate #2272 test-stability change.
 2026-10-25 12:10:47 +01:00
efitenandClaude Opus 5.5 2bb000f2aa feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262) 
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge
that first; until then this diff also shows its commit.

## Adaptive offline threshold

`userManagement.notifications.adaptiveOffline` — **off by default**:

```json
"notifications": {
  "enabled": true,
  "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 }
}
```

The fixed `infraSilentHours` (72 h) treats every repeater the same. With
this on, a watched node's `node.offline` window is learned from its own
**certain** traffic (the #2261 rule):

1. Certain packets per hour over the week **before its last certain
packet**. Ending there, not at now, keeps an outage from dragging the
rate down until the fixed window returns and the node reads "back
online".
2. Empty runs are classified: ≤ 6 h and recurring on another day at the
same hours = normal silence (kept; the window is never shorter than the
longest + 1 h); any other run = outage, left out of the learning so it
neither stretches the window nor blocks learning.
3. λ = 10th percentile of the remaining hourly counts, so quiet nights
don't alert.
4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson,
ln 10⁶ ≈ 13.8).
5. Clamped to [`floorMinutes`, the role's silent window].

Kept on the fixed window: λ < 1/hour, under 24 h of history in memory,
or a 10th-percentile hour heard by fewer than `minObservers` observers
(one observer dropping out must not read as the node going offline).
Silence is measured up to the newest packet in the store, so a stalling
feed stops every node's clock. A node stored offline comes back only on
evidence from the moment it went offline on. The notifier caches each
learned window for 10 minutes and drops unwatched nodes.

For the #2260 repeater (~840 packets/day) the mail would have gone out
about 30 minutes into the outage, instead of never (or 72 h with #2261
alone).

## Certain node analytics

- Each `activityTimeline` bucket carries `certain`: the part that can
only be this node (`certainFor`, the packet-level form of the
`certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other
endpoints are unchanged.
- Availability and longest silence use certain buckets only; the longest
silence now includes one **still running** (an outage in progress is the
silence worth seeing).
- The chart stacks certain (`--accent`) and uncertain (`--text-muted`) —
existing theme variables, so no new customizer entry. Descriptions
updated.

## Performance

`CertainActivity` walks a week of packets per watched node, at most once
per node per 10 minutes, with the read lock taken **per node**:

```
BenchmarkCertainActivity-16   12.1 ms/op   (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold)
BenchmarkNoteCertainRelays-16  121 ns/op   0 allocs
```

Node analytics adds one `certainFor` per packet of the requested node (a
prefix-map lookup, or the cached decoded JSON for its own packets).

## Tests

- `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet
nights set the rate; > 10 % empty hours, short history, too few
observers → fixed.
- `CertainActivity` ends at the last certain packet and counts only
certain packets — verified red by mutating the window to end at now.
- Notifier end to end: learned window passed → bad; inside → good; too
few observers / setting off → fixed 72 h; cache refresh and prune.
- `certainFor`, `longestSilence` (incl. running silence), config
defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket.
- Frontend: `NodeAnalyticsTimeline.series` (new
`tests/unit/test-node-analytics-certain-timeline.js`).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork.

## Validation

Server on the freshened e2e fixture: analytics buckets carry `certain`
(e.g. 89 of 90 packets certain for one repeater). **No browser
screenshot**: Playwright isn't installed on this machine.

## Later (rule 8)

`floorMinutes` / `minObservers` are server config, not theme settings;
an admin UI for notification settings would be the place, not the
customizer.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-22 21:20:42 +02:00
efitenandClaude Opus 5.5 a08f2f4e7c fix(liveness): count only evidence that can be the node alone (#2260) (#2261) 
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold
follows in a separate PR).

## Problem

A repeater silent since 2026-10-18 showed **Active — Last relayed 4m
ago** and **Last Heard 36m**, and its offline mail never fired. Every
hit after that date was the 1-byte hop `67`, shared by nine nodes:

- the live resolver's affinity tiebreak (`resolvePathForObs`) put those
transmissions in `byNode`, which fed `stats.lastHeard` and
`LastHeardMap`;
- `last_relayed` folds in the 1-byte prefix bucket on purpose (#662),
and drove the status in `roles.js` and the notify offline check.

## Fix

The store keeps `certainHeard` per node: the newest transmission that
can only be that node.

| Evidence | Counts? |
|---|---|
| Own packet (`pubKey`, `srcPubKey`) | ✅ |
| Flood relay hop whose prefix, at the packet's path hash size, matches
exactly one path-capable node | ✅ |
| 1-byte hop shared with other repeaters | ❌ |
| Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌
|
| TRACE (path bytes are SNR) | ❌ |
| Destination (`destPubKey`) | ❌ |

A multibyte repeater legitimately appears as a 1-byte hop (the
originator picks the hash size, `Mesh::routeRecvPacket` appends at that
size), so only uniqueness decides. The rule is applied in every load
path through `indexResolvedPathHops` and the new
`indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live
ingest, late observations.

Liveness reads only that:

- `/api/nodes` and the node detail: `last_heard` is the newer of
`last_seen` and the certain last heard. Before, it was a plain copy of
`last_seen`.
- health / bulk-health `stats.lastHeard`.
- `LastHeardMap` → notify offline check. The separate relay input
(`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed.
- `roles.js getEffectiveHeardMs`: newest known timestamp, without
`last_relayed`. The side pane no longer says "Last relayed"; the Last
Relayed row stays (analytics) with an updated tooltip.
- New status **`uncertain`** (repeaters and rooms): nothing certain
within the silent window, but the shared prefix (`last_relayed`) still
in paths. Grey badge with explanation, an *Uncertain* filter on the
nodes list and the map, dimmed like stale. The offline mail for such a
node says only its shared prefix was still seen.
- A prefix also held by a path-capable node in `inactive_nodes` is not
unique; adverts with `signatureValid: false` are no evidence;
`/api/nodes/search` reports the same `last_heard`.

Attribution for analytics (`byNode`, packet counts, `last_relayed`,
relay counters, usefulness) is unchanged.

## Performance

Hot path: `noteCertainRelays` runs per observation on ingest under the
write lock: two hex bytes of header parsing plus one prefix-map lookup
per resolved hop.

```
BenchmarkNoteCertainRelays-16   10307640   116.2 ns/op   0 B/op   0 allocs/op   (8-hop 2-byte path, 2,000 repeaters)
BenchmarkLastHeardMap-16            1902   0.77 ms/op                            (5,000 nodes; was 3.7 ms scanning byNode)
```

Memory: one string per node ever certainly heard, dropped together with
the node's `byNode` list on eviction.

## Tests

- `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk,
LoadChunked — each with and without resolved_path — live ingest, late
observations) feeds `certainHeard`; each of the 8 call sites verified
red with a nil prefix map.
- `certain_heard_test.go`: hop width per route/payload type; retired
namesake prefix; failed signature; uncertain offline mail; shared vs
unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load
path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`,
detail and health agree.
- Regression for #2260: last certain packet on day X, 20 shared-prefix
hops after it → `LastHeardMap` stays at X and `node.offline` goes bad
after 72 h. Verified red with the uniqueness check mutated out.
- Notify: `TestEvaluateCertainHeard` replaces the relay-input test.
- Frontend: `last_relayed` alone no longer keeps a repeater or room
active; newest timestamp wins (an old `_liveSeen` can't hide a fresh
certain relay).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork with 457 extra commits.

## Validation

Ran the server on the freshened e2e fixture: list, detail and health
report the same last heard; e.g. a repeater whose `last_relayed` is 3
min newer than anything certain keeps its certain time. **No browser
screenshot**: Playwright isn't installed on this machine; CI's E2E
covers the pages.

## Not changed

- The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a
shared hop only when exactly one candidate is adjacent in the neighbor
graph; it did not fire in this case. Left as is.
- The `lastHeard=` list filter still filters `last_seen` in SQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-22 19:11:30 +02:00
efitenandClaude Opus 5.5 0e3ce48cff feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268) (#2210) 
Refs #1268. #2209 is merged as `cab4f71c`, and master is merged into
this branch (`47ce8264`), so the diff against master is this PR's own
change: `ea4d1130` plus the fix in `5cbbe5d5`.

## What it does

Two nodes hear each other only when SF and bandwidth match and their
frequencies are close enough. `/api/rf-params` now also returns
`deviations`: nodes whose setting differs from a majority of the nodes
around them on the same band.

- **Peers:** nodes with GPS and RF params within
`rfParams.compareRadiusKm` (default 50) whose frequency is within 30
MHz.
- **Same channel:** frequencies within a quarter of the bandwidth count
as one channel (15.625 kHz at BW 62.5). The SX1262 tolerates a
transmitter/receiver offset of up to 25% of BW without sensitivity loss,
for SF5 to SF12 and all bandwidths. Source: the `FERR_L` row in the
[S62F module
datasheet](https://lora-alliance.org/wp-content/uploads/2020/07/S62F_SPEC_Ver.B.pdf),
an SX1262 module; I did not find the figure in a Semtech document
itself. That table also lists -50/-100/-200 ppm for SF12/SF11/SF10,
which at 869 MHz is wider than BW/4 at 62.5 and 125 kHz, so BW/4 is the
tighter bound there.
- **Flagged when:** a setting the node cannot hear is shared by more
than half of the group, the node itself counted, with at least 3 peers
and no other incompatible setting at the same support. So 5 against 5
flags nobody, and 2 of 4 does not count as a majority. Each deviation
carries the majority, how many peers are compatible with it, the peer
count, and which of freq/sf/bw differ.
- **Coding rate is not compared.** RadioLib starts SX126x radios with an
explicit LoRa header (`SX126x::begin` sets
`RADIOLIB_SX126X_LORA_HEADER_EXPLICIT` at `6d89348`, the commit the
firmware pins in `platformio.ini`), and nothing in `firmware/src`
switches to implicit. The header carries the CR, so a CR mismatch still
decodes. I checked SX126x only, not SX127x or LR11x0.

UI:
- Nodes list: a badge such as `LoRa SF7` next to the name of a deviating
node (the majority in its tooltip), and a `LoRa: differs from nearby
(N)` filter kept in the URL as `?lora=differs`.
- Node detail: one line, for example "Differs from 12 of 14 nearby nodes
on this band, which use 869.618 MHz · SF8 · BW 62.5 kHz".
- Badge and filter only exist while the server has map data.

## Why same-band only

On our instance (1,110 nodes with params and GPS, measured with the
first version of the rule), a plain neighbourhood majority flagged
exactly 15 nodes, all of them a deliberate 433 MHz network in Limburg
next to the 868 MHz one (names like `NL-ZAY-RP01 | 433 X 868`). The
same-band comparison flags none there. On our network this PR shows
nothing today. The flags are for instances where a node really is set to
a different SF, bandwidth or channel than the nodes around it.

## Cost

- Points are bucketed on a radius-sized grid (rows of radius/110.57
degrees, longitude columns widened by 1/cos(lat)) and compared with a
squared equirectangular distance.
- `BenchmarkRfDeviations1k` (1,100 nodes over Belgium and the southern
Netherlands): 4.5 ms. `BenchmarkRfDeviations10k` (10,000 nodes in the
same box): 388 ms, 1.3 MB. Both on the first version.
- After `5cbbe5d5`, on a slower machine, 3 runs each: 1k 4.4 to 9.5 ms,
10k 821 to 852 ms, 1.8 MB. On that machine the first version measured
7.6 to 9.3 ms and 819 to 861 ms, so the majority step adds no measurable
time. It is one pass over the distinct settings around a node, and those
are few (one per preset in use).
- Inside the radius the work is close to O(n²): 10,000 nodes in one
dense box means about 2,000 neighbours per node, and a request after the
5 minute TTL waits for that computation.
- The result is cached per map snapshot for 5 minutes; concurrent
requests wait for one computation.
- On staging `/api/perf` reports `/api/rf-params` at 9.9 ms average, 26
ms max (8 requests).

## Tests

- Go: deviation cases (SF, freq within band, BW, CR ignored, other band
ignored, no GPS, no params, far away), split vote and too few peers,
radius, a peer in another longitude column at 60°N, the cache (same
snapshot within TTL, new snapshot, after TTL), the route with four
inserted nodes, plus the two benchmarks.
- Go, added in `5cbbe5d5`: a 5/5 split and 2 of 4 flag nobody; a 1 and a
15 kHz offset are not deviations while 32 kHz is, and offset peers add
up into one majority; the BW/4 boundary; peers two grid columns away at
60°N; a peer at the north edge of the radius. Each fails on the first
version. The two-column test also fails with a ±1 column search, which
the existing longitude test did not catch.
- Full `cmd/server` suite and `test-all.sh` pass locally on `5cbbe5d5`.
- Frontend unit: 6 new assertions for `deviationBadge` and the detail
line (13 in the file).
- E2E: 4 new checks: badge only on the deviating node, filter count and
result, `?lora=differs` written and restored, detail line. The disabled
check now also asserts no filter and no badges. 8/8 locally.

## Validation

On staging with live map data, with the first version: 1,115 nodes
served, `deviations: {}`, and the filter shows `LoRa: differs from
nearby (0)` with no badges. Because no real deviation exists there, the
badge, filter and detail line were checked by serving one simulated
deviation through Playwright route interception against the staging
page.

## Not done

- No map marker for deviating nodes.
- `compareRadiusKm` is config-only; the customizer is a later step. The
other values are hardcoded and belong in that step too: at least 3
peers, the 30 MHz band, the more-than-half rule, BW/4 and the 5 minute
TTL.
- Not re-run on staging after `5cbbe5d5`, and the E2E suite was not
re-run locally. The fix changes only the server's choice of nodes, not
the UI.
- The detail line says the peers "use" the majority setting. With the
BW/4 tolerance, some of them may be a few kHz off it.
- Companions and room servers are compared too, not only repeaters.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 15:18:14 +02:00
efitenandClaude Opus 5.5 dc98886d92 feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239) 
Part of #2238. Path hops through such a repeater (`/api/nodes`,
`/api/resolve-hops`, the frontend hop resolver) still show a bare
prefix; that part of the issue stays open.

## Summary

A repeater that never sends an advert can still tell a nearby companion
its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and
publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`.
This PR stores those answers, resolves names with them, and shows these
repeaters on the map, in the nodes list and in search, marked as
self-reported.

### Ingestor: store the names
- `367ecde1` adds a new `names` subtopic.
  - It is gated on `clientNodeNames.enabled`, default off.
- Answers go into a new table, `client_node_names`: one row per
observation, `UNIQUE(target, rx_pubkey, observed_at)`.
- **Validation.**
  - `target` must be 64 hex.
- The name is trimmed and rejected when it is empty, over 31 runes
(firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8,
or contains a control character.
- **Retention.** `retention.clientNodeNamesDays` (0 = keep).
`config.example.json` sets 30, like the other client tables: a companion
only asks a repeater for its name when CoreScope has none (`nameMissing`
in CoreDrive's `src/names.js`), so pruning is also how a renamed
repeater gets its new name.
- **Compatibility.** An ingestor without this change drops `/names`
silently, so CoreDrive v1.20.0 works against either version.

### Server: resolve by the reported name
- `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and
`client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys`
(the coverage tooltips) both use it.
- **Which name wins.** The advert name, otherwise the newest reported
name by `observed_at`. A prefix that matches different pubkeys across
the two tables stays ambiguous.
- **Only repeaters heard directly.** A reported name counts only once
its target has at least one `discover` or `advert` reception in
`client_receptions`, keyed by the full pubkey, or by its 8-byte prefix
when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names`
starts with that prefix (`clientNameHeard`). A name for a pubkey nobody
heard adds no candidate, so it cannot make a real node's prefix
ambiguous. Without the uniqueness check, a made-up pubkey that shares a
heard repeater's first 8 bytes borrowed that repeater's receptions and
passed (fixed in `4c569f91`).
- **Filters.** The blacklist and the hidden name prefixes apply to both
sources.
- **Missing table.** A database without the table resolves from `nodes`
alone, and the server checks for the table again until the ingestor has
created it.
- **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?,
name?, ambiguous}`.

### Server: estimated position
- `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`.
It lists every repeater in `client_node_names` that never sent an advert
(no row in `nodes` or `inactive_nodes`) and was heard directly at least
once (see above). A node with an advert but no position stays out: it is
already in `/api/nodes` under its advert name.
- **Gated** like the other endpoints on `client_receptions`: 404 when
`clientRxCoverage.enabled` is off, because the counts and the estimate
come from companion GPS fixes.
- **Which receptions count.** Only direct ones:
- `discover` and `advert` rows, keyed by the full pubkey, or by its
8-byte prefix when no other known pubkey shares it;
  - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone.

  `geo` rows are left out.
- **How the estimate is made.**
- Receptions within 100 m form one spot, with the best SNR of that spot.
  - An estimate needs at least 3 spots, at least 200 m apart.
- The position is the SNR-weighted centroid (`10^(snr/20)`). The radius
is the distance to the farthest spot, at least 500 m.
- The response is cached for 5 minutes, and the compute time is logged.

### Map
- `5fcde554` adds `public/map-estimated.js`, which draws each estimate
on its own layer:
  - a hollow, dashed "≈" marker and a dashed uncertainty circle;
- a popup with an "Estimated position" badge, the method, the caveat
that one-sided driving skews the estimate, and a link to the coverage
map;
- a Display toggle, "Estimated positions": on by default, kept in
localStorage, and also hidden by the repeater role filter and in route
view.

### Nodes page and search
- `12e6b345` adds a "Self-reported repeaters" section under the nodes
table, on the All and Repeaters tabs, filtered by the search box.
- **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a
self-reported detail view replaces "Node not found":
  - who reported the name;
  - the receptions;
  - the estimate, or why there is none;
  - links to the map and the coverage map.

  An unknown key still shows "Node not found".
- **Header search** adds matching self-reported repeaters, tagged
"self-reported".
- **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as
they are. These repeaters have no packets, so the packet filters, the
live view and the channel sender lookup must not match them.

### Docs
- `03580437` corrects the `/regions` section of
`docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the
switch, but the code uses `clientRegions.enabled`.
- The new subtopic, endpoint and config keys are documented in the same
file.

## Trust

Any companion the broker lets publish can report a name for any
repeater, and the newest report wins. This is the same model as
`/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to
linked companions.

A report for a repeater nobody heard directly is stored but not used: it
is not listed and adds no prefix candidate. A companion can still report
a wrong name for a repeater that was heard. The same companion can also
publish a fake `discover` reception for a made-up pubkey, so the "heard
directly" check filters junk, not a malicious companion;
`requireLinkedCompanion` is the control for that (its gate runs before
the `/names` handler too).

## Performance

- **`lookupPrefixes`** adds one term per key on `client_node_names`.
- It matches the prefix as a range, `target >= ? AND target < ?` (upper
bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive
column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX
idx_cnn_target (target>? AND target<?)`, and the reception check is
`SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`.
- It is still one query per chunk of 200 keys: 400 compound terms, under
SQLite's default limit of 500.
  - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks.
- **`/api/nodes/reported`** makes three kinds of query:
  - one grouped query on `client_node_names`;
  - one `lookupPrefixes` call for the short hashes;
- one `IN` query per 400 keys on `client_receptions`, served by
`idx_client_recept_heard_geo`.

The spread check is O(spots²) per repeater, with at most one spot per
100 m visited. The result is cached for 5 minutes; concurrent misses
wait for one computation, and the response is written after the lock is
released.
- **The map does not wait for it.** It draws its markers first and adds
the estimates when the list arrives. The map and the nodes page share
one cached fetch (`ReportedNodes.load()`).
- **Not benchmarked.** On the instance where this runs,
`client_node_names` holds one row today. I expect at most tens of such
repeaters per instance, but have not measured that.

## Notes

- **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2
occurrences: the handler's `msg` parameter and its `gps` field. That is
the decoded MQTT message, passed in the same shape as
`handleClientPacket` and `handleClientRegions` already take it. No API
response or internal structure uses it.
- **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min
TTL. They should move to the customizer later (rule 8).
- **Map filters.** The estimates follow the "Estimated positions" toggle
and the repeater role filter, not the area, region or last-heard
filters. The toggle's tooltip says so.
- **Name characters.** Names are rejected for control characters but not
for Unicode format characters (bidi overrides, zero-width joiners),
because emoji sequences in node names use the zero-width joiner. All
output goes through `escapeHtml`.
- **Not done:** the home-page search box (`home.js`, `/api/nodes/search`
with a Claim button) does not list these repeaters. Claiming a node
without adverts would have nothing to show.

## Tests

On this branch (based on `550829af`):
- `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and
`cmd/ingestor`.
- **New Go tests:**
  - handler and validation cases;
  - idempotent redelivery;
  - newest `observed_at` wins;
  - prune;
  - the gate on and off, without falling through to the observer path;
- client-only node, nameless `nodes` row, advert name wins, ambiguity
across the two sources;
  - hidden and blacklisted nodes;
  - absent-then-created table;
  - spot merging, threshold, weighted centroid and radius floor;
- `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an
advert position and a missing table;
- an advert without a position, a never-heard name, a name heard only in
`rxlog` paths, and the `clientRxCoverage` gate (404). These fail on
`12e6b345` and pass at head.
- a made-up pubkey sharing a heard repeater's 8-byte prefix, and two
reported targets sharing one (`4c569f91`). Both fail without that
commit.
- **New frontend unit suites:** `tests/unit/test-map-estimated.js` and
`tests/unit/test-reported-nodes.js` pass. Both load the real
`escapeHtml` from `app.js`.
- **`test-all.sh`** passes, all suites (with Python available for the
XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the
four changed frontend files.
- **E2E**, against a static server with a mocked API:
- `test-map-estimated-e2e.js`: 6/6, including the repeater role filter
and route view hiding the estimates. Mutating either hide makes its step
fail.
  - `test-reported-nodes-e2e.js`: 6/6.
  - `test-issue-1150-404-state-e2e.js`: 5/5.
- **Real data, API only.** On the fork's staging, `/api/nodes/reported`
lists one repeater: 5 spots, 17 receptions, an estimate with a radius of
2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays
listed under the stricter rules. Not re-checked after `4c569f91`.
- **Not done:** a check of that repeater in a browser.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 15:17:58 +02:00
efitenandClaude Opus 5.5 efd2aeabb4 perf(neighbors): rebuild the neighbor graph only in the background (#2221) (#2222) 
Fixes #2221.

## Problem

`getNeighborGraph` rebuilt `Server.neighborGraph` on the request
goroutine once it passed its 5 min TTL. It held `neighborMu` for the
whole build, which takes 11.1-11.4 s on a production store.

The 5 min recomputer went through the same staleness check. A tick that
landed just before the TTL skipped the rebuild and left it to the next
request. As a result `/api/nodes/{pubkey}/neighbors` ran at p95 10.4 s
and max 11.0 s (57 requests).

## Change

- **`getNeighborGraph`** serves the current graph. It builds only when
there is none yet, which happens at startup.
- **`rebuildNeighborGraph`** builds from the packet store whatever the
age of the current graph, outside `neighborMu`, and then swaps the
pointer.
  - `recomputeNeighborGraphCache` calls it first on every tick.
- Without a store it keeps the current graph, so tests that inject a
graph are unaffected.
- **`neighborBuildMu`** serializes builds, so a startup request and the
first tick do not both build.

Freshness is now bounded by the recompute interval
(`neighborGraph.cacheRecomputeIntervalSeconds`, default 5 min) plus one
build. Before this change it was the 5 min TTL plus one build, whatever
the interval. So the bound is the same only at the default 300:
- with a longer interval (for example 3600), `/api/nodes/{pk}/neighbors`
can serve a graph up to that old, where before it was about 5 min;
- with a shorter interval (for example 30), every tick does a full
build, where before there was at most one build per 5 min.

The config comment and `config.example.json` now say this (commit
`de7ae981`).

`IsStale`/`neighborGraphTTL` stay, because `path_inspect.go` still uses
them for `store.graph`.

## Perf

- **Request path:** after the first build it does no build at all. It
takes one mutex and reads one pointer.
- **Background:** the recompute tick now always builds, once every 5
min. Before, it built only on the ticks that found the graph stale, and
a request built in the other cases.
- **Total:** the number of builds stays about the same.

Before: `/api/perf` on production shows p95 10404 ms. After: the
follow-up comment gives client-side curl timings from staging (40
requests, p95 0.204 s), not `/api/perf`. Staging builds the graph in
1.03 s against 11 s on production, so the two are different instances.

## Tests

`neighbor_graph_rebuild_test.go`:
- a stale graph is served without a rebuild;
- a graph is built when there is none yet;
- a recompute tick replaces a graph that is not yet stale;
- a recompute without a store keeps the injected graph.

The first and third tests fail on master. `go vet` and `go test .` pass
in `cmd/server`, and `-race` passes on the neighbor-graph tests.

No frontend change.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 12:17:37 +02:00
efitenandClaude Opus 5.5 e8f6d4165a fix(observers): an observer's own transmissions are not receptions (#667) (#2220) 
Refs #667 (M1, the first part).

## Problem

Observations with `direction = 'tx'` are packets the observer's own
radio sent, published by the MQTT bridge. Nothing in `cmd/server` or the
ingestor's neighbor builder filtered on `direction`, so these rows
counted as receptions.

Two shapes, both seen in a production DB:

- **Forwarded flood.** A repeater-observer appends its own hash before
retransmitting (firmware `Mesh.cpp`), so the tx copy's path ends on the
observer itself, for example tx `["2979","2029"]` next to rx `["2979"]`
for the same transmission.
- **Own advert.** The observer's own advert goes out with an empty path
and no rx copy.

Effects:

- **The node detail "Heard By" card listed the observer as having heard
itself.** On one repeater-observer that was the only row in the card
(2,107 packets, no SNR).
- **The neighbor graph counted a forwarded advert twice,** once from the
rx copy and once from the tx copy (originator ↔ path[0]). This happened
in both the in-memory build and the ingestor's `neighbor_edges` builder.

## Scale

Last 2,000,000 observations on that DB, about 7.5 days:

- 1,385 tx rows against 1,805,378 rx rows;
- all tx rows have SNR and RSSI NULL;
- every tx row with a non-empty path has an rx row from the same
observer for the same transmission.

## Change

- `StoreObs.isOwnTransmission()` (`store.go`).
- These now skip tx rows:
  - `buildDirectHeardIndex` (`direct_heard.go`);
  - phase 1 of the neighbor graph build (`neighbor_graph.go`);
- the ingestor's `buildAndPersistNeighborEdges` SQL, via `o.direction IS
NOT 'tx'` (NULL rows from older bridges still count).

## Deliberately not changed

- **Reach** (`node_reach.go`). A tx row with path `[A, X]` from observer
X is the only row that shows X received A and forwarded it, because X is
absent from the rx copy `[A]`. There it is valid evidence.
- **Home regions** (`computeNodeHomeRegions`). A tx row of the
observer's own zero-hop advert counts the observer's own IATA, which is
where it is.
- **Counters and region filters.** These include observation counts,
`byObserver`, `packet_count` and the SQL region joins. There the effect
is double counting of about 0.08% of rows, not a wrong fact. That is
left for a follow-up if wanted.
- **Existing `neighbor_edges` rows.** The builder is incremental from
its watermark, so edges persisted before this change keep their counts.
The Heard By card is rebuilt from the store, so it is correct after a
restart.

## Tests

- **Red then green:**
- `TestBuildDirectHeardIndex_SkipsOwnTransmissions`: before the fix the
observer was credited for itself;
- `TestBuildNeighborGraph_SkipsOwnTransmissions`: X↔R1 count 2 instead
of 1, plus a zero-hop edge from an empty-path tx row;
- `TestNeighborEdgesBuilderSkipsOwnTransmissions`: 2 `neighbor_edges`
rows from a single tx observation.
- `go vet` and `go test .` pass in `cmd/server` and `cmd/ingestor`.

No frontend change. I have not yet checked the "Heard By" card in a
browser; that will be on a staging deploy.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 12:17:33 +02:00
efitenandClaude Opus 5.5 abbe19531d perf(paths): batch the resolved-path fallback for /nodes/{pk}/paths (#2223) (#2224) 
Fixes #2223.

## Problem

`loadCanonicalResolvedPaths` reads the resolved path of each candidate's
longest observation in batches (#2146). A transmission whose longest
observation has no stored `resolved_path` then went through
`bestResolvedPath` on its own, with two queries:

- `fetchResolvedPathForObs` re-read that same observation, which the
batch had just read;
- `fetchResolvedPathsForTx` read all of the transmission's observations.

On a production database, 24% of the observations with a path have no
`resolved_path`, so most `/api/nodes/{pubkey}/paths` requests paid for
this once per such transmission.

## Change

- **`loadFallbackResolvedPaths` batches the fallback.** It reads those
transmissions' stored paths in one query per `resolvedPathBatchSize`
transmissions:

  ```sql
  SELECT id, transmission_id, resolved_path FROM observations
  WHERE resolved_path IS NOT NULL AND transmission_id IN (...)
  ```

On a production database the planner runs it as `SEARCH observations
USING INDEX idx_observations_tx_ts (transmission_id=?)`. It then picks
the observation `bestResolvedPath` would pick: the longest `path_json`
with a stored path, the first in snapshot order on a tie. The result
goes into the resolved-path LRU, as before.
- **`longestStoredResolvedPath`** holds that choice. `bestResolvedPath`
now uses it as well, so the rule exists in one place.
- **A failed batch still falls back.** If the first-stage batch fails,
it goes through `bestResolvedPath` per transmission, as before. A failed
fallback batch does the same for its transmissions.

The answer is the same as before, and no cache is added.

## Perf

Staging, 9 nodes, requests made from the host itself (`localhost:3001`),
so no proxy or TLS. Each node 3 times, after one warm-up round:

| | median | p90 | max | total for 27 requests |
|---|---|---|---|---|
| before | 0.174 s | 0.604 s | 0.866 s | 6.95 s |
| after | 0.079 s | 0.433 s | 0.484 s | 3.22 s |

CPU profile, 30 s each, under the same request loop:
- **Requests completed:** 180 before, 261 after.
- **CPU in `handleNodePaths` per request:** 154 ms before (27.77 s /
180), 107 ms after (27.90 s / 261).
- **Fallback cost:** `bestResolvedPath` took 13.56 s of
`loadCanonicalResolvedPaths` before. The batched
`loadFallbackResolvedPaths` takes 6.57 s.

Caveats:
- The before run was on a process that had been up longer than the after
run, so the two LRU states differ.
- One node (`060679ad…`, about 11k transmissions) did not change: 0.455
s against 0.451 s. Its time is elsewhere in the handler, and this PR
does not address it.

## Tests

- **`TestLoadCanonicalResolvedPathsBatchesFallback`:** 1,001
transmissions whose longest observation has no stored path.
  - On master: 2,002 per-transmission queries.
- Now: 0 per-transmission queries and 6 batched ones (3 for the longest
observations, 3 for the fallback), with every path taken from the
shorter observation.
- **Existing tests:**
`TestLoadCanonicalResolvedPathsMatchesBestResolvedPath` still passes. It
covers the longest observation without a path (#810), ties, nothing
stored, an empty stored path, and an observation absent from the
snapshot. The LRU and query-count tests also still pass.
- `go vet` and `go test .` in `cmd/server` pass.

No frontend change.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 11:58:28 +02:00
n30nex b9593f7174 fix(accounts): keep pending registration updates atomic (#2253) 
Pending-account re-registration previously changed the password and
display name in separate unconditional writes. A failed name update
could leave only the password changed, and a stale pending lookup could
authorize changes after activation.

Update both fields in one statement constrained to `status = 'pending'`.
If the account changed before the write, keep the generic registration
response and issue no new activation message. Failed updates preserve
the original credentials and activation link.

Validation: reproduced the partial-write bug through the HTTP handler
before the fix; 22 authentication tests and all 109 account-store tests
pass on native SQLite. The four pending-registration store tests passed
ten race-detector repetitions, covering competing registrations, a held
activation writer, and active/disabled/deleted accounts. The complete
server suite and full account-store race suite also pass. Local Chromium
passed all 28 account journeys. No schema, configuration, UI, or
dependency changes.

Closes #2228.
 2026-10-20 14:53:26 +02:00
efitenandClaude Opus 5.5 30d58a5e15 fix(store): resolve region and area sets before taking s.mu (#2219) (#2247) 
Fixes #2219.

`resolveRegionObservers` and `resolveAreaNodes` query SQL when their
cache is cold or a key is new. Nine callers looked them up while holding
`s.mu.RLock()`, which `store_lock_sql_invariant_test.go` allowed through
two explicit exemptions. One uncached region or area filter then blocked
the poller's writer (`s.mu.Lock()`) and every reader queued behind it,
`/api/healthz` included.

## What changes

All in `cmd/server/store.go`. The lookups now run before the lock is
taken:

- **`QueryPackets`, `QueryGroupedPackets`:** follow the #2147 `nodePK`
pattern. `resolvePacketFilterSets(q)` resolves node, region and area
into `packetFilterSets` before the lock, and `queryPacketsLocked` and
`filterPackets` take that struct instead of `nodePK`.
- A region that resolves to no observers still returns nothing, and the
area is then not looked up, as before.
- **`GetChannels`, `GetEncryptedChannels`, `GetChannelMessages`,
`computeAnalyticsChannels`, `computeAnalyticsRF`,
`computeAnalyticsSubpathsWindowed`:** the region lookup moved above
`s.mu.RLock()`. Their area lookups were already outside it.
- **`computeAnalyticsSubpaths`:** resolves the region first and passes
the set to `computeSubpathsSlow`, which no longer looks it up itself.

Results, cache TTLs and cache code are unchanged.

The other callers (`routes.go`, `clock_skew.go`, `neighbor_api.go`,
`region_nodes.go`, `retransmission_pressure.go`, and the topology,
distance, hash-size, collision and bulk-health paths in `store.go`)
already looked up outside the lock, and are not changed.

## Tests

**`store_lock_sql_invariant_test.go`:**
- The two exemptions are removed.
- With the exemptions removed and the old `store.go`, the test lists the
9 call sites. With this change it lists none.
- `getCachedNodesAndPM` keeps its exemption; it is outside this issue.

**`store_cold_lookup_lock_test.go`, new:**
- `TestColdRegionAreaLookupDoesNotStallHealthz` has 11 subtests: raw and
grouped packets, each with a region and with an area, plus the 7 region
callers. Each one:
  - blocks the cold lookup on the existing `cacheLoadHook`;
  - queues an `s.mu.Lock()` writer;
- requires `handleHealthz` to answer 200 within 1 s, and the writer to
finish, while the lookup is still blocked.
- `TestRegionSetResolvedBeforeLockSurvivesEviction` covers the eviction
case from the issue:
  - the cache entry is replaced while the caller waits for the lock;
  - the result must still match the set the caller resolved;
- a fix that only pre-warmed the cache and looked it up again under the
lock gets 0 rows here.

**How I checked them:**
- With the old `store.go`, all 11 subtests fail ("healthz … blocked for
more than 1s"), and so does the eviction test.
- Moving only the `GetEncryptedChannels` lookup back under the lock
fails
`TestColdRegionAreaLookupDoesNotStallHealthz/GetEncryptedChannels`.
- `go test ./...` in `cmd/server` passes; `go vet` and `gofmt -l` are
clean.

## Not done

- No `-race` run, no E2E, no browser check, and no latency measurement
on a live instance, so this PR makes no latency claim.
- `computeAnalyticsSubpaths` and `computeAnalyticsSubpathsWindowed`
still call `getCachedNodesAndPM` under the lock. The invariant test
still exempts that.
- This PR and #2246 both change the first lines of
`QueryGroupedPackets`. Whichever merges second needs a small rebase
there.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 11:04:07 +02:00
efitenandClaude Opus 5.5 fe58dbd7d9 fix(server): read StoreTx state under s.mu in grouped packets and other readers (#2237) (#2246) 
Fixes #2237.

Ingest appends to `tx.Observations` and rewrites a transmission's header
fields (`LatestSeen`, observer, path, counts) under `s.mu.Lock`. Several
readers touched that state without the lock. This PR follows the
approach of dborup/CoreScope#475, the reporter's fix: copy what each
reader needs under the RLock it already takes, and sort or encode
outside it. That fix does not apply to upstream as is (the fork has
diverged), so this is a rewrite against current master.

## What changes, per site

| Site | Before | After |
|---|---|---|
| `QueryGroupedPackets` (`store.go`) | Sort read `tx.LatestSeen` after
`RUnlock`; the page (`groupedTxsToPage`, `storeTxDistinctIatas`) was
built from the shared cache with no lock | The sort key is copied under
the existing RLock (`groupedEntry`). The cache holds those entries.
`groupedPage` copies only the requested page's rows under a short RLock;
the maps are built after it |
| `txGetParsedPath` (`store.go`) | Filled `tx.parsedPath` lazily under
RLock, so concurrent readers wrote the same fields |
`pickBestObservation` fills it through `setTxPath` where the path is
written, under the write lock or before the tx is shared |
| `BuildFromStoreWithOptions` (`neighbor_graph.go`) | Ranged
`tx.Observations` after releasing the lock | Each tx's `Observations`
slice header is copied with the packet list |
| `/api/audio-lab/buckets` (`routes.go`) | Read path, observer and count
after `RUnlock` | The sort stays outside the lock (`RawHex` never
changes); the picked rows are read under a second short RLock |
| `trackedBytes` (`/api/stats`, perf stats) | Read with no lock | Read
under `s.mu` |
| Observer analytics (`routes.go`, `observer_analytics.go`) | Read
`tx.Hash` after `RUnlock`; the content-hash migration rewrites it |
`txByID` holds copies made under the existing RLock
(`txAnalyticsFields`) |

`/api/nodes/{pk}/paths` and `GetNodeHopAnalytics`, items 2 in the issue,
are not changed: on current master they already copy under the lock
(`snapshotRPObs`) or run inside it.

Appending to `tx.Observations` never changes the elements an already
copied slice header covers, so copying the header is enough there.

## Tests

`store_read_race_2237_test.go` adds one `-race` regression test per
site. Each runs an ingest-style writer under `s.mu.Lock` against the
reader:

- `TestQueryGroupedPacketsIngestRace2237` (cold and cached paths)
- `TestTxGetParsedPathReadersRace2237`
- `TestBuildFromStoreIngestRace2237`
- `TestAudioLabBucketsIngestRace2237`
- `TestPerfStoreStatsTrackedBytesRace2237`
- `TestStatsTrackedBytesRace2237`
- `TestObserverAnalyticsHashMigrationRace2237`

Plus `TestGroupedPageReflectsHeaderAfterIngest2237` (no race detector):
header fields, `distinct_iatas`, count and order after ingest.

How I checked them:
- With the fix, `go test -race -count=3 -run 'Race2237'` passes.
- Each site reverted on its own, one at a time, makes its test report
`DATA RACE` (between 1 and 24 reports per site).
- `go test ./...` in `cmd/server` passes; `go vet` and `gofmt -l` are
clean.

## Performance

`QueryGroupedPackets` is on the packets page's hot path.
`BenchmarkQueryGroupedPackets2237`, 30K transmissions, median of 8
alternating runs per binary (Windows, noisy: single runs varied up to
2x):

| Case | Before (ns/op) | After (ns/op) | B/op before → after |
|---|---|---|---|
| cold, limit 50 | 1,015,512 | 1,022,879 (+1%) | 1.62 MB → 2.10 MB |
| cold, limit 1000 | 1,711,237 | 1,793,032 (+5%) | 2.93 MB → 3.64 MB |
| cached, limit 50 | 29,439 | 32,608 (+11%, about 3 µs) | 69 KB → 82 KB
|
| cached, limit 1000 | 606,970 | 662,204 (+9%) | 1.38 MB → 1.61 MB |

The extra cost is copying the page's rows. The lock is held for that
copy only (at most `limit` rows), never for the sort or for building the
maps.

## Not done

- `BuildFromStoreWithOptions` still calls `tx.ParsedDecoded()` outside
the lock, and `tx_rewrite_feed.go` resets `decodedOnce` /
`parsedDecoded` under the write lock. That pair is still a race. Fixing
it means parsing under the RLock on the first build, so I left it for a
follow-up rather than widen this PR.
- I ran the new tests under `-race`, not the whole package.
- No load test against a live server; backend only, no browser check.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 10:41:01 +02:00
nullrouten0nullroutenClaude Sonnet 5.5
 		7c00a42947 perf(reach): answer /nodes/{pk}/reach from a rolling hop index, not a path_json LIKE scan (#2214) 
Refs #2146 (reach item: "an indexed hop table would remove it"). Follows
#2155.

## What changes

Reach finds the observations that relayed through a node with `path_json
LIKE '%"XX"%'` over every observation in the window. The cost follows
the window, not the node: a quiet repeater costs about as much as a busy
one. When the database does not fit in the page cache this outlives the
60 s `WriteTimeout`, the request 500s, and nothing is cached. #2146
reports this endpoint at 14.9 s average and 74.7 s max on a 10.9 GB
instance. Its 1.9 s figure after #2155 is a warm-cache timing; the scan
reads the whole window, so cold it is far slower (below), and it holds
one of the 4 read-pool connections the whole time.

This PR adds an inverted index and uses it only when it is complete for
the requested window:

- `observation_hops(hop, timestamp, observation_id)`, `WITHOUT ROWID`. A
token lookup is a seek and the window bound is a range on the same
b-tree.
- Triggers on `observations` maintain it (insert, delete, `path_json`
update), so prune and path-repair need no changes.
- `observation_hops_coverage.since` marks how far back the index is
complete. If the window starts before `since`, or the table is missing
(older DB, backfill unfinished), reach runs the existing `LIKE` scan
unchanged.
- The ingestor owns the schema, a resumable newest-first backfill, and a
daily trim to `retention.hopIndexDays` (default 14). The server stays
read-only and `AssertReady` does not require the table.

The backfill starts after the ingest gate opens rather than inside
`OpenStore`. My first version did the latter and held live ingest in the
buffer for about 15 minutes on a 13.7 GB database (nothing dropped).
Chunks are 15 minutes because one-hour chunks held the writer for 2 to 7
s per commit on EBS.

## Results

### Real data: vanilla vs this branch

One snapshot of a real instance (11.5 GB, 10.7M observations over 14
days) served by both builds on the same machine (8 cores, 30 GB RAM,
NVMe, warm cache). One cold request per cell, seconds, repeaters from
busiest to quietest by 24 h relay count.

| node (relays / 24 h) | 7 d vanilla | 7 d branch | 14 d vanilla | 14 d
branch |
|---|---|---|---|---|
| 7,314 | 24.5 | 4.5 | 8.3 | 9.6 |
| 7,307 | 1.7 | 2.4 | 1.3 | 3.1 |
| 7,047 | 3.3 | 2.2 | 2.0 | 2.3 |
| 2,151 | 20.0 | 0.44 | 29.6 | 2.0 |
| 673 | 23.7 | 0.18 | 28.5 | 0.62 |
| 240 | 21.8 | 0.01 | 27.8 | 0.04 |
| 52 | 21.6 | 0.00 | 28.4 | 0.00 |

- Nodes with few relays drop from 20 to 30 s to under a second.
- The busiest relays are about the same, sometimes slower: both builds
spend their time loading tens of thousands of matching rows by primary
key, which the index does not speed up.
- Single runs, and page-cache state moves them a lot: an earlier run,
with the file freshly cached, had vanilla's quiet nodes at 4.5 to 5 s
for 7 d, close to #2155's warm figure. Read orders of magnitude, not
exact figures. Nothing failed on this fast machine.

**Same answers.** Unit tests check indexed rows equal `LIKE` rows,
fallback on partial coverage, and trigger upkeep. On real data, 21
node/window pairs (3 to 14 days) requested from both builds at the same
moment were all identical once links were compared by public key. `LIKE`
ignores case and the index does not; 0 of 10.1M observations in the
window have a lower-case hop. Above `reachScanRowLimit` both paths
truncate, and they can keep different subsets.

### Production instance

t4g.large (2 vCPU, 8 GB, EBS), 13.7 GB database, about 4.5M observations
per 7 days, so the cache held a fraction of it.

- Before: `days=7` on a busy node returned 500 (`context canceled`), and
`days=1` took 58.5 s.
- With the index (38 probes of four nodes over 18 hours): `days=7` on
that node returned 200 in 23 s. The busiest probe node had a median of
0.7 s and p95 18 s. The hottest (127k relay observations in 7 d) had a
median of 1.3 s, p95 52 s, and one probe past 60 s at evening peak. The
quietest had a median of 0.04 s.
- On 2026-10-04 the host was resized to 4 vCPU and 16 GB so the cache
holds the database. Same index afterwards (493 probes): hottest node
median 1.4 s, p95 1.7 s, max 38 s; quietest median 0.05 s. (From
2026-10-03 the instance also ran a local change, not in this PR, that
lets reach requests continue in the background, which only affects
requests over 45 s.)

So on a small cold host the index turned "always fails" into "usually
works" but did not make the hottest relays reliably fast. That needs
cheaper row loading (a compact side table measured 4 to 10x on cold
reads; kept out of this PR) or background computation.

### Benchmarks (included)

```
go test ./cmd/server -run xxx -bench ScanReachRows -benchtime 5x
  # 600k observations, 11 hops avg, 7 d window in 14 d, warm cache
  hot_token/like     816 ms     rare_token/like     486 ms
  hot_token/indexed  293 ms     rare_token/indexed  1.7 ms

go test ./internal/dbschema -run xxx -bench ObservationInsert -benchtime 20000x
  # one observation per transaction, WriterDSN (WAL, synchronous=FULL), 11 hops
  without_index  23 us/op      with_index  127 us/op
```

The triggers add about 100 us per observation, roughly 0.1% of a core at
the production instance's 8 observations per second. They cap one writer
at about 8k observations per second instead of 43k. The benchmark ran in
a temp directory, so it shows CPU and b-tree cost, not disk sync cost.
Over 13 days on the production instance, 0 observations were dropped and
newest-observation age was p99 4 s. Four of 623 samples were 20 s or
more: 91 s and 20 s came within six minutes of a container restart, 31 s
came 24 hours after one (when the ingestor's daily tickers fire), and 23
s had no restart nearby. Index coverage stayed between 14.0 and 15.2
days.

**Disk.** On the snapshot (`dbstat`) the index is 1.98 GB for 10.1M
observations in the window, about 196 bytes each, next to 6.81 GB for
`observations`. It scales with hops per observation and `hopIndexDays`.
Backfill of 77M rows took 15.5 min on the production instance.

**Perf justification (AGENTS.md).** Reach goes from a scan over the
whole window to a seek over the matching rows. Everything new is
bounded: the index by `hopIndexDays` and the daily trim, the backfill by
chunk size and a cancellable context, and the server adds no in-memory
state. The ingest hot path pays the trigger cost above.

## Design notes and open questions

- Why not the in-memory store: it defaults to 7 days (`retentionHours:
168`) and reach offers up to 30, so it could not answer the longer
windows. The other option listed in #2146, driving the query from
`observations.timestamp`, is what #2155 already did.
- SQLite only: the triggers use `json_each`. The optional PostgreSQL
backend in #2145 would need its own equivalent.
- No off switch: the table and triggers are always installed and
`hopIndexDays` only sets the window. If you would rather have this
opt-in, or want a smaller default for small disks, I'll change it.

## Operator notes

- Upgrade: the ingestor installs the table and triggers on start and
backfills. Until a window is covered, reach behaves as today.
- Rollback: an older binary keeps indexing new rows with no trim. To
remove it with the ingestor stopped: `DROP TRIGGER
trg_observation_hops_ai; DROP TRIGGER trg_observation_hops_ad; DROP
TRIGGER trg_observation_hops_au; DROP TABLE observation_hops; DROP TABLE
observation_hops_coverage;`
- New key `retention.hopIndexDays`, documented in `config.example.json`.

## Tests

`go test ./...` passes in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`. New: `node_reach_hops_test.go`,
`observation_hops_test.go`, `observation_hops_backfill_test.go`, and the
two benchmarks.

## Update after review (commit `2291bea2`)

The text above is what was reviewed at `95e17976`. Where they differ,
this section is current:

- **Opt-in.** `retention.hopIndexDays` unset or 0 (the default, and the
value in `config.example.json`) installs nothing. A value above 0
installs the table and triggers and backfills that many days. This
replaces "default 14", the "No off switch" note and the Upgrade note
above.
- **Changing the setting** applies at the next ingestor start: raising
it backfills the gap, lowering it trims, and 0 removes the triggers and
both tables. On a 10.5M-observation copy the removal took 45 to 71 s, at
startup before MQTT connects.
- **Case.** Hops are stored and matched upper-cased, so the index
ignores ASCII case the way LIKE does. This replaces the sentence on case
under "Same answers".
- **Rollback.** Set it to 0 and start the ingestor once, then roll back.
The manual `DROP` list above still works with the ingestor stopped.
- **Disk.** A full 14-day backfill on that copy built 2.17 GB for 9.68M
observations, about 224 bytes each, in 13 min 37 s.
- Backfill and trim are serialized, and the chunk SQL lives in
`dbschema` next to the triggers.

## Update after review, round 2 (commit `95690e8c`)

Where this differs from the two sections above, this one is current:

- **Startup order.** The index is installed or removed in the startup
block, after the MQTT subscription is up and before the ingest gate
opens, so packets arriving during a removal are buffered
(`ingestBufferSize`), not lost. This replaces "at startup before MQTT
connects" in the first update.
- **Mark moves cover their range.** Each backfill and trim transaction
indexes or deletes everything between the old mark and the new one, so a
row arriving between the chunk-picking read and the transaction is
accounted for either way.
- **Negative `hopIndexDays`** is a config error at startup; the ingestor
refuses to run. This replaces "−5 is off" in the first update.
- **Server.** If the index disappears between the coverage check and the
scan, the request is answered from the `LIKE` scan instead of a 500.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-19 00:36:51 +02:00
Alex BandClaude Opus 5.5 9529bf9d41 perf(store): read resolved_path pubkeys in place, one copy each (#2197) 
Refs #1870.

## Problem
`Load`, `loadChunk` and `scanAndMergeChunk` decode each observation's
`resolved_path` only to collect its non-empty pubkeys. On live data
that's a JSON array of pubkeys and nulls, about 420 characters, present
on most observations. Decoding into `[]*string` allocated a string and a
pointer for every hop, plus two slices.

## Change
- `resolvedPubkeysFromJSON` reads the usual array of escape-free strings
and nulls in place. Anything else falls back to decoding, so it returns
exactly `extractResolvedPubkeys(unmarshalResolvedPath(s))`.
- Relay pubkeys repeat across nearly every observation, so each one goes
through `internStr` (from #2183) and is kept once. A pubkey seen before
is found by substring, without copying, and the indexes it feeds
(`byNode` and friends) share that copy. Interned keys live for the
process, evicted nodes' included; the table is bounded at 64k entries.
- The three bulk-load call sites use it.
- **Eviction too.** `resolvedPubkeysForEvictionBatch` runs on every
eviction batch, about once a minute in steady state, and spent about 130
MB per five minutes on a busy week. It now reads each pubkey in place
and de-duplicates each packet's short list by scanning it instead of
building a map per packet.
- Other callers of `unmarshalResolvedPath` are unchanged.

## Measured
| | before | after |
|---|---|---|
| allocations for 3 known pubkeys | 13 | 1 |
| eviction, allocations per evicted packet (3 observations × 4 hops) |
51 | 15 |
| this path's allocations over startup (copy of a busy week) | 236 MB |
61 MB |
| server CPU over the run | 175.6 s | 170.6 s |

The startup rows need that dataset and are from a single run. The
benchmarks are rerunnable (`go test -run X -bench ResolvedPubkeys
-benchmem ./cmd/server`, medians of 3):

| | master | this PR |
|---|---|---|
| `BenchmarkResolvedPubkeysFromJSON` (5 relays + 3 nulls, pool of 2,000)
| n/a | 619 ns, 128 B, 1 alloc |
| `BenchmarkResolvedPubkeysForEvictionBatch` (1,000 packets × 3
observations) | 14.3 ms, 3.96 MB, 65,495 allocs | 6.7 ms, 1.36 MB,
18,501 allocs |

## Tests
- `TestResolvedPubkeysFromJSONAllocations`: red commit first, with the
bound tightened in green.
- `TestResolvedPubkeysForEvictionBatch`: keys, order, and an allocation
bound. It has its own red commit.
- `TestResolvedPubkeysFromJSONMatchesDecoding` compares against
decode-then-extract on 26 edge cases and 20,000 random inputs.
- `go test ./cmd/server` passes.

No config or UI changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 11:46:23 +02:00
Alex BandClaude Opus 5.5 cf342efc43 perf(store): key byNode's de-dup sets by transmission ID (#2199) 
## Problem
`nodeHashes` is the set behind `byNode`'s de-dup: for every node, the
transmissions already listed under it. It was keyed by each
transmission's 16-character hash string. On a week of a busy network
that's about 650k entries and 21.5 MB, four times the size of `byNode`
itself.

## Change
- The sets are keyed by transmission ID. That's the same membership,
since a store holds one transmission per hash.
- Eviction deletes by ID.
- The health endpoint's region filter matches regional observations by
transmission ID.
- `newNodeSets` and `nodeHasTx` wrap the map for callers and tests.

## Measured
| | before | after |
|---|---|---|
| node sets' live heap (copy of a busy week) | 21.5 MB | 13.2 MB |
| fixture, 500k (node, packet) pairs | 25.8 MB | 18.0 MB |

## Tests
- `TestNodeSetRetainedHeap`: red commit first.
- `TestAddToByNodeDedup`.
- The existing eviction, routes and resolved-index tests updated to the
ID-keyed sets.
- `go test ./cmd/server` passes.

No config or UI changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 11:46:18 +02:00
Adam 4fdd494d9b fix(nodes): read the transport-direct path byte at offset 5 (#2225) (#2234) 
Red commit: 3bb81254 (fails on assertions both ways: `zero-hop counts =
map[pk_relayed:map[SAT:1]], want map[pk_transport:map[AUS:1]]`)

Fixes #2225

## What this does

`nodeZeroHopRegionCounts` read the path byte at offset 1 for
`ROUTE_TYPE_TRANSPORT_DIRECT` adverts. Transport routes carry 4
transport-code bytes between the header and `path_length` (firmware
`docs/packet_format.md`), so offset 1 is the first byte of
`transport_code_1`. Two effects, both fixed:

- A transport-direct zero-hop advert counted as zero-hop only when that
code byte's low 6 bits happened to be 0, so the node usually got
`fewest_hops` instead of `zero_hop`, and #804 analytics missed it.
- A relayed transport-direct advert whose first code byte was `0x00` (or
`0x40`, `0x80`, …) counted as zero-hop.

## How

- New `packetpath.PathByte(rawHex) (routeType int, pathByte byte, ok
bool)` applies the transport-code offset in one place, next to the
existing `IsTransportRoute`, `HashSize` and `DecodePathFromRawHex`.
- Every reader of the path byte from `raw_hex` in `store.go` now uses
it:
  - `nodeZeroHopRegionCounts` (the bug);
  - `computeAnalyticsHashSizes`;
  - `computeNodeHashSizeInfo`;
  - `computeMultiByteCapability`.

The last three already had the right offset, each with its own copy of
the rule, so their behavior doesn't change. `packetpath.HashSize` uses
`PathByte` too.

## Performance

- `PathByte` runs once per advert on every recompute, so it must not
allocate. `hexByteAt` now decodes the two hex nibbles itself instead of
calling `hex.DecodeString`; `TestPathByteDoesNotAllocate` pins it at 0
allocs.
- Without that, `BenchmarkRecomputeNodeHomeRegionSets` went from 17,232
to 457,232 allocs/op. With it, the count is unchanged from master.
- Time on the same machine (Apple M5, Docker linux/arm64, `-count 3`):
~61 ms/op on master, ~54 ms/op here. Your hardware measured ~3x higher
on #2190, so expect proportionally more there; the change itself only
removes work.

## Tests

- `TestZeroHopCountsReadTransportDirectPathByte`: a transport-direct
zero-hop advert whose first code byte is `0xaa` counts, and a relayed
one (path byte `0x41`) whose first code byte is `0x00` doesn't. Checked
through both `nodeZeroHopRegionCounts` and `computeNodeHomeRegions`
(#804).
- `TestPathByte` covers all four route types, upper-case hex, and
too-short or malformed input. `TestPathByteDoesNotAllocate` pins the
allocation count.
- `go test` passes for `cmd/server`, `cmd/ingestor` and
`internal/packetpath` (`CGO_ENABLED=1`). `gofmt` and `go vet` are clean.

No API or UI change, no new `map[string]interface{}`, and no config.
 2026-10-16 10:53:10 +02:00
efitenandClaude Opus 5.5 927a8eec8a fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211) 
Refs #748 (M3a: correct numbers within the in-memory window; the SQL
history in M3b is a separate plan, see the end).

## Problem

The node analytics range picker offers 24h, 7d, 30d and All (365d), but
the store only holds its in-memory window (`retentionHours`, then
eviction). `GetNodeAnalytics` still divided availability and packets/day
by the requested range. Measured on our live instance (14-day window),
repeater BE-BRE-ON8AR:

| Range | Availability | Packets/day |
|---|---|---|
| 7d | 88.7% | 91 |
| 30d | 42.6% | 265 |
| All | 3.5% | 22 |

Nothing on the page said that the data stopped at the edge of the
window.

## What changes

- `coverageStart` returns the edge of the in-memory window: the latest
of `oldestLoaded` (background-load progress), `now - retentionHours`,
and a new `evictedThrough` that eviction sets to the newest `FirstSeen`
it removed, so memory-pressure eviction inside the retention window
moves the edge too.
- `s.packets[0]` is not that edge, and the first version of this PR used
it. Chunks are selected by `last_seen`, so a transmission first heard
weeks ago and heard again recently sits at the head with its old
`FirstSeen`. On staging (7-day window) it returned 21/07.
- When the edge lies more than a minute after the requested start, node
analytics counts only transmissions from the edge, computes availability
and packets/day over that span, and returns `timeRange.dataFrom` and
`timeRange.partial`.
- The minute of slack keeps a range equal to the retention window from
reading as partial. Without it every 7d request on staging (168h) was
partial, because the two edges came from separate clock reads.
- The analytics page shows "Data from 2026-10-07: the server keeps about
7 days of packets in memory. Availability and packets/day cover that
period, not the full 365 days." when partial.
- **UTC fix.** The window bounds of node analytics and node hop
analytics are formatted in UTC. They are compared as strings with UTC
`FirstSeen` values, so on a non-UTC host the 7d window lost its oldest
hours (2 at +02:00). Docker images run in UTC, which is why this did not
show in production.
- `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the
edge is kept.

## Tests

- Go, `node_analytics_coverage_test.go`:
  - full coverage within memory;
  - 30d over 3 days in memory;
  - empty store;
- a re-heard 85-day-old transmission at the head of `s.packets` (edge
stays at retention, packet not counted, no bucket before `dataFrom`);
  - memory-eviction edge;
  - eviction records `evictedThrough`;
  - retention 30 s shorter than the range is not partial, while 30d is;
  - window bounds under a +02:00 `time.Local`.
- Two tests were checked to fail on the old code: the local-zone test
(166 of 168 packets, a `+02:00` bound) and the retention test without
the slack (`dataFrom 09:57:08` vs `from 09:56:38`).
- Full `cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master). One run of five reported `FAIL` and four reruns were green; I
did not capture which test failed.
- Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js`
(5 assertions on the real `node-analytics.js` in `vm`), in
`test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js`
(registered in `deploy.yml` and `scripts/non-unit-tests.json`), against
the real fixture response with no interception: 7d is not partial and
shows no notice; All is partial and the notice names the real start
date. 2/2 locally.

## Validation

Staging (retentionHours 168), same repeater, after the background load
completed:

| Range | Before | After |
|---|---|---|
| 7d | 82.7% | 83.3%, not partial |
| 30d | 19.3% | 83.3%, partial from 2026-10-07 |
| All | 1.6% | 83.3%, partial from 2026-10-07 |

In the browser the notice appears under All and not under 7d.

## Not done

- M3b, older history from SQLite. In the database only the sender is
indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay
hops only in `observations.resolved_path` (filled for about 27% of
observations on our live instance, not indexed). A fallback today would
show only a node's own packets for older days, a different meaning than
the in-memory part. Doing it properly needs a per-node index written by
the ingestor (pubkey, transmission, role); that is a separate plan.
- The other analytics endpoints with a range (observer analytics, RF)
were not checked for the same dilution.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:27:21 +02:00
efitenandClaude Opus 5.5 cab4f71c64 feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) (#2209) 
Fixes #1268.

Frequency, spreading factor, bandwidth and coding rate are per-device
settings that never travel in a packet. The public MeshCore map is the
only source. This PR adds them to the node detail views, opt-in, and
answers the open policy questions from the triage comments.

## Policy decisions

| Question (triage) | Answer in this PR |
|---|---|
| Opt-out for self-hosted instances | Opt-**in**: `rfParams.nodesUrl`
empty (default) = nothing fetched, `/api/rf-params` returns `enabled:
false`. Same model as `knownChannelsUrl`. |
| 33 MB every 6 h, thundering herd | Random 0-60 s delay before the
first fetch, then every `refreshHours` (default 6). The map sends gzip
(17 MB on the wire). It sends no ETag or Last-Modified, so conditional
requests are not possible. |
| Size cap | 128 MB on the decoded body (it is 51.5 MB today). Exceeding
it fails the fetch instead of parsing a truncated body. |
| Field shapes at the parse boundary | 64-hex key; 100-1100 MHz; SF
5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows
are dropped. A body that is not a well-formed array fails the fetch. |
| Keep stale cache on anomaly | Any failure keeps the previous snapshot;
a failed presets fetch keeps the previous presets. `fetched_at` is
returned so an outage is visible rather than hidden. |
| Freshness | Only adverts within `maxAgeDays` (default 7) are kept at
fetch time and re-checked at serve time. |
| TLS pinning | Not done: Go's certificate validation plus the field
validation above. |

## What changes

- `cmd/server/rf_params_cache.go`: stream decoder, validation, cap
reader, preset parser (presets with identical values are merged, titles
joined), background refresh. In memory only, no DB writes, so the
read-only server invariant holds.
- `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`;
`nodes` holds only nodes this instance knows, keyed by lower-case
pubkey. `/api/nodes` is unchanged, so its payload does not grow for
every consumer and the 500-row paging is unaffected.
- Node detail (side panel and full page): `LoRa` row with the four
values, the preset name when all four match one, and the age of the map
advert. Shown only when the feature is on and the server has fetched the
map at least once.
- No column in the Nodes table, on purpose: on our network such a column
showed one preset title in 814 of 1,366 rows and nothing in 443. Showing
only the nodes that differ from their region is the follow-up PR.
- `config.example.json` documents the block with the two URLs to use.

## Cost

- Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52
s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days).
Runs in a background goroutine every 6 h.
- The endpoint walks the cached node list once with map lookups:
O(nodes).
- Frontend: one bulk fetch of `/api/rf-params` when a node detail opens
(client TTL 5 min), not one per node.

## Tests

- Go (`rf_params_cache_test.go`): parse with every rejection reason,
malformed bodies (not an array, truncated, wrong type, empty), preset
merge and invalid presets, cap reader (one byte over refused, exactly at
the cap accepted), fail-soft against an `httptest` upstream (503, HTML
body, presets-only failure), config defaults, route (disabled shape,
only known nodes, entry aged past max age dropped at serve time). Full
`cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master).
- Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions
on the real file in `vm`), in `test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in
`deploy.yml` and `scripts/non-unit-tests.json`). It checks the real
disabled state on the fixture server (no detail row), then serves
`/api/rf-params` through `page.route`: no table column, full detail page
with preset, side panel with raw values, and the hint for a node that is
not on the map. 4/4 locally, plus the existing node table and node
detail E2E files.

## Validation

On a staging instance with `rfParams` set to the map and preset URLs:

- Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s
after start.
- `/api/rf-params` returned 1,115 of our nodes: 988 on
869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on
SF7.
- In the browser the detail row shows in the side panel and on the full
page, and the Nodes table keeps its original columns.

## Not done

- Highlighting nodes whose settings differ from their region's majority
(a badge in the Nodes list plus a filter) is the follow-up (separate
PR). In that follow-up a CR-only difference stays informational until it
is confirmed whether MeshCore radios use the explicit LoRa header (then
CR travels in the header and a mismatch still decodes).
- `maxAgeDays` is config-only; the customizer is a later step.
- No data is written to the DB, so the RF parameters are gone after a
restart until the next fetch (at most 60 s plus the fetch time).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:26:45 +02:00
Adam 6d56f55168 fix(nodes): place nodes in regions by zero-hop home regions (#1879) (#2190) 
Red commit: cd546b23 (fails on assertions, e.g. `region "SAT": got
[pk_aus pk_edge pk_far], want [pk_edge pk_far]`)
Red commit (fewest-hops fallback): a66a7f36 (fails on assertions, e.g.
`region "AUS": got [pk_cap pk_gulf pk_near pk_tie], want [pk_cap]`)
Red commits (review round 1): d4154ee1 (`region_placement` missing),
862a4510 (`zero: RegionFallbackMaxHops() = 3, want 0`), 16dcd8dd
(`pk_mixed:{fewest_hops [AUS] hops=0}`, want `[SAT] hops=2`), 95f2aca4
(`pk_late not placed after the startup load finished`)

Fixes #1879

## What this does

`/api/nodes?region=` put a node in every region where any observer heard
any of its adverts. Adverts flood, so a node showed up in every region
its adverts got relayed into, not just where it actually is.

Now, as agreed in the issue:

- **Nodes heard zero-hop are placed by their home regions.** Same rule
as #804: a zero-hop DIRECT advert can't have been relayed, so the
observer that heard it is in RF range. This is the tally
`computeNodeHomeRegions()` already does, pulled out into
`nodeZeroHopRegionCounts()` so analytics and the filter share one
definition.
- **It's a set, not a winner.** A node heard zero-hop in two regions is
in both. So per-region counts don't add up to the node total anymore, on
purpose.
- **Everything else goes by fewest hops.** A node never heard zero-hop
is in the region(s) whose observers heard it in the fewest hops, and in
no region when even that is more than `regionFallback.maxHops` (config,
default 3); it still shows unfiltered. The old fallback (every region
that heard it at all) turned out to be most of the leak: see the numbers
at the bottom. Hearings with no `path_json` (legacy NULL rows) are
unknown, not zero hops, and are skipped.
- **Only nodes the last recompute didn't cover** (first advert newer
than it, or no hearing with a path) still use the old observer check.
- **`region_placement` on every node row** (list and detail) says which
rule placed it, so an operator can tell the four outcomes apart:

  ```json
  {"basis": "zero_hop", "regions": ["AUS"]}
  {"basis": "fewest_hops", "regions": ["HOU", "SAT"], "hops": 2}
  {"basis": "none", "hops": 5}
  {"basis": "observer_scan"}
  ```

It's a named struct (`NodeRegionPlacement`), documented as its own
OpenAPI component. It replaces the earlier `home_regions` field.

## How

- `node_home_regions.go`: a recomputer in the `bridge_recomputer.go`
shape (sync prewarm, ticker on `AnalyticsDefaultRecomputeInterval()`,
atomic snapshot, panics swallowed), started from `main.go` next to the
bridge one. It also recomputes once when `StartupLoadDone()` closes,
like `recomputeWhenLoaded` does for the analytics recomputers, because
the prewarm only sees the first chunk.
- Each recompute builds one snapshot, pubkey → `NodeRegionPlacement`,
from `nodeZeroHopRegionCounts` plus `nodeFewestHopsByRegion`. Hop counts
come from `path_json` without decoding (`pathHopCount`: hop hashes hold
no commas).
- `RegionNodePubkeys` loads the snapshot (atomic, no lock), skips placed
nodes in the observer scan, then adds the ones whose regions intersect
the request. The 30 s region cache is unchanged.
- `computeNodeHomeRegions()` keeps its plurality output for analytics;
only the counting moved. The #804 / #1025 tests are unchanged and pass.
- No new `map[string]interface{}`: the node rows set `region_placement`
to the struct, the OpenAPI component uses a small typed `openAPISchema`,
and the old `enrichNodeHomeRegions` helper is gone. Touched files are at
the upstream count.

## Performance

- Request path: one atomic load plus map lookups. `RegionNodePubkeys`
adds one pass over the snapshot per cache miss, so O(nodes), well under
the existing advert scan it sits next to. `BenchmarkRegionNodePubkeys`
stays ~19 ms.
- Recompute, off the request path, once per interval under `s.mu.RLock`:
`BenchmarkRecomputeNodeHomeRegionSets` times the whole function at 220k
adverts × 8 observations over 0-7 hop paths (one in eight zero-hop
direct) plus 1M channel messages: **~54 ms/op** on an Apple M5 (Docker,
linux/arm64), of which the fewest-hops pass
(`BenchmarkNodeFewestHopsByRegion`) is ~40 ms.
- That's after two small speedups in 3dc3cb7d: `nodeZeroHopRegionCounts`
walks `byPayloadType[ADVERT]` instead of every packet (same counts), and
the fewest-hops pass looks up a node's map once per advert instead of
once per observation. Before them the whole recompute was ~72 ms on the
same machine.

## Tests

- `TestRegionNodePubkeysUsesZeroHopHomeRegions`: a node whose relayed
advert reached SAT stays out of SAT; the boundary node is in both; the
never-zero-hop node is placed by the fallback.
- `TestBoundaryNodeStaysInEachHomeRegion`: the two-region case on its
own, so a later refactor can't quietly collapse it back to one winner.
- `TestNodeRegionPlacementsZeroHop`: the snapshot for zero-hop and
fallback nodes.
- `TestHandleNodesRegionPlacement`: `/api/nodes` returns `zero_hop`,
`none` and `observer_scan` placements for seeded nodes.
- `TestRegionFallbackUsesFewestHops`: fewest hops wins, ties keep both,
exactly the cap is placed, past it is in no region (and the observer
scan doesn't re-add it), a closer later advert counts.
- `TestRegionFallbackMaxHopsConfig`: the config accessor (unset or
negative keeps 3, 0 allowed, parsed from JSON) and the recompute
honoring it.
- `TestRegionFallbackIgnoresUnknownPaths`: an empty `path_json` doesn't
count as zero hops; `[]` still does.
- `TestRegionFilterObserverScanForNodesNewerThanRecompute`: the
observer-scan path for a node newer than the snapshot.
- `TestRegionFilterListingEveryRegionOmitsUnplacedNodes`: naming every
region explicitly still leaves out a node in no region.
- `TestNodeRegionsRecomputeWhenStartupLoadDone`: placements cover a
later chunk once the startup load is done.
- `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test
./...` pass (`CGO_ENABLED=1`). `gofmt` and `go vet` are clean.

No UI change; the API change is covered by the handler test above.

## Config

- `regionFallback.maxHops` (default 3), documented in
`config.example.json`.
- Per AGENTS.md rule 8, this should be exposed in the customizer in a
later milestone. Not done here.

## Not in this PR

- **Transport-direct zero-hop adverts aren't counted as zero-hop**
(pre-existing, `nodeZeroHopRegionCounts`): for
`ROUTE_TYPE_TRANSPORT_DIRECT`, `RawHex[2:4]` is the first transport-code
byte, not `path_len`. The fallback still places such nodes (their
`path_json` is `[]`), but as `fewest_hops`, not `zero_hop`. I'll send
that as its own PR since it changes #804 analytics too.
- **Memory window.** The answer only covers adverts still in the packet
store, so quiet nodes fall back. That's the long-tail point from the
thread. A `node_regions` table kept by the ingestor (the
`UpdateNodeDefaultScope` pattern) would remove the window; happy to do
that as a follow-up if you want it.
- **One-off long-range hearings.** A single zero-hop hearing during a
ducting event still adds that region. On our deployment we run the same
rule with a 20% share threshold per region (ignore a region holding
under a fifth of a node's direct hearings), which drops those. Easy to
add here if you'd like it.
- **Flood adverts heard with an empty path** are direct evidence too,
and much more common than zero-hop DIRECT adverts. Left out of the
zero-hop rule to keep one definition with #804; the fallback places them
at 0 hops.
- **Moving #804 / #1025 analytics to the set version**, as you
suggested, after this lands.

**Fallback numbers.** On our network only ~27% of nodes are ever heard
directly, so the fallback decides most of the filter. With "every region
that heard it", our San Antonio filter showed 241 nodes whose GPS puts
them outside SAT against 149 inside, and Austin picked up Louisiana and
Gulf Coast nodes heard 3-10 hops out. We've been running the
fewest-hops/cap-3 rule in production since 2026-10-14; nodes with GPS
outside the selected region went SAT 241 → 7, DFW 76 → 7, HOU 47 → 2,
AUS 226 → 48. About 240 nodes are in no region (they still show
unfiltered).

For context, on our network (Texas, ~1,500 nodes) the same idea with the
broader "heard directly" rule took San Antonio from 543 nodes to 482.
The 61 removed were all Austin-area nodes whose adverts get relayed
south, and only 3 nodes statewide are heard directly in two regions.
 2026-10-15 16:31:00 +02:00
Alex BandClaude Opus 5.5 2cdb18e1dd perf(store): the decoded-payload cache drops fields nothing reads (#2198) 
## Problem
`ParsedDecoded` caches each packet's decoded payload for as long as the
packet is stored, and the analytics passes fill it for every packet. It
kept fields that no server code reads:
- the ciphertext and MAC of encrypted payloads;
- advert signatures (and their validity flag);
- ISO copies of timestamps;
- raw path data;
- control-packet details.

Together those are about a third of the cached JSON.

## Change
- After decoding, the cache deletes those keys
(`decodedCacheDroppedKeys`).
- The live feed was the only place that sent the whole map.
`broadcastPayload` now passes the stored `DecodedJSON` through as a
`json.RawMessage` (after `json.Valid`), so clients still get every
field, byte for byte. Nothing in the server reads the broadcast payload
as a map; it is only serialised. This replaces the shallow copy of the
cache, so no JSON is decoded or map copied under the store's write lock
in either ingest path, and no nested map (such as `flags`) is shared
with the cache.
- `TestDecodedCacheDroppedKeysAreUnread` is a guard: it fails if any
server code starts naming a dropped key, so a future reader can't
silently get nothing.

## Measured
| | before | after |
|---|---|---|
| decoded cache's live heap (copy of a busy week) | 46 MB | 39.5 MB |

The saving is smaller than the JSON share suggests, because the
per-packet map overhead dominates. A typed struct would be the next
step, but that touches 27 call sites.

## Tests
- `TestParsedDecodedDropsUnreadKeys` and
`TestBroadcastPayloadKeepsEveryKey` (every key present in the broadcast,
and invalid JSON left out as before): red commit first.
- The guard test above.
- `go test ./cmd/server` passes. `map[string]interface{}` in
`cmd/server`: 933 on master, 932 here.

No config or UI changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 10:30:28 +02:00
Alex BandClaude Opus 5.5 92dd172f6e perf(store): bulk loaders reuse their row scan targets (#2201) 
## Problem
`Load`, `loadChunk` and `scanAndMergeChunk` declared their ~20
`sql.Null*` scan targets, and built the `Scan` argument slice, inside
the row loop. Each target's address goes into the `[]interface{}`, so
each one escaped and was heap-allocated again on every row, along with
the slice and its appends. On a week of a busy network that was the
biggest flat cost inside `loadChunk`, about 330 MB.

## Change
- The targets and the argument slice are declared once, before the loop.
- `Scan` overwrites every target on each row.
- Nothing else takes their addresses; the `null*` helpers copy their
argument.

## Measured
| | before | after |
|---|---|---|
| allocations per row, fixture (`Load` / `LoadChunked` / `loadChunk`),
on current master | 85.9 / 87.1 / 86.0 | 63.9 / 65.1 / 64.0 |
| allocated over startup (copy of a busy week) | 5.41 GB | 5.05 GB |
| server CPU over the run | 175.6 s | 172.4 s |

The per-row figures are from the test on current master. The startup
figures were measured before #2183 and #2185 landed; this change saves
the same ~22 allocations per row on top of them.

**Scale:** this is the cold-load hot path, once per observation row
(about 400k rows on that week). The change removes per-row work and adds
none; complexity is unchanged.

## Tests
- `TestBulkLoadersReuseScanTargets` sets an allocations-per-row bound
for each loader, between the before and after values. Red commit first,
then green.
- `go vet ./...` is clean and `go test ./...` passes in `cmd/server`;
`-race -count=2` passes on the loader and observer-string tests.
- Local SonarQube: quality gate passed, no issues on added lines.
- `map[string]interface{}` in `cmd/server`: 934 on master and here.

No config or UI changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 10:30:08 +02:00
efitenandClaude Opus 5.5 1ae4c74913 feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) (#2208) 
Fixes #661 (the narrowed scope from the 2026-10-14 comment).

## What changes

- `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and
`rx_airtime_pct_1h` per observer: the airtime share over the hour before
now, whatever `window` is, from the cumulative `tx_air_secs` /
`rx_air_secs` counters in `observer_metrics`.
- A counter that goes down marks a reboot; only the stretch after the
last reset counts.
- Under 15 minutes of samples (one sample, or a reboot late in the hour)
returns `null`.
  - Clamped to 0..100, rounded to 2 decimals.
- The response also carries `duty_cycle_warn_pct`, from the new config
key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit).
Added to `config.example.json`.
- RF Health tiles show `TX 1h: x%`; the observer list gets a sortable
numeric `TX 1h` column. Both use the shared `public/observer-airtime.js`
helper and flag values at or over the threshold with `.airtime-over`
(`var(--status-red-text)`).
- The handler now returns a named `MetricsSummaryResponse` struct
instead of `map[string]interface{}` (one occurrence fewer).

## Why the warning matters

Repeater and room-server firmware default to `airtime_factor = 1.0`
(`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26`
turns into a 50% TX budget per 1 h window. The device therefore does not
keep itself under the EU 10% limit; this view is where an operator can
see it.

## Cost

- Server: one extra query per summary request, `SELECT observer_id,
timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE
timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`,
then one O(rows) pass. On our instance that is 35 observers x about 12
samples (5 min interval).
- Observer list: one extra bulk call `summary?window=1h` per load
(client TTL 30 s, failure ignored), not a call per observer.

## Tests

- Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample,
span under 15 min, reboot leaving under 15 min, nil counters, clamp,
rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h
ignored inside a 24 h window, single-sample observer gets null),
`TestMetricsSummaryDutyCycleWarnPct` (default 10, config override,
`observers` is `[]` not `null`). Full `cmd/server` suite passes locally
(skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows
on clean master too).
- Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8
assertions, real file in `vm`), added to `test-all.sh`.
`test-observers-headings.js` updated for the new column. Full
`test-all.sh` passes locally.
- E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in
`deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no
`observer_metrics` table, so the summary is served through `page.route`
with known values for the fixture's own observer ids. It checks the
column header, over/under/unknown cells, sort order and the RF Health
tile. 4/4 locally. It caught a real bug during development:
`loadRFHealthData` declares a local `const window`, so the tile calls
the helper as the global `ObserverAirtime`.

## Validation

Deployed to a staging instance with live MQTT data (35 observers with
metrics) and checked in the browser:

- The observer list shows the column for 42 rows (33 with a value, 9
without enough samples). Sorting puts the highest share (5.3%) first.
- RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line
wrapped inside items on narrow tiles; the stats row now wraps between
items only.
- No observer was over 10% at the time, so the red state was checked by
rendering one cell client-side at 12.5% (computed colour `rgb(185, 28,
28)` in light mode), and by the E2E test.

## Not done

- The threshold is config-only; exposing it in the customizer is a
follow-up.
- No airtime comparison across observers (that is #671) and no
per-packet airtime (Relay Airtime Share covers it).
- Dark mode was not checked in the browser; `--status-red-text` is
defined for both themes.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-14 21:07:20 +02:00
Alex BandClaude Opus 5.5 b8e571c9c5 perf(store): build the relay-filtered candidate lists once (#2195) 
## Problem
`relayCandidates` runs once per hop of every observation the cold loader
resolves, and again from the direct-heard and neighbor analytics. Once
any listener-only observer was marked (#1290), every call built a
filtered copy of the candidate `nodeInfo` structs, even when none of the
candidates was a listener. On a week of a busy regional network that
accounted for 1.1 GB of the 5.4 GB allocated over one startup, the
biggest single source of startup churn.

## Change
`markNonRelay` now builds a map once, holding the filtered list for each
prefix whose candidates include a listener. A listener can only be in
the lists of its own prefixes and its full key, so only those are
filtered: at most 9 lists per listener, not every list in the prefix
map. `relayCandidates` returns either that list or the prefix map's own
list, without copying. Every caller only reads the slice (checked:
direct-heard, neighbor persist, node hop analytics and
`resolveWithContext`).

## Measured on a copy of that week
| | before | after |
|---|---|---|
| allocated over startup | 5.44 GB | 4.32 GB |
| server CPU over the run | 175.6 s | 169.1 s |
| peak RSS | 1074 MB | 1077 MB (unchanged; the GC limit sets it) |

## Cost per rebuild against cost per lookup
`markNonRelay` now does once per node cache rebuild (as often as every
30 s, `nodeCacheTTL`) the filtering that master did on every
`relayCandidates` call. 5,000 repeaters and 3 listeners, median of 6
runs (`BenchmarkMarkNonRelay5000`, `BenchmarkRelayCandidates256`):

| | master | first version of this PR | this PR now |
|---|---|---|---|
| `markNonRelay`, once per rebuild | 0.59 µs, 320 B, 3 allocs | 14.8 ms,
6.57 MB, 36,285 allocs | 20 µs, 20.6 KB, 33 allocs |
| one `relayCandidates` lookup | ~7.8 µs, 6.7 KB | ~30 ns, 0 B | ~30 ns,
0 B |
| 256 lookups (one per 2-character hop) | 2.0 ms, 1.71 MB, 1,453 allocs
| 7.7 µs, 0 B | 7.7 µs, 0 B |

The extra ~20 µs per rebuild is paid back after about 3 lookups. Every
ingested observation looks up each hop of its path, so a running server
makes far more than that between rebuilds, and a cold load makes
millions (the 1.1 GB above).

## Tests
- `TestRelayCandidatesDoesNotAllocate`: zero allocations per call, and
listeners still filtered.
- `TestRelayCandidatesAfterSecondMark`.
- `TestMarkNonRelayVisitsOnlyListenerPrefixes`: at most 80 allocations
for 3 listeners among 5,000 repeaters (36,285 before the fix). It has
its own red commit.
- `BenchmarkMarkNonRelay5000` and `BenchmarkRelayCandidates256` give the
table above.
- `TestMarkNonRelayMatchesFilteringEachList`: every prefix of the
5,000-repeater map matches filtering its list on the spot.
- Red commit first, then green.
- `go test ./cmd/server` passes.

No config or UI changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 15:42:18 +02:00
Alex BandClaude Opus 5.5 8d2ae405e8 perf(store): cache an observation's parsed time in 8 bytes (#2196) 
Refs #2148.

## Problem
`StoreObs` is the store's most numerous struct: 400k of them on a week
of a busy regional network. Its lazy timestamp cache (#1481) is a
`time.Time`, a `sync.Once` and a `bool`, together 40 bytes. On current
master that makes the struct 200 bytes, which puts every observation in
the 208-byte size class.

## Change
The cache is now a single `atomic.Int64`:
- `0` before the first call;
- the parsed time as UTC nanoseconds after it;
- or one of two sentinels: one for empty or unparseable strings, and one
for times it can't hold exactly (a non-UTC offset, or outside
`UnixNano`'s range). Those are re-parsed on each call so their location
is preserved.

Concurrent first calls may both parse; they store the same value.
`StoreObs` is now 160 bytes, in the 160-byte size class: 48 bytes × 400k
observations ≈ **19 MB** less heap. The first parse costs the same;
cached reads now build the `time.Time` from an int.

## Tests
- `TestStoreObsFitsIn160Bytes`: red commit first.
- `TestStoreObsParsedTimeMatchesParse`: Z, nanosecond, offset, SQL,
empty, garbage and epoch inputs give the same time and location as
`time.Parse`, on the first and the cached call.
- `TestStoreObsParsedTimeConcurrentFill`.
- The existing `storeobs_parsedtime_test.go` still passes.
- `go test ./cmd/server` passes, and `-race` passes on the ParsedTime,
retransmission and observer analytics tests.

## This PR or #2149
#2149 fixes #2148 by reordering these same three fields. The two
conflict, so here is how they compare. Sizes are
`unsafe.Sizeof(StoreObs{})` on each branch, with Go's heap size class
beside them:

| | struct | heap per observation | saved per observation | saved at
400k observations |
|---|---|---|---|---|
| master | 200 B | 208 B | – | – |
| #2149 (field reorder) | 192 B | 192 B | 16 B | ~6.4 MB |
| this PR (one 8-byte field) | 160 B | 160 B | 48 B | ~19 MB |

- **#2149:** a pure field reorder with no change in behaviour. It is the
smallest, safest change and the easiest to review. It saves a third of
what this PR does.
- **This PR:** three times the saving, but more code. It adds two
sentinel values, and re-parses on each call any timestamp with a non-UTC
offset. The store writes UTC (`Z`), so that path is rare. It is tested
against `time.Parse` on edge cases, concurrently, and under `-race`.

Options:
1. **Take this PR.** #2149 then has nothing left to reorder.
2. **Take #2149 now, and this PR after it** as a follow-up for the
further 32 bytes per observation. I'll rebase onto #2149 if it lands
first.
3. **Take only #2149** if the smaller, simpler change is preferred.

No config or UI changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 09:01:57 +02:00
nullrouten0andClaude Opus 5 478f74a452 fix(config): write the geo filter through a symlinked config.json (#2137) 
Found while reworking #2125, unrelated to that PR's change.

In the Docker image the entrypoint symlinks `/app/config.json` to the
bind-mounted `/app/data/config.json`. `SaveGeoFilter` takes the first
path `os.Stat` resolves, so `configPath` stays the symlink, and the
tmp+rename then **replaces the link with a regular file inside the
container**:

- the mounted `config.json` keeps the old `geo_filter`, so the filter
the operator just saved appears nowhere on the host;
- server and ingestor read the detached file from then on, so the UI
keeps showing the new filter and nothing looks wrong;
- the next container start reverts it: the entrypoint's `ln -sf` puts
the link back, and re-creating the container drops the file anyway.

**Fix:** resolve a symlink at `configPath` before the write, so the save
lands in the file it points at and the link stays a link. Only a regular
file is followed — a link to a device, fifo or directory is left alone
and handled as before.

This is deliberately the opposite of `writeStatsAtomic` (refs #1170),
where replacing a destination symlink is the point because the default
stats path can be world-writable `/tmp`. `config.json` is different: the
link is one the image itself creates, and the directory is not
world-writable. Happy to narrow it further (e.g. only follow a link that
stays inside `configDir`) if you'd rather keep the two paths consistent.

**Verification:** `TestSaveGeoFilterSymlinkedConfig` fails on master —

```
--- FAIL: TestSaveGeoFilterSymlinkedConfig
    config_symlink_test.go:47: .../config.json is no longer a symlink (mode -rw-------) — the rename replaced it
    config_symlink_test.go:60: geo_filter missing from .../data/config.json: {"port":3000,"_comment":"keep me"}
```

— and passes here. `TestSaveGeoFilterKeepsOwner` covers keeping the
file's owner, and `TestSaveGeoFilterFifoSymlinkTarget` checks that only
regular files are followed. `go test ./...` in `cmd/server` passes.

**Single-file mounts:** when `config.json` is mounted as a single file,
the save now fails with 500 (`EBUSY`) instead of writing a copy that the
next start discards.

Our own instance has no geo filter configured, which is why this went
unnoticed there; its `/app/config.json` is still a symlink.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-10-13 08:44:49 +02:00
Alex BandClaude Opus 5.5 e53c055d6e perf(store): share one copy of each observer string across observations (#2183) 
Part of #1870: this interns the three fields the issue names
(`ObserverID`, `ObserverName`, `ObserverIATA`), plus `Direction`.

## Problem
Each `StoreObs` kept the row scan's own copy of its observer ID (64 hex
characters for most observers), observer name, IATA code and direction.
A busy regional network's 7-day window holds about 400k observations
from a few hundred observers, so a few hundred distinct values were
stored hundreds of thousands of times.

## Change
- `internStr` returns the canonical copy of each value from a map
guarded by an `RWMutex`, so every observation shares one copy.
- The copies are held for the life of the process. That is what makes
the sharing survive garbage collections. An earlier version of this PR
used `unique.Make`, which holds its copies weakly, so a value interned
again after a GC came back as a second copy (thanks to the review for
catching that).
- Observer strings number in the hundreds. The map is capped at 65,536
entries as a guard (AGENTS.md: no unbounded structures); past the cap a
string is returned as it came, so nothing breaks.
- A hit takes only the read lock and allocates nothing. A new value is
cloned, so it doesn't pin a larger buffer it may be a slice of.
- All five places that build a `StoreObs` use it: `Load`, `loadChunk`,
`scanAndMergeChunk`, and both ingest paths.
- The ingest paths intern while scanning rows, **before** taking `s.mu`,
so interning adds nothing to the time API readers wait on the store.
- `cmd/server/go.mod` stays at `go 1.22`. The earlier 1.27 bump was only
there for `unique`, and is reverted here; the module upgrade belongs in
its own PR.

## Scaling
`BenchmarkInternStr` (a hit, the steady state):

| | alone | 12 cores (per call) | allocs |
|---|---|---|---|
| plain `Mutex` map (the first version of this PR) | 20 ns | 57 ns | 0 |
| `unique` (the second version, withdrawn) | 48 ns | 8 ns | 0 |
| **`RWMutex` map (this PR)** | 17 ns | 26 ns | 0 |

## Measured on a copy of that window (`GOMEMLIMIT=750MiB`)
The same database and harness, one run each, the `unique` version
against this one:

| | `unique` version | this PR |
|---|---|---|
| live heap at the last GC after load | 581 MB | **535 MB** |
| string bytes allocated by row scans still live (`bytealg.MakeNoZero`)
| 106 MB | **80 MB** |
| peak RSS | 777 MB | 781 MB |
| load to ready | 15 s | 15 s |

Peak RSS is set by the load itself, not by these strings. Against
master, the first measurement of this PR found the strings retained from
row scans went from 84 MB to 58 MB.

An earlier version of this PR (the `unique` one) also ran on a staging
instance (live MQTT ingest, `GOMEMLIMIT=1GiB`) through five back-to-back
restarts and a two-hour soak, with no OOM kills and every request
succeeding. This version has not had a staging soak yet.

## Tests
- `observer_strings_intern_test.go`:
- one copy of each value across 50 observations, through `Load`,
`LoadChunked` and `loadChunk`;
- **a value interned again after `runtime.GC()` is the same copy** (new:
red against the `unique` version, green now);
  - 16 goroutines interning concurrently get the same copy;
  - a hit allocates nothing;
  - `BenchmarkInternStr` (serial and parallel).
- Red commits first, then green.
- `cd cmd/server && go test ./...` passes at `go 1.22`, and `-race`
passes on the intern tests.

Backend only: no UI, API shape or config change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 08:37:54 +02:00
SaarMesh-BotSaarMesh-BotClaude
 		b43fd4a470 feat(#2107): backfill unresolved channel/scope rows when the key set grows (#2133) 
Part 1 of #2107, scoped as discussed there: the backfill on its own, no
catalogue promotion.

## Problem

Ingest decides a GRP_TXT's channel and a transport packet's region scope
once, at insert time. A row stored before its key was known keeps
`channel_hash = 'enc_XX'` / `scope_name = ''` forever, even after the
key is added — via `hashChannels` plus a restart, or at runtime when the
`autoRegionKeys` tier picks up a newly declared region. On
live.saarmesh.de we have been closing that gap with an out-of-tree
script that rewrites the DB and restarts the container; this moves it
into the ingestor and removes the restart.

## What this does

**Ingestor** — `cmd/ingestor/key_backfill.go`

- Runs after the write path is ready, and again whenever the key set
grows (kicked after every `autoRegionKeys` refresh; a no-op when nothing
is new).
- Touches **only unresolved rows**: `enc_*` GRP_TXT and transport rows
with `scope_name = ''`. Every `UPDATE` carries the unresolved value in
its `WHERE`, so a row resolved meanwhile by live ingest keeps its value.
A decrypted channel or named scope is never rewritten.
- **Same decision as ingest**: `DecodePacket` with the full channel key
map → `PayloadJSON`, and `regionKeySnapshot.match()` with the full
snapshot — including abstaining on an ambiguous collision (#1609). The
scope-match counters are not touched; they describe live traffic.
- **Bounded**: each batch covers a fixed window of 5000 transmission
ids. Unresolved rows are sparse (hundreds in a million), and the
ingestor has a single SQLite connection, so a "next N matching rows"
query would scan most of the table in one statement while ingest waits.
A fixed id window keeps every read a short primary-key range scan. The
writer is yielded between windows.
- **Resumable and transactional**: one transaction per window holds the
rewrites, their change-feed rows and the advanced cursor
(`key_backfill_state`). A cancelled or crashed run resumes at its last
commit; nothing is applied twice.
- **Key-set growth**: a run is tied to the fingerprint of the key set it
started with. If the set changes mid-run, the run restarts from the
beginning, so a key added halfway still sees all of history. Each key
gets one complete pass (`key_backfill_done`); a restart with an
unchanged key set does no work. A key that leaves the set forgets its
pass and is replayed if it returns.
- **Derived field**: a named ADVERT fills `nodes.default_scope` /
`inactive_nodes.default_scope` only where it is still empty. A known
`default_scope` is newer evidence than a backfilled advert and is never
overwritten.

**Change feed** — `internal/dbschema/key_backfill.go`,
`cmd/server/tx_rewrite_feed.go`

- Each rewrite appends a row to `tx_rewrite_feed` in the same
transaction as the rewrite.
- The server tails it exactly like `advert_route_evidence`: cursor
captured before `Load`, absent table re-probed (not required by
`AssertReady`, so an older ingestor simply means no feed), polled from
`IngestNewObservations`. It refreshes `DecodedJSON` / `ScopeName` of
loaded transmissions, resets the `ParsedDecoded` cache, recharges
tracked bytes, and invalidates the channel caches. **No restart
needed.**
- The feed carries only the transmission id; current values are read
from `transmissions`, so replaying a row is harmless. Rows older than
24h are pruned.

## Tests

Ingestor (`key_backfill_test.go`) — fixtures are real wire frames:
GRP_TXT encrypted AES-128-ECB with the firmware MAC, transport frames
with the real region HMAC.

- decrypts only unresolved rows; already-decrypted and unknown-key rows
untouched
- backfilled `decoded_json` / `channel_hash` are byte-identical to what
`BuildPacketData` stores at ingest
- unique match named; a constructed two-name collision on the same
payload stays unmatched; named and non-transport rows untouched
- ingest scope-match counters unchanged
- ADVERT fills only an empty `default_scope`
- idempotent: second pass writes nothing, done keys recorded, state
cleared
- cancel between commits → resume from the committed cursor, every row
exactly once, in order
- stale run fingerprint (key added mid-run) → restart reaches rows below
the old cursor
- a later key only adds its own rows; a removed key is replayed over the
gap when it returns
- a row resolved between decide and commit keeps its value and gets no
feed entry
- channel hash byte derivation matches the builtin Public key (0x11)
- old feed rows pruned

Server (`tx_rewrite_feed_test.go`)

- rewrite reaches the loaded store through the normal
`IngestNewObservations` poll, including a previously parsed
`ParsedDecoded` (verified to fail without the `sync.Once` reset)
- cursor starts at the load watermark
- absent feed is a no-op and is picked up once created, without restart
- failed read keeps the cursor and retries
- feed row for an unloaded transmission advances the cursor and is
ignored

`gofmt` clean, `go vet` clean, no `go.mod` / `go.sum` changes.

## Validation against production data

Ran the backfill on a snapshot of live.saarmesh.de taken **before** last
night's run of our out-of-tree script (13,100 unresolved rows, 164
channel keys, 24 region keys incl. 2 derived), with the current config.
It rewrote **the same 31 transmissions with the same channel names** as
the script did on production (compared by id), named 0 scopes (script:
0), and a second pass wrote nothing.

Browser validation (AGENTS rule 2): this changes no UI. The visible
effect — a decrypted message or named scope appearing without a restart
— is covered by the server feed tests; I have not yet run it against a
deployed instance.

## Notes for review

- Window size (5000 ids), inter-batch yield (10 ms) and feed retention
(24h) are constants for now; per rule 8 they can move to
config/customizer later if anyone needs to tune them.
- `client_rx_observations.scope_name` is not backfilled here; happy to
add it in a follow-up if wanted.
- The orphaned "scope-repair" mention in `region_keys.go`'s
`scopeMatch.Candidates` comment predates this PR; I left it alone.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BKaogNXgy8dEQruqzR5foP

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-13 08:35:20 +02:00
efitenandClaude Opus 5.5 abc6ab9b59 fix(analytics): apply nodeBlacklist and hiddenNamePrefixes to Route Patterns (#2184) (#2189) 
Fixes #2184.

## Problem

`nodeBlacklist` and `hiddenNamePrefixes` did not apply to Route
Patterns:

- **`subpaths` and `subpaths-bulk`:** `filterBlacklistedFromSubpaths`
asserted `data["subpaths"].([]interface{})`, while the store builds
`[]map[string]interface{}`, so it returned its input unchanged. It then
read `hops` as a list, which is an `int` (the run length). It also ran
only when `nodeBlacklist` was set, so `hiddenNamePrefixes` alone never
filtered anything.
- **`subpath-detail`:** the check compared each raw hop (a prefix) with
full keys, so it matched only a client that sent a full key.
- **Parent paths** in the detail response were not filtered.

## Change

All in `cmd/server/routes.go`; `store.go` is untouched, so this doesn't
overlap #2182.

- **Lists:** an entry is dropped when a name in its resolved `path` is
the name of a blacklisted node (names looked up in the store's node
cache) or has a hidden prefix. A subpath carries names and prefixes, not
keys, and the store picks a node per prefix from context, so
re-resolving a prefix in the handler would not be guaranteed to land on
the same node. Matching the names the reader would see avoids that.
Trade-off: another node with exactly the same name as a blacklisted one
is hidden too.
- **Detail:** 404 when a hop resolved to a blacklisted node (by the
resolved key in `nodes[].pubkey`, so a node without a name is caught and
one that only shares its name is not) or to a hidden-prefix name; parent
paths that show one are dropped (by name, like the lists). The existing
raw-hop check stays.
- The filter runs when either `nodeBlacklist` or `hiddenNamePrefixes` is
set.
- `dropHiddenPaths` replaces `filterBlacklistedFromSubpaths` and returns
a copy: the input can be the store's cached response, which the old
function mutated in place.
- `totalPaths` and `pct` are unchanged, so percentages stay relative to
every stored path, the filtered ones included.

**Perf:** per request, one pass over the node cache (about 2k nodes on
the deployment below) to collect blacklisted names, then at most 8
lookups per entry on at most `limit` entries. It runs outside the store
lock and only when a filter is configured. `map[string]interface{}`
count in `routes.go`: 3 removed, 3 added.

## Tests

- `subpaths_blacklist_2184_test.go`: on the seeded path `["aa","bb"]`
(first hop resolves to TestRepeater), for `subpaths`, `subpaths-bulk`
and `subpath-detail`:
  - unfiltered, the repeater is listed (so the test is not vacuous);
- a blacklisted key, also in capitals, removes it and the detail returns
404;
  - a hidden name prefix removes it;
  - a different key with the same 1-byte prefix leaves it listed;
- clearing the blacklist afterwards shows it again (the cached response
was not changed).
- subpath-detail: a blacklisted node without a name returns 404, a
different node that only shares its name does not hide the detail.
- Red commits first (`0ebee485`, `4acfb70a`, failing on assertions),
then green.
- `cd cmd/server && go test ./...` passes
(`TestSaveGeoFilterPreservesFileMode` skipped, a Windows-only failure on
master too); `go test -race -run 2184` passes; `go vet` clean.

## Validation on a live-fed staging instance

`nodeBlacklist` temporarily set to the key of the node in the top Route
Patterns entry (5848 occurrences), then restored:

| | blacklist set | restored |
|---|---|---|
| `subpaths?limit=200` entries showing the node | 0 (185 entries left) |
15 (200 entries) |
| `subpaths-bulk` groups 2-2 / 3-3 / 5-8 showing it | 0 / 0 / 0 | |
| `subpath-detail` on a path through it | 404 | |
| `subpath-detail` on a path without it | 200 | |

## Not done

- Other analytics responses that carry node names (for example topology)
were not audited for the same gap.
- The UI still shows `pct` relative to all paths; whether a filtered
list should renormalise is left open.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 08:35:16 +02:00
efitenandClaude Opus 5.5 60af9bd469 fix(hops): narrow only the directly heard hop to the observer region (#2098) (#2188) 
Fixes #2098.

## Problem

`HopResolver.resolve()` narrowed every hop of a path to the observer's
IATA region (300 km around the code's coordinates). Only the last hop
was heard by the observer directly; every earlier hop can be anywhere
the mesh reaches, so the region says nothing about it.

This stayed invisible on deployments whose observer codes were missing
from `/api/iata-coords`: there `nodeInRegion()` returns `null` and
nothing is filtered. A Belgian deployment's 42 observers use 10 codes
and none was in the table. Filling the table alone would have switched
on a filter that is wrong for every hop but the last.

## Why only the last hop

#2098 lists three options and argues against simply repopulating the
table, because the filter it would switch on is not sound. This PR does
fill the table, but only after limiting the filter to the one hop where
it is sound:

- In a MeshCore path the last hop is the repeater the observer received
the packet from, so it is within radio range of the observer. Measured
below: p99 30 km for directly heard multi-byte hops.
- An earlier hop says nothing about the observer's position. On the
Belgian deployment nearly every observer hears nearly every flood, so a
region filter on earlier hops would drop the right candidate whenever
the packet started outside the observer's region.

This was decided while preparing the PR, not on the issue thread, so the
reasoning is recorded here.

## Change

- **`public/hop-resolver.js`:** `resolve()` takes an optional
`opts.directHops`, the hops the observer heard directly. Without it, the
input is one path and its last hop is the direct one. The region check
runs only on direct hops. An earlier hop gets `regional: false` and,
with an observer, `regionSkipped: true`. `globalFallback` now means
checked against the region with no candidate in it, so it is only set on
a direct hop.
- **`public/hop-display.js`:** the badge counts every candidate for
`globalFallback` or `regionSkipped`. The dashed underline and the
popover's "(global fallback)" stay for `globalFallback` only, so an
earlier hop is not labelled as matched outside the region when no region
check ran on it.
- **`public/packets.js`:** `resolveHopsForPackets()` resolves the last
hop of each path apart from the earlier hops. `hopCacheKey(h,
observerId, direct)` keeps the two in separate cache entries, because
the same prefix can be the last hop in one packet and a relayed hop in
another. `renderPath()` reads the direct entry for the path's last hop,
and the detail-pane resolve writes it. The region-narrowed entry never
goes to the bare (observer-less) key, in the list or in the detail pane.
The detail pane resolves and caches the shown observation's path with
that observation's observer (`effectivePkt.observer_id`), the observer
`renderPath` already used; before, it resolved with the packet row's
observer, so when the first observation came from another observer the
last hop missed its direct entry and showed unfiltered. The route button
uses the same observer.
- **`cmd/server/routes.go`:** `iataCoords` gains 104 entries from
OurAirports `airports.csv` (public domain): every IATA code in Belgium,
the Netherlands and Luxembourg, and the airports with scheduled service
in Germany and France. AMS, CDG, FRA and MUC were already present and
keep their values.

## Measured before building

On 5000 packets from the Belgian deployment (3661 with a path):

- 1-byte last hops: 1212, all ambiguous, 1204 of them without a server
`resolved_path`, so they go through the client resolver.
- Distance from the observer to a directly heard node, on 1892 unique
multi-byte last hops: p95 25 km, p99 30 km, max 82 km.
- With this change, on 906 1-byte last hops from observers whose code is
now in the table: the badge counts 6.1 candidates on average instead of
6.6. The displayed name changes for 52, all from observers without their
own lat/lon, where distance to the region now orders the candidates
instead of list order. For observers with a position, no displayed name
changes, because the #2097 observer anchor already picked the nearest
candidate.

## Tests

- `tests/unit/test-issue-2098-last-hop-region.js` (new, in
`test-all.sh`, 25 assertions):
- HopResolver: the last hop of a path is narrowed and an earlier hop is
not; an earlier hop keeps every candidate and reports `regionSkipped`,
not `globalFallback`; a direct hop with no candidate in region reports
`globalFallback`; `directHops` overrides the position; a one-hop path is
narrowed.
- HopDisplay (real file): an earlier hop keeps a badge with every
candidate but no fallback class or popover text; a direct hop outside
the region keeps both.
- packets.js (real file, real resolver and display): with one observer,
a prefix that is the last hop of one packet and an earlier hop of
another renders narrowed in the first and with a badge in the second;
after the detail pane resolves a path, the observer-less render does not
show the narrowed last hop; when the shown observation comes from
another observer than the packet row, its last hop is narrowed to the
region of the observer that heard it.
- `TestIataCoordsIncludesBelgianObserverCodes`: ANR, BRU, GNE, KJK, LGG,
OBL, OST and MST with their OurAirports coordinates.
- Red commits first (`26000a22`, `bb91ff9d`, `189f55d7`, failing on
assertions), then green. Mutation check on the packets.js tests:
inverting `isDirect` fails 3 assertions, swapping the `directHops`
argument fails 4.
- `test-issue-2097-hop-ambiguity-badge.js`: two structural regexes now
allow the new `direct` argument; every assertion still holds.
- All 198 `node tests/unit/*` suites in `test-all.sh` pass. `cd
cmd/server && go test ./...` passes (with
`TestSaveGeoFilterPreservesFileMode` skipped, a Windows-only failure on
master too); `go vet` clean.

## Browser validation

Staging (this branch on top of the deployment's current build), headless
Chrome on `#/packets/<hash>` for packets whose path ends in a 1-byte
hop: the last hop's popover lists only candidates within the region with
their distances (for example `7E`: 3 candidates at 46, 207 and 296 km),
earlier hops list every candidate (`1D`: 13). The same prefix `7E`
showed 3 in region as the last hop of one path and 4 as a relayed hop of
another. No page errors. `/api/iata-coords` returns 158 codes.

## Browser validation of the follow-up

Staging again (this branch at `4e129b49` on top of the deployment's
build), headless Chrome:

- Packets list, 41 paths of 2+ hops: the global-fallback class appears
on 2 last hops and on 0 earlier hops.
- Detail pane, 5 packets whose path ends in a 1-byte hop (1 to 13 hops
shown): no earlier hop carries the class or "(global fallback)" in its
popover, and every ambiguous earlier hop keeps its badge. Every last hop
was checked against the region: 4 show "N candidates in region", 1 shows
"(global fallback)" with the class.
- Before the observer fix, 2 of 5 detail panes showed a last hop with
neither label, because the shown observation came from another observer
than the packet row (`b66a8f4c`: row from one observer with 13 hops,
shown observation from another with 9).
- No page errors.

## Not done

- The 300 km radius is unchanged. The measured direct-hearing distances
suggest it is wide, but a smaller radius would also have to hold for
less dense networks, and I have no data for those.
- `live.js` passes no observer, so the region filter does not run there,
before or after this change. `map.js` passes the observer of the chosen
path in path order, so once the resolver knows the observer regions it
narrows that path's last hop only.
- The "Cross-observer aggregate" line in the detail pane renders the
longest path with the packet's own observer, so its last hop is narrowed
to that observer's region even if another observer reported that path.
The per-observer keying there predates this PR (#2097).
- Two of the Belgian deployment's codes, HEP and NRW, are not airport
codes and are not in OurAirports; those observers stay without a region
filter.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 15:24:13 +02:00
Alex BandClaude Opus 5.5 38ed9d6ce6 perf(store): count path hops without decoding the JSON (#2187) 
## Problem
`pathLen` is how `pickBestObservation` chooses each transmission's best
(longest-path) observation, so it runs for every observation on every
load and ingest, and again from the resolved-path index. It decoded the
path JSON into a `[]interface{}` only to take its length.

## Change
- `countStringArray` counts the usual array of plain hex strings in
place, with no allocation (helpers `skipJSONSpace` and
`skipPlainJSONString`).
- Anything else falls back to the decoder exactly as before: escapes,
non-string elements, `null`, or malformed input. So the result is always
what decoding gives, including 0 for invalid JSON.

## Perf justification (AGENTS.md rule 0)
- **Complexity.** O(len(path_json)), a single pass with no allocation.
The old version was also linear but went through reflection and
allocated per element.
- **Scale.** It runs once per observation on every load and ingest (400k
observations in a busy regional network's 7-day window), plus each time
the best observation is re-picked.

Throwaway benchmark (not committed), current master vs this branch:

| `path_json` | master | this PR |
|---|---|---|
| `[]` | 215 ns, 32 B, 2 allocs | 4 ns, 0 allocs |
| 6 hops, 1-byte | 1825 ns, 488 B, 18 allocs | 27 ns, 0 allocs |
| 10 hops, 3-byte | 2815 ns, 936 B, 27 allocs | 72 ns, 0 allocs |

On a copy of that 7-day window:

| | master | this PR |
|---|---|---|
| allocated over startup (the whole server) | 5.41 GB | 5.10 GB |
| server CPU over the run | 175.6 s | 169.2 s |

`pathLen` accounted for about 275 MB of the startup allocations.

## Tests
- `TestPathLenDoesNotAllocate`: zero allocations for typical paths. Red
commit first (2–11 allocations), then green.
- `TestPathLenMatchesDecoding` compares the result with `json.Unmarshal`
into `[]interface{}` on 27 edge cases and 20,000 random inputs built
from JSON punctuation, escapes, `null`, objects and numbers.
- `cd cmd/server && go test ./...` passes.
- SonarQube: quality gate passed, no issues on added lines.

Backend only: no UI, API shape or config change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 12:45:41 +02:00
Alex BandClaude Opus 5.5 a48dc32b8a perf(store): build hop context without a map or lowered copies (#2186) 
## Problem
`buildHopContextPubkeys` (#1197) runs for every transmission in each
topology and distance pass, inside `buildAggregateHopContextPubkeys` for
every subpath query, and for each matched packet in subpath detail. On
every call it:
- made a de-dup map sized for 16 keys, to de-duplicate what's usually
2–5 keys;
- lowered every hop into a new string just to look it up in the prefix
map. Hops are upper-case hex on live data, so `strings.ToLower` always
allocates;
- lowered the observer ID twice, once to check it's hex and once to add
it.

## Change
- Duplicates are found by scanning the result slice.
- Hops are lowered into a 64-byte stack buffer for the prefix-map
lookup. `m[string(buf)]` doesn't allocate, and non-ASCII input falls
back to `strings.ToLower`, so lookups are unchanged.
- The observer ID is checked as hex in either case (`isHexAnyCase`) and
lowered once.
- The result is the same keys in the same order: sender, observer, then
unambiguous hop anchors.

## Perf justification (AGENTS.md rule 0)
- **Complexity.** The de-dup scan is O(k²) for the k keys of one
transmission. k is at most 2 + path hops; MeshCore paths are under 64
hops and real values are 2–10. It replaces a map allocated on every
call, and nothing grows with the number of transmissions or nodes.
- **Scale.** It runs for every stored transmission in each analytics
pass (topology and distance) and per request in subpath analytics, so
the saving recurs on every recompute, not just at startup.

Benchmarks, current master vs this branch, same machine:

| | master | this PR |
|---|---|---|
| `BenchmarkBuildHopContextPubkeys` | 830 ns, 1192 B, 4 allocs | 507 ns,
128 B, 1 alloc |
| `BenchmarkBuildAggregateHopContextPubkeys` | 5.6 ms, 6.65 MB, 20.4k
allocs | 3.7 ms, 1.33 MB, 5.3k allocs |

On a copy of a busy regional network's 7-day window (64k transmissions):

| | master | this PR |
|---|---|---|
| this function's allocations over startup and the first analytics pass
| 317 MB | 80 MB |
| server CPU over the same run | 175.6 s | 165.6 s |

## Tests
- `TestBuildHopContextPubkeysAllocatesOnlyItsResult` uses a fixture
cased like live data (upper-case hops and observer ID): at most 2
allocations, the result and the lowered observer ID. Red commit first (9
allocations), then green.
- `TestBuildHopContextPubkeysOrderAndDedup` pins the same keys in the
same order.
- The existing hop-context and disambiguation tests (#1198, #1202) pass.
- `cd cmd/server && go test ./...` passes.
- SonarQube: quality gate passed, no issues on added lines.

Backend only: no UI, API shape or config change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 12:42:17 +02:00
Alex BandClaude Opus 5.5 8d39a9d465 perf(store): count subpaths per request instead of indexing every run (#2182) 
## Problem
`spTxIndex` kept every 2–8 hop run of every stored path as a map key,
with one transmission pointer per occurrence. On a busy regional
network's 7-day window (59k transmissions) that's about 0.9–1.07 M keys
and 130–160 MB of live heap, roughly a quarter of the store. It was also
rebuilt at the end of every startup, which is when memory peaks.

Two smaller issues came with it:
- Eviction never decremented `spTotalPaths`, so the percentages were
divided by paths that were no longer stored.
- Route Patterns took 1.2–3.6 s uncached, because the fast path still
split, resolved and re-joined about a million raw keys per request.

## Change
- **Count on request.** Subpaths are counted when the analytics are
asked for, from the stored paths. Results stay cached for `rfCacheTTL`,
as before.
- **Pointer-free tallies.** `tallyNamedSubpaths` counts runs in one
pointer-free `map[uint64]subpathTally`, keyed by a `maphash` of the
resolved name built in a reused buffer. The GC doesn't scan the map, and
counting allocates nothing but the map.
- **Names only for the top entries.** Each tally records the first
transmission and offset where its run appeared, so names and raw hops
are built only for each group's top entries. `topSubpathTallies` picks
those with a `container/heap` min-heap of `limit` entries instead of
sorting every name, and the existing `rankSubpaths` formats the
response, so no new response maps are added.
- **Hash collisions.** If two names shared a hash their counts would
merge; the odds are about 1 in 10⁷ for a few million names.
- **Subpath detail.** `GetSubpathDetail` finds the run with
`txsWithSubpath`, a scan of the stored paths.
- **Counts unchanged:** runs of 2–8 hops, counted once per occurrence,
case-folded. The tests pin this.
- **Totals.** `spTotalPaths` stays, and eviction now decrements it.
- **Memory estimate.** The O(path²) subpath term is dropped from the
estimate.

## Perf justification (AGENTS.md rule 0)
- **Complexity.** A cold Route Patterns request is O(Σ runs), about 1.5
M for this window: one pass over the stored paths, with one map lookup
per run against a reused buffer. Ranking is O(tallies · log limit).
Results are cached for `rfCacheTTL`, so this runs at most once per TTL
per group set.
- **Subpath detail** is O(stored hops), one pass with no allocation per
packet. It's an on-click endpoint, not a hot path.
- **Ingest and eviction** no longer touch a subpath index at all.
Removal used to be a linear search of every key's transmission list.

**Measured on a copy of that window**, upstream master vs this branch,
same queries:

| | master | this PR |
|---|---|---|
| peak RSS (`GOMEMLIMIT=750MiB`) | 948 MB | 767 MB |
| in-use heap after load (750MiB) | 592 MB | 456 MB |
| `/api/analytics/subpaths` cold (750MiB) | 3.56 s | 0.64 s |
| `/api/analytics/subpaths-bulk` cold (750MiB) | 3.33 s | 0.70 s |
| `/api/analytics/subpaths` cold (1GiB) | 1.18 s | 0.61 s |
| `/api/analytics/subpath-detail` | 0.6 ms | 8 ms |
| cached responses | 0.6 ms | 0.5 ms |

At 750 MiB, master's request time is mostly GC pressure from the index.

On the production-shaped fixture that `subpath_index_memory_test.go`
builds (41k transmissions):

| | master | this PR |
|---|---|---|
| retained by the store for subpaths | 197 MB | 0 MB |
| allocated per bulk request | 336 MB | 111 MB |

This has also run on a staging instance with live MQTT ingest for about
nine hours, with Route Patterns answering in about 1 s.

## Tests
- `subpaths_on_demand_test.go`:
  - per-occurrence, case-folded counts;
  - the 8-hop cap;
  - bulk matches single;
  - detail counts, including repeats, case and the length limits;
  - eviction drops evicted paths from the totals.
- `TestSubpathIndexRetainedHeap`: the bound is now 5 MB (it was 225 MB).
- `TestSubpathRequestAllocations`: 150 MB per request.
- Each change has its red commit first, failing on assertions.
- Tests that reached into `spTxIndex` now assert through
`txsWithSubpath` and the public counts.
- `cd cmd/server && go test ./...` passes.
- SonarQube: quality gate passed, no issues on added lines.

**Overlap with #2181:** both edit the same memory-estimate comment block
in `store.go`. Whichever lands second needs a trivial rebase there;
there's no code dependency.

Backend only: no UI, API shape or config change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 12:41:46 +02:00
Alex BandClaude Opus 5.5 ca62790c81 fix(store): load the observer IATA in the background chunk loader (#2185) 
## Problem
With `hotStartupHours` set, startup loads the hot window first.
`loadChunk` then fills the rest of the retention window in the
background. Its query selected the observer ID and name but not the IATA
code, and scanned no IATA, so every packet it loaded had an empty
`ObserverIATA`. On a 7-day window with a 24 h hot start, that's six days
of packets after every restart that:
- have no `observer_iata` in the packets API;
- never match the `iata` filter (#1188).

`Load`, `scanAndMergeChunk` (the hot window) and both ingest paths
already join `observers` to get the IATA code, so only background-loaded
packets were affected.

**Seen on a live instance** (`hotStartupHours: 24`, restarted about 18 h
earlier), sampling `/api/packets?until=…`:

| packets | with `observer_iata` |
|---|---|
| 20–30 h old (hot window) | 200 / 200 |
| 46 h old (background-loaded) | 0 / 200 |
| 60 h old (background-loaded) | 0 / 200 |

With this fix on a staging instance, packets older than its hot window
are 300 / 300.

## Change
The chunk query selects `COALESCE(obs.iata, '')` for both schemas: the
v3 query through the observers join it already had, and the legacy query
through a new join on the observer ID, as `Load` does. `loadChunk` scans
the column into `ObserverIATA`.

## Tests
- `TestLoadChunkKeepsObserverIATA`: red commit first (fails on its
assertion), then green.
- `cd cmd/server && go test ./...` passes.
- SonarQube: quality gate passed, no issues on added lines.

**Overlap with #2183:** both edit the same lines of `loadChunk`'s
`StoreObs` literal (#2183 interns the observer fields). Whichever lands
second needs a trivial rebase; if this lands first, #2183 should intern
`ObserverIATA` there too. There's no code dependency.

Backend only: no UI, API shape or config change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 11:13:53 +02:00