dandri/meshcore-analyzer
1
0 Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-11-11 11:37:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,126 Commits 85 Branches 42 Tags
417c8dd00a243f29822f34e5c9aaadbd2e522213
Commit Graph
23 Commits
Author SHA1 Message Date
TeTeHackoandClaude Opus 5.5 417c8dd00a feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300) 
Closes #2270, closes #2271.

The ingestor took only the confirmed region scopes out of an observer's
`/neighbors` report (#1865) and dropped the rest. Nothing could
therefore say when an observer last reported, or whom it heard.

## Ingestor

`observer_neighbor_reports` (created in `dbschema.Apply`) keeps the
latest report of each observer, one row per observer, replaced by every
live report.

The report is unauthenticated, so `neighbors[]` is stored sanitised:

- only 64-hex keys
- `snr` and `heard_secs_ago` finite and in range
- `status` limited to `[a-z_]`
- `scopes` clamped
- at most 500 rows

`total_neighbors` is believed only when it is not below the number of
rows that arrived.

A **retained replay** only fills a missing row, and it is dated by the
report's own `timestamp`. After an ingestor restart, a long-dead
observer therefore does not look as if it just reported.

## API

The table is optional for the server (no `AssertReady` change); without
it the fields are simply absent.

- `GET /api/observers`: `neighbors_report_at` and `neighbors_count`,
from one batch query in the existing cached builder.
- `GET /api/observers/{id}`: also `neighbor_report`, which is
`{received_at, reported_at, neighbor_count, queried_neighbors,
self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr,
heard_secs_ago, status, scopes}]}`. Names and roles come from the node
table in one query.

## Frontend

`public/observer-neighbors.js` is shared by both pages and escapes
everything.

- **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h
ago · 12`, or `—` when the observer never reported. The same value
appears in the slide-over.
- **#2270, observer page:** **Neighbors Report** and **Neighbors** stat
cards.
- **#2271, observer page:** a **Neighbors** table listing each neighbor
by name (a link to the node) or key prefix, with SNR, heard, scope query
status and scopes. The heading says when the list is shorter than the
count, e.g. `(2 of 3 listed)`.

## Tests

- **ingestor:** sanitising, the cap, live replace, the retained rules
(fill only, dated by the report, undated ignored) and the MQTT topic end
to end. The existing #1865 scope tests are unchanged and still pass.
- **server:** the list summary, the detail with names, and both
endpoints without the table.
- **frontend:**
  - unit tests of the helpers
- an XSS case for every rendered field; I checked that it fails when
escaping is removed
- `test-observers-headings` now expects the new column after `Last
Packet`
- **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer
endpoints, since the fixture has no reports. It is listed in
`scripts/non-unit-tests.json` and on shard 2.

Gates that pass:

- `go test -race` in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`
- `sh test-all.sh`
- `eslint@8`, with 0 errors
- the XSS diff gate
- these E2E suites: the new one,
`test-issue-1639-observers-sort-e2e.js`,
`test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js`

Performance: the list adds one primary-key-sized query per cache fill,
and the detail adds two small queries per request.

Also fixes #2330 (folded in from #2337, second commit).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 22:36:53 +01:00
TeTeHackoandClaude Opus 5.5 77a5f14925 feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292) 
Part of #1869.

## Problem

`observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer
reported. Once an observer is retuned (say from SF7 to the EU narrow SF8
setting to listen to a neighbouring mesh), every packet it ever heard
reads as heard on the new setting, and there is no way to tell an SF7
packet from an SF8 one.

## What this does

As #1869 proposes, the packet data model is untouched: the setting of an
observation is resolved from its observer and time.

**Ingestor** (all schema in `dbschema.Apply`)

- `observer_radios`: one row per stretch of time an observer, and each
radio of a multi-radio node, stayed on one setting. Every live status
extends the current row and a change starts a new one; retained replays
never extend a row. On the first run it is seeded with every observer's
current radio, which is what the UI already showed for the past.
- Settings are stored canonical through `dbschema.CanonicalRadio`, so
`869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting.
- `observation_radios` (`observation_id → rx_radio_id`): multi-radio
openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id,
radio}]` in their status, so their observations resolve per radio.
- The table is sparse: single-radio nodes never send the field, so
`observations` and every query that scans it stay untouched.
  - The row id comes from the observation upsert via `RETURNING id`.
- Rows are pruned with their transmissions, and a purged observer takes
its history along.

**Server** (read-only)

- No `AssertReady` change: an absent table reads as empty and is probed
again later, the same rule as `advert_route_evidence`.
- The history is held in memory. The live poller checks it on every tick
(two primary-key lookups), so a retune or a new `rx_radio_id` is known
before the observation is broadcast. A full reload runs every 10 min.
- `radio` is added to observations, packets, grouped rows (taken from
the header observation) and the live feed.
- `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting.
- Combined with `observer=`, it means *that* observer heard it on that
setting.
- A radio query is served from memory instead of falling back to SQL,
which would return the window unfiltered.
- `GET /api/radios` (named struct) and `radio_history` on `GET
/api/observers/{id}`.

**Frontend**

- Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8`
or `freq > 869.5 && bw == 62.5`.
- A Radio row in the packet detail and a Radio column in its
observations table (only when the server reports one).
- A radio history table on the observer page once there is more than one
stretch.
- `RfParams.radioText` formats the setting. It escapes all five
characters and returns non-numeric input as it came, escaped.

## Performance

`txMatchesRadio` runs once per observation inside `filterPackets`.

- Each observation costs one map lookup by observer id.
- Observers with a single stretch (nearly all) answer from that lookup
directly.
- Retuned observers add a binary search over their few stretches.
- Multi-radio observers add one lookup by observation id.
- The snapshot is loaded once per filter call.

`BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20
observers, 2 of them retuned):

```
BenchmarkTxMatchesRadio-24    602    2056924 ns/op    0 B/op    0 allocs/op
```

Memory is one entry per stretch plus one per multi-radio observation;
nothing is added per observation otherwise.

## Tests

- **dbschema:** seed runs once, canonical spellings, the tables exist
without `observers.radio`.
- **ingestor:**
  - stretches extend and split; canonical spellings do not split
- two radios of one node hearing one transmission over two paths each
keep their own radio
  - a retained status does not write history
  - prune and purge
  - `rx_radio_id` reaches `PacketData`
- **server:**
- resolution before, between and after stretches, and with an unknown
radio id
  - `observer` + `radio`, and grouped rows carry `radio`
  - incremental refresh, `/api/radios`, `radio_history`, missing tables
  - the benchmark above
- **frontend:** the new filter fields in
`tests/unit/test-packet-filter.js`, and XSS cases through the real
`RfParams.radioText` and the history renderer.
- **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped
rows' radio, `sf ==` on the packets page, the detail Radio row and the
observer history. It is added to `scripts/non-unit-tests.json` and to
shard 2.

`go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema`
passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS
diff gate. The new E2E test and `test-e2e-playwright.js` pass against
the fixture.

## Browser validation

This runs on a live instance with six observers, one of them retuned to
869.618/62.5/SF8/CR8:

- The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR
4/5`.
- `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107;
the SF8 observer has not heard anything yet.
- `/api/radios` lists both settings.
- No page errors.

## Notes

- **The customizer (rule 8)** gets nothing: the only constants are
server-side refresh intervals.
- **Attribution timing:** packets heard between a retune and the
observer's next status message (minutes) are attributed to the previous
setting, because a stretch starts when the status is handled. This is
documented at `RecordObserverRadios`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 21:08:40 +01:00
n30nex e61ff6e473 fix(observers): keep detail responses tied to the current view (#2323) 
Fixes #2316 and #2320. Related to #1778; progressive loading remains
separate.

Delayed observer responses could replace a newer observer or period,
overwrite its error/status/calibration, and leave detached charts alive.
Each load now captures its observer and days and checks ownership before
success or failure can update the view. Leaving the page invalidates
pending loads, including same-observer revisits.

New visits also reset the request window to the selector's existing
seven-day default. Previously, a new page could display seven days while
requesting the preceding visit's 30-day window.

Production change: 12 added lines and three replacements in one frontend
file. No new requests, caches, settings or dependencies; the ownership
check is O(1). Existing request deduplication, comparison controls,
chart cleanup and current-request errors are preserved.

Validation: 11 unit checks, all 219 standalone suites, 32 controlled
browser cases across desktop/mobile and both themes, comparison 9/9, and
observer-IATA 5/5. Core browser run exited successfully with 149 passes
and three declared fixture skips; existing conditional subchecks remain.
The unchanged backend tree passed both full backend suites. The parent
independently reran the new unit/browser tests.

Test-only RED `a73c227f` produced [nine unit and 28 browser assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40239824064).
Three independent reviews passed. [Final-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40240673369)
passes on `ba52d873` integrated with upstream `11ce1235`, including all
browser shards and architecture builds.
 2026-10-26 15:42:56 +01:00
efitenandClaude Opus 5.5 dc98886d92 feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239) 
Part of #2238. Path hops through such a repeater (`/api/nodes`,
`/api/resolve-hops`, the frontend hop resolver) still show a bare
prefix; that part of the issue stays open.

## Summary

A repeater that never sends an advert can still tell a nearby companion
its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and
publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`.
This PR stores those answers, resolves names with them, and shows these
repeaters on the map, in the nodes list and in search, marked as
self-reported.

### Ingestor: store the names
- `367ecde1` adds a new `names` subtopic.
  - It is gated on `clientNodeNames.enabled`, default off.
- Answers go into a new table, `client_node_names`: one row per
observation, `UNIQUE(target, rx_pubkey, observed_at)`.
- **Validation.**
  - `target` must be 64 hex.
- The name is trimmed and rejected when it is empty, over 31 runes
(firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8,
or contains a control character.
- **Retention.** `retention.clientNodeNamesDays` (0 = keep).
`config.example.json` sets 30, like the other client tables: a companion
only asks a repeater for its name when CoreScope has none (`nameMissing`
in CoreDrive's `src/names.js`), so pruning is also how a renamed
repeater gets its new name.
- **Compatibility.** An ingestor without this change drops `/names`
silently, so CoreDrive v1.20.0 works against either version.

### Server: resolve by the reported name
- `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and
`client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys`
(the coverage tooltips) both use it.
- **Which name wins.** The advert name, otherwise the newest reported
name by `observed_at`. A prefix that matches different pubkeys across
the two tables stays ambiguous.
- **Only repeaters heard directly.** A reported name counts only once
its target has at least one `discover` or `advert` reception in
`client_receptions`, keyed by the full pubkey, or by its 8-byte prefix
when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names`
starts with that prefix (`clientNameHeard`). A name for a pubkey nobody
heard adds no candidate, so it cannot make a real node's prefix
ambiguous. Without the uniqueness check, a made-up pubkey that shares a
heard repeater's first 8 bytes borrowed that repeater's receptions and
passed (fixed in `4c569f91`).
- **Filters.** The blacklist and the hidden name prefixes apply to both
sources.
- **Missing table.** A database without the table resolves from `nodes`
alone, and the server checks for the table again until the ingestor has
created it.
- **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?,
name?, ambiguous}`.

### Server: estimated position
- `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`.
It lists every repeater in `client_node_names` that never sent an advert
(no row in `nodes` or `inactive_nodes`) and was heard directly at least
once (see above). A node with an advert but no position stays out: it is
already in `/api/nodes` under its advert name.
- **Gated** like the other endpoints on `client_receptions`: 404 when
`clientRxCoverage.enabled` is off, because the counts and the estimate
come from companion GPS fixes.
- **Which receptions count.** Only direct ones:
- `discover` and `advert` rows, keyed by the full pubkey, or by its
8-byte prefix when no other known pubkey shares it;
  - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone.

  `geo` rows are left out.
- **How the estimate is made.**
- Receptions within 100 m form one spot, with the best SNR of that spot.
  - An estimate needs at least 3 spots, at least 200 m apart.
- The position is the SNR-weighted centroid (`10^(snr/20)`). The radius
is the distance to the farthest spot, at least 500 m.
- The response is cached for 5 minutes, and the compute time is logged.

### Map
- `5fcde554` adds `public/map-estimated.js`, which draws each estimate
on its own layer:
  - a hollow, dashed "≈" marker and a dashed uncertainty circle;
- a popup with an "Estimated position" badge, the method, the caveat
that one-sided driving skews the estimate, and a link to the coverage
map;
- a Display toggle, "Estimated positions": on by default, kept in
localStorage, and also hidden by the repeater role filter and in route
view.

### Nodes page and search
- `12e6b345` adds a "Self-reported repeaters" section under the nodes
table, on the All and Repeaters tabs, filtered by the search box.
- **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a
self-reported detail view replaces "Node not found":
  - who reported the name;
  - the receptions;
  - the estimate, or why there is none;
  - links to the map and the coverage map.

  An unknown key still shows "Node not found".
- **Header search** adds matching self-reported repeaters, tagged
"self-reported".
- **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as
they are. These repeaters have no packets, so the packet filters, the
live view and the channel sender lookup must not match them.

### Docs
- `03580437` corrects the `/regions` section of
`docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the
switch, but the code uses `clientRegions.enabled`.
- The new subtopic, endpoint and config keys are documented in the same
file.

## Trust

Any companion the broker lets publish can report a name for any
repeater, and the newest report wins. This is the same model as
`/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to
linked companions.

A report for a repeater nobody heard directly is stored but not used: it
is not listed and adds no prefix candidate. A companion can still report
a wrong name for a repeater that was heard. The same companion can also
publish a fake `discover` reception for a made-up pubkey, so the "heard
directly" check filters junk, not a malicious companion;
`requireLinkedCompanion` is the control for that (its gate runs before
the `/names` handler too).

## Performance

- **`lookupPrefixes`** adds one term per key on `client_node_names`.
- It matches the prefix as a range, `target >= ? AND target < ?` (upper
bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive
column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX
idx_cnn_target (target>? AND target<?)`, and the reception check is
`SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`.
- It is still one query per chunk of 200 keys: 400 compound terms, under
SQLite's default limit of 500.
  - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks.
- **`/api/nodes/reported`** makes three kinds of query:
  - one grouped query on `client_node_names`;
  - one `lookupPrefixes` call for the short hashes;
- one `IN` query per 400 keys on `client_receptions`, served by
`idx_client_recept_heard_geo`.

The spread check is O(spots²) per repeater, with at most one spot per
100 m visited. The result is cached for 5 minutes; concurrent misses
wait for one computation, and the response is written after the lock is
released.
- **The map does not wait for it.** It draws its markers first and adds
the estimates when the list arrives. The map and the nodes page share
one cached fetch (`ReportedNodes.load()`).
- **Not benchmarked.** On the instance where this runs,
`client_node_names` holds one row today. I expect at most tens of such
repeaters per instance, but have not measured that.

## Notes

- **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2
occurrences: the handler's `msg` parameter and its `gps` field. That is
the decoded MQTT message, passed in the same shape as
`handleClientPacket` and `handleClientRegions` already take it. No API
response or internal structure uses it.
- **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min
TTL. They should move to the customizer later (rule 8).
- **Map filters.** The estimates follow the "Estimated positions" toggle
and the repeater role filter, not the area, region or last-heard
filters. The toggle's tooltip says so.
- **Name characters.** Names are rejected for control characters but not
for Unicode format characters (bidi overrides, zero-width joiners),
because emoji sequences in node names use the zero-width joiner. All
output goes through `escapeHtml`.
- **Not done:** the home-page search box (`home.js`, `/api/nodes/search`
with a Claim button) does not list these repeaters. Claiming a node
without adverts would have nothing to show.

## Tests

On this branch (based on `550829af`):
- `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and
`cmd/ingestor`.
- **New Go tests:**
  - handler and validation cases;
  - idempotent redelivery;
  - newest `observed_at` wins;
  - prune;
  - the gate on and off, without falling through to the observer path;
- client-only node, nameless `nodes` row, advert name wins, ambiguity
across the two sources;
  - hidden and blacklisted nodes;
  - absent-then-created table;
  - spot merging, threshold, weighted centroid and radius floor;
- `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an
advert position and a missing table;
- an advert without a position, a never-heard name, a name heard only in
`rxlog` paths, and the `clientRxCoverage` gate (404). These fail on
`12e6b345` and pass at head.
- a made-up pubkey sharing a heard repeater's 8-byte prefix, and two
reported targets sharing one (`4c569f91`). Both fail without that
commit.
- **New frontend unit suites:** `tests/unit/test-map-estimated.js` and
`tests/unit/test-reported-nodes.js` pass. Both load the real
`escapeHtml` from `app.js`.
- **`test-all.sh`** passes, all suites (with Python available for the
XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the
four changed frontend files.
- **E2E**, against a static server with a mocked API:
- `test-map-estimated-e2e.js`: 6/6, including the repeater role filter
and route view hiding the estimates. Mutating either hide makes its step
fail.
  - `test-reported-nodes-e2e.js`: 6/6.
  - `test-issue-1150-404-state-e2e.js`: 5/5.
- **Real data, API only.** On the fork's staging, `/api/nodes/reported`
lists one repeater: 5 spots, 17 receptions, an estimate with a radius of
2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays
listed under the stricter rules. Not re-checked after `4c569f91`.
- **Not done:** a check of that repeater in a browser.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 15:17:58 +02:00
efitenandClaude Opus 5.5 d43f3c3777 feat(theme): ?theme=dark|light URL parameter for one page load (#2236) (#2251) 
Fixes #2236.

A display that can only be given a URL (digital signage, an iframe
without cookies or login) had no way to choose dark or light. This adds
`theme=dark` or `theme=light` to the hash query, next to `embed=1`:

```
https://<instance>/#/map?embed=1&theme=dark
```

## What changes

**`public/index.html`, the pre-paint script**
- Reads `theme` from the hash query with `URLSearchParams`, as
`shouldEmbedRoute` does for `embed`. Only `dark` and `light` count;
anything else is ignored.
- A valid value wins over the saved choice and the OS preference, and is
set before first paint, so there is no light flash.
- The value is handed to `app.js` as `window.__themeOverride`.
- When `localStorage` throws (a sandboxed iframe), the URL value is
still applied.

**`public/app.js`, the dark-mode setup**
- On load, a URL value is applied through `applyTheme(theme, false)`,
which does not write `meshcore-theme`. The saved choice, and the account
settings that `settings-sync.js` syncs from that key, stay as they were.
- Theme changes from other tabs or from settings sync (the `storage`
listener) are ignored while the URL value holds. Otherwise a logged-in
display would flip within a minute.
- A click on the dark-mode toggle ends the URL value and is saved as
before.

It works on every page, not only on the two `embed=1` pages. The value
is read once per page load, so later hash changes (navigation, map
viewport) keep it.

## Tests

| File | Assertions | Covers |
|---|---|---|
| `tests/unit/test-theme-url-param-2236.js` | 8 | Runs the real
pre-paint script from `index.html`: `dark` and `light` win over a saved
choice and the OS, nothing is written to `localStorage`, any page and
position in the query, an unknown value, no param, `localStorage`
throwing |
| `tests/e2e/test-theme-url-param-2236-e2e.js` | 6 steps |
`#/map?embed=1&theme=dark` over a saved "light" (dark, embed, saved
still "light"), a reload without the param, `theme=light` over a saved
"dark", a storage event while the override holds, the toggle ending it,
`theme=blue` |

How I checked them:
- With `index.html` and `app.js` from master, 6 of the 8 unit assertions
fail, and the E2E fails at step 1 ("expected dark + embed + saved
light").
- With only the `app.js` change reverted, the E2E fails at step 1. With
only the `if (themeOverride) return;` line in the `storage` listener
removed, it fails at step 4.
- `sh test-all.sh` passes, except `test-issue-1956-release-routing.js`,
which I skipped: it needs the workflow's `GITHUB_REF_NAME` and does not
run in local git-bash. `npx eslint public/app.js` reports 0 errors (18
warnings, the same count as master).
- E2E on a fixture seeded as in CI:

  | Suite | Result |
  |---|---|
  | `test-e2e-playwright.js` | 148 passed, 3 skipped |
  | `test-customize-theme-e2e.js` | 8/8 |
  | the new #2236 suite | 6/6 |

- `test-logo-theme-e2e.js` fails locally with "wordmark `<text>`
overflows SVG viewBox", with the same numbers on master. It measures
font metrics, which differ on this Windows machine.

The new E2E is registered in `scripts/non-unit-tests.json` and
`deploy.yml` (shard 2, next to the logo theme suites); the unit test is
in `test-all.sh`.

## Not done

- No screenshot: the E2E checks `data-theme` in headless Chromium, not
how the page looks.
- `live.js` Matrix mode forces dark and, when switched off, saves the
theme it restores. With a URL value active, turning Matrix off saves
that value as the stored choice. I left this as is, because a display
that only shows a URL does not toggle Matrix mode.
- Only `theme=`; the `?darkMode=1` form from the issue text is not
supported, so there is one form to document.
- I placed the E2E in shard 2 without measuring suite times.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 11:40:25 +02:00
efitenandClaude Opus 5.5 3960fadd96 fix(pages): drop late responses that belong to an earlier page visit (#2218, #2230, #2200) (#2248) 
Fixes #2218, fixes #2230, fixes #2200.

Node analytics, the account page and analytics await their data, then
write into the shared `#app` container. A response that lands after the
user has navigated away replaces the new page. In #2218's reproduction,
the Nodes list, with 200 rows, became an Analytics heading.

## The shared part

`public/app.js` keeps one view generation, `viewGeneration()`:
- `navigate()` bumps it before the old page's `destroy()`, so every page
init counts as a new visit, including a switch to another node of the
same page type.
- A page captures the value at init, and after each `await` drops the
result when the value has moved on.

`.eslintrc.json` declares it as a global.

## Per page

**#2218, `node-analytics.js`**
- `loadAnalytics` and `loadBatteryChart` check the generation on both
success and error. A page-local `loadSeq` makes a time-range click a new
load on the same visit, so an older range can't land over a newer one.
- `NodeHopAnalytics.load` already checked `container.isConnected`, so it
is unchanged.

**#2230, `account.js`**
- The render after `CSAuth.ready()` runs only while the generation is
unchanged.
- `destroy()` was empty. It now clears `shown.app` and `shown.userId`.

**#2200, `analytics.js`**
- A page-local `_loadSeq` sits next to the generation. Results and
errors of an older window/region selection or an earlier visit are
dropped, so a late failure of an old load no longer shows an error over
newer results.
- The bulk load renders only the six tabs drawn from its data. A tab
that loads its own data (Prefix Tool, Nodes, Roles, ...) is no longer
rebuilt or replaced by the bulk result: a deep-linked one renders at
init, and a window, region or area change re-renders it directly.
- `renderTab` guards the six tabs drawn from the bulk data (overview,
rf, topology, channels, hashsizes, collisions). Until the data arrives
they show "Loading analytics…", or the current load's error. Overview no
longer dereferences an undefined `rf`.
- Three related fixes in the same lines:
  - The error message is now escaped with `esc()`; it was inserted raw.
- `destroy()` removes the RegionFilter and AreaFilter listeners. They
piled up on every visit and called `loadAnalytics` from other pages.
  - `destroy()` resets the stored error.

## Tests

| File | Assertions | Covers |
|---|---|---|
| `tests/unit/test-issue-2218-node-analytics-stale-render.js` | 12 |
Loads the real router: the generation rises on every init, including
another node. Late success and error after leaving, a stale visit of the
same node, an older range landing last, late battery success and error.
The current response, battery and error still render |
| `tests/unit/test-issue-2230-account-stale-render.js` | 6 | Leaving
before ready (login view and "not enabled"), re-entering before ready,
the current render, an auth change after `destroy()` |
| `tests/unit/test-issue-2200-analytics-stale-load.js` | 19 | An older
selection landing last, a late rejection of an older selection, leave
and re-enter (success and error), the current error rendered escaped and
kept on a tab click, Overview and RF clicked before the data |
| `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js` | 4 steps |
Holds `/api/nodes/*/analytics` with `page.route`, returns to `#/nodes`,
then releases the response as a 200 and as a 500. The 200 rows stay and
no Analytics heading appears; a normal visit still renders |

How I checked them:
- With the production files reverted and the tests kept, 7 of 12, 4 of 6
and 7 of 12 unit assertions fail. With the fix, all pass.
- The E2E fails on the unfixed frontend with "Nodes rows 200 before
release, 0 after".
- Single mutations, each reverted:
- removing the success check in `loadAnalytics` (analytics) fails 2
assertions;
  - removing the check after `CSAuth.ready()` fails 3.

Other test changes:
- `test-analytics-theme-refresh-before-data.js` and
`test-user-management-ui.js` get a `viewGeneration: () => 0` stub.
- The new E2E is registered in `scripts/non-unit-tests.json` and
`deploy.yml` (shard 2, next to the #748 suite). The unit tests are in
`test-all.sh`.
- The broad suite's "Node side panel Details link navigates" test
selected an anchor that now opens Analytics, as #2218 notes. It now
clicks `#nodesRight .node-detail-btn`.

Results:
- `sh test-all.sh` passes, and `npx eslint public/*.js` reports 0
errors.
- E2E on a fixture seeded as in CI:

  | Suite | Result |
  |---|---|
  | `test-e2e-playwright.js` | 148 passed, 3 skipped |
  | `test-user-management-e2e.js` | 28/28 |
  | the new #2218 suite | 4/4 |
  | `test-issue-748-node-analytics-coverage-e2e.js` | 2/2 |
  | `test-analytics-fluid-charts.js` | 8/8 |
  | `test-issue-1657-analytics-channels-group-sprites-e2e.js` | 5/5 |

## Not done

- Two tabs that each fetch their own data can still overwrite each other
after a fast tab switch between them. #2200 does not ask for that.
- `nodes.js` `navigateToNode` calls its own `init` without the router,
so it does not bump the generation. None of the three pages fixed here
depends on that.
- I placed the E2E in shard 2 without measuring suite times, which the
`deploy.yml` comment asks for.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 10:39:15 +02:00
CoderNemesisandClaude Opus 5.5 e55c95db24 1-Byte Hop Path Inference for Packet and Map Analysis (#2245) 
## Summary

On meshes with 1-byte path hashes, most hop prefixes are shared by
several repeaters. The packet detail pane currently resolves each hop on
its own and stores the result by prefix. Two things go wrong as a
result:

- A prefix that appears twice in one path resolves to the same node both
times.
- Nothing checks that the chosen hops form a path a packet could
actually take.

This PR adds `HopResolver.inferPath()`, which infers the most likely
node behind every hop of **one observation's path**, scoring the path as
a whole and anchoring it on the observer that heard it. The packet
detail pane and the map route view now use it, so both show the same
nodes for the selected observation (`?obs=`).

## The problem, on a real packet

Packet `57da37d1636f895a`, observation `1388126`: a 46-hop flood with
1-byte hashes. The server's `resolved_path` resolved 1 hop of 46, so the
client resolver did the rest. Eight prefixes appear twice in the path.

| | Current resolver | `inferPath()` |
|---|---|---|
| Hops longer than 120 km | 13 (21 longer than 60 km, two over 950 km) |
0 |
| Distance from the last hop to the observer | 59 km | 0.1 km |
| Hops reported as unlisted (no listed node fits) | 0, every hop guessed
| 13 |
| Agreement on the shared path prefix across the packet's 23
observations | — | 97.8% |

The current resolver picks badly for three reasons:

1. **Results are keyed by prefix.** Hop 0 and hop 45 share a prefix, so
both show the node near the start of the path, 59 km from the observer.
2. **Picks are greedy.** Each hop looks only at its nearest resolved
neighbours, so nothing penalises a 1,100 km jump.
3. **The detail pane read the wrong `resolved_path`.** It used the
packet row's, not the selected observation's, and when the lengths
matched it never fell back for the hops that were `null`.

## Firmware basis

The model relies on two facts from `firmware/src/Mesh.cpp`:

- **The last hop is the one the observer heard.**
`Mesh::routeRecvPacket` appends the relaying node's own hash to the end
of a flood path.
- **A node appears at most once in a flood path.** A node forwards a
given packet once: `_tables->wasSeen()` / `markSeen()`.

## How the model works

Each hop is either one of the **located** repeaters or room servers
sharing its prefix, or **unlisted**: a node missing from the node list,
or one without GPS. With 1-byte prefixes, unlisted is common, and naming
it beats drawing a 300 km hop.

| Term | Cost |
|---|---|
| A hop between two located nodes | distance / 50 km; past 100 km per
hop, the cost rises 4× faster |
| A hop with a neighbour-graph link | the distance cost × 0.5 |
| An unlisted hop | 1.6, flat (about one 80 km hop) |
| The last hop to the observer | the same distance cost; falls back to
the observer's IATA region centre (with 50 km slack) when the observer
reports no GPS or 0,0 |
| The first hop to the sender | the same distance cost, when the
sender's position is known (e.g. an ADVERT) |

- **Search:** dynamic programming over (hop, candidate) finds the
cheapest sequence. When the best path uses a node twice, the node is
banned from whichever position costs less to give up.
- **Confidence:** each hop's confidence comes from the min-marginal
costs of its alternatives, put through a softmax. A node the path
already uses at another hop is not counted as an alternative.
- **Server hint:** the observation's own `resolved_path` restricts a hop
to the named node, but that hop can still be reported unlisted if the
node doesn't fit.

The tuning values live in `PATH_MODEL` in `hop-resolver.js`, with a TODO
to expose them in the customizer (AGENTS rule 8).

## What changes in the UI

**Packet detail pane** (`packets.js`):

- The Path row, the cross-observer aggregate row and the byte breakdown
render hops **by position**, from the inferred entries.
- The detail pane no longer writes into the list's shared per-prefix
cache (`hopNameCache`), so it can't change what the packets list shows.

**Hop display** (`hop-display.js`):

| Hop state | Display |
|---|---|
| Confident (≥ 70%) | The node's name; the tooltip says `NN% likely` and
how many nodes share the prefix |
| A guess (< 70%) | Dashed underline, plus a candidate badge when there
are alternatives |
| Unlisted | The hex prefix in muted text; the badge lists the rejected
candidates |
| The only known node, without GPS | The name, with `(no GPS)` |

The candidate popover lists each candidate with its likelihood, or `hop
N` when the path uses that node elsewhere. The "Path symbols" legend
gains an entry for unlisted hops.

**Map route view** (`map.js`, `route-view.js`):

- The `#/map?packet=…&obs=…` deep link infers the chosen observation's
hops with the same shared helper, so the map and the detail pane agree.
- Unlisted hops render as `unlisted` placeholders rather than as the
nearest node that happens to share the prefix.
- When inferred hops are available, the `/api/resolve-hops` request is
skipped.

**Shared helpers** (`packet-helpers.js`):

- `inferObservationPath(obs, observerPos, originPos)`
- `ensureHopAffinity()`: loads the neighbour graph once per page, cached
by `api()` for 5 minutes.
- `routeHopFromInferred(hop, entry, role)`

**Cleanup:** removed dead hop resolution from the "View route on map"
button handler. It computed hops, sender and recipient, then navigated
by URL and discarded them.

**Unchanged:** the packets list, live view, node pages and
`HopResolver.resolve()`.

## Performance

AGENTS rule 0 asks for proof of performance claims; the figures follow.

- **Complexity:** O(hops × 8 × candidates²). Hops more than 8 positions
apart aren't compared directly; a longer run of unlisted hops costs a
flat amount, tracked with a running minimum. Hop-to-hop costs are
computed once and reused by every pass.
- **Where it runs:** once per detail render (plus once for the aggregate
row) and once per map deep link. Nothing runs in the list, live feed or
WebSocket paths.
- **Measured** (Node 22, warm JIT):

| Case | Time |
|---|---|
| 64 hops × 16 candidates, synthetic worst case | 9–14 ms median |
| Real 39–48 hop paths, 1,153 nodes (23 observations) | 1–12 ms each |

- **Enforced:** `test-hop-path-inference.js` asserts a median under 40
ms (target 20 ms) for the 64 × 16 case. It times the module outside
Node's `vm` sandbox, because the sandbox makes global lookups about 10×
slower than a browser.
- **Network:** one extra bulk request per page life
(`/api/analytics/neighbor-graph`, cached server-side, TTL 5 min). No
per-hop or per-packet requests.

## Tests

**New:**

- `tests/unit/test-hop-path-inference.js` (37 assertions). The model
itself:
  - a repeated prefix resolves per position, and no node appears twice;
- the observer anchors the last hop, with the IATA fallback when the
observer has no position or reports 0,0;
  - a far-away lone candidate is reported unlisted;
  - a neighbour-graph link breaks a tie;
  - the server hint and the origin anchor are applied;
  - a node without GPS is handled;
  - output shape, and the performance gate.
- `tests/unit/test-hop-path-detail.js` (32 assertions). Rendering and
wiring:
  - inferred-hop rendering and the popover;
  - positional `renderPath`, including with the 1-byte hop filter on;
- `renderDetail` infers the shown observation from its own
`resolved_path` and leaves the list cache alone;
  - the neighbour graph is fetched once;
  - the shared map helpers, and source checks on the map wiring.
- `tests/e2e/test-hop-path-inference-e2e.js` (6 steps, shard 1). Runs on
fixture packet `57224a4a01c9c682`, observation 147, a 22-hop path that
repeats one prefix three times:
  - every hop is inferred;
  - the repeats resolve per position;
  - the popover shows likelihoods;
  - the map's route list matches the detail pane hop for hop.

**Updated, because they pinned the old behaviour:**

- `test-issue-1418-deeplink-hops-channels.js`: it tested a copy of the
old map priority logic. It now tests the real `inferObservationPath()`,
including `resolved_path` as a hint.
- `test-issue-2098-last-hop-region.js`: the detail pane no longer writes
the prefix cache. The test now reads the region-anchored last hop from
the rendered Path row.
- `test-issue-2097-hop-ambiguity-badge.js`: the source check allows the
new `entries` option while still asserting that the detail pane never
uses the summarised form.
- `.eslintrc.json`: declares the three new shared helpers as globals.

**Results on this branch:**

- `sh test-all.sh`: all standalone frontend suites pass.
- `npx eslint public/*.js`: 0 errors. All 85 warnings are pre-existing;
none come from code this PR adds.
- The "Run Playwright E2E tests (fail-fast)" step was run locally with
its script taken verbatim from `deploy.yml` (`bash -eo pipefail`,
fixture prepared as in CI):
  - shard 1: 27 of 27 passed;
  - shard 2: 47 of 47 passed;
  - shard 3: 50 of 50 passed.

Note for reviewers: `test-issue-1692-packets-init-parallel-e2e.js` only
passes while the freshened fixture's newest packets are less than 15
minutes old. The test sets the time window in `localStorage` after the
page has already read it. CI meets that condition because shards start
right after freshening. Unmodified `master` fails the same way when run
late.

**Browser validation:** the local server ran on the fixture DB. Checked
in the browser:

- the Path row, the candidate popover and the byte breakdown;
- the map route view, which matched the detail pane hop for hop with
unlisted placeholders;
- the console showed no errors.

## Known limitations and follow-ups

- **TRACE packets are not modelled yet.** A TRACE's path is its
*planned* route. The header path byte counts the hops actually forwarded
(repeaters append SNR, not their hash, `Mesh.cpp`), and a trace may
revisit a node on its way back (see the comment in
`Packet::calculatePacketHash`). A follow-up will add a `heardAfter` /
`allowRepeats` option, so a trace is anchored after the hops it actually
completed and may repeat nodes.
- **Customizer (AGENTS rule 8):** the `PATH_MODEL` tuning values are
hardcoded for now. The unlisted cost, typical and maximum hop length,
and the confidence threshold should become customizer settings.
- **Server-side `resolved_path`:** the ingestor writes a hop as resolved
whenever its prefix is unique in the node table, even when that node is
hundreds of km from the neighbouring hops. `inferPath()` treats these as
hints and can overrule them, but the stored data is still misleading.
- **The first hop without a sender position is often a close call.** On
encrypted channel messages, hop 0 is constrained only by what follows
it, so 55/45-style splits between a long chain of known nodes and
"unlisted" are expected. The badge and the percentage show this rather
than hide it.

## Screenshots

This is what a normal corescope instance would show for a 1 byte path.
<img width="318" height="338" alt="image"
src="https://github.com/user-attachments/assets/4056bc27-0b37-4107-9b25-31ac0f7864ec"
/>

<img width="430" height="601" alt="image"
src="https://github.com/user-attachments/assets/01da5ee8-babf-4e41-a9ab-836b5d640549"
/>

Now the same packet but with the logic to infer each hope along the path
different nodes are chosen as they're ranked and selected based on
proximity to the next hop,

<img width="436" height="646" alt="image"
src="https://github.com/user-attachments/assets/bcca73b8-daf0-4031-8ea1-8c96d716c8b3"
/>

<img width="436" height="646" alt="image"
src="https://github.com/user-attachments/assets/fc926d9e-158c-48ce-bee6-93086afb0775"
/>

The higher the confidence level then there's no need to show the
exclamation mark to show that multiple nodes share the same prefix but
it does continue to share that

Examples of it currently running can be found at,
https://corescope.baymc.org - modified instance

https://analyzer.comchan.net - unmodified instance
https://analyzer-stage.comchan.net - modified instance

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-19 20:00:07 +02:00
efitenandClaude Opus 5.5 2d09335cfb fix(packets): keep the node filter pick on a slow click or late search (#2243) (#2244) 
Fixes #2243.

## Problem

The packets node filter selected a suggestion on `click`, while the
input's `blur` handler hides the dropdown 200 ms after focus leaves. A
press held longer than that, or a search response that rebuilt the list
between mousedown and mouseup, lost the pick: the typed text stayed and
no `node=` filter was applied.

## Change

- `public/packets.js`: suggestions select on `mousedown` with
`preventDefault`, so the input keeps focus and the pick happens before
any rebuild. `live.js` already does this for its node filter.
- `public/packets.js`: a sequence counter drops search responses for an
older query, bumped on every search and on a pick, so a late response
can't replace or reopen the list.
- `tests/e2e/test-packets-node-filter-e2e.js` (new), registered in
`scripts/non-unit-tests.json` and on shard 1 in `deploy.yml`.

## Tests

The new E2E has three cases: a normal click, a 400 ms press, and a
search response released mid-click.

| Code | Normal click | 400 ms press | Response mid-click |
|---|---|---|---|
| master | pass | fail | fail |
| this PR without the sequence guard | pass | pass | fail (list reopens)
|
| this PR | pass | pass | pass |

Run locally against the freshened e2e fixture. `sh test-all.sh` passes
(locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and
`PYTHONUTF8=1`, unrelated to this change).

## Not done

- Full `test-e2e-playwright.js` not run locally.
- Could not reproduce on the reporter's instance with the browser tool,
whose clicks are shorter than 200 ms. The 400 ms Playwright press
reproduces it on the fixture.
- `live.js` and `packets.js` still carry two separate node-autocomplete
implementations. Merging them is out of scope here.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 21:30:57 +02:00
nullrouten0nullroutenClaude Sonnet 5.5
 		127dd511a7 fix(health): judge favorites and home status cards by the node's role (#2215) 
## The problem

A starred repeater can show two different health statuses. The My
Repeaters tab (#1761) and `getNodeStatus` judge it by the infra
thresholds (degraded after 24 h, silent after 72 h). The favorites
dropdown and the home page cards judge every node by the node thresholds
(degraded after 1 h, silent after 24 h), because they never look at the
node's role.

| repeater last heard | favorites / home cards (before) | My Repeaters
tab | after this PR |
|---|---|---|---|
| 5 h ago | degraded | healthy | healthy |
| 30 h ago | silent | degraded | degraded |
| 80 h ago | silent | silent | silent |

If you set `healthThresholds` in `config.json`, every view follows your
values.

## The change

Three call sites now use `getHealthThresholds(role)`, the helper the My
Repeaters tab already uses:

- `renderFavDropdown` in `app.js`
- the "My Nodes" grid card in `home.js`
- the hero search-result card in `home.js`

The search card's fallback text, "Not heard in 24+ hours", was only true
for non-infra nodes, so it now reads "Not heard recently".

## Scope

- I kept the existing three-state healthy / degraded / silent logic and
changed only which thresholds apply.
- Not included: counting a repeater that is relaying traffic as alive
(#1598, which `getNodeStatus` does). That is a bigger behavior change
and would be its own PR.

## Risk

Low. It is 9 lines in two files, and the helper and thresholds already
exist and are used elsewhere.

## Tests

New `tests/unit/test-health-card-role-thresholds.js`, added to
`test-all.sh`. It checks that infra roles get looser thresholds than
nodes, and that each of the three call sites goes through
`getHealthThresholds(role)` with no node-only comparison left. On
current `master` 5 of its 7 assertions fail; with this change all pass.

`sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning from an
a11y self-test, which it also does on `master`.

## Update after review (commit `0f2376df`)

The text above is the description efiten reviewed on 2026-10-15. This
section describes the code as it is now.

### The change

- New `getHealthStatus(role, heard, now)` in `roles.js`. It returns
`healthy`, `degraded` or `silent` by the role's thresholds, or `null`
when there is no usable timestamp. It only applies the cut-offs; each
caller picks the timestamp.
- All six places that compared an age against `degradedMs` / `silentMs`
now call it:
  - the favorites dropdown (`app.js`);
  - the "My Nodes" card and the home health card (`home.js`);
  - the analytics network status and My Repeaters (`analytics.js`);
  - the live node panel (`live.js`).

Each keeps its own labels and timestamp, so the last three show the same
as before.

`analytics.js` and `live.js` no longer guard with `typeof
getHealthThresholds`. `roles.js` loads before both, and both already
used it without a guard elsewhere: the old network status called
`getHealthThresholds` directly, and `live.js` reads `ROLE_COLORS`.
- The favorites dropdown and both home cards now fall back to the node's
`last_seen` when the packet store has no packet for it. My Repeaters
(`last_heard || last_seen`) and the node page (`nodes.js:729`) already
do this. Without it, with a packet store window under 72 h, a repeater
could be silent on the cards and degraded on My Repeaters. The "ago"
text uses the same timestamp.

With the default thresholds, the fallback changes a status only when the
packet store holds less than 72 h of packets. `retentionHours` unset
(unlimited) or at the example's 168 h holds more than that. Otherwise a
node with no packet in the window is silent either way, and the views
now show how long ago it was last seen instead of "never" or nothing.
- With no timestamp at all, the home health card said "Not heard in 24+
hours". It now says "No packets in the current window".

### Why a new helper and not `getNodeStatus`

`getNodeStatus` has two states (active / stale) and counts a repeater
that relays traffic as alive (#1598). Using it here would change more
than the thresholds. `getHealthStatus` keeps these views' three states.

### Scope

- Not included: counting a repeater that relays traffic as alive (#1598)
in these views. That is a bigger behavior change and would be its own
PR.
- The live node panel still judges by `last_seen` alone, and the node
page stays relay-aware. Both now use the shared thresholds; aligning
their timestamps would also be its own change.

### Tests

- `tests/unit/test-health-status.js` runs `roles.js` in `vm`. It covers:
- repeater, room, companion, sensor, an unknown role and a missing role,
at ages either side of 1 h, 24 h and 72 h;
  - ISO and epoch input;
  - null, empty and invalid timestamps;
  - a `healthThresholds` override through `/api/config/client`.

One source guard remains: no age-vs-threshold comparison outside
`roles.js`.
- `tests/unit/test-my-repeaters-dashboard.js` now uses the real helper
instead of a stub, which gave repeaters the node thresholds. It also
checks that a repeater heard 30 h ago is Degraded.
- New `tests/e2e/test-health-status-e2e.js` mocks
`/api/nodes/<pk>/health` for six nodes. It checks five views: the
favorites dropdown, the "My Nodes" card, the home health card, the
analytics network status and the live node panel. It fails on `master`
and on this PR's first commit. I put it on shard 2, which was the
shortest shard in the last CI run.
- Mutation check: I made 33 deliberate breaks. In the helper they cover
each comparison and boundary. At each of the six call sites they cover
the timestamp field, the role argument and at least one label mapping.
Each one makes the suite fail.
- I checked a `config.json` with `infraDegradedHours: 2` and
`infraSilentHours: 6` in Chromium. Repeaters heard 3 h and 7 h ago show
Degraded and Silent.
- `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning, which
comes from a deliberate fixture in the a11y self-test.

### Cost

The analytics network status loops over every node. In headless
Chromium, 30,000 synthetic nodes take 9.0 to 9.2 ms before this change
and 9.1 ms after (median of 21 runs, on one machine).

### Risk

Low. The change is frontend only, with no API, config or data change.
Rolling back means reverting the commits.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-17 09:38:11 +02:00
efitenandClaude Opus 5.5 927a8eec8a fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211) 
Refs #748 (M3a: correct numbers within the in-memory window; the SQL
history in M3b is a separate plan, see the end).

## Problem

The node analytics range picker offers 24h, 7d, 30d and All (365d), but
the store only holds its in-memory window (`retentionHours`, then
eviction). `GetNodeAnalytics` still divided availability and packets/day
by the requested range. Measured on our live instance (14-day window),
repeater BE-BRE-ON8AR:

| Range | Availability | Packets/day |
|---|---|---|
| 7d | 88.7% | 91 |
| 30d | 42.6% | 265 |
| All | 3.5% | 22 |

Nothing on the page said that the data stopped at the edge of the
window.

## What changes

- `coverageStart` returns the edge of the in-memory window: the latest
of `oldestLoaded` (background-load progress), `now - retentionHours`,
and a new `evictedThrough` that eviction sets to the newest `FirstSeen`
it removed, so memory-pressure eviction inside the retention window
moves the edge too.
- `s.packets[0]` is not that edge, and the first version of this PR used
it. Chunks are selected by `last_seen`, so a transmission first heard
weeks ago and heard again recently sits at the head with its old
`FirstSeen`. On staging (7-day window) it returned 21/07.
- When the edge lies more than a minute after the requested start, node
analytics counts only transmissions from the edge, computes availability
and packets/day over that span, and returns `timeRange.dataFrom` and
`timeRange.partial`.
- The minute of slack keeps a range equal to the retention window from
reading as partial. Without it every 7d request on staging (168h) was
partial, because the two edges came from separate clock reads.
- The analytics page shows "Data from 2026-10-07: the server keeps about
7 days of packets in memory. Availability and packets/day cover that
period, not the full 365 days." when partial.
- **UTC fix.** The window bounds of node analytics and node hop
analytics are formatted in UTC. They are compared as strings with UTC
`FirstSeen` values, so on a non-UTC host the 7d window lost its oldest
hours (2 at +02:00). Docker images run in UTC, which is why this did not
show in production.
- `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the
edge is kept.

## Tests

- Go, `node_analytics_coverage_test.go`:
  - full coverage within memory;
  - 30d over 3 days in memory;
  - empty store;
- a re-heard 85-day-old transmission at the head of `s.packets` (edge
stays at retention, packet not counted, no bucket before `dataFrom`);
  - memory-eviction edge;
  - eviction records `evictedThrough`;
  - retention 30 s shorter than the range is not partial, while 30d is;
  - window bounds under a +02:00 `time.Local`.
- Two tests were checked to fail on the old code: the local-zone test
(166 of 168 packets, a `+02:00` bound) and the retention test without
the slack (`dataFrom 09:57:08` vs `from 09:56:38`).
- Full `cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master). One run of five reported `FAIL` and four reruns were green; I
did not capture which test failed.
- Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js`
(5 assertions on the real `node-analytics.js` in `vm`), in
`test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js`
(registered in `deploy.yml` and `scripts/non-unit-tests.json`), against
the real fixture response with no interception: 7d is not partial and
shows no notice; All is partial and the notice names the real start
date. 2/2 locally.

## Validation

Staging (retentionHours 168), same repeater, after the background load
completed:

| Range | Before | After |
|---|---|---|
| 7d | 82.7% | 83.3%, not partial |
| 30d | 19.3% | 83.3%, partial from 2026-10-07 |
| All | 1.6% | 83.3%, partial from 2026-10-07 |

In the browser the notice appears under All and not under 7d.

## Not done

- M3b, older history from SQLite. In the database only the sender is
indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay
hops only in `observations.resolved_path` (filled for about 27% of
observations on our live instance, not indexed). A fallback today would
show only a node's own packets for older days, a different meaning than
the in-memory part. Doing it properly needs a per-node index written by
the ingestor (pubkey, transmission, role); that is a separate plan.
- The other analytics endpoints with a range (observer analytics, RF)
were not checked for the same dilution.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:27:21 +02:00
efitenandClaude Opus 5.5 cab4f71c64 feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) (#2209) 
Fixes #1268.

Frequency, spreading factor, bandwidth and coding rate are per-device
settings that never travel in a packet. The public MeshCore map is the
only source. This PR adds them to the node detail views, opt-in, and
answers the open policy questions from the triage comments.

## Policy decisions

| Question (triage) | Answer in this PR |
|---|---|
| Opt-out for self-hosted instances | Opt-**in**: `rfParams.nodesUrl`
empty (default) = nothing fetched, `/api/rf-params` returns `enabled:
false`. Same model as `knownChannelsUrl`. |
| 33 MB every 6 h, thundering herd | Random 0-60 s delay before the
first fetch, then every `refreshHours` (default 6). The map sends gzip
(17 MB on the wire). It sends no ETag or Last-Modified, so conditional
requests are not possible. |
| Size cap | 128 MB on the decoded body (it is 51.5 MB today). Exceeding
it fails the fetch instead of parsing a truncated body. |
| Field shapes at the parse boundary | 64-hex key; 100-1100 MHz; SF
5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows
are dropped. A body that is not a well-formed array fails the fetch. |
| Keep stale cache on anomaly | Any failure keeps the previous snapshot;
a failed presets fetch keeps the previous presets. `fetched_at` is
returned so an outage is visible rather than hidden. |
| Freshness | Only adverts within `maxAgeDays` (default 7) are kept at
fetch time and re-checked at serve time. |
| TLS pinning | Not done: Go's certificate validation plus the field
validation above. |

## What changes

- `cmd/server/rf_params_cache.go`: stream decoder, validation, cap
reader, preset parser (presets with identical values are merged, titles
joined), background refresh. In memory only, no DB writes, so the
read-only server invariant holds.
- `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`;
`nodes` holds only nodes this instance knows, keyed by lower-case
pubkey. `/api/nodes` is unchanged, so its payload does not grow for
every consumer and the 500-row paging is unaffected.
- Node detail (side panel and full page): `LoRa` row with the four
values, the preset name when all four match one, and the age of the map
advert. Shown only when the feature is on and the server has fetched the
map at least once.
- No column in the Nodes table, on purpose: on our network such a column
showed one preset title in 814 of 1,366 rows and nothing in 443. Showing
only the nodes that differ from their region is the follow-up PR.
- `config.example.json` documents the block with the two URLs to use.

## Cost

- Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52
s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days).
Runs in a background goroutine every 6 h.
- The endpoint walks the cached node list once with map lookups:
O(nodes).
- Frontend: one bulk fetch of `/api/rf-params` when a node detail opens
(client TTL 5 min), not one per node.

## Tests

- Go (`rf_params_cache_test.go`): parse with every rejection reason,
malformed bodies (not an array, truncated, wrong type, empty), preset
merge and invalid presets, cap reader (one byte over refused, exactly at
the cap accepted), fail-soft against an `httptest` upstream (503, HTML
body, presets-only failure), config defaults, route (disabled shape,
only known nodes, entry aged past max age dropped at serve time). Full
`cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master).
- Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions
on the real file in `vm`), in `test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in
`deploy.yml` and `scripts/non-unit-tests.json`). It checks the real
disabled state on the fixture server (no detail row), then serves
`/api/rf-params` through `page.route`: no table column, full detail page
with preset, side panel with raw values, and the hint for a node that is
not on the map. 4/4 locally, plus the existing node table and node
detail E2E files.

## Validation

On a staging instance with `rfParams` set to the map and preset URLs:

- Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s
after start.
- `/api/rf-params` returned 1,115 of our nodes: 988 on
869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on
SF7.
- In the browser the detail row shows in the side panel and on the full
page, and the Nodes table keeps its original columns.

## Not done

- Highlighting nodes whose settings differ from their region's majority
(a badge in the Nodes list plus a filter) is the follow-up (separate
PR). In that follow-up a CR-only difference stays informational until it
is confirmed whether MeshCore radios use the explicit LoRa header (then
CR travels in the header and a mismatch still decodes).
- `maxAgeDays` is config-only; the customizer is a later step.
- No data is written to the DB, so the RF parameters are gone after a
restart until the next fetch (at most 60 s plus the fetch time).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:26:45 +02:00
efitenandClaude Opus 5.5 1ae4c74913 feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) (#2208) 
Fixes #661 (the narrowed scope from the 2026-10-14 comment).

## What changes

- `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and
`rx_airtime_pct_1h` per observer: the airtime share over the hour before
now, whatever `window` is, from the cumulative `tx_air_secs` /
`rx_air_secs` counters in `observer_metrics`.
- A counter that goes down marks a reboot; only the stretch after the
last reset counts.
- Under 15 minutes of samples (one sample, or a reboot late in the hour)
returns `null`.
  - Clamped to 0..100, rounded to 2 decimals.
- The response also carries `duty_cycle_warn_pct`, from the new config
key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit).
Added to `config.example.json`.
- RF Health tiles show `TX 1h: x%`; the observer list gets a sortable
numeric `TX 1h` column. Both use the shared `public/observer-airtime.js`
helper and flag values at or over the threshold with `.airtime-over`
(`var(--status-red-text)`).
- The handler now returns a named `MetricsSummaryResponse` struct
instead of `map[string]interface{}` (one occurrence fewer).

## Why the warning matters

Repeater and room-server firmware default to `airtime_factor = 1.0`
(`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26`
turns into a 50% TX budget per 1 h window. The device therefore does not
keep itself under the EU 10% limit; this view is where an operator can
see it.

## Cost

- Server: one extra query per summary request, `SELECT observer_id,
timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE
timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`,
then one O(rows) pass. On our instance that is 35 observers x about 12
samples (5 min interval).
- Observer list: one extra bulk call `summary?window=1h` per load
(client TTL 30 s, failure ignored), not a call per observer.

## Tests

- Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample,
span under 15 min, reboot leaving under 15 min, nil counters, clamp,
rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h
ignored inside a 24 h window, single-sample observer gets null),
`TestMetricsSummaryDutyCycleWarnPct` (default 10, config override,
`observers` is `[]` not `null`). Full `cmd/server` suite passes locally
(skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows
on clean master too).
- Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8
assertions, real file in `vm`), added to `test-all.sh`.
`test-observers-headings.js` updated for the new column. Full
`test-all.sh` passes locally.
- E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in
`deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no
`observer_metrics` table, so the summary is served through `page.route`
with known values for the fixture's own observer ids. It checks the
column header, over/under/unknown cells, sort order and the RF Health
tile. 4/4 locally. It caught a real bug during development:
`loadRFHealthData` declares a local `const window`, so the tile calls
the helper as the global `ObserverAirtime`.

## Validation

Deployed to a staging instance with live MQTT data (35 observers with
metrics) and checked in the browser:

- The observer list shows the column for 42 rows (33 with a value, 9
without enough samples). Sorting puts the highest share (5.3%) first.
- RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line
wrapped inside items on narrow tiles; the stats row now wraps between
items only.
- No observer was over 10% at the time, so the red state was checked by
rendering one cell client-side at 12.5% (computed colour `rgb(185, 28,
28)` in light mode), and by the E2E test.

## Not done

- The threshold is config-only; exposing it in the customizer is a
follow-up.
- No airtime comparison across observers (that is #671) and no
per-packet airtime (Relay Airtime Share covers it).
- Dark mode was not checked in the browser; `--status-red-text` is
defined for both themes.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-14 21:07:20 +02:00
Alex BandClaude Opus 5.5 352a0ccd19 feat(region-filter): configurable region quick picks (#2170) 
## Summary

**Independent of every other open PR.** Choosing a pick saves the
selection, redraws the control and notifies the page's listeners itself,
exactly as a click in the control does, so it re-queries straight away
whether or not #2168 (`setSelected` notifies) is merged; with both in,
nothing fires twice.

Region quick picks are named groups of region (IATA) codes, offered as
one-tap choices inside the region filter, e.g. "Northern California" for
a dozen airport codes instead of ticking each box. One deployment has
been injecting these into the page through an nginx `sub_filter` and a
script that wraps `window.RegionFilter`; this makes them a configured
feature.

```jsonc
"regionQuickPicks": [
  { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] }
]
```

- **Server:** `GET /api/config/region-quick-picks` returns the groups
cleaned up: names and codes trimmed, codes upper-cased and de-duplicated
in their configured order, groups without a name or codes dropped, at
most 20 groups of 200 codes, always a list. `/api/config/regions` is
unchanged.
- **Region filter:** loads the groups with the regions and offers every
group with at least one observer today, inside the region control:
- dropdown layout: rows of the same kind as **All** (checkbox and bold
name), directly under All with a divider before the single regions; the
row is ticked while that pick is the selection, ticking it selects the
pick and closes the menu, unticking goes back to all;
- pill layout: the picks lead the same bar, set off from the single
regions by a thin divider.
- A tap selects the group's codes that have an observer and re-queries
straight away.
- The selector then reads the group's name ("South Bay ▾") instead of "2
Regions", and the button shows as pressed. Selecting the same codes by
hand does the same.
  - Tapping the active group again goes back to all regions.
- A failed fetch, or no configured groups, leaves the filter exactly as
before.
- **Mobile:** the rows take the menu's 44 px row height under `(pointer:
coarse)`; the pill-bar picks reuse `.region-pill` and its touch target.
Inside `.filter-bar` (Packets) the existing touch rule for inputs made
every region checkbox a 44 px square; the checkbox is now 20 px and the
44 px row is the tap target.
- **Semantics:** like every region filter, a group means "heard by
observers in these regions", not "located there"; `config.example.json`
says so.

No existing issue covers this.

## Screenshot

Nodes page with two picks configured (example config below), after
tapping **South Bay**: the pick and its codes (SJC, MRY) are selected,
and the list re-queried for them. Taken against
`test-fixtures/e2e-fixture.db`.

![Region quick picks on the Nodes
page](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/region-quick-picks.png)

## Config example

```json
"regionQuickPicks": [
  { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] },
  { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] }
]
```

Each pick needs a `name` and at least one code; `description` is
optional (shown as the tooltip). Codes are trimmed, upper-cased and
de-duplicated; a pick with no code that has an observer today is not
shown. At most 20 picks of 200 codes.

## Browser validation

The branch's server was run locally against a migrated copy of
`test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with
three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North
Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless
Chromium:

| check | desktop 1280 px | phone 390 px (touch) |
|---|---|---|
| groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) |
same |
| tap South Bay selects | SJC, MRY (SCZ has no observer) | same |
| next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` |
| selector label | "South Bay ▾" | "South Bay ▾" |
| second tap | back to all regions | same |
| button height | 24 px | 44 px |
| page wider than viewport | no | no |
| page errors | none | none |

Checked on Packets (dropdown layout) and Nodes (pill layout). Three
layout problems found this way on the phone were fixed before opening: a
sideways-scrolling row clipped at the screen edge, the selector squeezed
into a narrow column (the filter group is a non-wrapping flex row), and
the two wrapped rows spread apart on Nodes (`align-content`).

**Revision 1:** the first version put the picks in a separate row beside
the selector; they moved inside the dropdown. Re-checked with a
14-region list and three picks on Packets: desktop 1280 px dark and
phone 390 px light both keep the menu on screen (menu 340 px wide, page
not wider than the viewport), the pick closes the menu and the trigger
names the pick.

**Revision 2:** picks are checkbox rows like **All** instead of chips.
Re-checked on Packets at 1280 px dark and 390 px light: rows render bold
under All, the divider separates the single regions, ticking a pick
ticks its codes and names the trigger, no sideways scroll.

Unrelated and not changed: on Nodes at 390 px the region row (existing
pills included) starts flush with the screen edge.

## Tests

- Red commit `81ea67d6`: the config type, a stub normaliser and endpoint
returning nothing, and tests.
- Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups),
caps, nil config, and the endpoint's JSON.
- Frontend, `tests/e2e/test-region-quick-picks.js` (moved from
`tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job
like `test-table-sort.js`): the real `region-filter.js` in **jsdom**
against faked config endpoints. Nine cases: rendering and hiding groups
with no observer, selecting and re-querying, the selector label,
toggling back, a hand-made selection marked active, the pill layout, no
groups, a failed fetch, HTML escaping. Seven fail on the stub, all on
assertions.
- Green commit `0e33210e`: the implementation, styles,
`config.example.json` (`regionQuickPicks` +
`_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route
list (`TestOpenAPICompleteness` requires it). The jsdom test also had a
cross-realm comparison bug fixed in this commit (`Array.from` on arrays
coming out of the jsdom window).
- `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the
one failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails the same way on `master` locally.

- Red commit `f991060d` (revision): picks live inside the dropdown menu
before the checkboxes, choosing one closes the menu, and in the pill
layout they share the region bar; two fail on assertions against the old
layout. Green commit `1f720e35`. Frontend 195 of 196 (same
environment-only failure).

- Red `134f5256` / green `da0adb50` (second revision): picks are
checkbox rows between All and the single regions with a divider, no
chips in the menu; ticking and unticking a pick row. The green commit
also repairs `style.css`: `1f720e35` had prepended its block to the top
of the file (its edit matched an earlier `.region-pill {`) and left the
previous rules in place. The file now differs from `master` by 13 lines.

## Review notes

- Groups are offered only when at least one of their codes appears in
`/api/config/regions`, and only those codes are selected. The stored
selection is pruned to known codes on load anyway, so selecting absent
codes would not survive a reload.
- Customizer (AGENTS.md rule 8): groups are operator configuration; a
customizer editor could come later.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `b84ace4e`:
- `normalizeQuickPickCodes` split out of the config normaliser (S3776,
17 > 15);
  - a comment on why a failed quick-pick fetch is ignored (S2486);
  - `for…of` in `activePick` (S4138);
- in the tests: `node:` specifiers, `includes()`, and an explicit string
comparator for sorted selections (S7772, S7765, S2871).
- **Kept on purpose:**
- S2486 is still reported on that `catch`. Swallowing the error is the
intended behaviour: the filter works as before, just without picks.
  - S1523 on the `vm` test loader, as above.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 21:23:05 +02:00
efitenandClaude Opus 5.5 f3004d977e feat(rx-coverage): show where a companion drove and received nothing (#2172) 
The Mobile RX coverage page (`#/rx-coverage`) shows only cells where a
companion received something. Where a companion drove and received
nothing is just as useful for operators, and the data is already there:
the RF sample track (`client_rf_samples`, #1905/#1906) and the
receptions (`client_receptions`) use the same hex grid (`hexCellAt`,
`zoomToHexRes`). This PR adds those cells as a hatched layer under the
reception cells.

## Changes in this PR

**Server**
- `GET /api/rx-coverage?...&gaps=1` adds a `gaps` member to the
FeatureCollection: cells with moving (non-stationary) track samples and
no reception in the same `days` window and `bbox`, each with `samples`.
`gaps_truncated` is set when more than 5000 cells qualify (the
most-sampled are kept, same pattern as `coverageFeatureCap` and
`rfNoiseFeatureCap`).
- Gaps are only returned when `clientRfSamples` is on and there is no
`node` filter: with `?node=` the question would be "this node was not
heard", which these cells do not answer. `?rx=` limits both the track
and the receptions to that companion.
- Without `gaps=1` the payload is unchanged: the `gaps` member is
omitted.
- New code lives in `cmd/server/rx_coverage_gaps.go` (pure
`aggregateGaps` plus `queryTrackPoints`); the handler change in
`rx_dashboard.go` is 14 lines.
- `/api/rx-coverage` is now documented in `openapi.go` and removed from
`openapi_known_gaps.json` (#1670).

**Frontend** (`public/rx-coverage.js`, `public/node-reach-coverage.css`)
- A "Nothing received" toggle on the signal layer, on by default, only
rendered with `MC_CLIENT_RF_SAMPLES`. Hidden on the noise layer.
- Gap cells use a 45-degree hatch (an SVG pattern in a dedicated
renderer, so they sit under the reception cells). The hatch is denser
for cells driven through more often (1 / 2-3 / 4+ samples). Tooltip:
sample count. Legend entry: "driven, nothing received".
- Off state is deep-linkable as `gaps=0`.
- New token `--nq-cov-gap` for light and dark. Plain grey
(`--nq-cov-grey`) already means "received, no SNR", so the hatch keeps
the two apart.

## Threshold: one sample

Measured on a live network (analyzer.on8ar.eu, Belgium, September to
October 2026), with cells of about 250 m: a cell with track samples and
no reception in one half of a month, driven again in the other half. How
often did it receive something the second time?

| Minimum samples | Cells "nothing received" | Driven again | Received
something later |
|---|---|---|---|
| 1 | 3971 | 1263 | 29% |
| 2 | 1565 | 592 | 32% |
| 3 | 619 | 243 | 35% |
| 5 | 142 | 63 | 40% |

A higher threshold hides cells without making the rest more reliable, so
the threshold is 1 and the wording is "nothing received in this period",
never "no coverage". Using every decoded packet as reception evidence
(including 1-byte path hashes) gave 25% to 32%, so the capture rule for
1-byte hashes is not what drives this. The median cell has 2 samples.

## Verification

- Go: 7 new tests in `rx_coverage_gaps_test.go` (gap/no gap, stationary
excluded, single sample, cap keeps the most-sampled, endpoint, `?rx=`
filter, no `gaps` member when not asked / with `?node=` / with RF
samples off). The endpoint tests fail without the handler change. Full
`cmd/server` suite green locally, `go vet` and `gofmt` clean.
- E2E: `tests/e2e/test-rx-coverage-gaps-e2e.js` (4 steps: no toggle with
RF samples off, on by default with hatched cells under the reception
cells, toggle off with `gaps=0` and deep link, noise layer hides it). 2
steps fail without the frontend change. Existing
`test-rx-coverage-noise-e2e.js` and `test-rx-coverage-viewport-e2e.js`
still pass. Listed in `scripts/non-unit-tests.json`.
- Frontend unit suite (`test-all.sh`) green.
- Real data on a staging instance (7 days, bbox 50.8,4.9,51.3,5.8,
z=12): 478 reception cells and 379 gap cells, no cell in both, reception
features identical with and without `gaps=1`, 0.48 s to 0.52 s per
request both ways.
- Browser: checked on staging with real data (light theme), and light,
dark and 390 px wide with mocked data.

## Not in this PR

- A per-node variant on the Reach page ("this repeater transmitted while
a companion was nearby, and was not heard"). That needs the transmit
moments from `observations` and is a heavier query.
- `deploy.yml` does not get a line for the new E2E file, like the other
entries that are only in `non-unit-tests.json` (#2037).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 17:39:33 +02:00
efitenandClaude Opus 5.5 d232072f85 feat: optional user accounts (part A: foundation) (#2129) 
Part A of #2128: optional, off-by-default user accounts. With the
feature off nothing changes; with it on, visitors can register and log
in, and admins manage users and use the operator actions without the API
key.

PR #2130 (settings sync) builds on this one. The two are meant to be
merged together.

## The situation

- Operator actions (geofilter save and prune, backup, perf reset) need
the shared `apiKey`. There is no per-person right.
- Nothing in CoreScope knows who a visitor is, so the requests in #2128
that need that (#1835, #2092, #1508, #730) have nothing to build on.

## What this PR adds

**Two new Go modules**
- `internal/users`: a separate `users.db` (SQLite through
`modernc.org/sqlite`) with users, sessions, single-use tokens, an audit
log and a mail log. Passwords use argon2id.
- `internal/mailer`: a `Mailer` interface with a Brevo client (send,
delivery events, webhook parsing) and an in-memory fake for tests.

**Server (`cmd/server`)**, active only with `userManagement.enabled`
- 24 routes, all documented in OpenAPI under the `users` tag
([`auth_routes.go`](https://github.com/efiten/CoreScope/blob/feat/user-management/cmd/server/auth_routes.go)):
  - auth: register, activate, login, logout, me, forgot, reset;
- account: profile, password, email change with confirmation, sessions,
self-delete;
- admin: list, detail, disable, enable, delete, role, resend activation,
manual activation, mail status refresh;
  - a Brevo webhook, registered only when `mail.webhookSecret` is set.
- `requireAdmin` replaces `requireAPIKey` at the 7 operator call sites:
the API key **or** an admin session. With the feature off it is the old
API-key gate (`TestRequireAdminWithoutUserManagementIsAPIKeyGate`).
- `/api/config/client` gets `userManagement: {enabled: true}` only when
the service started; with the feature off the response is
byte-identical.

**Frontend**
- `auth.js` (header account control, request helper that adds the CSRF
header), `account.js` (login, register, activate, forgot, reset, confirm
email, my account), `admin-users.js` (`#/admin/users`, deep-linked
filters), `account.css` (theme tokens only).
- On phones the top-bar control is hidden, so a conditional entry goes
into the bottom-nav "More" sheet and the nav drawer.
- The customizer geofilter tab and the Perf "Reset stats" button use the
admin session when there is one.

**Config.** A `userManagement` block (`config.example.json`,
[`docs/user-guide/accounts.md`](https://github.com/efiten/CoreScope/blob/feat/user-management/docs/user-guide/accounts.md)).
The Brevo key can come from `CORESCOPE_BREVO_API_KEY`. The server
refuses to start when the block is enabled but incomplete.

## Security choices

- Session cookie `cs_session`: HttpOnly, SameSite=Lax, Secure when
`publicBaseUrl` is https. Every cookie-authenticated state change needs
the `X-CS-CSRF` header and a matching Origin.
- Activation needs the token **and** the account password. Without the
password, an attacker who keeps re-registering a known address could get
the owner to activate an account that carries the attacker's password.
- Register, forgot and email change answer identically for known and
unknown addresses. A password reset ends all sessions, a password change
ends all other sessions, and both end outstanding email-change links.
- Rate limits: login 10 per 15 minutes, register and forgot 5 per hour,
per IP and per address. The bucket count is capped. `trustedProxies`
makes the per-IP limits see real client IPs behind a proxy.
- Server logs carry `#<user id>`, never addresses, tokens or passwords;
mail-provider error texts are redacted before logging.

## Performance

No change to an existing hot path with the feature off. With it on:
- One `users.db` lookup per authenticated request (session by token
hash).
- The admin user table rebuilds its `tbody` on each filter change.
`users.List` caps the result at 1000 rows (`internal/users/users.go`),
which bounds the rebuild.
- `map[string]interface{}` in `openapi.go`: 79 before, 78 after.

## Verification

- `internal/users`, `internal/mailer` and `cmd/server`: `go vet` and `go
test -race` pass locally. 121 new Go tests.
- `cmd/server` with `-tags e2etest`: vet and the e2e hook tests pass.
- `sh test-all.sh` exits 0. `tests/unit/test-user-management-ui.js`: 67
passing (vm, real modules).
- `tests/e2e/test-user-management-e2e.js` (6 steps) passed locally
against an `e2etest` build with the fake mailer and against a
feature-off build. CI builds the `e2etest` binary and runs the suite on
a second server (`deploy.yml`).
- On a staging instance with a real Brevo key: register, activation mail
delivered, activate, admin table, "Refresh status" showing sent,
deferred, delivered, opened and clicked.

## Not in this PR

- Settings sync (#2130), the admin dashboard, approval flows and
notifications (parts B to E of #2128).
- A `requireReadAuth` mode (#1835). Sessions from this PR are what such
a mode would accept.
- Binary size and build time with `modernc.org/sqlite` linked next to
`mattn/go-sqlite3` were not measured. Their driver names do not collide.
#1992 discusses the driver choice.
- No Brevo webhook was configured on staging; delivery status there came
from "Refresh status".

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-07 09:25:29 +02:00
efitenandClaude Opus 5.5 000d9ab030 feat(coverage): RF noise-floor layer on the Mobile RX coverage page (#2113) 
The ingestor already stores the noise floor that CoreDrive RX companions
report with each GPS fix (`client_rf_samples`, opt-in through
`clientRfSamples`, `cmd/ingestor/client_rf_sample.go:62`), but nothing
reads it back. An operator who enables it collects the data and cannot
see it. This adds the read side: `GET /api/rf-noise` and a Signal/Noise
toggle on the Mobile RX coverage page.

## What it does

- **`GET /api/rf-noise?bbox=&z=&days=`** returns a GeoJSON hex grid with
the median, quietest and noisiest noise floor per cell, in the same
shape as `/api/rx-coverage`. It is registered always and 404s unless
`clientRfSamples.enabled` is true, the same pattern as the coverage
routes.
- **Stationary samples are excluded.** A parked companion logs hundreds
of readings at one point, which would otherwise define its cell.
- **Coverage page:** a Signal/Noise toggle, rendered only when
`/api/config/client` reports `clientRfSamples: true`. The noise layer
reuses the coverage colour tokens with the axis inverted, because a
lower dBm is quieter. Tiers are at -115 and -108 dBm.
- **Deep link:** `#/rx-coverage?layer=noise` opens on the noise layer.
- **Empty and failed answers** are labelled on the map ("No RF samples
in this view yet", or a retry hint), so a blank map never reads as
"feature off".

No new configuration key: it reads the existing `clientRfSamples`
section that the ingestor already uses. Default off, so nothing changes
for an instance that has not opted in.

## Where it comes from

Ported from the efiten/CoreScope fork, where it has run on
analyzer.on8ar.eu since September (fork commits `42d09f0e`, `80581c43`,
`cde95078`). The cherry-picks conflicted with upstream's newer
`routes.go`, `types.go` and `rx-coverage.js`, so the final state was
ported by hand. The fork-only `/scopes` route that sat next to it in the
same hunk is deliberately left out.

## Performance

The query is bounded by `sampled_at` (indexed, `idx_crf_prune`) and the
bbox, aggregation is one pass plus a per-cell sort, and the response is
capped at 5000 cells. Measured on analyzer.on8ar.eu, all of Belgium
(`bbox=49.4,2.4,51.6,6.5&z=9`), from a client in Belgium, so network
time included:

| window | samples | cells | response time |
|---|---|---|---|
| 7 days | 10,835 | 241 | 0.37 s |
| 30 days | 43,303 | 429 | 0.39 s |

That table holds 45,270 rows in total. It is only read when someone
opens the noise layer, never on ingest or WebSocket paths.

## Tests

- `cmd/server/rf_noise_test.go`: 8 tests for the aggregation (median and
extremes, stationary exclusion, the cell cap, empty input) and the gate
(404 when off, even with data present). All pass, and the full
`cmd/server` suite passes locally.
- `tests/unit/test-rx-coverage-noise.js` (new, in `test-all.sh`): the
colour axis runs the right way, including both tier bounds and a string
median from the API. Slices the real function out of `rx-coverage.js`
and fails on master's copy, which has no thresholds.
- `tests/e2e/test-rx-coverage-noise-e2e.js` (new, wired in `deploy.yml`
and `scripts/non-unit-tests.json`): no toggle when the flag is off;
Noise fetches `/api/rf-noise`, draws the cells, swaps legend and
subtitle, puts `layer=noise` in the hash; a deep link opens on the noise
layer and an empty answer shows the message. 3/3 locally; against
master's `rx-coverage.js` and `roles.js`, 1/3 (only the "flag off" case
passes, as it should).
- `test-rx-coverage-viewport-e2e.js` still passes. ESLint 8 reports 0
errors on the changed files.

## Not done

- The thresholds (-115 / -108 dBm) are fitted to the fork's own data
(1,241 moving samples at the time) and are constants in
`rx-coverage.js`. Per AGENTS.md rule 8 they belong in the customizer
eventually; not in this PR.
- The E2E suite mocks the API. The real endpoint is covered by the Go
tests and by the measurements above, not by a fixture with seeded
samples.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 22:11:34 +02:00
Sylvain Rabot a4a447acea fix(packets): pack short columns, one expand arrow, Full Names toggle (#2090) 
The packets table wastes most of its width on short columns, draws two
arrows on grouped rows, and cuts every observer and hop name short. This
fixes all three and adds a way to read the full path when it does not
fit.

## Why the columns were wide

`makeColumnsResizable` measured each column once on load and stored the
widths as **percentages** of the table. Percentages grow with the
screen: on a 2000px display "17s ago" sat in a 180px column. The
measurement also read the virtual-scroll spacer row — a single `colspan`
cell as wide as the table — as column 0, so the expand column came out
160px wide locally and about 950px on a live instance with grouped rows.

Measured on the fixture at 2000px:

| Column | Before | After |
|---|---|---|
| Expand | 162–230px | 32px |
| Time | 111px | 67px |
| Size | 64px | 46px |
| HB | 44px | 29px |
| Scope | 80px | 48px |
| Path / Details | ~180px each | ~650px each |

## The change

**Column packing.** A new `fitColumnsToContent` (`app.js`) sizes every
column except Path and Details to its widest cell, in px; those two
split what is left. Every visible column gets an explicit width and the
table gets their sum: a spacer row keeps a slot for each hidden column,
and leaving any column `auto` hands those slots a share of the slack as
a blank strip at the right edge.

When space is short (detail panel open, narrow window) the widest fixed
columns give way first, then Path and Details shrink to 60px, and only
then does the wrapper scroll. The split itself is a pure function,
`distributeColumnWidths`, with unit tests.

It re-lays out when the container resizes (detail panel), when
TableResponsive hides or reveals columns (a new `table-columns-changed`
event), and after sorting or a column toggle. After each render `grow()`
widens columns for the rows just inserted and never narrows them, so
nothing jitters while scrolling. Drag handles store px; double-clicking
one resets it. Path and Details have no handle.

`makeColumnsResizable`, still used by the nodes, observers and analytics
tables, now skips `colspan` rows when measuring; other rows are measured
as before.

**One expand arrow.** A CSS `::before` "▶" sat next to the SVG caret,
which pointed up when collapsed. The glyph is gone and the caret points
right, then down.

**Full Names.** A toolbar toggle next to Hex Paths shows observer and
hop names untruncated. It is saved per browser; a link can carry
`fullNames=1`, which applies to that page without overwriting the
visitor's saved preference. Paths stay on one line: the virtual scroller
positions rows at one fixed height, so wrapping would make scrolling
jump.

**The "+N" path pill, and a full-path popover.** Hops past the column
edge were meant to go behind a "+N" pill (#1124), but the pill was
appended after the overflowing chips and so sat past the clipped edge:
it never showed. It is now sticky at the right edge, opaque on hover,
and counts hop chips only (it used to count warning icons too). Hovering
or focusing it shows the full path, one hop per line and numbered;
click, Enter or Space pins it until an outside click or Escape, and it
closes when the page changes or its row scrolls away. Clicking the pill
no longer selects the row underneath.

**Two related fixes.**
- Rows drawn before `/api/observers` resolves show raw 64-char pubkeys.
`loadObservers` now re-renders and re-measures, instead of leaving
Observer at the pubkey's width (~600px under Full Names).
- Group child rows indented every cell by 20px, which misaligned them
with the headers and widened every column on expand. Only the Time cell
is indented now.

Anyone who had dragged packet columns gets the new defaults once: the
old percentage key is dropped.

## Performance

`grow()` runs after every `renderVisibleRows`, so it is on the hot path.
Measured in Chromium on the fixture, 300 iterations, three runs:

| Case | Cost |
|---|---|
| Full render, 63 rows measured | +0.88–0.93ms over the forced reflow
alone |
| Scroll step (incremental render) | ~73 `Range` rect reads, only the
inserted rows |
| Large jump / re-render | ~569 rect reads |

Before narrowing `grow()` to inserted rows, one scroll step with 92
rendered rows made 5,472 rect reads. The DOM row count is bounded by the
virtual-scroll window, so the cost does not grow with the packet count.
`distributeColumnWidths` is a binary search over at most ~10 columns.

## Tests

- `tests/e2e/test-packets-compact-columns-e2e.js`, 16 cases: short
columns stay under fixed px bounds at 1920px; Path and Details take the
freed width; hiding both still fills the wrapper; no blank strip and no
horizontal scroll, with and without the detail panel, at 1920, 1100,
1000 and 375px; responsive reveal and re-hide refit; stale percentage
widths dropped; drag follows the pointer and double-click resets;
expanding a group leaves widths alone; Full Names untruncates, keeps
paths on one line, persists and round-trips through the URL; the
Observer column shrinks back when observer names arrive late; the pill
popover lists every hop vertically, closes when the pointer leaves, pins
on click without selecting the row, closes on Escape and on navigation,
and opens on keyboard focus. Registered in `scripts/non-unit-tests.json`
and `deploy.yml`.
- On `master` the suite fails where it should, and restoring each fix's
old behaviour (child-row indent, the late-observer refit) turns its case
red.
- `tests/unit/test-packets.js`: `distributeColumnWidths` (room to spare,
rounding, widest-first capping, floors, dragged columns, the 60px
minimum, overflow), the single chevron and no `::before` glyph, Full
Names on flat and grouped observer cells, and the Full Names CSS.
- `tests/unit/test-xss-escape-sinks.js` now builds the real
`obsCellName` from source: escaping moved into it. Removing its
`escapeHtml` turns both observer-cell tests red.
- `test-issue-1189-composed-cell.js` and
`test-issue-2012-clear-filters-selection.js` evaluate packets.js
fragments with injected helpers and needed `obsCellName` /
`showFullNames` added.
- Existing packets E2E suites pass on this branch and on master against
a CI-prepared fixture: scope column, #1128 (layout and multi-viewport),
#1122, #1188, #1648, table sort, #1692, #1486.
- Unit: 182 of 183 suites pass locally;
`test-issue-1956-release-routing.js` could not create a temp directory
in my sandbox and is unrelated.

## Browser validation

Checked in Chromium against the fixture at 2000, 1920, 1280 (panel open
and closed), 1100 and 375px, and with the seeded grouped row expanded
and collapsed. Screenshots taken at each step; the table fills its
container exactly, grouped rows show one arrow, and the pill popover
lists full names.

## Not done, deliberately

- **No width cap on Observer under Full Names.** A cap brings back the
truncation the toggle removes, and Observer is already the first column
to give way when space is short.
- **Customizer.** The floors (60px flex minimum, 64px shrink floor, 32px
expand, 58px Rpt) are hardcoded, per AGENTS rule 8; exposing them in the
customizer can follow.
- **Wrapping paths.** Rejected because of the fixed row height above;
the pill popover is the way to read a long path.
 2026-10-03 12:15:52 +02:00
efitenandClaude Opus 5 25f8426d32 test: make every suite in tests/e2e run, and tell the truth when it does (#2053) 
Closes #2037.

Step 1 (#2045) wired in the nine suites that already passed. This is
steps 2 and 3: the four that ran and failed, and the five that could not
run at all. After this, the count of suites in `tests/e2e` invoked by
nothing goes from 18 to 0.

## Step 2 — the four that ran and failed

Triaged against a fixture server in CI with full output kept, not the
four-line tail the first probe saved.

| suite | verdict |
|---|---|
| `test-packets-scope-column.js` | passes on master today. It failed on
2026-09-18, so something between the two fixed it. Wired in unchanged
rather than investigated. |
| `test-node-reach-e2e.js` | test wrong. It waited for the reach map
whenever any *link* had GPS; `public/node-reach.js` builds the map only
when the *node* has coordinates. Guaranteed 10s timeout on a node with
positioned neighbours and no position of its own. |
| `test-channel-modal-e2e.js` | both failures test-side. The Add
button's visible label was shortened to "+ Add" with the accessible name
moved to `aria-label` (`public/channels.js:746`); and
`.ch-section-mychannels` is conditional on the visitor having added a
channel, which has not happened at that point in the suite. |
| `test-touch-targets.js` | three test-side, two a product finding. |

The three test-side touch-target failures were the harness measuring
controls that are not shown: `.compare-btn` (the CTA was removed in
#1646, and `style.css` says so), `.ch-back-btn` (`display:none` outside
the mobile channels layout), and `.filter-toggle-btn` (`display:none` on
mobile since #1461; the control shown is the navbar mirror, which
`mobile-page-actions.js:70` builds as a `.nav-btn`, so it was already
measured). All three are dropped from the table with the reason recorded
in the file.

The remaining two are **not** a test problem: `.nav-btn` and
`.ch-icon-btn` are each declared twice in `public/style.css`, 48px in
the touch-target block and 44px in their own component rule, and the
later one wins. Rather than lower the blanket or hide the failures, the
suite now has `DEFAULT_MIN = 48` plus a `MIN_OVERRIDES` table holding
those two at their effective 44, so a third selector dropping to 44
still fails the build. The contradiction is **#2052**, with both ways
out costed; the override entries should go when it is settled.

## Step 3 — the five that could not run

None is deleted. I checked each selector and seam against the product
before deciding, and every one still targets a surface that exists and
that nothing else covers.

Four were written against `@playwright/test`, a runner the project
neither installs nor uses anywhere else. Adopting a second runner for
twelve tests costs more than porting them, and the precedent is already
set: `test-path-inspector-coverage-e2e.js` exists, as its own header
says, because `test-path-inspector-e2e.js` could not run. So they are
ported to the plain-node Chromium pattern the other 109 suites use.

- **`test-issue-1522-trace-url-sync-e2e.js`** — the trace hash in the
URL, both directions. `test-e2e-playwright.js` covers that the page
loads and searches; it never looks at the URL, which is the whole of
#1522.
- **`test-marker-outline-weight.js`** — the canvas pulse ring never
thins below 2px. There is no CSS rule to read and axe cannot see inside
a canvas, so sampling the seam is the only way. Added a guard that the
ring was actually visible, so the weight check cannot pass vacuously on
a pulse that never rendered.
- **`test-pr-1490-live-map-gpu-animations-e2e.js`** — the queue drains,
the engine sleeps again, the fading trails stay under the cap of 5, and
the canvas sits on `animationsPane` rather than under the markers.
- **`test-path-inspector-e2e.js`** — reduced to what nothing else
covers: the map side pane, the `/#/traces/<hash>` redirect, the tools
landing. Its standalone-page test duplicated the wired coverage suite
and is dropped. Its "switching candidate clears prior polyline" case
ended after the click with a comment and no assertion, which is the same
green-but-empty problem this issue is about; it now compares path
counts, and skips loudly when the fixture yields too few candidates.

The fifth, **`test-table-sort.js`**, needed `jsdom`, which was declared
nowhere. It is a unit test of `public/table-sort.js` filed under
`tests/e2e`, so: `jsdom` is a devDependency (lockfile updated, `npm ci`
stays consistent), the file moved to `tests/unit/`, and the
`domIntegration` group in `scripts/non-unit-tests.json` is gone with its
only member. It runs 22 tests. 20 passed immediately; 2 had rotted,
because #1648 M2 replaced the up/down glyphs with Phosphor sprites and
the direction moved out of `textContent` into the `<use href>`. Those
two now read the sprite ref and the `aria-sort` value, so they also
guard the accessible announcement.

## Verification

`tests/unit/test-table-sort.js` 22/22 and `test-test-inventory.js` pass
locally; the E2E suites need a fixture server, which I cannot build here
(no cgo toolchain since #1992), so CI is their first run as committed.
The triage above was measured in CI, not assumed.

## Not done

The per-assertion skips named in the second comment on #2037 are
untouched: the two flaky packet-detail cases, the fixture-data ones, and
the two `clientRxCoverage` suites that skip wholesale while reporting
success. Those need a fixture deployment with coverage enabled, which is
its own change. I have not opened it.

Option 2 from the issue, making `test-test-inventory.js` require a
`deploy.yml` line for every `tests/e2e` file, is also not here. It is
the right guard and it is now enforceable, since the list is finally at
zero, but it belongs in its own change where a red build means what it
says.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-22 11:45:11 +02:00
efitenandClaude Opus 5 5c016a210a fix(analytics): show a building state on the distance index's 202, and stop caching it (#2051) 
Closes #1997.

## What was wrong

`/api/analytics/distance` answers `202 {status:"building",
retry_after_seconds:5}` with no `summary` until the lazy index (#1011)
has been built.

1. `renderDistanceTab` read `data.summary.totalHops` straight away. The
TypeError is caught by the tab's own try/catch, so it never reaches
`window.onerror`: it is painted into the tab as `Failed to load distance
analytics: Cannot read properties of undefined (reading 'totalHops')`.
2. `api()` caches any `res.ok` body, and `res.ok` is true for 202, so
the placeholder was stored for the `analyticsRF` TTL. Even a correct
retry read the cached "building" body back. That is the half that made
the broken state outlast the index build.

## What changed

- `public/app.js:173` skips the cache write when `res.status === 202`. A
200 still caches, unchanged.
- `public/analytics.js` renders a building notice and retries itself,
honouring `retry_after_seconds` clamped to [1s, 30s]. The timer is
cleared on tab switch and in `destroy()`, and a new render supersedes a
pending retry, so two renders cannot write into the same tab.

The notice uses `.text-center`/`.text-muted` rather than the `.spinner`
class used at `analytics.js:1683`, because `.spinner` has no CSS
anywhere in the repo and renders nothing.

## Tests

`tests/unit/test-issue-1997-distance-building.js` (7 assertions, wired
into `test-all.sh`) pins the two pure decisions the renderer makes and
`api()`'s refusal to cache a 202 while still caching a 200. Red-run on
the unfixed sources: 6 of 7 fail, and the "a 200 is still cached"
control stays green.

`tests/e2e/test-issue-1997-distance-building-e2e.js` (classified in
`scripts/non-unit-tests.json`, invoked from `deploy.yml` with
`CHROMIUM_REQUIRE=1`) serves both responses by route interception, so it
does not depend on whether the server under test has an index built. It
asserts: the building state appears, the tab does not paint the error
text, a retry arrives with no interaction, the retry replaces the
placeholder once the server answers 200, and no retry fires after
leaving the tab.

Check (2) deliberately asserts on the rendered text and not on
`pageerror`: the TypeError is caught, so a `pageerror` assertion would
pass on the broken build too.

## Verification

No local cgo toolchain here since #1992, so I could not build a server
to run the E2E against. Instead I ran its five steps in Playwright
against a live instance with this branch's `public/app.js` and
`public/analytics.js` injected in place of the deployed ones (both files
are byte-identical between that instance and upstream master, so the
injection is faithful):

| check | deployed build | this branch |
|---|---|---|
| (1) building state shown | fail | pass |
| (2) not rendered as data | fail | pass |
| (3) retried on its own | fail (1 request) | pass (2 requests) |
| (4) real payload after retry | fail | pass |
| (5) no retry after leaving the tab | pass | pass |

(5) passes on the broken build too: it schedules no retry at all, so it
is a control and only means anything together with (3).

The committed E2E suite has not been run as committed. CI is its first
real run.

## Not done

- The server still recomputes on every 202 poll rather than signalling
readiness.
- No other analytics tab was audited for the same assume-a-summary
pattern.
- One pre-existing unit suite (`test-preflight-xss-gate.js`) fails on
this Windows machine with a cp1252 `UnicodeEncodeError` from its Python
helper, on a clean tree as well as with this change. Unrelated, and
green on Linux CI.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-20 17:04:37 +02:00
liquidraver 84eba41592 fix(live): anchor the legend toggle to .live-page so the VCR bar stops eating its clicks (#2049) 
The PACKET TYPES legend could not be dismissed on Live: clicking the palette toggle did nothing, while document.querySelector('#legendToggleBtn').click() from the console worked. That asymmetry was the diagnosis, since a synthesised click skips hit testing.

Residual half of #1833. That fix corrected the button's offset to calc(var(--vcr-bar-height) + 10px) but left the anchor: .legend-toggle-btn was position: fixed, so the offset resolved against the viewport, while the .vcr-bar it clears is absolute inside .live-page, whose height subtracts --bottom-nav-reserve (56px + safe-area at <=768). The two anchors disagreed by exactly the reserve, dropping the button into the bar's band, and .vcr-bar at z-index 1000 against the button's 500 took every real click. The window is 641-768px: below 641 both buttons are display:none, above 768 the reserve is 0, which is why neither the desktop nor the phone layout test saw it.

Both buttons are now position: absolute, resolving against the same containing block as the bar. .feed-show-btn carried the identical defect but missed the bar by 16px because --legend-toggle-stack parks it a row higher, so that change prevents a break rather than repairs one.

Three review rounds, each verified rather than argued:
- The new E2E was classified in non-unit-tests.json but had no deploy.yml line, so it would never have run (#2037). Now wired in, and it covers both buttons.
- The '.feed-show-btn is display:none at <=768' claim was wrong (it is <=640), which the author measured and corrected.
- Its first real execution failed on its own setup: getComputedStyle().getPropertyValue() on an unregistered custom property returns the unevaluated declaration ('calc(56px + 0px)'), so the non-zero check could not work. It now asserts the measured gap instead, and deliberately does not pin 56 because the gap measures 58 on CI.

Final run: 14 passed, 0 failed. The discriminating assertion is (f), each button's offset net of the bar being identical at 720 and 1440 (11px and 123px slack); under the old anchor those differ by the reserve. (g) pins the desktop layout unmoved.

Merged by the interim maintainer without a second human reviewer: CI and the review above are the independent checks.

Fixes #1833
 2026-09-20 14:00:45 +02:00
liquidraver bbf54cfe1b fix(rx-coverage): open at the configured map default, with its own saved viewport (#2033) 
The coverage page opened at a hardcoded [51.0, 4.8] zoom 8 regardless of deployment, ignoring /api/config/map (#2032). It now follows the same precedence as the main map (URL hash, then saved position, then /api/config/map, then [37.6, -122.1] zoom 9) and persists its own position across visits, syncing lat/lon/zoom into the hash so a view is shareable.

The saved position lives under its own key, rx-coverage-view, and the page never reads or writes the main map's map-view: sharing the configured default was the bug, sharing the session position was not. Both suites assert map-view stays untouched after a pan, so reintroducing a shared write fails instead of passing quietly.

Also fixed here: selectedRx is now percent-encoded into the hash, and a generation counter stops a late /api/config/map response or a stale 150ms layout timer from building a map for a page that was already left.

Reviewed twice. Verified by mutation rather than by reading: writing map-view too, ignoring the saved coverage position, and dropping the /api/config/map fetch each make the unit suite exit 1, so it covers the feature, the fix and the rejected alternative. The deploy.yml invocation was confirmed to land inside Run Playwright E2E tests (fail-fast) by parsing the workflow, and CI run 35261689694 is the E2E suite's first real execution: Go prints 'RX coverage viewport regressions OK' and Playwright prints 'RX coverage viewport browser regressions OK'.

Worth recording for the next reviewer: that E2E asserted localStorage.getItem('map-view'), so wiring it into deploy.yml without updating the assertion would have turned the job red on its first ever run. It was registered in scripts/non-unit-tests.json but invoked by nothing, the gap tracked as #2037.

Merged by the interim maintainer without a second human reviewer: CI and the mutation checks above are the independent checks.

Fixes #2032
 2026-09-17 22:02:36 +02:00
Alex B 893773338e chore(tests): move root test-*.js into tests/unit and tests/e2e (#2036) 
Moves 290 root test-*.js into tests/unit (177, listed in test-all.sh) and tests/e2e (113, classified in scripts/non-unit-tests.json), per #1981 and PR-D of #1385. Root goes from 348 entries to 48. test-all.sh and test-fixtures/ stay put. The inventory guard now fails if a test reappears in the root or sits in the wrong folder.

Verified independently of the diff: the invoked sets are unchanged (test-all.sh 177 before and after, deploy.yml 96 before and after, both identical as sets), and a full local run of test-all.sh on master and on the branch produced 4702 output lines each whose only differences are absolute paths, stack-trace line numbers shifted by the REPO_ROOT line, the inventory wording and two perf ratios. The guard was mutation-checked: a test back in the root, a unit suite in tests/e2e, and a suite dropped from test-all.sh each make it exit 1. CI run 35246304316 ran 97 suites from tests/e2e and is green.

Follow-up 9335c51d finished the instruction files: no bare root test command is left in AGENTS.md, the squad charters, .github or docs, and every tests/ path they name resolves.

Merged by the interim maintainer without a second human reviewer: CI and the local runs above are the independent checks.

Known and deliberately out of scope: 18 of the 113 files in tests/e2e are invoked by no runner at all, and one of them cannot run anywhere because it requires jsdom, which is not a declared dependency. Tracked separately.
 2026-09-17 19:06:07 +02:00
n30nex cd9b4c04d0 test: unify frontend test runs and prevent inventory drift (#1965) 
Make `test-all.sh` the authoritative standalone frontend runner for npm
and CI. Restore stale assertions and reject missing, duplicate, removed
or undocumented inventory entries.

Fixes #1858.

Rebased onto `a2f039d4`. Retains release-routing, map scope-state and
Scope Audit stylesheet tests, adds `test-packets-local-channels.js` to
the sorted runner, and classifies `test-neighbor-map-btn-clip-e2e.js`
under browser. Its separate CI browser step is preserved. Inventory: 280
root suites, 167 standalone, 113 requiring separate setup.

The icon repair fixes two suites red on master:
`test-issue-1648-m2-emoji-scan.js` and
`test-issue-1648-m6-final-sweep.js`. Node/live configured-scope
confirmations now use the existing accessible Phosphor check sprite.
Values, visibility conditions and scanner assertions are preserved.

- Red evidence: `89e45a9` inventory assertions; `4e255df` accessible
confirmation assertion. The latest rebase also reproduced both
unclassified-file failures before adding their entries. This follow-up
only changes runner/classification configuration and counts; no test
files modified.
- Local validation: all 167 standalone suites; 27 M2 browser checks and
16 neighbor geometry checks in Chromium. Syntax, whitespace, PII,
CSS-variable and XSS checks passed.
- Browser coverage includes populated/empty/null configured scopes in
node and live views.
- No new dependencies, requests, application settings or Go changes.
Workflow outside the unit step matches master.
- Windows validation uses process-local UTF-8 settings. Encoding and
node-reach confirmation follow-ups remain separate, as requested.

## Preflight override

External `run-all.sh` is unavailable; applicable repository checks were
run directly.
 2026-09-13 19:19:39 +02:00