diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index f207e5b7..f25ff206 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -806,9 +806,12 @@ jobs: # locally, reported success, and pushed nothing. # # That went unnoticed until v3.10.0, where the tagged commit was - # documentation-only. Documentation-only commits skip this workflow (see the - # paths-ignore above), so no :edge image was ever built for it, the label - # comparison failed, and the fallback ran for the first time. + # documentation-only. At that time a trigger-level paths-ignore skipped this + # workflow for such commits, so no :edge image was ever built for it, the label + # comparison failed, and the fallback ran for the first time. That filter is + # gone: the `changes` job now forces code=true for anything that is not a pull + # request, so every master commit gets an image and a documentation commit is + # safe to tag. See the comment on the `on:` block. build-and-publish: name: "🏗️ Build & Publish Docker Image" needs: [e2e-test, changes] diff --git a/CHANGELOG.md b/CHANGELOG.md index 1bc1459b..bc970c3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,31 @@ ## [Unreleased] +## [3.12.0] - 2026-09-26 + +See [docs/release-notes/v3.12.0.md](docs/release-notes/v3.12.0.md) for the full notes. 29 commits since v3.11.0: 15 fix, 5 test, 3 perf, 3 feat, 2 ci, 1 chore. + +### Highlights +- **The first start after this release runs `ANALYZE`, and ingest stalls while it does** (#2058) - measured at 3m43.9s on a 9.4 GB database, once per database, buffered with nothing dropped. It buys 25% fewer pages read on the region-filtered channel query (143,442 to 107,429). Set `db.analysisLimit` negative to skip it. **Operator awareness required**, and worth picking the moment on a busy instance. +- **`maxMemoryMB` eviction actually triggers** (#2035) - path, decode-cache and dedup-key bytes were unaccounted, so the measured footprint stayed under the configured limit and eviction never fired. Instances relying on that limit will evict where they previously grew. +- **Reception is attributed more honestly** (#2057, #2063, #2064) - zero-hop adverts count as direct reception, Heard By lists only observers that heard the node on air, and its empty state no longer claims the node is out of range. +- **Each observation carries its own wire bytes** (#1999) - the packet detail API handed every observation the same canonical frame. +- **CoreDrive RX region answers are stored** (#2047) - `node_declared_regions` was read but created by nothing, so the Scope Audit described a source that never arrived. Opt-in through `clientRegions`, kept with position, repeater clock and per-collector history. +- **Eight E2E suites had no runner at all** (#2045, #2053) - the aggregate reported a pass while skipping them. + +No manual migration step: `node_declared_regions` (#2047) and `sqlite_stat1` (#2058) are both created at boot. + +## [3.11.0] - 2026-09-16 + +Released and deployed, but it never received a changelog entry at the time; this one is written after the fact and there is no `docs/release-notes/v3.11.0.md`. 52 commits since v3.10.1: 27 fix, 16 feat, 4 test, 2 perf. + +### Highlights +- **Network-wide Scope Audit page** (#1976), with declared-region verification against a repeater's own traffic (#1990), unnameable-traffic accounting (#1987), and opt-in region keys derived from what nodes declare (#1989). +- **SQLite driver swapped** from `modernc.org/sqlite` to `mattn/go-sqlite3`, cross-built with zig (#1992). This makes the build cgo-dependent. +- **Map colours and filters repeaters by scope-configuration state** (#2006), and filters repeaters by region name (#1862). +- Analytics additions: retransmission pressure over time (#1699), scope adverts by node role (#1979), per-node hop-count statistics (#1812). +- The scope-match tally survives ingestor restarts (#2002). + ## [3.10.1] - 2026-09-04 v3.10.0 was tagged and withdrawn before any container image or release asset was published; nothing was ever available under that number. Same release, next number. diff --git a/docs/release-notes/v3.12.0.md b/docs/release-notes/v3.12.0.md new file mode 100644 index 00000000..30820eeb --- /dev/null +++ b/docs/release-notes/v3.12.0.md @@ -0,0 +1,159 @@ +# v3.12.0 + +29 commits since v3.11.0: 15 fixes, 5 test, 3 perf, 3 feat, 2 ci, 1 chore. + +## Read this before upgrading + +**The first start after this release runs `ANALYZE` on your database, and ingest +stalls while it does.** + +SQLite's query planner had no statistics at all, because `ANALYZE` had never run +against these databases and nothing in the code ever called it. Without +`sqlite_stat1` the planner guesses, and on the channel queries it guessed wrong: +it drove from the plain `idx_transmissions_payload_type` instead of +`idx_tx_channel_hash`, the partial index the schema already carries for exactly +that filter (#2058, reported with measurements by @anieto). + +The ingestor now builds those statistics, once, on the first start against a +database that has none. Measured on a 9.4 GB database with 1,250,489 +transmissions and 14,169,329 observations: + +| | | +|---|---| +| `ANALYZE` duration, cold page cache | **3m43.9s** | +| ingest during that time | stalled, buffered, nothing dropped | +| rows written in the minute it finished | 1027, against ~130 either side | + +It holds the single SQLite write connection throughout, so observations stop +arriving and queue in the ingest buffer, then land in one burst when it +completes. On the deployment measured, the ingest buffer absorbed all of it and +the log reported `0 dropped`. The log now warns before starting, so this is +distinguishable from a hang. + +This happens **once per database**, not once per restart: `sqlite_stat1` is an +ordinary table and stays in the file. Every later start costs one query. + +A deploy already restarts the service, so the stall overlaps an interruption you +were having anyway. It is still worth picking the moment on a busy instance. + +To skip it entirely, set `db.analysisLimit` to a negative number. The statistics +then never get built and the channel queries keep the old plan. + +What it buys, measured on the same database, region-filtered `GetChannels`: + +| | pages read | +|---|---| +| before | 143,442 | +| after | 107,429 | + +25% fewer pages, about 147 MB less per query at a 4 KB page. Wall time when the +OS page cache is warm does not change (0.80s either way); the gain is on the cold +path, which measured 56.7s on that file. The 3-4x improvement reported in #2058 +was measured on a smaller database (464,667 transmissions) and did not reproduce +at this size. + +`db.analysisLimit` defaults to 10000 rather than the 400 SQLite's documentation +suggests. 400 and 1000 were measured to leave the plan unchanged; 10000 produces +the same plan as an unbounded `ANALYZE` while costing 2.0s warm against that +one's 242.9s. + +## Features + +- **CoreDrive RX region answers are stored** (#2047) - `node_declared_regions` + was read by `region_keys.go` and `config.go` but created by nothing, so the + Scope Audit page described a source that never arrived: the client-topic switch + handled packets and rf only and dropped `/regions` without a log line. The + table is now created at boot, and the answer is kept with position, the + repeater's own clock, and per-collector history bounded by age rather than one + row per target. Opt in through `clientRegions`, separate from + `clientRxCoverage`, so region answers do not require GPS-tagged reception + uploads. Measured on the instance CoreDrive RX publishes to, 2026-08-18 to + 2026-09-19: 1840 answers about 275 repeaters from 51 collectors, of which 138 + repeaters have answers from more than one collector and 40 have collectors that + disagree, which is the disagreement the Scope Audit exists to surface. +- **The running version is in the nav drawer** (#2068) - read from + `/api/health`, with commit and build time in the tooltip. A version-less or + failing health endpoint leaves the plain label rather than "CoreScope + undefined". +- **The IATA whitelist says when it drops a region** (#2067) - one throttled + warning per rejected region, so a misconfigured `observerIATAWhitelist` is + visible instead of silent. + +## Reception and node health + +- **Zero-hop adverts count as direct reception** (#2064) - a node's own advert + with an empty path is credited to the observer that heard it, without needing + prefix resolution. +- **Heard By lists only observers that heard the node on air** (#2057) - a + relayed packet no longer implies its observer heard the originator directly. +- **The Heard By empty state stopped claiming the node is out of range** (#2063) + - no direct reception recorded is not the same as out of range. +- **The node-detail advert section is called Recent Adverts** (#2042, #2071) - + it never held anything else, because `transmissions.from_pubkey` is populated + for adverts only. The heading now says so in its tooltip instead of promising + every packet type. +- **Blacklist retention is queried by `from_pubkey`** (#2069). + +## Data correctness + +- **Each observation carries its own wire bytes** (#1999, #2055) - the packet + detail API was handing every observation the same canonical frame, so + per-observer differences were invisible. +- **Background chunks merge in order** (#2050) - they were prepended, which put + history out of sequence. +- **Pagination continues past a page that post-LIMIT filtering shortened** + (#2061) - a filtered page that came back short ended the walk early. +- **`maxMemoryMB` eviction actually triggers** (#2035) - path, decode-cache and + dedup-key bytes were not accounted, so the measured footprint sat below the + configured limit and eviction never fired. Instances relying on that limit + will now evict where they previously grew. +- **The distance index shows a building state on its 202** (#1997, #2051) - the + Analytics distance tab treated the lazy index's 202 as data and crashed, and + the 202 was being cached, so the tab stayed broken until the cache expired. + +## Interface + +- **Live legend toggle is clickable again** (#2049) - the VCR bar was eating its + clicks. +- **Map inspector controls stay clickable, stale loads are ignored** (#2048). +- **RX coverage opens at the configured map default** (#2033), with its own saved + viewport. +- **Scope Audit observed-region chips meet 4.5:1 in light theme** (#1996, #2070). + +## Performance + +- **Concurrent `GetChannels` cache misses coalesce** (#2059) - the second and + later callers wait for the first flight instead of each running the query. +- **Planner statistics** (#2058, #2072, #2074) - see the top of this file. + +## Tests, CI and release tooling + +- **Every suite in `tests/e2e` runs** (#2053, #2045) - eight suites had no runner + at all, and the aggregate reported a pass while skipping them. +- Each E2E suite is named in the log before it runs (#2046). +- Root `test-*.js` moved into `tests/unit` and `tests/e2e` (#2036). +- RX-coverage viewport assertions budgeted in pixels rather than a flat 0.001° + (#2040). +- The #1239 lock-hold threshold set from measurement, 150µs to 5ms (#2039) - the + old value sat inside the healthy band and failed at random. +- RF-sample fixtures anchored to now instead of a calendar date (#2034). +- Ingestor test stores wait for boot migrations before being handed over (#2065, + #2066) - this was a real data race under `-race`. +- Re-tagging `:edge` as a release keeps every platform (#2031) - v3.11.0's tags + were flattened to amd64 and had to be repaired. + +## Upgrade notes + +No manual migration step. #2047 adds the `node_declared_regions` table, created +at boot with `CREATE TABLE IF NOT EXISTS`, and #2058 creates `sqlite_stat1` the +same way; nothing else touches the schema. + +`clientRegions` is opt-in and off by default. + +The `ANALYZE` described at the top needs no action, only awareness. Set +`db.analysisLimit` to a negative number to disable it. + +One behaviour change worth repeating here: #2035 makes `maxMemoryMB` eviction +fire where it previously did not, because the footprint it compares against the +limit was undercounted. An instance that set the limit and never saw eviction +will start seeing it.