From 6d5686dbb449db6f4150398a9c6bde70d5511ee2 Mon Sep 17 00:00:00 2001 From: efiten Date: Sat, 26 Sep 2026 12:03:31 +0200 Subject: [PATCH] docs: release notes for v3.12.0, and the 3.11.0 entry that was never written (#2075) Prepares the v3.12.0 release. No tag is pushed by this PR: the release procedure requires the tagged commit to carry an `:edge` image whose revision label matches it, so that check belongs after this merges, against the merge commit. ## Why minor, not patch 29 commits since v3.11.0: 15 fix, 5 test, 3 perf, **3 feat**, 2 ci, 1 chore. The feats are #2047, #2067 and #2068. ## What the notes lead with Two changes alter what a running instance does without anyone asking it to, so they are at the top rather than in a list: - **#2058**: the first start against a database that has never been `ANALYZE`d builds planner statistics and stalls ingest while it does. Measured at 3m43.9s on 9.4 GB, once per database, buffered with nothing dropped. `db.analysisLimit` set negative skips it. - **#2035**: `maxMemoryMB` eviction now fires where it previously did not, because the footprint compared against the limit was undercounted. An instance that set the limit and never saw eviction will start seeing it. ## The 3.11.0 gap `CHANGELOG.md` had no entry for 3.11.0 and `[Unreleased]` was empty, so 52 shipped commits were undocumented. Added as a short entry that says outright it was written after the fact and has no notes file. The alternative of reconstructing 52 commits for a superseded version is error-prone work with little value, and leaving the gap silent is worse than naming it. ## One fix beyond documentation `deploy.yml` carried a comment that would mislead the next person cutting a release. It still said documentation-only commits skip the workflow "(see the `paths-ignore` above)", which is exactly how v3.10.0 lost its `:edge` image and then its tag name permanently. That filter no longer exists: the `changes` job forces `code=true` for anything that is not a pull request (lines 61-73), so every master commit gets an image and a documentation commit is safe to tag. The history stays in the comment; the false present tense does not. ## Verified rather than asserted Both went into the notes as upgrade advice, so both were checked in the tree: - `node_declared_regions` is created at boot with `CREATE TABLE IF NOT EXISTS` (`cmd/ingestor/db.go:435`), added by #2047, which found the table was read by `region_keys.go` and `config.go` and created by nothing. So "no manual migration step" is accurate. - The cgo dependency attributed to #1992 in the 3.11.0 entry is the one that stops this repo building with `CGO_ENABLED=0` today. ## Not done - No tag, no release. Next steps, after this merges: confirm the merge commit's pipeline is green, confirm `:edge`'s revision label is that commit, then tag `v3.12.0` annotated, push the tag only, dispatch `CI/CD Pipeline` on the tag ref, and verify the published digests in the registry rather than in a green job. - No `docs/release-notes/v3.11.0.md`, deliberately, as above. Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/deploy.yml | 9 +- CHANGELOG.md | 25 ++++++ docs/release-notes/v3.12.0.md | 159 ++++++++++++++++++++++++++++++++++ 3 files changed, 190 insertions(+), 3 deletions(-) create mode 100644 docs/release-notes/v3.12.0.md diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index f207e5b7..f25ff206 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -806,9 +806,12 @@ jobs: # locally, reported success, and pushed nothing. # # That went unnoticed until v3.10.0, where the tagged commit was - # documentation-only. Documentation-only commits skip this workflow (see the - # paths-ignore above), so no :edge image was ever built for it, the label - # comparison failed, and the fallback ran for the first time. + # documentation-only. At that time a trigger-level paths-ignore skipped this + # workflow for such commits, so no :edge image was ever built for it, the label + # comparison failed, and the fallback ran for the first time. That filter is + # gone: the `changes` job now forces code=true for anything that is not a pull + # request, so every master commit gets an image and a documentation commit is + # safe to tag. See the comment on the `on:` block. build-and-publish: name: "🏗️ Build & Publish Docker Image" needs: [e2e-test, changes] diff --git a/CHANGELOG.md b/CHANGELOG.md index 1bc1459b..bc970c3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,31 @@ ## [Unreleased] +## [3.12.0] - 2026-09-26 + +See [docs/release-notes/v3.12.0.md](docs/release-notes/v3.12.0.md) for the full notes. 29 commits since v3.11.0: 15 fix, 5 test, 3 perf, 3 feat, 2 ci, 1 chore. + +### Highlights +- **The first start after this release runs `ANALYZE`, and ingest stalls while it does** (#2058) - measured at 3m43.9s on a 9.4 GB database, once per database, buffered with nothing dropped. It buys 25% fewer pages read on the region-filtered channel query (143,442 to 107,429). Set `db.analysisLimit` negative to skip it. **Operator awareness required**, and worth picking the moment on a busy instance. +- **`maxMemoryMB` eviction actually triggers** (#2035) - path, decode-cache and dedup-key bytes were unaccounted, so the measured footprint stayed under the configured limit and eviction never fired. Instances relying on that limit will evict where they previously grew. +- **Reception is attributed more honestly** (#2057, #2063, #2064) - zero-hop adverts count as direct reception, Heard By lists only observers that heard the node on air, and its empty state no longer claims the node is out of range. +- **Each observation carries its own wire bytes** (#1999) - the packet detail API handed every observation the same canonical frame. +- **CoreDrive RX region answers are stored** (#2047) - `node_declared_regions` was read but created by nothing, so the Scope Audit described a source that never arrived. Opt-in through `clientRegions`, kept with position, repeater clock and per-collector history. +- **Eight E2E suites had no runner at all** (#2045, #2053) - the aggregate reported a pass while skipping them. + +No manual migration step: `node_declared_regions` (#2047) and `sqlite_stat1` (#2058) are both created at boot. + +## [3.11.0] - 2026-09-16 + +Released and deployed, but it never received a changelog entry at the time; this one is written after the fact and there is no `docs/release-notes/v3.11.0.md`. 52 commits since v3.10.1: 27 fix, 16 feat, 4 test, 2 perf. + +### Highlights +- **Network-wide Scope Audit page** (#1976), with declared-region verification against a repeater's own traffic (#1990), unnameable-traffic accounting (#1987), and opt-in region keys derived from what nodes declare (#1989). +- **SQLite driver swapped** from `modernc.org/sqlite` to `mattn/go-sqlite3`, cross-built with zig (#1992). This makes the build cgo-dependent. +- **Map colours and filters repeaters by scope-configuration state** (#2006), and filters repeaters by region name (#1862). +- Analytics additions: retransmission pressure over time (#1699), scope adverts by node role (#1979), per-node hop-count statistics (#1812). +- The scope-match tally survives ingestor restarts (#2002). + ## [3.10.1] - 2026-09-04 v3.10.0 was tagged and withdrawn before any container image or release asset was published; nothing was ever available under that number. Same release, next number. diff --git a/docs/release-notes/v3.12.0.md b/docs/release-notes/v3.12.0.md new file mode 100644 index 00000000..30820eeb --- /dev/null +++ b/docs/release-notes/v3.12.0.md @@ -0,0 +1,159 @@ +# v3.12.0 + +29 commits since v3.11.0: 15 fixes, 5 test, 3 perf, 3 feat, 2 ci, 1 chore. + +## Read this before upgrading + +**The first start after this release runs `ANALYZE` on your database, and ingest +stalls while it does.** + +SQLite's query planner had no statistics at all, because `ANALYZE` had never run +against these databases and nothing in the code ever called it. Without +`sqlite_stat1` the planner guesses, and on the channel queries it guessed wrong: +it drove from the plain `idx_transmissions_payload_type` instead of +`idx_tx_channel_hash`, the partial index the schema already carries for exactly +that filter (#2058, reported with measurements by @anieto). + +The ingestor now builds those statistics, once, on the first start against a +database that has none. Measured on a 9.4 GB database with 1,250,489 +transmissions and 14,169,329 observations: + +| | | +|---|---| +| `ANALYZE` duration, cold page cache | **3m43.9s** | +| ingest during that time | stalled, buffered, nothing dropped | +| rows written in the minute it finished | 1027, against ~130 either side | + +It holds the single SQLite write connection throughout, so observations stop +arriving and queue in the ingest buffer, then land in one burst when it +completes. On the deployment measured, the ingest buffer absorbed all of it and +the log reported `0 dropped`. The log now warns before starting, so this is +distinguishable from a hang. + +This happens **once per database**, not once per restart: `sqlite_stat1` is an +ordinary table and stays in the file. Every later start costs one query. + +A deploy already restarts the service, so the stall overlaps an interruption you +were having anyway. It is still worth picking the moment on a busy instance. + +To skip it entirely, set `db.analysisLimit` to a negative number. The statistics +then never get built and the channel queries keep the old plan. + +What it buys, measured on the same database, region-filtered `GetChannels`: + +| | pages read | +|---|---| +| before | 143,442 | +| after | 107,429 | + +25% fewer pages, about 147 MB less per query at a 4 KB page. Wall time when the +OS page cache is warm does not change (0.80s either way); the gain is on the cold +path, which measured 56.7s on that file. The 3-4x improvement reported in #2058 +was measured on a smaller database (464,667 transmissions) and did not reproduce +at this size. + +`db.analysisLimit` defaults to 10000 rather than the 400 SQLite's documentation +suggests. 400 and 1000 were measured to leave the plan unchanged; 10000 produces +the same plan as an unbounded `ANALYZE` while costing 2.0s warm against that +one's 242.9s. + +## Features + +- **CoreDrive RX region answers are stored** (#2047) - `node_declared_regions` + was read by `region_keys.go` and `config.go` but created by nothing, so the + Scope Audit page described a source that never arrived: the client-topic switch + handled packets and rf only and dropped `/regions` without a log line. The + table is now created at boot, and the answer is kept with position, the + repeater's own clock, and per-collector history bounded by age rather than one + row per target. Opt in through `clientRegions`, separate from + `clientRxCoverage`, so region answers do not require GPS-tagged reception + uploads. Measured on the instance CoreDrive RX publishes to, 2026-08-18 to + 2026-09-19: 1840 answers about 275 repeaters from 51 collectors, of which 138 + repeaters have answers from more than one collector and 40 have collectors that + disagree, which is the disagreement the Scope Audit exists to surface. +- **The running version is in the nav drawer** (#2068) - read from + `/api/health`, with commit and build time in the tooltip. A version-less or + failing health endpoint leaves the plain label rather than "CoreScope + undefined". +- **The IATA whitelist says when it drops a region** (#2067) - one throttled + warning per rejected region, so a misconfigured `observerIATAWhitelist` is + visible instead of silent. + +## Reception and node health + +- **Zero-hop adverts count as direct reception** (#2064) - a node's own advert + with an empty path is credited to the observer that heard it, without needing + prefix resolution. +- **Heard By lists only observers that heard the node on air** (#2057) - a + relayed packet no longer implies its observer heard the originator directly. +- **The Heard By empty state stopped claiming the node is out of range** (#2063) + - no direct reception recorded is not the same as out of range. +- **The node-detail advert section is called Recent Adverts** (#2042, #2071) - + it never held anything else, because `transmissions.from_pubkey` is populated + for adverts only. The heading now says so in its tooltip instead of promising + every packet type. +- **Blacklist retention is queried by `from_pubkey`** (#2069). + +## Data correctness + +- **Each observation carries its own wire bytes** (#1999, #2055) - the packet + detail API was handing every observation the same canonical frame, so + per-observer differences were invisible. +- **Background chunks merge in order** (#2050) - they were prepended, which put + history out of sequence. +- **Pagination continues past a page that post-LIMIT filtering shortened** + (#2061) - a filtered page that came back short ended the walk early. +- **`maxMemoryMB` eviction actually triggers** (#2035) - path, decode-cache and + dedup-key bytes were not accounted, so the measured footprint sat below the + configured limit and eviction never fired. Instances relying on that limit + will now evict where they previously grew. +- **The distance index shows a building state on its 202** (#1997, #2051) - the + Analytics distance tab treated the lazy index's 202 as data and crashed, and + the 202 was being cached, so the tab stayed broken until the cache expired. + +## Interface + +- **Live legend toggle is clickable again** (#2049) - the VCR bar was eating its + clicks. +- **Map inspector controls stay clickable, stale loads are ignored** (#2048). +- **RX coverage opens at the configured map default** (#2033), with its own saved + viewport. +- **Scope Audit observed-region chips meet 4.5:1 in light theme** (#1996, #2070). + +## Performance + +- **Concurrent `GetChannels` cache misses coalesce** (#2059) - the second and + later callers wait for the first flight instead of each running the query. +- **Planner statistics** (#2058, #2072, #2074) - see the top of this file. + +## Tests, CI and release tooling + +- **Every suite in `tests/e2e` runs** (#2053, #2045) - eight suites had no runner + at all, and the aggregate reported a pass while skipping them. +- Each E2E suite is named in the log before it runs (#2046). +- Root `test-*.js` moved into `tests/unit` and `tests/e2e` (#2036). +- RX-coverage viewport assertions budgeted in pixels rather than a flat 0.001° + (#2040). +- The #1239 lock-hold threshold set from measurement, 150µs to 5ms (#2039) - the + old value sat inside the healthy band and failed at random. +- RF-sample fixtures anchored to now instead of a calendar date (#2034). +- Ingestor test stores wait for boot migrations before being handed over (#2065, + #2066) - this was a real data race under `-race`. +- Re-tagging `:edge` as a release keeps every platform (#2031) - v3.11.0's tags + were flattened to amd64 and had to be repaired. + +## Upgrade notes + +No manual migration step. #2047 adds the `node_declared_regions` table, created +at boot with `CREATE TABLE IF NOT EXISTS`, and #2058 creates `sqlite_stat1` the +same way; nothing else touches the schema. + +`clientRegions` is opt-in and off by default. + +The `ANALYZE` described at the top needs no action, only awareness. Set +`db.analysisLimit` to a negative number to disable it. + +One behaviour change worth repeating here: #2035 makes `maxMemoryMB` eviction +fire where it previously did not, because the footprint it compares against the +limit was undercounted. An instance that set the limit and never saw eviction +will start seeing it.