mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-26 02:25:44 +00:00
c283f42c1fd1e43fc3dbd78ae6ed4ef4178bd067
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 efitenandSaarMesh-Bot
|
9d6f08c144 |
feat(#1865): ingest observer /neighbors as confirmed scope evidence (#1971)
Carries SaarMesh-Bot's implementation from the closed #1867 forward onto current master, 67 commits later, and surfaces the result on the per-node Reach report. The declared region list a repeater answers with is now stored on the node as configured_scope, normalised to the same leading-# syntax default_scope already uses so the two are directly comparable. That normalisation is the point @cwichura raised on #1865 and @dborup agreed with before the original PR closed. Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> |
||
|
|
40f664c587 |
chore(#1859): gofmt sweep + gofmt/go vet CI gate (rebase of #1881) (#1941)
Rebase of #1881 by @SaarMesh-Bot onto current master. Their three commits are preserved, two of them cherry-picked with authorship intact; the sweep itself had to be regenerated. Opened as a new PR rather than force-pushing their branch. Closes #1881 once merged. Addresses parts 1 and 3 of #1859; part 2 landed as #1937. ## Why regenerated rather than merged The sweep in #1881 was cut on 2026-09-02 07:13 and roughly forty PRs landed after it, so it went `CONFLICTING/DIRTY`. Re-running `gofmt` on current master is cheaper and less error-prone than resolving 72 conflicts that are all whitespace. The drift it fixes also grew in the meantime: 66 files now, against 72 then, but spread differently. ## The three commits 1. **`style(#1859)`** — `gofmt -w` across the 14 modules. 66 files. 2. **`test(#1859)`** — @SaarMesh-Bot's fix for the one `go vet` copylocks finding, `cmd/ingestor/coverage_boost_test.go`: the range variable copied a `Config` embedding `sync.Once`. Cherry-picked unchanged. 3. **`ci(#1859)`** — @SaarMesh-Bot's CI step that fails on gofmt drift or vet findings, plus `.git-blame-ignore-revs`. Cherry-picked with one change, noted in the commit message: the ignore file pointed at `04bc80ee`, the sweep commit on their branch, which does not exist on this base and would make `git blame --ignore-revs-file` error. Repointed at `d3a02599`, the sweep here. ## Verification The claim "formatting only" is checked twice rather than asserted: - Every changed file is byte-identical to `gofmt(previous content)`. 0 of 66 deviate. - With line comments and all whitespace stripped, 0 of 66 files differ, so no code outside comments changed. 14 of the 66 also show doc-comment reflow. Since Go 1.19 `gofmt` re-indents indented comment blocks to tabs and inserts a blank comment line before them; the behavior matrix above `resolveHopWithContext` in `cmd/ingestor/path_resolver.go` is a clear example. That is gofmt's own output, not an edit, but it is worth naming because it makes the diff look larger than "whitespace" suggests. The gate was run locally exactly as the workflow runs it: `gofmt` clean, and `go vet` clean in all 14 modules, including `cmd/ingestor` which is what commit 2 fixes. Suites: `cmd/server` ok (80.7s), `internal/packetpath` ok (2.3s), `cmd/ingestor` passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails identically on bare master with "A required privilege is not held by the client" (Windows symlink privilege on my host, not code). ## Sequencing This should go last in the queue. The sweep touches 66 files, so merging it before the remaining open Go PRs gives each of them a conflict about nothing but formatting. After it lands the gate is active, and any PR with drift fails CI until it runs `gofmt -w`. Excluded from the sweep: the misnamed `Dockerfile.go`, which is a Dockerfile that gofmt cannot parse (the workflow excludes it too), and `docs/DEPLOYMENT.md`, which a case-insensitive filesystem surfaces as a spurious modification against `docs/deployment.md` and is unrelated. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandJonathan Herlin
|
89544b1d08 |
perf: index, cache, and deflake /api/channels queries (rebase of #1887) (#1936)
Continues #1887 by @Jonher937. The commit is theirs, authorship unchanged; I only rebased it onto master. It went CONFLICTING because #1934 (prepared statements, originally @Joel-Claw's #1878) landed in the same `DB` struct. Both PRs add fields there and this one also replaces the single-slot channels cache. Resolution: kept this PR's keyed caches (`channelsCache`, `encChannelsCache`, `msgCache` plus their entry types and TTL constants) and kept master's thirteen prepared-statement fields alongside them. The old single-slot `channelsCacheKey`/`channelsCacheRes`/`channelsCacheExp` trio is gone, which is the point of this PR. Nothing else touched. Verified: `cmd/server` builds and the **full suite passes**, not just the channel tests. My review stands: approve, with two questions that do not block and are worth a look at some point. 1. `msgCache` is keyed by `hash|limit|offset|region`, and `offset` grows without bound as someone pages through a channel. Each entry also holds a full page of message maps, so a full 256-entry cache at `limit=50` holds around 12,800 maps. The other two caches are keyed by region only and genuinely low-cardinality as your comment says; this one is the odd one out. 2. `getMsgCache` returns the cached slice directly, so every hit hands the caller the same message maps. If any handler mutates one before serialising, it corrupts the cache for the next ten seconds. Same class as the finding on #1871, which was fixed there by copying at the two broadcast sites. Co-authored-by: Jonathan Herlin <jonte@jherlin.se> |
||
|
efitenandClaude Opus 5
|
176bb53335 |
fix(#1784): ship pathTrust default 1, not 2 (#1929)
Follows #1841. Moves the pathTrust default from 2 back to 1. ## Why #1784's first acceptance criterion is **"Default behaviour remains backward-compatible"**, and its example config shows `minHashBytesForMapping: 1`. What shipped is 2. The problem is not the value. It is that **there is no way to undo it from the UI.** #1841 adds no control for the threshold: it is `config.json` only, and changing it needs a restart. The customizer gains a hint that says exactly that. So an instance that upgrades without touching config switches to the stricter rule, and the only visible symptom is that the neighbour graph and the resolved paths quietly get smaller. The existing "Hide 1-byte path hops" toggle (#1633) is a *display* filter and does not change what counts as evidence, so it is not an escape hatch either. The two are easy to confuse. ## How much this actually moves Measured on a live instance via `/api/analytics/hash-sizes`, not estimated: | path-hop observations | count | share | |---|---|---| | 1-byte prefix | 116,923 | **56.0%** | | 2-byte prefix | 86,031 | 41.2% | | 3-byte prefix | 5,753 | 2.8% | | repeaters by observed hash size | count | |---|---| | 1-byte | **645 (41%)** | | 2-byte | 865 | | 3-byte | 63 | At threshold 2 the 1-byte column stops counting as mapping evidence. `MeetsPathTrust` also drops the legacy bucket-0 observations with it (pre-#1638 persisted neighbor edges that carry no per-mode breakdown), so already-stored edges lose their evidence status on upgrade too. ## What this does not change The knob works and is untouched. Operators who want the stricter behaviour set `minHashBytesForMapping` to 2 or 3, which is the opt-in #1784 describes. Only the default moves. Nothing about storage changes; packets and paths were never affected either way. ## Also fixes an inconsistency inside #1841 Five frontend consumers already fall back to **1** when `MC_getPathTrustThreshold()` is unavailable: `analytics.js`, `live.js`, `map.js`, `nodes.js`, `route-view.js`. Two fell back to **2**: `hop-filter.js` (the getter itself) and `customize-v2.js`. They now all agree. ## Tests - `internal/packetpath`: `TestMeetsPathTrust_ZeroValueOptIn` was asserting the old default *through behaviour*, so it would need rewriting on any future default change. It now asserts the property instead: an absent JSON field resolves to `DefaultMinHashBytesForMapping`, behaves identically to naming that value outright, and an explicit stricter setting still wins. Package tests pass. - `test-issue-1633-hide-1byte-hops.js`: the case pinning the getter's default is updated, with the reasoning in a comment so the next person sees why it is 1. **37 passed, 0 failed** (master baseline: 37 passed, 0 failed). - `cmd/server` config tests pass. ## One thing I want to flag rather than paper over The test I changed was named `default is 2 (operator-confirmed)`. I am overriding something that was confirmed with an operator, and I am not claiming that confirmation was wrong. My reading is that it was about the threshold being a *useful* value, which it is, rather than about it being the default in a build with no UI to change it. If the intent really was "2 out of the box for everyone", say so and I will close this. @Bjorkan as the issue author, @nullrouten0 and @Saarlandpower since you have touched adjacent code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jesper B
|
8c48f2b49a |
feat(#1784): wire display consumers to path trust threshold (#1841)
Fixes #1784 ## Summary Step 4 of the #1784 multi-PR project — wires all frontend display consumers to respect the configured `pathTrust.minHashBytesForMapping` value. **Depends on:** #1840 (must be merged first — provides `MC_meetsPathTrust` / `MC_pathBelowTrust` / `MC_getPathTrustThreshold` helpers and `PATH_TRUST` global). ## Changes ### `map.js` — trust-gated route display - `drawPacketRoute` checks `MC_pathBelowTrust` before drawing polylines - When all path hops are below the configured threshold, shows a "Route not displayed" message with config guidance instead of speculative polylines - Cleans up the trust message control when a new route is drawn ### `analytics.js` — subpath trust filtering - `renderTable` in `renderSubpaths` filters route patterns whose hops are below trust threshold - Combined filter logic: both the 1-byte hide toggle AND trust threshold are applied together - Info line shows which filters are active ("1-byte hide + 2-byte trust", etc.) - No-data message explains which filters caused exclusion ### `route-view.js` — speculative path annotation - Path picker groups tagged with `belowTrust` flag when any hop doesn't meet the configured threshold - Speculative paths show `(speculative, <N-byte hops)` annotation with tooltip explaining the trust threshold - Tooltip includes guidance on changing `pathTrust.minHashBytesForMapping` in config.json ### `live.js` — Paths Through widget - `_pathHopsBelowTrust()` helper checks whether all hops in a path are below trust threshold - Widget fallback message distinguishes between "1-byte filtered" (display toggle) and "N-byte trust threshold" (server config) - Message includes the config key for operators to adjust ### `nodes.js` — confidence weight adjustment - `modeWeight` initialization considers the trust threshold - Hash modes below threshold get zero confidence weight - Bucket-0 (legacy/unknown) excluded at threshold >= 2 per #1784 bucket-0 policy ## Testing `test-issue-1633-hide-1byte-hops.js`: - 5 new source-grep guards verifying each consumer references the trust threshold helpers - All 26 tests passing (21 existing + 5 new) ## Files changed (6 files, +136/-6) ``` public/analytics.js | 28 ++++++++++++++++++--- public/live.js | 19 ++++++++++++++- public/map.js | 21 ++++++++++++++++ public/nodes.js | 8 ++++++ public/route-view.js | 16 +++++++++++- test-issue-1633-hide-1byte-hops.js | 50 ++++++++++++++++++++++++++++++ ``` --- **Depends on:** #1840 **Written by:** DeepSeek V4 Pro in Max Mode |
||
|
|
57956712e7 |
fix(#1768): Relay Airtime Share uses LoRa Time-on-Air (preamble-aware) — partial fix (#1776)
Partial fix for #1768 — Relay Airtime Share now uses closed-form LoRa Time-on-Air instead of a payload-bytes-only proxy, removing the ~3-4× bias against small frames (preamble + fixed-symbol intercept). cross-stack: justified — backend score formula needs a frontend caption change (`public/analytics.js` dumbbell preset banner + tooltip) so operators can interpret the assumed PHY block. Both move together or the metric is misleading. ## Red commit `8da57062` — failing test asserts ToA-based score (~83.48 % ADVERT share on the locked acceptance fixture) instead of the byte proxy's 95.24 %. `internal/lora.TimeOnAir` was a zero-returning stub at the red commit; tests failed with assertion errors, not build errors. ## Green commit `dd402edd` — implements `lora.TimeOnAir` (Semtech AN1200.13 / SX126x §6.1.4 closed form, cross-checked against RadioLib), wires `score = TimeOnAir(payloadBytes, preset) × distinctRelays` in `cmd/server/relay_airtime_share.go`, surfaces the preset in the JSON response and analytics caption. ## Config (per AGENTS Config Documentation Rule) New keys under existing `analytics` block: ```json "loraPreset": { "freq": 869600000, "bw": 62.5, "sf": 8, "cr": 5 } ``` Defaults match the deployment's actual `get radio` (869.6 MHz / BW 62.5 kHz / SF 8 / CR 4/5). `CRC=1`, `IH=0`, `DE = (T_sym ≥ 16 ms)`, and the SF-dependent preamble (32 for SF≤8 else 16, per firmware `preambleLengthForSF` / MeshCore PR #1954) are firmware-fixed constants in `internal/lora/toa.go` and intentionally NOT surfaced as config (per re-triage). ## Scope In-scope files (6): - `internal/lora/toa.go` (new package — closed-form ToA) - `internal/lora/toa_test.go` (table-driven preset tests) - `cmd/server/relay_airtime_share.go` (wire ToA into score) - `cmd/server/relay_airtime_share_test.go` (recomputed expected values) - `cmd/server/config.go` + `config.example.json` (preset config keys) - `public/analytics.js` (preset caption on dumbbell chart + tooltip) Plus `cmd/server/go.mod` (replace directive for the new internal module). ## Deferred to v2 (separate issues per re-triage) - Per-observation SF/BW + radio-settings-aware dedup (blocked: ingestor stores SNR/RSSI only, no SF/BW on observations). - CR-per-hop dual-point sensitivity band (CR scales only the payload symbol term `(CR+4)`, not the preamble/header; second-order accuracy gain). - Cross-SF bridge accounting. ## Tests ``` cd internal/lora && go test ./... → PASS cd cmd/server && go test -run RelayAirtime → PASS ``` ## Preflight overrides - `check-branch-clean` (cross-stack): justified above — score formula change requires matching caption update; both files trace to the same issue. --------- Co-authored-by: kpa-clawbot <kpa-clawbot@users.noreply.github.com> Co-authored-by: Kpa-clawbot <bot@openclaw.local> Co-authored-by: bot <bot@meshcore> |
||
|
Kpa-clawbotandclawbot
|
e465e1c6c6 |
perf(#1740): replace idx_tx_last_seen with partial index WHERE last_seen=0 (#1756)
Fixes #1740. ## What Replace the full `idx_tx_last_seen` with a partial index `WHERE last_seen=0`. Two ordered, sequential migrations in `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn`: - **(a)** `CREATE INDEX IF NOT EXISTS idx_tx_last_seen_zero ON transmissions(id) WHERE last_seen=0` - **(b)** `DROP INDEX IF EXISTS idx_tx_last_seen` — gated after (a) succeeds (sequential `Exec`; (b) never runs if (a) errors) ## Why The only consumer is `chunkedTxLastSeenBackfill`'s `WHERE last_seen=0` scan + `MAX(id)` lookup. The full index covers ALL rows including the long tail where `last_seen != 0` after backfill converges (71K+ rows in prod per carmack's #1740 note). The partial index degenerates to ~the count of un-backfilled rows (0 in steady state, bounded by ingest rate during ops) and stops competing for page cache. ## Migration cost Both migrations are sync and annotated `PREFLIGHT: async=false`: - (a) `CREATE INDEX` on partial subset `WHERE last_seen=0` is bounded by un-backfilled rows — a small superset of the inflight ingest window, not a full table scan. - (b) `DROP INDEX` is a metadata-only schema rewrite in SQLite — no row scan at any size. `dbschema/` has no access to `Store.RunAsyncMigration` (that helper lives in `cmd/ingestor/` and is the wrong layer for the schema source-of-truth per #1321), so sync is the only path here regardless. ## TDD - **RED** `a529f0f4`: `EXPLAIN QUERY PLAN` test asserting the backfill `MAX(id) WHERE last_seen=0` query uses `idx_tx_last_seen_zero` + a second test asserting the legacy `idx_tx_last_seen` is dropped post-Apply. Both failed on assertion (planner picked the full index; partial index didn't exist). - **GREEN** `208fde8a`: add migrations (a) and (b). Both tests pass. ## Files touched - `internal/dbschema/dbschema.go` — swap the index - `internal/dbschema/dbschema_test.go` — `EXPLAIN QUERY PLAN` pin + DROP assertion - `cmd/ingestor/db.go` — comment refresh only (idx_tx_last_seen → idx_tx_last_seen_zero) ## Acceptance - ✅ New partial index created - ✅ Old full index dropped via gated migration (sequential, order preserved) - ✅ Query plan test asserts partial-index usage - ✅ Existing migration tests still green (`internal/dbschema`, `cmd/ingestor` TestIssue1690 + applySchema) --------- Co-authored-by: clawbot <bot@openclaw> |
||
|
|
048143f54f |
fix(#1690): cold-load uses last_seen (effective recency) instead of first_seen (#1691)
## #1690 — cold-load uses wrong time axis (RED → GREEN) The on-disk DB has thousands of long-lived hashes with recent traffic. Prod's cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a column that is set once at insert time and never updated — so re-observation of an old hash does not move it into the hot window. Result: prod cold-loaded ~0.3% of the on-disk rows and flipped `backgroundLoadComplete=true` without ever walking the retention window (the `retentionHours - hotStartupHours <= 0` short-circuit at line 1353 of `cmd/server/store.go`). ### Three sub-fixes **A) Denormalize `transmissions.last_seen`** so cold-load can window on effective recency. - `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn` adds the column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both PREFLIGHT-annotated as cheap metadata-only ops). - `cmd/ingestor/db.go::OpenStoreWithInterval` schedules `tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` — `UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod). - Writer-side: `InsertTransmission` seeds `last_seen` on initial insert, and every observation insert bumps `last_seen = ?` via prepared statement `stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest never goes backwards). - Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and `cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY clauses to `t.last_seen` when the column is present (PRAGMA-detected via `DB.hasLastSeen`). Test/legacy DBs without the column fall back to `first_seen` so existing fixtures stay green. **B) Honest `backgroundLoadComplete` gating.** - Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod runs with both at 12h, which flipped Done=true immediately. - After the chunk loop, query `SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor` and compute `loadCoverageRatio = inMem / inDB`. Done=true only when `ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` + `backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000 rows (1000 in memory)"`). - `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the perf endpoint can read them without blocking the writer. **C) Perf exposure.** `PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`, `LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of the on-disk DB the in-memory store currently reflects, so operators can see the 0.3% case in `/api/perf` without reading the logs. ### TDD trail - **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests + 21-line store.go scaffolding. CI on this commit failed on assertions (intended). - **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite: `Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`, `Test1690_PerfStats_NewFields`, `TestHotStartup_*`, `TestIssue1690_LastSeenUpdatedOnObservation` — all pass. Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()` guard around `backgroundLoadDone.Store(true)` — `Test1690_BackgroundLoadHonesty` fails on the assertion `"backgroundLoadDone=true with only 1000/5000 packets loaded; must be false until coverage ≥ 90%"`. Restored. ### Async-migration preflight - `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both `// PREFLIGHT: async=true reason="..."` annotated. - `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`. - `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level UPDATE BY PRIMARY KEY, not a migration. ### Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=5K - check-async-migration: justified for `cmd/server/issue1690_cold_load_test.go` CREATE TABLE/INDEX statements — these build an in-memory test fixture DB (≤5000 rows, runs in <1s in CI), not a prod migration. Fixes #1690. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: bot <bot@example.com> |
||
|
|
a4776557ae |
feat(#1290): use firmware repeat:on|off hint to exclude listener-only observers from disambiguator (#1624)
Closes #1290. cross-stack: justified — backend persists firmware-side `repeat` hint to a new observers column, frontend surfaces the listener/repeater status as a badge on the observers list and node-detail Heard By table per the issue's UI acceptance criterion. ## What Firmware 1.16 publishes a `repeat: on|off` flag in the MQTT `/status` JSON (confirmed by @cwichura on the issue thread — see [`MQTTMessageBuilder.cpp:58`](https://github.com/agessaman/MeshCore/blob/b45373a31f111fb0de98bb3b168226d09ceadc47/src/helpers/MQTTMessageBuilder.cpp#L58) in `agessaman/MeshCore mqtt-bridge-implementation-flex`). Listener-only observers (`repeat:off`) by firmware contract never relay packets, so they cannot legitimately be a hop in someone else's resolved path. This PR plumbs the hint end-to-end so the disambiguator stops considering them. ## How * **`internal/dbschema`**: idempotent `can_relay INTEGER DEFAULT 1` migration on `observers`, plus `AssertReady` probe (server fatal-logs if absent). Mirrored in `cmd/ingestor/db.go` `CREATE TABLE` for fresh DBs. Annotated `PREFLIGHT: async=true` — `DEFAULT 1` is constant so SQLite does this as a metadata-only schema rewrite. * **`cmd/ingestor`**: `extractObserverMeta` accepts `repeat` as bool, case-insensitive string (`on|off|true|false|yes|no`), or numeric `0|1`. Missing field → `nil` → `COALESCE` preserves the existing column value (back-compat with legacy observers). Plumbed through `UpsertObserverAt` and the prepared upsert statement. * **`cmd/server`**: `GetNonRelayObserverPubkeys` + new `prefixMap.markNonRelay` drop matching candidates inside `pm.resolveWithContext` at the top of the resolver, so all 4 tiers see the pruned candidate set. `ObserverResp.CanRelay` is surfaced on `/api/observers` and `/api/observers/{id}`. `GetNodeHealth` enriches per-observer rows with `can_relay` so the node-detail badge renders. Probe-and-fall-back when the `can_relay` column is absent (legacy test fixtures). * **`public/`**: listener vs repeater pill on observers list, observer detail `Relay` stat card, and node-detail `Heard By` table. CSS uses existing theme vars. ## Test Added `TestResolveWithContext_ExcludesNonRelayObservers_Issue1290` in `cmd/server/resolve_non_relay_1290_test.go` covering all three required cases: * `repeat:off` pubkey → not a candidate (assertion failed in red commit `5f7fdb96`, passes after green `f12911dc`) * `repeat:on` pubkey → still a candidate (regression guard) * legacy obs (no field) → still a candidate (back-compat) Red→green proof: ``` $ git log --oneline origin/master..HEAD |
||
|
|
bc1822e46c |
perf(load): chunked Load with early HTTP readiness (#1009) (#1596)
## What Switches the server's startup from a synchronous full-scan `PacketStore.Load()` to a chunked `LoadChunked(chunkSize)` that: 1. Streams transmissions+observations from SQLite in id-ordered chunks (default `chunkSize=10000`, configurable via `db.load.chunkSize`). 2. Closes `FirstChunkReady()` after the first chunk is merged — `main.go` binds the HTTP listener on that signal instead of blocking on the full multi-minute load. 3. Stamps `X-CoreScope-Load-Status: loading; progress=<rows>` on every response while LoadChunked is in flight, flipping to `ready` once it completes (via `loadStatusMiddleware`). 4. Preserves the existing retention/`hotStartupHours`/`maxMemoryMB` clamps and the post-load index rebuild (`pickBestObservation` / `buildSubpathIndex` / `buildPathHopIndex` / `buildDistanceIndex`). ## Why Per #1009: at 5M+ observations (Cascadia scale) the synchronous Load blocked HTTP for ~80s with a 2–3× steady-state RAM peak. With chunked load the listener binds within seconds; dashboards and probes can read partial data and see the `loading` status header until the background load finishes. ## Notes - `/api/healthz` readiness gate (`readiness` atomic, init `WaitGroup`) is unchanged — it still waits for neighbor-graph build + initial `pickBestObservation` before reporting `ready:true`. `LoadChunked` only changes when the listener BINDS, not when it advertises ready. - `cmd/server/main.go` waits for `FirstChunkReady` (or the full load on a tiny DB) before proceeding, and drains the load goroutine in the background with a logged error path. - Config Documentation Rule: `config.example.json` now documents `db.load.chunkSize` with a nested `_comment` describing the trade-off. ## Tests - `cmd/server/chunked_load_test.go` asserts: - (a) `FirstChunkReady` fires before `LoadChunked` returns - (b) `X-CoreScope-Load-Status` transitions `loading; progress=...` → `ready` - (c) `chunkSize` honored (2500 rows @ 1000 → 3 chunks via `OnChunkLoaded`) - (d) `Config.DBLoadChunkSize()` default 10000 + override - Red commit (`102a4c84`) lands the tests with stubs that fail on assertion — verified locally before the green commit. - Green commit (`35cecf16`) makes all four pass; full `cmd/server` suite green (47s locally). Closes #1009 ## TDD red-commit exemption The original red commit `f878e15e` ("test(load): failing tests for chunked Load + early HTTP readiness") fails to **compile** rather than failing on an assertion, because it references symbols (`store.LoadChunked`, `store.FirstChunkReady`, `store.OnChunkLoaded`, `Config.DBLoadChunkSize`, `loadStatusMiddleware`) that do not exist on master. Per `AGENTS.md` the bar is "MUST fail on an assertion ... A compile error is NOT a valid red commit." This is claimed under the **net-new surface** exemption with the following justification: - LoadChunked / FirstChunkReady / loadStatusMiddleware / DBLoadChunkSize are all introduced by this PR — no prior implementation existed to refactor. There is no behaviour on master that the red commit could meaningfully assert against without first declaring the new symbols. - The cheapest "proper" alternative (split the red into two commits: stub-first + assertion-fail) was deferred because the test file unambiguously fails on missing-symbol — there is no risk of the test becoming a tautology against a pre-existing stub. - **Behaviour gating IS proven elsewhere on this branch.** Commit `799bde49` ("test(load): red — LoadChunked must mark indexes ready + not flip Complete on error") is a proper assertion-fail red against the same package, and commit `92cadd1d` is the matching green. Reviewers can verify the red→green pattern there. If a future reviewer wants the strict pattern, the follow-up is mechanical: split `f878e15e` into a stub-only commit followed by the assertion commit. Not done here to keep the rework cost proportional to the risk (zero, in this case). ## Preflight overrides - check-async-migrations: justified — the flagged `CREATE TABLE`/`CREATE INDEX` statements live in `cmd/server/chunked_load_id_zero_test.go` and `cmd/server/chunked_load_oldest_test.go` only. They run against per-test `t.TempDir()` SQLite files (in-process, ~10 rows, lifetime = single test) — they are NOT production schema migrations. No prod table is touched. PREFLIGHT-MIGRATION-SCALE: <30s N=10 (per-test tempdir fixture). --------- Co-authored-by: CoreScope Bot <bot@corescope.local> Co-authored-by: clawbot <bot@noreply.example.com> Co-authored-by: Kpa-clawbot <bot@example.com> Co-authored-by: Kpa-clawbot <bot@kpa-clawbot> |
||
|
Kpa-clawbotandopenclaw
|
43b93c6bb9 |
feat(observers): surface naive-clock observers as ⚠️ chip + detail banner (#1478) (#1480)
## Summary Issue #1478 — surface observers whose envelope timestamps are being clamped because they're emitting zone-less local-time strings (UTC-N observers showed up perpetually as "Stale" before #1466, and per-packet rxTime is still clamped to ingest time for them, muddying propagation-delay analytics). Now the UI tells operators which observers are misconfigured + how to fix it. ## What changed ### Ingestor (cmd/ingestor) - New `observers_clock_naive_v1` migration adds three columns to `observers`: - `clock_skew_seconds INTEGER` (signed: negative = behind UTC, positive = ahead) - `clock_skew_count_24h INTEGER` (rolling 24h event count) - `clock_last_naive_at TEXT` (RFC3339 timestamp of last clamp) - `resolveRxTime` now returns `(rxTime, naiveSkewSec)`. The packet-handler call site invokes `store.RecordNaiveSkew(observerID, deltaSec)` whenever a naive envelope is clamped (the existing >15 min naive-tolerance path). The counter resets to 1 if no event in the prior 24h, else increments. Single INSERT-or-UPDATE round trip per clamp. ### Server (cmd/server) - `Observer` struct + `GetObservers` / `GetObserverByID` extended to scan the three new columns. - `ObserverResp` gains four JSON fields exposed by `/api/observers` and `/api/observers/{id}`: - `clock_naive` (bool, derived from `clock_last_naive_at` being within 24h) - `clock_skew_seconds`, `clock_skew_count_24h`, `clock_last_naive_at` - Decay is **read-side**: a stale event yields `clock_naive=false` with zero counts. No background sweep, no writes from the read-only server, no race with the ingestor. ### Frontend (public) - `window.ObserversNaiveChip.render(o)` — total render helper, returns ⚠️ chip HTML when `o.clock_naive===true`, `""` otherwise. Used inline in the observers-list `name` cell and in the row-detail slide-over. Tooltip explains magnitude + direction + count + fix. - `window.ObserverDetailNaiveBanner.render(obs)` — yellow alert banner at the top of the observer-detail page with the skew magnitude, last-event timestamp, and the actionable fix ("Set host clock to UTC, OR emit Z-suffixed/offset-aware timestamps from the observer script"). ## TDD trail - `5ddd5b42` red: backend `cmd/server/observer_naive_clock_1478_test.go` (3 tests asserting JSON fields + 24h decay) + frontend `test-observer-naive-clock-1478.js` (8 jsdom-style tests asserting helpers exist and render correctly). Both failed on master with field-missing / export-missing assertions. - `4ecc79c8` green backend: schema + Observer / GetObservers / ObserverResp / handler decay. - `2137ab81` green frontend: chip + banner helpers and call sites. ## Tests - `cd cmd/server && go test ./...` → all green (full suite, 46s) - `cd cmd/ingestor && go test ./...` → all green (full suite, 98s) - `node test-observer-naive-clock-1478.js` → 8/8 pass - `node test-frontend-helpers.js` → unchanged from master (pre-existing failures only) ## Acceptance (issue #1478) - ✅ Observer running with `python datetime.now().isoformat()` (naive, off by N hours) → `clock_naive=true` after the next clamp → UI shows ⚠️ chip + banner. - ✅ Observer with `datetime.now(timezone.utc).isoformat()` (Z-suffixed) → never clamped → never flagged. - ✅ Observer that fixed its clock → `clock_naive` returns to `false` 24h after the last clamp event (read-side decay). Closes #1478. --------- Co-authored-by: openclaw <bot@openclaw.local> |
||
|
|
0b35c7eef3 |
feat(server): persist multi-byte capability across restart + O(1) per-key lookup (#903) (#1324)
## Summary Follows the reconciliation recommendation in #916 — extracts only the NET-NEW persistence layer from that PR (which is now superseded by #1002 for the overlay UI) into a focused 6-file change against current master. **What this adds:** - `multibyte_sup_v1` migration: `multibyte_sup INTEGER NOT NULL DEFAULT 0` + `multibyte_evidence TEXT` on `nodes`/`inactive_nodes` so capability survives restart - `hasMultibyteSupCols` schema detection gates the persist/load paths - `loadMultibyteCapFromDB()`: pre-populates `mbCapSnapshot`/`mbCapIndex` at startup — cold starts serve last-known capability without waiting for the first ~15s analytics cycle - `maybePersistMultibyteCapability()` + `persistMultibyteCapability()`: after each analytics cycle; TryLock-gated (concurrent cycles coalesce); skips `sup==0` entries (data-destruction guard) - `GetMultibyteCapFor(pk)`: O(1) map lookup; both `handleNodes` and node-detail call sites updated from the O(N)-alloc `GetMultiByteCapMap()` **What this explicitly does NOT change:** - API field names (`multi_byte_status`, `multi_byte_evidence`, `multi_byte_max_hash_size`) - `EnrichNodeWithMultiByte` — unchanged - `GetMultiByteCapMap` — still present for any external callers - `public/map.js`, `public/live.css`, `Dockerfile`, `docs/` — zero frontend churn ## Test plan - [x] `TestMultibyteCapPersistRoundTrip` — confirmed values survive persist → fresh-store load - [x] `TestMultibyteCapPersistSkipsUnknown` — data-destruction guard: `sup==0` entry does not overwrite DB-confirmed value - [x] `TestMultibyteCapMaybePersistCoalesces` — TryLock coalesces 10 concurrent callers without deadlock - [x] `TestMultibyteCapGetMultibyteCapForO1` — O(1) index returns correct entry / false for unknown pubkey - [x] `TestMultibyteCapLoadFromDB` — only `sup>0` rows loaded; `sup==0` row excluded - [x] `TestSchemaMultibyteSupColumns` — migration adds columns to both tables; idempotent on second `OpenStore` - [x] All existing `TestMultiByteCapability_*` tests pass unchanged - [x] Full ingestor test suite: `ok` in 27s - [x] `go build ./cmd/server/ && go build ./cmd/ingestor/` clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
|
c7ab5f3eb9 |
fix(#1366): channels view shows latest message time — backend emits LatestSeen, not FirstSeen (#1368)
Red commit:
|
||
|
Kpa-clawbotandopenclaw-bot
|
d9ba9937a6 |
fix(dbschema): canonical source for optional column migrations — fixes startup race (closes #1321) (#1322)
Red commit `2a8102b9` (failing test) → green commit `bb957c9f`. CI: https://github.com/Kpa-clawbot/CoreScope/actions/workflows/ci.yml?query=branch%3Afix%2Fissue-1321 Fixes #1321. ## Why On staging `/api/scope-stats` 500'd with `scope_name column not present` despite the ingestor adding the column ~0.5s after server startup. `cmd/server/db.go detectSchema()` runs in `OpenDB` and caches `hasScopeName`/`hasDefaultScope`/`hasObsRawHex` booleans. With supervisord launching server + ingestor simultaneously, the server's PRAGMA can fire BEFORE the ingestor's `ALTER TABLE` completes — and the boolean stays false until the server restarts. Same race class as #1283; #1289 moved server-side ensures to `dbschema` but the optional columns the ingestor still owned were left out. ## Fix — option (c) from the issue Made `internal/dbschema/dbschema.go` the single source of truth for the optional columns the server detects. **Migrations moved from `cmd/ingestor/db.go applySchema` into `dbschema.Apply`:** - `transmissions.scope_name` + `idx_tx_scope_name` partial index - `nodes.default_scope` - `inactive_nodes.default_scope` - `observations.raw_hex` **`AssertReady` now asserts** every one of those columns. The server cannot start with stale-false booleans because `AssertReady` will fatal first if the columns are missing. The ingestor's old gated blocks are replaced with pointer comments so anyone hunting for them lands in `dbschema.go`. The `_migrations` marker rows are preserved (`INSERT OR IGNORE`) to keep legacy DBs idempotent. **Documented invariant** in the package doc: any new optional column the server PRAGMA-detects belongs in `internal/dbschema/dbschema.go`, NOT in `cmd/ingestor/db.go applySchema`. ## Tests Added `internal/dbschema/dbschema_test.go` (RED in `2a8102b9`): - `TestApplyAddsOptionalColumns_CanonicalSource` — post-`Apply`, all four columns must exist. - `TestAssertReady_RequiresOptionalColumns` — `AssertReady` must refuse a DB missing them AND pass after full `Apply`. `cmd/ingestor` and `cmd/server` full suites green. --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
efitenandClaude Sonnet 4.6
|
51f823bf7e |
feat: one-click prune nodes outside geofilter (#669 M4) (#738)
## Summary - Adds `POST /api/admin/prune-geo-filter` endpoint — dry-run by default, `?confirm=true` to permanently delete nodes outside the current geofilter polygon + buffer. Requires `X-API-Key` header. - Adds **Prune nodes** section inside the GeoFilter customizer tab (write-access only, same `writeEnabled` gate as PUT). **Preview** lists affected nodes; **Confirm delete** removes them. - Adds `GetNodesForGeoPrune` and `DeleteNodesByPubkeys` DB helpers. - Updates `docs/user-guide/geofilter.md` — documents the UI button as primary workflow, CLI script as alternative. > **Depends on M3** (`feat/geofilter-m3-customizer`, PR #736). Merge M3 first. ## Test plan - [x] `cd cmd/server && go test ./...` — all pass - [x] Customizer GeoFilter tab without `apiKey` — Prune section not visible - [x] With `apiKey` + polygon active — Prune section visible - [x] **Preview** returns list of nodes outside polygon (no deletions) - [x] **Confirm delete** removes nodes, list clears - [x] `POST /api/admin/prune-geo-filter` without `X-API-Key` → 401 - [x] `POST /api/admin/prune-geo-filter` with no polygon configured → 400 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
9383201c07 |
refactor(db): finish #1283 — Option 4: ingestor owns neighbor-graph + schema migrations; server is read-only (fixes #1287) (#1289)
Red commit: https://github.com/Kpa-clawbot/CoreScope/commit/eae179b99b5fd34924547632aa8f8025c405aa53 (CI: pending — opens with this PR) Finishes #1283. RED test `TestServerSourceHasNoCachedRWCalls` goes from failing (13 writer call-sites) to GREEN (zero). Per #1287 Option 4 (https://github.com/Kpa-clawbot/CoreScope/issues/1287#issuecomment-4485099992): ingestor owns the neighbor graph build + persist; server reads the snapshot. **Category A — Schema migrations** → new `internal/dbschema` package. `dbschema.Apply(rw)` runs in `cmd/ingestor` startup (in `OpenStore`). `dbschema.AssertReady(ro)` runs in `cmd/server/main.go` and FATAL-LOG-EXITS if any expected column/index/table is missing — the operator must restart the ingestor first. Covers indexes, `neighbor_edges`, `observations.resolved_path`, `observers.{inactive,last_packet_at,iata}`, `(inactive_)nodes.foreign_advert`, `transmissions.from_pubkey`. **Category B — Backfill** → ingestor. `BackfillFromPubkey` and observer-blacklist soft-delete moved to `cmd/ingestor/maintenance.go`. Server keeps an inert `fromPubkeyBackfillSnapshot` stub for `/api/healthz` API compatibility. **Category C — Neighbor-graph persistence (Option 4)** → ingestor writes, server reads. - Ingestor (`cmd/ingestor/neighbor_builder.go`): every 60s scans `observations + transmissions`, extracts edges (originator↔first-hop for ADVERTs; observer↔last-hop for all), resolves hop prefixes via a node-table prefix index, upserts into `neighbor_edges`. - Server (`cmd/server/neighbor_recomputer.go`): every 60s re-reads `neighbor_edges` and atomic-swaps the resulting `NeighborGraph` into `s.graph`. Initial load is synchronous on startup. All server-side incremental edge writers (the two `asyncPersistResolvedPathsAndEdges` paths in `cmd/server/store.go`) are gone. - Neighbor-edge daily prune (`PruneNeighborEdges`) moved to ingestor. **Why Option 4**: clean read/write separation, no startup CPU spike (server loads existing snapshot instead of rebuilding from history), no IPC/delta-protocol churn. Staleness budget ~60s — same model as the analytics recomputers in #1240 / #1248 / #672 axis 2. **Recomputer interval default for neighbor graph**: 60s (`NeighborGraphRecomputerDefaultInterval`, `NeighborEdgesBuilderInterval`). **Invariants added**: - `TestServerSourceHasNoCachedRWCalls` (RED commit eae179b9): grep enforces zero `cachedRW(`, `mode=rw`, or `sql.Open(_journal_mode=WAL…)` in non-test `cmd/server/` sources. - `TestServerStartupRequiresMigratedSchema`: server refuses to start against an unmigrated DB. - `TestNeighborGraphRecomputerLoadsSnapshot`: post-write snapshot is picked up on the next refresh. - `TestNeighborEdgesBuilderUpsertsFromObservations`: end-to-end pipeline writes the expected edge. `grep cachedRW cmd/server/*.go | grep -v _test.go` → 0 matches. Fixes #1287. --------- Co-authored-by: MeshCore Bot <bot@meshcore.local> Co-authored-by: Kpa-clawbot <Kpa-clawbot@users.noreply.github.com> Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
|
f4cf2acbc0 |
perf: cancelled writes + ingestor I/O + threshold tests (#1120 follow-up) (#1167)
Red commit:
|
||
|
Kpa-clawbotandyou
|
dd2f044f2b |
fix: cache RW SQLite connection + dedup DBConfig (closes #921) (#982)
Closes #921 ## Summary Follow-up to #920 (incremental auto-vacuum). Addresses both items from the adversarial review: ### 1. RW connection caching Previously, every call to `openRW(dbPath)` opened a new SQLite RW connection and closed it after use. This happened in: - `runIncrementalVacuum` (~4x/hour) - `PruneOldPackets`, `PruneOldMetrics`, `RemoveStaleObservers` - `buildAndPersistEdges`, `PruneNeighborEdges` - All neighbor persist operations Now a single `*sql.DB` handle (with `MaxOpenConns(1)`) is cached process-wide via `cachedRW(dbPath)`. The underlying connection pool manages serialization. The original `openRW()` function is retained for one-shot test usage. ### 2. DBConfig dedup `DBConfig` was defined identically in both `cmd/server/config.go` and `cmd/ingestor/config.go`. Extracted to `internal/dbconfig/` as a shared package; both binaries now use a type alias (`type DBConfig = dbconfig.DBConfig`). ## Tests added | Test | File | |------|------| | `TestCachedRW_ReturnsSameHandle` | `cmd/server/rw_cache_test.go` | | `TestCachedRW_100Calls_SingleConnection` | `cmd/server/rw_cache_test.go` | | `TestGetIncrementalVacuumPages_Default` | `internal/dbconfig/dbconfig_test.go` | | `TestGetIncrementalVacuumPages_Configured` | `internal/dbconfig/dbconfig_test.go` | ## Verification ``` ok github.com/corescope/server 20.069s ok github.com/corescope/ingestor 47.117s ok github.com/meshcore-analyzer/dbconfig 0.003s ``` Both binaries build cleanly. 100 sequential `cachedRW()` calls return the same handle with exactly 1 entry in the cache map. --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
56ec590bc4 |
fix(#886): derive path_json from raw_hex at ingest (#887)
## Problem Per-observation `path_json` disagrees with `raw_hex` path section for TRACE packets. **Reproducer:** packet `af081a2c41281b1e`, observer `lutin🏡` - `path_json`: `["67","33","D6","33","67"]` (5 hops — from TRACE payload) - `raw_hex` path section: `30 2D 0D 23` (4 bytes — SNR values in header) ## Root Cause `DecodePacket` correctly parses TRACE packets by replacing `path.Hops` with hop IDs from the payload's `pathData` field (the actual route). However, the header path bytes for TRACE packets contain **SNR values** (one per completed hop), not hop IDs. `BuildPacketData` used `decoded.Path.Hops` to build `path_json`, which for TRACE packets contained the payload-derived hops — not the header path bytes that `raw_hex` stores. This caused `path_json` and `raw_hex` to describe completely different paths. ## Fix - Added `DecodePathFromRawHex(rawHex)` — extracts header path hops directly from raw hex bytes, independent of any TRACE payload overwriting. - `BuildPacketData` now calls `DecodePathFromRawHex(msg.Raw)` instead of using `decoded.Path.Hops`, guaranteeing `path_json` always matches the `raw_hex` path section. ## Tests (8 new) **`DecodePathFromRawHex` unit tests:** - hash_size 1, 2, 3, 4 - zero-hop direct packets - transport route (4-byte transport codes before path) **`BuildPacketData` integration tests:** - TRACE packet: asserts path_json matches raw_hex header path (not payload hops) - Non-TRACE packet: asserts path_json matches raw_hex header path All existing tests continue to pass (`go test ./...` for both ingestor and server). Fixes #886 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
c233c14156 |
feat: CLI tool to decrypt and export hashtag channel messages (#724)
## Summary
Adds `corescope-decrypt` — a standalone CLI tool that decrypts and
exports MeshCore hashtag channel messages from a CoreScope SQLite
database.
### What it does
MeshCore hashtag channels use symmetric encryption with keys derived
from the channel name. The CoreScope ingestor stores **all** GRP_TXT
packets, even those it can't decrypt. This tool enables retroactive
decryption — decrypt historical messages for any channel whose name you
learn after the fact.
### Architecture
- **`internal/channel/`** — Shared crypto package extracted from
ingestor logic:
- `DeriveKey()` — `SHA-256("#name")[:16]`
- `ChannelHash()` — 1-byte packet filter (`SHA-256(key)[0]`)
- `Decrypt()` — HMAC-SHA256 MAC verify + AES-128-ECB
- `ParsePlaintext()` — timestamp + flags + "sender: message" parsing
- **`cmd/decrypt/`** — CLI binary with three output formats:
- `--format json` — Full metadata (observers, path, raw hex)
- `--format html` — Self-contained interactive viewer with search/sort
- `--format irc` (or `log`) — Plain-text IRC-style log, greppable
### Usage
```bash
# JSON export
corescope-decrypt --channel "#wardriving" --db meshcore.db
# Interactive HTML viewer
corescope-decrypt --channel wardriving --db meshcore.db --format html --output wardriving.html
# Greppable log
corescope-decrypt --channel "#wardriving" --db meshcore.db --format irc | grep "KE6QR"
# From Docker
docker exec corescope-prod /app/corescope-decrypt --channel "#wardriving" --db /app/data/meshcore.db
```
### Build & deployment
- Statically linked (`CGO_ENABLED=0`) — zero dependencies
- Added to Dockerfile (available at `/app/corescope-decrypt` in
container)
- CI: builds and tests in go-test job
- CI: attaches linux/amd64 and linux/arm64 binaries to GitHub Releases
on tags
### Testing
- `internal/channel/` — 9 tests: key derivation, encrypt/decrypt
round-trip, MAC rejection, wrong-channel rejection, plaintext parsing
- `cmd/decrypt/` — 7 tests: payload extraction, channel hash
consistency, all 3 output formats, JSON parseability, fixture DB
integration
- Verified against real fixture DB: successfully decrypts 17
`#wardriving` messages
### Limitations
- Hashtag channels only (name-derived keys). Custom PSK channels not
supported.
- No DM decryption (asymmetric, per-peer keys).
- Read-only database access.
Fixes #723
---------
Co-authored-by: you <you@example.com>
|
||
|
copelajeandyou
|
922ebe54e7 |
BYOP Advert signature validation (#686)
For BYOP mode in the packet analyzer, perform signature validation on advert packets and display whether successful or not. This is added as we observed many corrupted advert packets that would be easily detectable as such if signature validation checks were performed. At present this MR is just to add this status in BYOP mode so there is minimal impact to the application and no performance penalty for having to perform these checks on all packets. Moving forward it probably makes sense to do these checks on all advert packets so that corrupt packets can be ignored in several contexts (like node lists for example). Let me know what you think and I can adjust as needed. --------- Co-authored-by: you <you@example.com> |
||
|
efitenandClaude Sonnet 4.6
|
fe314be3a8 |
feat: geo_filter enforcement, DB pruning, geofilter-builder tool, HB column (#215)
## Summary
Several features and fixes from a live deployment of the Go v3.0.0
backend.
### geo_filter — full enforcement
- **Go backend config** (`cmd/server/config.go`,
`cmd/ingestor/config.go`): added `GeoFilterConfig` struct so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by
both the server and ingestor
- **Ingestor** (`cmd/ingestor/geo_filter.go`, `cmd/ingestor/main.go`):
ADVERT packets from nodes outside the configured polygon + buffer are
dropped *before* any DB write — no transmission, node, or observation
data is stored
- **Server API** (`cmd/server/geo_filter.go`, `cmd/server/routes.go`):
`GET /api/config/geo-filter` endpoint returns the polygon + bufferKm to
the frontend; `/api/nodes` responses filter out any out-of-area nodes
already in the DB
- **Frontend** (`public/map.js`, `public/live.js`): blue polygon overlay
(solid inner + dashed buffer zone) on Map and Live pages, toggled via
"Mesh live area" checkbox, state shared via localStorage
### Automatic DB pruning
- Add `retention.packetDays` to `config.json` to delete transmissions +
observations older than N days on a daily schedule (1 min after startup,
then every 24h). Nodes and observers are never pruned.
- `POST /api/admin/prune?days=N` for manual runs (requires `X-API-Key`
header if `apiKey` is set)
```json
"retention": {
"nodeDays": 7,
"packetDays": 30
}
```
### tools/geofilter-builder.html
Standalone HTML tool (no server needed) — open in browser, click to
place polygon points on a Leaflet map, set `bufferKm`, copy the
generated `geo_filter` JSON block into `config.json`.
### scripts/prune-nodes-outside-geo-filter.py
Utility script to clean existing out-of-area nodes from the database
(dry-run + confirm). Useful after first enabling geo_filter on a
populated DB.
### HB column in packets table
Shows the hop hash size in bytes (1–4) decoded from the path byte of
each packet's raw hex. Displayed as **HB** between Size and Type
columns, hidden on small screens.
## Test plan
- [x] ADVERT from node outside polygon is not stored (no new row in
nodes or transmissions)
- [x] `GET /api/config/geo-filter` returns polygon + bufferKm when
configured, `{polygon: null, bufferKm: 0}` when not
- [x] `/api/nodes` excludes nodes outside polygon even if present in DB
- [x] Map and Live pages show blue polygon overlay when configured;
checkbox toggles it
- [x] `retention.packetDays: 30` deletes old transmissions/observations
on startup and daily
- [x] `POST /api/admin/prune?days=30` returns `{deleted: N, days: 30}`
- [x] `tools/geofilter-builder.html` opens standalone, draws polygon,
copies valid JSON
- [x] HB column shows 1–4 for all packets in grouped and flat view
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|